From patchwork Fri Feb 21 00:09:21 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 13984662 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id AE8E4C021B3 for ; Fri, 21 Feb 2025 00:09:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=16tsa6nu0NypQcavo+jM81OxEAyEksqgRnQ4X9P7nXs=; b=CLuUADGrBw1hoG +EJ6Qywm5/Yp5ZQhgksClHL7DipgmsfzuO4EBYqQD4LXkzMZzadu9xM2L6Ma6RFeavHY4FCBr9qQv 6621f7zVo6kKNoIxH9/0noNDudXn13nuRZ8IhLdfMoQsaxInjuNSl3J1YSu1Hw7lP4K/x8XwkU81A C9T+Z+PAoosOFJnJGSSQNMQHVE/iIhBjZJ0B9DHwwc3iC78nt0CQRw1dXa5UXXHVKdZfBfbduEQC6 W/26PrFyETT1buIaJ6zIRMQ/I6dC1KraYQFBu386fX5RvoMF4EuJ09uhIT6PhgV2GapvqK7rPBt0z 5VI5HHqzaoH9h9uVgGBw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tlGbm-00000003SK2-2tOe; Fri, 21 Feb 2025 00:09:34 +0000 Received: from mail-oi1-x22d.google.com ([2607:f8b0:4864:20::22d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tlGbg-00000003SCw-3PxO for linux-riscv@lists.infradead.org; Fri, 21 Feb 2025 00:09:33 +0000 Received: by mail-oi1-x22d.google.com with SMTP id 5614622812f47-3f3da35555eso868084b6e.1 for ; Thu, 20 Feb 2025 16:09:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1740096567; x=1740701367; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ynQVfR/dFk01f+alSHWcouno+EVl3KD0yb8OXJAhnGc=; b=ZVdLVSyGcAwDNiBW1YQCSXbQo8ZSHsNxlbSaSEw4zVNC+hIICUlWWZw+kniIH87IA1 Yf+Ko1WI3ale9jmtlyoRr8a4gS09BXc/weD9ogXFoo3pF9wD4w67uYb4UyzNYFoCWr0d Pg37mhYwHTZwmAA4urMX+vk9UkUezZ7Mo3w5cuS6IwtBs0sYBd6VrHovj/CTGyHEpR4C 7CoPCb83yPUhhGaNDP5r4prcaEyzRYhXOhiei1e99HzK04Vk+JCXO6G5eBaTA+qYZ/Hb OasZ4c3BWBQ6tN+P/uj9CEmb+lGxjp99fCra0FBrggsDmfaCrOVDmOBXGyemg9mWydK8 Tarw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740096567; x=1740701367; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ynQVfR/dFk01f+alSHWcouno+EVl3KD0yb8OXJAhnGc=; b=uOejOa6daVg17GSHfIFkbnzBN+rNTfEEeQr5/z4zsQQ3Gu1Q3+PLyejgJMTPmjh95w xtOr87llBmWByw24GJhCo+w2l/bg/S84Nxv7DCpcv+a60y0lx9wxX2t9UjXoDPYYnxF2 bVcdwnbKVIwFu1HhvOmOLkXWVRaoVCoX5dTGQz/aoDEtZoAtMI3mIcp2g6E5BVaL4WSm iPIzRSy8Ql3rwkujMRNRhL8cICUy8KNPYhLiYJ9xPPxWOvk9argL5sWlYIRubUHu3G6V JWj834TfJZLhO9EZGIy1Mu8MHMYx2lBZKUfi6Ify+0L8orHCXCoQWWTuRV5CfNitrc14 tU1g== X-Gm-Message-State: AOJu0YzO901uSbAXM1cxHH8RZEeD9vymKdOqKMTeh70CB29VMHUlcbpe 8uRqbFXzb56yPdvC6eTKFqLcQuiskP2xenVPwO2Oh1fhkwHUXK75dSCZbRmTDQ== X-Gm-Gg: ASbGncuj+rODwbIyjlA5UpT/RNWNtzqOb4wm63urabebqcEZadBUW3153FPCm25DQyR y0/cEIGQSmVQavNf0QbYUzikjCL/0akoq5ttogzLcpJpmD9ZTjKH1J4aTkViUPMgWVT5Xuo3/0j O3kAdGUspTiKuNlQ8R+nqX+lAa1miU5Y4b+a9/vibA1DYB5MBKPyf4xIwNwbwrCuIPWn9gUAKVN BD5VY0UDyk/yWa69voINU1Cr0f2LY9iLnr42L42pIDHzPrFkh6mLgYqhIAAU86uWbc8WmIPbm01 KP2W/ognyXUXLZe4N0wAX0DxdNTBMvTo8Q== X-Google-Smtp-Source: AGHT+IFhyE2QRwXal+GO2Cm8WYvshkYyOzXF4EFDI4EcmkmeGlVmtYav3txhL7PD6lMOsijhkF4Ggg== X-Received: by 2002:a05:6808:384b:b0:3f4:b0c:ab6 with SMTP id 5614622812f47-3f42469dfc3mr1403840b6e.1.1740096567630; Thu, 20 Feb 2025 16:09:27 -0800 (PST) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3f40b027906sm1573401b6e.42.2025.02.20.16.09.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2025 16:09:26 -0800 (PST) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v3 1/4] riscv: implement user_access_begin() and families Date: Fri, 21 Feb 2025 00:09:21 +0000 Message-Id: <20250221000924.734006-2-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250221000924.734006-1-cyrilbur@tenstorrent.com> References: <20250221000924.734006-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250220_160928_853924_0138B910 X-CRM114-Status: GOOD ( 15.14 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang Currently, when a function like strncpy_from_user() is called, the userspace access protection is disabled and enabled for every word read. By implementing user_access_begin() and families, the protection is disabled at the beginning of the copy and enabled at the end. The __inttype macro is borrowed from x86 implementation. Signed-off-by: Jisheng Zhang Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 63 ++++++++++++++++++++++++++++++++ 1 file changed, 63 insertions(+) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index fee56b0c8058..43db1d9c2f99 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -61,6 +61,19 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne #define __disable_user_access() \ __asm__ __volatile__ ("csrc sstatus, %0" : : "r" (SR_SUM) : "memory") +/* + * This is the smallest unsigned integer type that can fit a value + * (up to 'long long') + */ +#define __inttype(x) __typeof__( \ + __typefits(x,char, \ + __typefits(x,short, \ + __typefits(x,int, \ + __typefits(x,long,0ULL))))) + +#define __typefits(x,type,not) \ + __builtin_choose_expr(sizeof(x)<=sizeof(type),(unsigned type)0,not) + /* * The exception table consists of pairs of addresses: the first is the * address of an instruction that is allowed to fault, and the second is @@ -368,6 +381,56 @@ do { \ goto err_label; \ } while (0) +static __must_check __always_inline bool user_access_begin(const void __user *ptr, size_t len) +{ + if (unlikely(!access_ok(ptr,len))) + return 0; + __enable_user_access(); + return 1; +} +#define user_access_begin(a,b) user_access_begin(a,b) +#define user_access_end() __disable_user_access() + +static inline unsigned long user_access_save(void) { return 0UL; } +static inline void user_access_restore(unsigned long enabled) { } + +/* + * We want the unsafe accessors to always be inlined and use + * the error labels - thus the macro games. + */ +#define unsafe_put_user(x, ptr, label) do { \ + long __err = 0; \ + __put_user_nocheck(x, (ptr), __err); \ + if (__err) goto label; \ +} while (0) + +#define unsafe_get_user(x, ptr, label) do { \ + long __err = 0; \ + __inttype(*(ptr)) __gu_val; \ + __get_user_nocheck(__gu_val, (ptr), __err); \ + (x) = (__force __typeof__(*(ptr)))__gu_val; \ + if (__err) goto label; \ +} while (0) + +#define unsafe_copy_loop(dst, src, len, type, label) \ + while (len >= sizeof(type)) { \ + unsafe_put_user(*(type *)(src),(type __user *)(dst),label); \ + dst += sizeof(type); \ + src += sizeof(type); \ + len -= sizeof(type); \ + } + +#define unsafe_copy_to_user(_dst,_src,_len,label) \ +do { \ + char __user *__ucu_dst = (_dst); \ + const char *__ucu_src = (_src); \ + size_t __ucu_len = (_len); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u64, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u32, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u16, label); \ + unsafe_copy_loop(__ucu_dst, __ucu_src, __ucu_len, u8, label); \ +} while (0) + #else /* CONFIG_MMU */ #include #endif /* CONFIG_MMU */ From patchwork Fri Feb 21 00:09:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 13984663 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 972BCC021B4 for ; Fri, 21 Feb 2025 00:09:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=U8BBnsoc/kk0dSozfADzEC4gWWUWspuRXhpkI23qWaI=; b=16eDI6qmW788Rl 6msu/TygWcMmcsCnQhsEgnTy1Xh7y/UT8GXYZeySgxW/krMI/8UqpvdctWBy8/RB+ly+kwY7VUUEy A5Yd8v1Wm1oSWfphnOSW4KnMp7ghhF8FhmtGXgajNvSIQPMgVd7p8lt/oIlvpq2cqmK3x507X6DzU uAt1gePkJDKsz4bMaHUbvONm5xgn3+hpTmwwvSEPZTiOeAIOr5bBxkrVSRY347/McosK0pOuHhbOl K9V8EHA9VFNi29PyiCs5HR71PbQ7hF9j07PfkNUqMNIrlLBqXwQSB2+onr9JJUTrv/Vha3Ocmd3Ar IwpALiF4IzoC++AbkTsg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tlGbq-00000003SNd-0Gsq; Fri, 21 Feb 2025 00:09:38 +0000 Received: from mail-oi1-x231.google.com ([2607:f8b0:4864:20::231]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tlGbi-00000003SEP-0kD8 for linux-riscv@lists.infradead.org; Fri, 21 Feb 2025 00:09:34 +0000 Received: by mail-oi1-x231.google.com with SMTP id 5614622812f47-3f3f4890596so848227b6e.2 for ; Thu, 20 Feb 2025 16:09:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1740096569; x=1740701369; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=AVS2gJ1tGF7KfC44fJzPB6s+5bKduKUUF483rWDGJdM=; b=Uc/8Mf4Q06IbxHlo+cGHpvJB1AIBnLe3mcYN2zFlAQOWk4xgDNM6O8K+6Vstea6kL2 G9uPxCVdAk8o7yskNORi1u+gOxgpOK2r2ALpDwvGKnQrxtL4YjC2Hi4d3qucGshmLKR5 Dmv74IkGxZDJMUp6WsJDVSUS334kIpURZYjnAqdD3cLsNtpVWvlquRTSAPYHFfGEGLZu SRk1M6r1ZW5484Zrfvdr4ujyCl8b3GOiLSqD5x6sRK0nSewFunpo4V765bfIZBFt4FfB FFgRSXNc5oIqE+XJyVay3mR2H9z592GcoIDnbKOTquDtvaY+wsE4oRTdtZCC8LiR06xI 6HOQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740096569; x=1740701369; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AVS2gJ1tGF7KfC44fJzPB6s+5bKduKUUF483rWDGJdM=; b=jOSrq48KFZCgbSw3olknNiZKi9BmMt53RkmZku303xDoEZV8eMdUiVZ0zkmOZDelhI gfStGiFH1qYOLdNOcHoTCriP3qpg+C86W4gZtsRMYssFR1MJgfFtIuKDs5vfdImD3CJU DtntCWSAUeI+Wcv/67U3sSMwKzWAy0Bee+OfEtI9K+tweTfPbq2mGSjFbFQDYTQBJ+wE ICsDFveZ0SHybaTB/heV9iK5ORDfpG6KNtNKpW79KeQa/N1PV8gNX+Gh1rtVLbN8J/0/ Rve4/BiRcIUd9zN/iTCUx8D61rDs4DJpKEz1c2/aSEyJvGEHm/Y5HyX8XPx6ZyPaqYhO TvEQ== X-Gm-Message-State: AOJu0YxSv3GBjT0cU0ralzZehQD7Rqh5psHBZbEvWCFHduKEq+IBahQZ wbJjJm5VaOCWMxbwPsGmd6aKW+ANiBzeyMHYiXCA/XCYGtnP9qcyeFvuAjvIzg== X-Gm-Gg: ASbGncuuhdzf299so3poPaiJFOmIrY9Gp847Lp9nvlWjslrFYMrIMkpsmuUk/n20+vE GJvwXe4T3hkUZxNO1uGdouYvdT2DYOkCtLgBbdJdeuOfIr7P7qzuwOewMeuv0Pz6UKyILYkLDbb I1/97YNLQO6j52x5Ty1q6nl5VJyoXp8UJQqrXEfAt31MtD4vXtAkTZoc3H8Xns2twOUSIhdNA5w qrtS5FlUVPAa8OiJNgECeyfPg0PaexnxRPtnaryRQgvTqY4mBigHiTLhg5wsP3ExMyvr2GdJgQw 9eClE1YbrM4DhYGXeoVBKgX/XIo0/uKXMQ== X-Google-Smtp-Source: AGHT+IFKRfnHqdiP+z9IB6oatR7TvVXNmSxjSJMg1TAS9ThRlHl4CaTM35ogp20h33uxLtf3x9nOIw== X-Received: by 2002:a05:6808:1490:b0:3f3:db88:841c with SMTP id 5614622812f47-3f425a75607mr599965b6e.9.1740096568994; Thu, 20 Feb 2025 16:09:28 -0800 (PST) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3f40b027906sm1573401b6e.42.2025.02.20.16.09.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2025 16:09:28 -0800 (PST) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v3 2/4] riscv: uaccess: use input constraints for ptr of __put_user() Date: Fri, 21 Feb 2025 00:09:22 +0000 Message-Id: <20250221000924.734006-3-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250221000924.734006-1-cyrilbur@tenstorrent.com> References: <20250221000924.734006-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250220_160930_233646_E781D214 X-CRM114-Status: GOOD ( 10.52 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang Putting ptr in the inputs as opposed to output may seem incorrect but this is done for a few reasons: - Not having it in the output permits the use of asm goto in a subsequent patch. There are bugs in gcc [1] which would otherwise prevent it. - Since the output memory is userspace there isn't any real benefit from telling the compiler about the memory clobber. - x86, arm and powerpc all use this technique. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index 43db1d9c2f99..fb3961ecf38b 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -219,11 +219,11 @@ do { \ __typeof__(*(ptr)) __x = x; \ __asm__ __volatile__ ( \ "1:\n" \ - " " insn " %z2, %1\n" \ + " " insn " %z1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err), "=m" (*(ptr)) \ - : "rJ" (__x)); \ + : "+r" (err) \ + : "rJ" (__x), "m"(*(ptr))); \ } while (0) #ifdef CONFIG_64BIT @@ -236,16 +236,16 @@ do { \ u64 __x = (__typeof__((x)-(x)))(x); \ __asm__ __volatile__ ( \ "1:\n" \ - " sw %z3, %1\n" \ + " sw %z1, %3\n" \ "2:\n" \ - " sw %z4, %2\n" \ + " sw %z2, %4\n" \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err), \ - "=m" (__ptr[__LSW]), \ - "=m" (__ptr[__MSW]) \ - : "rJ" (__x), "rJ" (__x >> 32)); \ + : "+r" (err) \ + : "rJ" (__x), "rJ" (__x >> 32), \ + "m" (__ptr[__LSW]), \ + "m" (__ptr[__MSW])); \ } while (0) #endif /* CONFIG_64BIT */ From patchwork Fri Feb 21 00:09:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 13984664 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2606EC021B6 for ; Fri, 21 Feb 2025 00:09:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7Vcn6bO9kVXq+cSz1oQLlSFkt3FA+IlEC4H/jnjuw9Q=; b=SciN+XS2tFKHZk lk6RE9E8FvoxLFcaL02dymVsRuKlwVssctOOxwAyNxVjrh6QlopCpCJa4XT+o0vu9/D2uOwIpKVhm zReKbh9yCxjNf1gnScAOa0HjP5SoGk1BMZAE4brjhRmExiXBIjdiO9euAUKsk456RHGBgDCUuj3uJ FsDJPA6V/hhbOcwAf0i2yuB/RgmOQg2lSLzQKjKATfn23p2spFBFv4r19gDiPr9z0VTAVus/CQaIH b7UETnAxgmW3kdAWtwumv6ktsy/vikMBalniRSPHLJ0fryfVb75QAjURANhiGY8wxbbOo8GgFjB52 Deaj91tdePicJvXpVg2A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tlGbp-00000003SMV-0dne; Fri, 21 Feb 2025 00:09:37 +0000 Received: from mail-oi1-x229.google.com ([2607:f8b0:4864:20::229]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tlGbj-00000003SFF-05x0 for linux-riscv@lists.infradead.org; Fri, 21 Feb 2025 00:09:34 +0000 Received: by mail-oi1-x229.google.com with SMTP id 5614622812f47-3f4104f0c4cso824126b6e.0 for ; Thu, 20 Feb 2025 16:09:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1740096570; x=1740701370; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=l5hM23iRroiayTpV3ONpzy5pooTVMbj1TF8BFXhb4r4=; b=hUzoy3qUc/fuiwLrJzh08LaYsx5+8EgFghy+gpthzPcgUSbyTdo95Lzwuc7o0ZVO8d 70BGACTliKGJuQZ9GisAo0TtNjeEyxvvNLePQCNKSXH0X9d8yfyJbJCXG8dCuc87SOmd x2opeB2wfwpbt3tSo36LiddjK9FbE4p3C4VAMAxe7rvPWGv9iGX88J87MFu/CRXYpMIx KMeFLgH094oa1xjaOHBwxjbf9cYOI7lqoR/ghLnHgtkNDFYjNuPNMHmk0gM/m4Z/oNif W93Jsbs0AP2iIFXq6VOU+0ayJPrg24RikvZp9BivinO0eFls2QBA6C4Jhmv+t9j7U6kR OdxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740096570; x=1740701370; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l5hM23iRroiayTpV3ONpzy5pooTVMbj1TF8BFXhb4r4=; b=sO+isOmDxyXDX8bJYC5uGnQ7BZQN8MKowUhkZZZrblCk4wrN8G0u+TXrZeMRY2NrM0 AWyE5vJolqEV1gdjQSsvV/K2XAl7MTAWk+VDqBEloIPq8ERYHoHg84BAx++2lRVJYMw/ gIn2a2CCxWkwMob97Ga3XJD3S9mwNqRMMKzzBNPtiwDm+Agoje4ijjXJ6MKEKEwdQfzW VKPPURsgWUnycYUI7fDDbNPqHOZF5eHBkaX1H1pP1x0ej55nWITztjSmV29yjYxQLkp+ BpVbNmfOiUJDJ+015vDhnij0LKMaxOj+1XLaZAxu+Dt5xikxeDcTX7JURqDWRotbYJdB bdBQ== X-Gm-Message-State: AOJu0Ywlmdqp3Iu0l9F4dkG6QaSTklFUMJVsA3bVqei+lrjlJpTjI55P WDamt44/T+eV7uxcLBPP/0MKr2bUaxcqKeuuK9+RDRfwIEA/BCY1Wxn9TFaZ5w== X-Gm-Gg: ASbGncvfO+wYiaUIxsUNdjxa3jb8bXI5CwU3W2awu/5/QSM2PrRZfv3xUYv4peOAqWG zAdfWYs5EBJncV04TDxYwD6oFudgueLPP8EBzs8S3IJ5Q+JVYxXbx2Zh3F4xhPovwjoatWWzWPa 7u7ITd4DeO2p0A/IQdRtpV3e+jqykqNg7YdCSgbcHMvL9ekwxfQTnmaO3YfhmJjvYcRGGNbqov2 O7DYKKs6EZ3m5GOF7ahHx80xT77U1zyvkrVDaniAK9RYgRq/rf17kixa4GpaGVor2QmRcjLUuX3 +NOlSI7GQElczPXrW8J2x4p0J/N3ZZFppw== X-Google-Smtp-Source: AGHT+IF2JM2LKC17wpZxVEWOnlCgEtBUNVIC6uqX4gRfMQHIPsGUXAqsCDrHVc3eY40FDoWxYpcitw== X-Received: by 2002:a05:6808:3191:b0:3f4:133d:f18 with SMTP id 5614622812f47-3f4247ecff3mr1177279b6e.34.1740096569876; Thu, 20 Feb 2025 16:09:29 -0800 (PST) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3f40b027906sm1573401b6e.42.2025.02.20.16.09.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2025 16:09:29 -0800 (PST) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v3 3/4] riscv: uaccess: use 'asm goto' for put_user() Date: Fri, 21 Feb 2025 00:09:23 +0000 Message-Id: <20250221000924.734006-4-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250221000924.734006-1-cyrilbur@tenstorrent.com> References: <20250221000924.734006-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250220_160931_065200_12BC0CA8 X-CRM114-Status: GOOD ( 13.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Because there are no output clobbers which could trigger gcc bugs [1] the use of asm_goto_output() macro is not necessary here. Not using asm_goto_output() is desirable as the generated output asm will be cleaner. Use of the volatile keyword is redundant as per gcc 14.2.0 manual section 6.48.2.7 Goto Labels: > Also note that an asm goto statement is always implicitly considered volatile. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 70 +++++++++++++++----------------- 1 file changed, 33 insertions(+), 37 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index fb3961ecf38b..1b926b61af66 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -214,61 +214,66 @@ do { \ ((x) = (__force __typeof__(x))0, -EFAULT); \ }) -#define __put_user_asm(insn, x, ptr, err) \ +#define __put_user_asm(insn, x, ptr, label) \ do { \ __typeof__(*(ptr)) __x = x; \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " " insn " %z1, %2\n" \ - "2:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 2b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "m"(*(ptr))); \ + " " insn " %z0, %1\n" \ + _ASM_EXTABLE(1b, %l2) \ + : : "rJ" (__x), "m"(*(ptr)) : : label); \ } while (0) #ifdef CONFIG_64BIT -#define __put_user_8(x, ptr, err) \ - __put_user_asm("sd", x, ptr, err) +#define __put_user_8(x, ptr, label) \ + __put_user_asm("sd", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __put_user_8(x, ptr, err) \ +#define __put_user_8(x, ptr, label) \ do { \ u32 __user *__ptr = (u32 __user *)(ptr); \ u64 __x = (__typeof__((x)-(x)))(x); \ - __asm__ __volatile__ ( \ + asm goto( \ "1:\n" \ - " sw %z1, %3\n" \ + " sw %z0, %2\n" \ "2:\n" \ - " sw %z2, %4\n" \ - "3:\n" \ - _ASM_EXTABLE_UACCESS_ERR(1b, 3b, %0) \ - _ASM_EXTABLE_UACCESS_ERR(2b, 3b, %0) \ - : "+r" (err) \ - : "rJ" (__x), "rJ" (__x >> 32), \ + " sw %z1, %3\n" \ + _ASM_EXTABLE(1b, %l4) \ + _ASM_EXTABLE(2b, %l4) \ + : : "rJ" (__x), "rJ" (__x >> 32), \ "m" (__ptr[__LSW]), \ - "m" (__ptr[__MSW])); \ + "m" (__ptr[__MSW]) : : label); \ } while (0) #endif /* CONFIG_64BIT */ -#define __put_user_nocheck(x, __gu_ptr, __pu_err) \ +#define __put_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __put_user_asm("sb", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sb", (x), __gu_ptr, label); \ break; \ case 2: \ - __put_user_asm("sh", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sh", (x), __gu_ptr, label); \ break; \ case 4: \ - __put_user_asm("sw", (x), __gu_ptr, __pu_err); \ + __put_user_asm("sw", (x), __gu_ptr, label); \ break; \ case 8: \ - __put_user_8((x), __gu_ptr, __pu_err); \ + __put_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) +#define __put_user_error(x, ptr, err) \ +do { \ + __label__ err_label; \ + __put_user_nocheck(x, ptr, err_label); \ + break; \ +err_label: \ + (err) = -EFAULT; \ +} while (0) + /** * __put_user: - Write a simple value into user space, with less checking. * @x: Value to copy to user space. @@ -299,7 +304,7 @@ do { \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __put_user_nocheck(__val, __gu_ptr, __pu_err); \ + __put_user_error(__val, __gu_ptr, __pu_err); \ __disable_user_access(); \ \ __pu_err; \ @@ -373,13 +378,7 @@ do { \ } while (0) #define __put_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err = 0; \ - \ - __put_user_nocheck(*((type *)(src)), (type *)(dst), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) static __must_check __always_inline bool user_access_begin(const void __user *ptr, size_t len) { @@ -398,11 +397,8 @@ static inline void user_access_restore(unsigned long enabled) { } * We want the unsafe accessors to always be inlined and use * the error labels - thus the macro games. */ -#define unsafe_put_user(x, ptr, label) do { \ - long __err = 0; \ - __put_user_nocheck(x, (ptr), __err); \ - if (__err) goto label; \ -} while (0) +#define unsafe_put_user(x, ptr, label) \ + __put_user_nocheck(x, (ptr), label) #define unsafe_get_user(x, ptr, label) do { \ long __err = 0; \ From patchwork Fri Feb 21 00:09:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Cyril Bur X-Patchwork-Id: 13984665 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0FAC6C021B5 for ; Fri, 21 Feb 2025 00:09:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=QNGULMRvQBKQlaCGwCbl/lfa7A13cPeL9YlnnEbaoG0=; b=zdYF4ae6kPwZBg 4a/vXW6mcO/vgAIsOua7fn/brpicBJtaGBFfqUmgLhKQjG6Xl6MxtH0+KeAppsHyMF4OiRP8QTP5N pb0f2mOzvpuHtO2y8spdSkg0wZmsupI8IIWYzR/s1lD378AWz4rLXJG6CegTpXBZIs0yUzm3AITlj DVp8rum+GhHxCWYYphIrydr5iNx+vZw4AwJn7FEvzjIP2XAycdbJkl+SjDg/RClclIEDbMoXyVLzw Rh3OpLO65lJY9QCVMekhPmr9tIqcyq4xJ5fMwX8AKiXPOXo0yvzVrT3iDLDh6a1I0e17OW0x2oFUP +aSqEZt+IgpeSHBUIKRw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tlGbo-00000003SLk-1yCX; Fri, 21 Feb 2025 00:09:36 +0000 Received: from mail-oi1-x232.google.com ([2607:f8b0:4864:20::232]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tlGbk-00000003SGw-23IL for linux-riscv@lists.infradead.org; Fri, 21 Feb 2025 00:09:34 +0000 Received: by mail-oi1-x232.google.com with SMTP id 5614622812f47-3f3ecfa7a76so749537b6e.2 for ; Thu, 20 Feb 2025 16:09:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tenstorrent.com; s=google; t=1740096571; x=1740701371; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+knCqqnB60be5uWSQ10xJ1S6sb+4Be8XtAoVkEDcmKA=; b=DOQZbDy9vsbAicmugkhdc86WOb0PAbLT9AUfVsCJ/XXajolHquXKqObwE5Us7R6NWx PSS/+Vlj81sFldZO0b67GbH+iswI8oaSDG0wi4qCdqymGSQkPeJz8LZiP2/Vr5/nriuN 5u+h5xmswc8TZg1ag39jApXhb5+ZqzvLPVhw5K6mb/ObiI2kJw322d9iOzaS//VEzSiJ nc76kvxdtbpgJKKB7CYFe+beF8sG6Fb47Ia/RFxybzjBbDcbymXN8vvZGpQDA1JSMwjv ZyVZEZ7WqgX8gdFPGejRyB8CTNyc4CeKBr3ibgk2gI4ApodMk3V5SaKB8x0K9yBkyWiq NqxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1740096571; x=1740701371; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+knCqqnB60be5uWSQ10xJ1S6sb+4Be8XtAoVkEDcmKA=; b=sVdHBPsLuU3sqxO3tC+6JdYKHO/7yJnKxuwH3NW3Av0vCUSPF9hA4dDZl2QExkECwc 5AePtUAFCz8MD6oAa6NUoEjm36QxHETZ8S8/L5Qz7dhKOGAZG2nZbTBcSkisKvG8swxB PnCODJqz92oVzatn8wTsv8H6iq9WkgDLWMvNfTzI/AnTKC90Sezo0gZp986nZpwp30CW wnDban4TwVWZY6QkW7niOiOG+czhebRAyOkiaGfmVqo4A/vL6JlTaFHPU8EloRZSbZt1 s80+FYb2KmbdLXR1R6jWihr+al86nVx3Hc/TbSR9JeZmAymN4CDU0+dkWEYXW9sQwJKI WAig== X-Gm-Message-State: AOJu0YzX6h5UeAsXqhPGowL9c7MzDAac+nzc8CyoyA+PDX4eluqSgg7y 0gDmJdnr7GX/4HiwXKrhHzdnvn+HcqPG7PoDFVUdi1cLjLyJ5u8etoxEoFm3Xg== X-Gm-Gg: ASbGnctNjV+mTBprFUJa7DY2sjDPpUDqFji3XEnmMM3KctTJ9JQ8oXtDtABsMPEJYxa EqMAWJU7yUjSuzfupQbSuiv+p6vTCKt5+5dxXd3iFh6lqd4ixHYRtedt1Ma5eyr06AvtCMGzyz+ LVwQxXmGLRZChKYEVWyyZP48CspPKuB2/a8QICywtI1dmOnMaopX+sy45jPGxZmR8lsUxIWk3PG XXuBhtK26wRb4tLZ9Af2L4Mgu4+eMithclrSnLE3nK8g6Fh8H5C6bL+87oZH5ka2z+/zFw/oGQI Q7IMj8eb+FfQq5dYwGfHDq16mHSx99QZ3g== X-Google-Smtp-Source: AGHT+IEMGnDVwYsV62cBocsB6VRb+qRff3a2UiPYmr/fXmxdBlq3NZLsTutH+x4Vzk0cSqwcF1F8Ow== X-Received: by 2002:a05:6808:384e:b0:3f4:d61:36cf with SMTP id 5614622812f47-3f4247a204fmr1275951b6e.30.1740096571285; Thu, 20 Feb 2025 16:09:31 -0800 (PST) Received: from aus-ird.tenstorrent.com ([38.104.49.66]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3f40b027906sm1573401b6e.42.2025.02.20.16.09.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 20 Feb 2025 16:09:30 -0800 (PST) From: Cyril Bur To: palmer@dabbelt.com, aou@eecs.berkeley.edu, paul.walmsley@sifive.com, charlie@rivosinc.com, jrtc27@jrtc27.com, ben.dooks@codethink.co.uk Cc: linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, jszhang@kernel.org Subject: [PATCH v3 4/4] riscv: uaccess: use 'asm_goto_output' for get_user() Date: Fri, 21 Feb 2025 00:09:24 +0000 Message-Id: <20250221000924.734006-5-cyrilbur@tenstorrent.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20250221000924.734006-1-cyrilbur@tenstorrent.com> References: <20250221000924.734006-1-cyrilbur@tenstorrent.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250220_160932_537443_96E30C75 X-CRM114-Status: GOOD ( 13.30 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org From: Jisheng Zhang With 'asm goto' we don't need to test the error etc, the exception just jumps to the error handling directly. Unlike put_user(), get_user() must work around GCC bugs [1] when using output clobbers in an asm goto statement. Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=113921 # 1 Signed-off-by: Jisheng Zhang [Cyril Bur: Rewritten commit message] Signed-off-by: Cyril Bur --- arch/riscv/include/asm/uaccess.h | 90 +++++++++++++++++++++++--------- 1 file changed, 65 insertions(+), 25 deletions(-) diff --git a/arch/riscv/include/asm/uaccess.h b/arch/riscv/include/asm/uaccess.h index 1b926b61af66..ab48ef57565e 100644 --- a/arch/riscv/include/asm/uaccess.h +++ b/arch/riscv/include/asm/uaccess.h @@ -96,27 +96,56 @@ static inline unsigned long __untagged_addr_remote(struct mm_struct *mm, unsigne * call. */ -#define __get_user_asm(insn, x, ptr, err) \ +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_asm(insn, x, ptr, label) \ + asm_goto_output( \ + "1:\n" \ + " " insn " %0, %1\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l2, %0) \ + : "=&r" (x) \ + : "m" (*(ptr)) : : label) +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_asm(insn, x, ptr, label) \ do { \ - __typeof__(x) __x; \ + long __gua_err = 0; \ __asm__ __volatile__ ( \ "1:\n" \ " " insn " %1, %2\n" \ "2:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 2b, %0, %1) \ - : "+r" (err), "=&r" (__x) \ + : "+r" (__gua_err), "=&r" (x) \ : "m" (*(ptr))); \ - (x) = __x; \ + if (__gua_err) \ + goto label; \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ #ifdef CONFIG_64BIT -#define __get_user_8(x, ptr, err) \ - __get_user_asm("ld", x, ptr, err) +#define __get_user_8(x, ptr, label) \ + __get_user_asm("ld", x, ptr, label) #else /* !CONFIG_64BIT */ -#define __get_user_8(x, ptr, err) \ + +#ifdef CONFIG_CC_HAS_ASM_GOTO_OUTPUT +#define __get_user_8(x, ptr, label) \ + u32 __user *__ptr = (u32 __user *)(ptr); \ + u32 __lo, __hi; \ + asm_goto_output( \ + "1:\n" \ + " lw %0, %2\n" \ + "2:\n" \ + " lw %1, %3\n" \ + _ASM_EXTABLE_UACCESS_ERR(1b, %l4, %0) \ + _ASM_EXTABLE_UACCESS_ERR(2b, %l4, %0) \ + : "=&r" (__lo), "=r" (__hi) \ + : "m" (__ptr[__LSW]), "m" (__ptr[__MSW]) \ + : : label) + +#else /* !CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ +#define __get_user_8(x, ptr, label) \ do { \ u32 __user *__ptr = (u32 __user *)(ptr); \ u32 __lo, __hi; \ + long __gu8_err = 0; \ __asm__ __volatile__ ( \ "1:\n" \ " lw %1, %3\n" \ @@ -125,35 +154,51 @@ do { \ "3:\n" \ _ASM_EXTABLE_UACCESS_ERR_ZERO(1b, 3b, %0, %1) \ _ASM_EXTABLE_UACCESS_ERR_ZERO(2b, 3b, %0, %1) \ - : "+r" (err), "=&r" (__lo), "=r" (__hi) \ + : "+r" (__gu8_err), "=&r" (__lo), "=r" (__hi) \ : "m" (__ptr[__LSW]), "m" (__ptr[__MSW])); \ - if (err) \ + if (__gu8_err) { \ __hi = 0; \ + goto label; \ + } \ (x) = (__typeof__(x))((__typeof__((x)-(x)))( \ (((u64)__hi << 32) | __lo))); \ } while (0) +#endif /* CONFIG_CC_HAS_ASM_GOTO_OUTPUT */ + #endif /* CONFIG_64BIT */ -#define __get_user_nocheck(x, __gu_ptr, __gu_err) \ +#define __get_user_nocheck(x, __gu_ptr, label) \ do { \ switch (sizeof(*__gu_ptr)) { \ case 1: \ - __get_user_asm("lb", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lb", (x), __gu_ptr, label); \ break; \ case 2: \ - __get_user_asm("lh", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lh", (x), __gu_ptr, label); \ break; \ case 4: \ - __get_user_asm("lw", (x), __gu_ptr, __gu_err); \ + __get_user_asm("lw", (x), __gu_ptr, label); \ break; \ case 8: \ - __get_user_8((x), __gu_ptr, __gu_err); \ + __get_user_8((x), __gu_ptr, label); \ break; \ default: \ BUILD_BUG(); \ } \ } while (0) +#define __get_user_error(x, ptr, err) \ +do { \ + __label__ __gu_failed; \ + \ + __get_user_nocheck(x, ptr, __gu_failed); \ + err = 0; \ + break; \ +__gu_failed: \ + x = 0; \ + err = -EFAULT; \ +} while (0) + /** * __get_user: - Get a simple variable from user space, with less checking. * @x: Variable to store result. @@ -178,13 +223,16 @@ do { \ ({ \ const __typeof__(*(ptr)) __user *__gu_ptr = untagged_addr(ptr); \ long __gu_err = 0; \ + __typeof__(x) __gu_val; \ \ __chk_user_ptr(__gu_ptr); \ \ __enable_user_access(); \ - __get_user_nocheck(x, __gu_ptr, __gu_err); \ + __get_user_error(__gu_val, __gu_ptr, __gu_err); \ __disable_user_access(); \ \ + (x) = __gu_val; \ + \ __gu_err; \ }) @@ -369,13 +417,7 @@ unsigned long __must_check clear_user(void __user *to, unsigned long n) } #define __get_kernel_nofault(dst, src, type, err_label) \ -do { \ - long __kr_err = 0; \ - \ - __get_user_nocheck(*((type *)(dst)), (type *)(src), __kr_err); \ - if (unlikely(__kr_err)) \ - goto err_label; \ -} while (0) + __get_user_nocheck(*((type *)(dst)), (type *)(src), err_label) #define __put_kernel_nofault(dst, src, type, err_label) \ __put_user_nocheck(*((type *)(src)), (type *)(dst), err_label) @@ -401,11 +443,9 @@ static inline void user_access_restore(unsigned long enabled) { } __put_user_nocheck(x, (ptr), label) #define unsafe_get_user(x, ptr, label) do { \ - long __err = 0; \ __inttype(*(ptr)) __gu_val; \ - __get_user_nocheck(__gu_val, (ptr), __err); \ + __get_user_nocheck(__gu_val, (ptr), label); \ (x) = (__force __typeof__(*(ptr)))__gu_val; \ - if (__err) goto label; \ } while (0) #define unsafe_copy_loop(dst, src, len, type, label) \