From patchwork Thu Jan 13 23:39:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Oskolkov X-Patchwork-Id: 12713191 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B887C43217 for ; Thu, 13 Jan 2022 23:40:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DDE146B0074; Thu, 13 Jan 2022 18:40:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id D67066B0075; Thu, 13 Jan 2022 18:40:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BE1D36B0078; Thu, 13 Jan 2022 18:40:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0237.hostedemail.com [216.40.44.237]) by kanga.kvack.org (Postfix) with ESMTP id B01E36B0074 for ; Thu, 13 Jan 2022 18:40:08 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 6D001181D12E2 for ; Thu, 13 Jan 2022 23:40:08 +0000 (UTC) X-FDA: 79026884496.12.C8D9F43 Received: from mail-yb1-f201.google.com (mail-yb1-f201.google.com [209.85.219.201]) by imf18.hostedemail.com (Postfix) with ESMTP id 000011C0008 for ; Thu, 13 Jan 2022 23:40:07 +0000 (UTC) Received: by mail-yb1-f201.google.com with SMTP id s89-20020a25aa62000000b00611afc92630so10514541ybi.17 for ; Thu, 13 Jan 2022 15:40:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=wwJdGAN5osUC9HM7kKzj2Htmsov37Cc7r1NsvxZMlRg=; b=EqRlBPehovANSdBzY2evi/EKQYY+q7HLOecxR/MagzYR2LxVFJ2uK4VkjufpfGFEzx KIeCSUfIoiL1qoE5OPQTr4qRMnceLv7+32QM4Hl7gPahtlj2cTemKA31nN2Df8LfZhZa El2EqLCJCO1yQ/3YpQ/0XwnC8W2XEBXUm3dbCBExBLp/IGMMCLUgBq+B94A1I13rRSAN ukHIOytgnjWR7LSixoc/8BtU4PvqqLx30PpXNj3fgwOdLAk53TWQvhTQkjw3G0wKpYC8 PUt8wipSgOqEFQ+Y2EWyJvcrby1YZ8yF2OF7TvyETom5WyTQmwBDMAfStsNe41S0zUdC 0UTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=wwJdGAN5osUC9HM7kKzj2Htmsov37Cc7r1NsvxZMlRg=; b=fhpF33jnVdPobIQoiIuzyj1rmUHpy298hpUgD/b9IhtRlQO9jh/ftx6iDQ8w+5L55y 00xPvmUEu6tUkBon0ptO2yJw62CFhJ0YByZe3qDktVfg7udqCvdOTxbSXUUoNqsrOzTM PEnGAFw29I/ooGZAFqbBNdjiLXbtVbjSnsNDJaDZAPJu2Vbk8ZRW6szCh5AXkxHswLC2 EsUodg8UBEJRkhiCwGD0VYmxzVSPTRogHIMGgcnkIr7q96YmR3ja4uQAlApZkxvhKwgl UOX2MJD1/m+1IXGi1bVEB5ORMSCpOmo/KtPB97SKSnRj0UExjAdetQ1gX9NZT1J989Rh JNDQ== X-Gm-Message-State: AOAM532fM9iGQlyBzfitXiJb/OQA5q1dq2Ae/fPtJuOz0I+jyfqehzqG /imWON8Zw55NAApppzbDs1c9/FXS X-Google-Smtp-Source: ABdhPJzzGWHCGjZGF5k35EtO1RjDkQh8pxMnUpvwv8nSbHJHv8giYUEy9UlBMk+wUGWLFmwcfXW/lLPr X-Received: from posk.svl.corp.google.com ([2620:15c:2cd:202:c548:e79f:8954:121f]) (user=posk job=sendgmr) by 2002:a25:6d09:: with SMTP id i9mr4958623ybc.703.1642117206900; Thu, 13 Jan 2022 15:40:06 -0800 (PST) Date: Thu, 13 Jan 2022 15:39:37 -0800 In-Reply-To: <20220113233940.3608440-1-posk@google.com> Message-Id: <20220113233940.3608440-3-posk@google.com> Mime-Version: 1.0 References: <20220113233940.3608440-1-posk@google.com> X-Mailer: git-send-email 2.34.1.703.g22d0c6ccf7-goog Subject: [RFC PATCH v2 2/5] x86/uaccess: Implement unsafe_try_cmpxchg_user() From: Peter Oskolkov To: Peter Zijlstra , mingo@redhat.com, tglx@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, bristot@redhat.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-api@vger.kernel.org, x86@kernel.org, pjt@google.com, posk@google.com, avagin@google.com, jannh@google.com, tdelisle@uwaterloo.ca, posk@posk.io X-Stat-Signature: eugh5uftdre4cqje8199oqih1wpd8csz Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=EqRlBPeh; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf18.hostedemail.com: domain of 3VrjgYQQKCF8MLPHDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--posk.bounces.google.com designates 209.85.219.201 as permitted sender) smtp.mailfrom=3VrjgYQQKCF8MLPHDLLDIB.9LJIFKRU-JJHS79H.LOD@flex--posk.bounces.google.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 000011C0008 X-HE-Tag: 1642117207-485152 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Peter Zijlstra Do try_cmpxchg() loops on userspace addresses. Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/uaccess.h | 57 ++++++++++++++++++++++++++++++++++ 1 file changed, 57 insertions(+) diff --git a/arch/x86/include/asm/uaccess.h b/arch/x86/include/asm/uaccess.h index ac96f9b2d64b..8277ec05be02 100644 --- a/arch/x86/include/asm/uaccess.h +++ b/arch/x86/include/asm/uaccess.h @@ -342,6 +342,24 @@ do { \ : [umem] "m" (__m(addr)) \ : : label) +#define __try_cmpxchg_user_asm(itype, _ptr, _pold, _new, label) ({ \ + bool success; \ + __typeof__(_ptr) _old = (__typeof__(_ptr))(_pold); \ + __typeof__(*(_ptr)) __old = *_old; \ + __typeof__(*(_ptr)) __new = (_new); \ + asm_volatile_goto("\n" \ + "1: " LOCK_PREFIX "cmpxchg"itype" %[new], %[ptr]\n"\ + _ASM_EXTABLE_UA(1b, %l[label]) \ + : CC_OUT(z) (success), \ + [ptr] "+m" (*_ptr), \ + [old] "+a" (__old) \ + : [new] "r" (__new) \ + : "memory", "cc" \ + : label); \ + if (unlikely(!success)) \ + *_old = __old; \ + likely(success); }) + #else // !CONFIG_CC_HAS_ASM_GOTO_OUTPUT #ifdef CONFIG_X86_32 @@ -407,6 +425,30 @@ do { \ : [umem] "m" (__m(addr)), \ "0" (err)) +#define __try_cmpxchg_user_asm(itype, _ptr, _pold, _new, label) ({ \ + int __err = 0; \ + bool success; \ + __typeof__(_ptr) _old = (__typeof__(_ptr))(_pold); \ + __typeof__(*(_ptr)) __old = *_old; \ + __typeof__(*(_ptr)) __new = (_new); \ + asm volatile("\n" \ + "1: " LOCK_PREFIX "cmpxchg"itype" %[new], %[ptr]\n"\ + CC_SET(z) \ + "2:\n" \ + _ASM_EXTABLE_TYPE_REG(1b, 2b, EX_TYPE_EFAULT_REG, \ + %[errout]) \ + : CC_OUT(z) (success), \ + [errout] "+r" (__err), \ + [ptr] "+m" (*_ptr), \ + [old] "+a" (__old) \ + : [new] "r" (__new) \ + : "memory", "cc"); \ + if (unlikely(__err)) \ + goto label; \ + if (unlikely(!success)) \ + *_old = __old; \ + likely(success); }) + #endif // CONFIG_CC_HAS_ASM_GOTO_OUTPUT /* FIXME: this hack is definitely wrong -AK */ @@ -501,6 +543,21 @@ do { \ } while (0) #endif // CONFIG_CC_HAS_ASM_GOTO_OUTPUT +extern void __try_cmpxchg_user_wrong_size(void); + +#define unsafe_try_cmpxchg_user(_ptr, _oldp, _nval, _label) ({ \ + __typeof__(*(_ptr)) __ret; \ + switch (sizeof(__ret)) { \ + case 4: __ret = __try_cmpxchg_user_asm("l", (_ptr), (_oldp), \ + (_nval), _label); \ + break; \ + case 8: __ret = __try_cmpxchg_user_asm("q", (_ptr), (_oldp), \ + (_nval), _label); \ + break; \ + default: __try_cmpxchg_user_wrong_size(); \ + } \ + __ret; }) + /* * We want the unsafe accessors to always be inlined and use * the error labels - thus the macro games.