From patchwork Sat Oct 22 11:14:15 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Zijlstra X-Patchwork-Id: 13015940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7EE02C04A95 for ; Sat, 22 Oct 2022 11:48:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 461E28E0008; Sat, 22 Oct 2022 07:48:37 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 065728E000D; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CECE38E0008; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 800F08E0007 for ; Sat, 22 Oct 2022 07:48:36 -0400 (EDT) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 61D901C3042 for ; Sat, 22 Oct 2022 11:48:36 +0000 (UTC) X-FDA: 80048413032.06.54E08D2 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf23.hostedemail.com (Postfix) with ESMTP id 0BAAD140002 for ; Sat, 22 Oct 2022 11:48:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: Subject:Cc:To:From:Date:Message-ID:Sender:Reply-To:Content-Transfer-Encoding: Content-ID:Content-Description:In-Reply-To; bh=vmo888klJOYTK6NXdYxj1ylLTVw6+U7xUyde2QpBL2o=; b=bYr1reyz4HyDuluM2y+c4SgJM0 PBJN8uYkZxEcRHT3GePK91yBuUwfsfegkY9kOgusA3DzeACZX107P9rwRsGS+B5vmcl2jCFpRqPPq dY+JhH8JW06eGjLRNrjQfsUO/3WvAUW0Mnr0U8/5MItJ1KB06PnuMLz9AKqkVjcZ+3ImKa4B4PPbx MXfbYcCiAe9q7O28LJ1ssPhDG1tuqtRmAMRil+8BvHeMkUevTn5g569pTJJaTks3FEbOAnHuEgDx5 13zre+JP7+tEWJSrresAW1tKhUL7Mnjcjod2QJFzkI1sVeA4Id2JzSbvBWj8QCFumXkoCwc6YQj9L ka4hjhhw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1omCzR-00Dtgv-LC; Sat, 22 Oct 2022 11:48:33 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id CA46F303106; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 0) id 46C8E28B8E519; Sat, 22 Oct 2022 13:48:26 +0200 (CEST) Message-ID: <20221022114425.233481884@infradead.org> User-Agent: quilt/0.66 Date: Sat, 22 Oct 2022 13:14:15 +0200 From: Peter Zijlstra To: x86@kernel.org, willy@infradead.org, torvalds@linux-foundation.org, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, peterz@infradead.org, linux-mm@kvack.org, aarcange@redhat.com, kirill.shutemov@linux.intel.com, jroedel@suse.de, ubizjak@gmail.com Subject: [PATCH 12/13] x86/mm/pae: Get rid of set_64bit() References: <20221022111403.531902164@infradead.org> MIME-Version: 1.0 ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bYr1reyz; spf=none (imf23.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1666439316; a=rsa-sha256; cv=none; b=F2se0EfL/NPKsYwjfWUfNTqSeKfYxioasNqfzdtLKhu7zgVZvBReI+7TUnsMU7tVqoMyPQ aIpDWNjTW5p/SnYWySkWEogvcpgGJ2fJv3/PD06uOtp866vDCK6pPKQ8OetW3ULB2fBcR0 7OT2R+z+uIOey9jlyqhjV5ZnV8m8Tb0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1666439316; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:references:dkim-signature; bh=vmo888klJOYTK6NXdYxj1ylLTVw6+U7xUyde2QpBL2o=; b=uCWH19SKbxFOaZdLRVWf7wnSFeLYLszwi9ly6UgE/BrucOBqjCTN5fgAfSIZN9SJP+GELG a2Z5ywjGh4SdNk/HvLb4lSalmcrzDzK2DVCESHy5zjUgNyBfxquStWshaVGmam+iYqE4R/ zD+IpSzKiiQ07E6a8CQaU/L4zrdmV/A= Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=bYr1reyz; spf=none (imf23.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none X-Stat-Signature: 7ic5fugjj5hstgpmdx9xgc3oikhx8rpu X-Rspamd-Queue-Id: 0BAAD140002 X-Rspamd-Server: rspam02 X-Rspam-User: X-HE-Tag: 1666439315-212092 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Recognise that set_64bit() is a special case of our previously introduced pxx_xchg64(), so use that and get rid of set_64bit(). Signed-off-by: Peter Zijlstra (Intel) --- arch/x86/include/asm/cmpxchg_32.h | 28 ---------------------------- arch/x86/include/asm/pgtable-3level.h | 23 ++++++++++++----------- 2 files changed, 12 insertions(+), 39 deletions(-) --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -7,34 +7,6 @@ * you need to test for the feature in boot_cpu_data. */ -/* - * CMPXCHG8B only writes to the target if we had the previous - * value in registers, otherwise it acts as a read and gives us the - * "new previous" value. That is why there is a loop. Preloading - * EDX:EAX is a performance optimization: in the common case it means - * we need only one locked operation. - * - * A SIMD/3DNOW!/MMX/FPU 64-bit store here would require at the very - * least an FPU save and/or %cr0.ts manipulation. - * - * cmpxchg8b must be used with the lock prefix here to allow the - * instruction to be executed atomically. We need to have the reader - * side to see the coherent 64bit value. - */ -static inline void set_64bit(volatile u64 *ptr, u64 value) -{ - u32 low = value; - u32 high = value >> 32; - u64 prev = *ptr; - - asm volatile("\n1:\t" - LOCK_PREFIX "cmpxchg8b %0\n\t" - "jnz 1b" - : "=m" (*ptr), "+A" (prev) - : "b" (low), "c" (high) - : "memory"); -} - #ifdef CONFIG_X86_CMPXCHG64 #define arch_cmpxchg64(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \ --- a/arch/x86/include/asm/pgtable-3level.h +++ b/arch/x86/include/asm/pgtable-3level.h @@ -19,7 +19,15 @@ pr_err("%s:%d: bad pgd %p(%016Lx)\n", \ __FILE__, __LINE__, &(e), pgd_val(e)) -/* Rules for using set_pte: the pte being assigned *must* be +#define pxx_xchg64(_pxx, _ptr, _val) ({ \ + _pxx##val_t *_p = (_pxx##val_t *)_ptr; \ + _pxx##val_t _o = *_p; \ + do { } while (!try_cmpxchg64(_p, &_o, (_val))); \ + native_make_##_pxx(_o); \ +}) + +/* + * Rules for using set_pte: the pte being assigned *must* be * either not present or in a state where the hardware will * not attempt to update the pte. In places where this is * not possible, use pte_get_and_clear to obtain the old pte @@ -34,12 +42,12 @@ static inline void native_set_pte(pte_t static inline void native_set_pte_atomic(pte_t *ptep, pte_t pte) { - set_64bit((unsigned long long *)(ptep), native_pte_val(pte)); + pxx_xchg64(pte, ptep, native_pte_val(pte)); } static inline void native_set_pmd(pmd_t *pmdp, pmd_t pmd) { - set_64bit((unsigned long long *)(pmdp), native_pmd_val(pmd)); + pxx_xchg64(pmd, pmdp, native_pmd_val(pmd)); } static inline void native_set_pud(pud_t *pudp, pud_t pud) @@ -47,7 +55,7 @@ static inline void native_set_pud(pud_t #ifdef CONFIG_PAGE_TABLE_ISOLATION pud.p4d.pgd = pti_set_user_pgtbl(&pudp->p4d.pgd, pud.p4d.pgd); #endif - set_64bit((unsigned long long *)(pudp), native_pud_val(pud)); + pxx_xchg64(pud, pudp, native_pud_val(pud)); } /* @@ -91,13 +99,6 @@ static inline void pud_clear(pud_t *pudp } -#define pxx_xchg64(_pxx, _ptr, _val) ({ \ - _pxx##val_t *_p = (_pxx##val_t *)_ptr; \ - _pxx##val_t _o = *_p; \ - do { } while (!try_cmpxchg64(_p, &_o, (_val))); \ - native_make_##_pxx(_o); \ -}) - #ifdef CONFIG_SMP static inline pte_t native_ptep_get_and_clear(pte_t *ptep) {