From patchwork Sun Jan 31 00:11:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12057477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.5 required=3.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED,DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN, FREEMAIL_FROM,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 05082C43381 for ; Sun, 31 Jan 2021 00:16:48 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AD2AD64E19 for ; Sun, 31 Jan 2021 00:16:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AD2AD64E19 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9A9CF6B0087; Sat, 30 Jan 2021 19:16:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 931EC6B0088; Sat, 30 Jan 2021 19:16:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 75DEE6B0089; Sat, 30 Jan 2021 19:16:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0131.hostedemail.com [216.40.44.131]) by kanga.kvack.org (Postfix) with ESMTP id 5F8D86B0087 for ; Sat, 30 Jan 2021 19:16:33 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 2F4A6181AEF23 for ; Sun, 31 Jan 2021 00:16:33 +0000 (UTC) X-FDA: 77764153866.19.cats14_1017169275b5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 178C61AD1B9 for ; Sun, 31 Jan 2021 00:16:33 +0000 (UTC) X-HE-Tag: cats14_1017169275b5 X-Filterd-Recvd-Size: 6807 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf38.hostedemail.com (Postfix) with ESMTP for ; Sun, 31 Jan 2021 00:16:32 +0000 (UTC) Received: by mail-pf1-f174.google.com with SMTP id y205so9069313pfc.5 for ; Sat, 30 Jan 2021 16:16:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tkrptTBK3N3R404MIQhBTsLhuUAmLX1yl6cqYDzKwyA=; b=WjebMTziYJW52tJp0ecHzhl0k1PuuNiWVAV+79HZYfxsnt5v4az2VBXEzvPkAHS6Q9 hjT1LxAy2ED9H+NVTdABs2daiWJLGsg9Jbp97LsFT8TPB7fPnMns14St8gL9pl0KJdJi jUHg+3yPAd7aKlu2mg/YH19B3bejOWyhMlyqDbilE6I0G3ZJjiDWjqnZz+fO7sTPp6dZ vyqe6gKiUcV6hZJVcwAui8L71cdM9pgzHchPndGr1ook+9Gd92I7wU2/zhlZwv7AdTha 4O0kAUXxZJ8u+f+mDWzqbWK+EYI4SOOmL3F1q3aMvRo6Mtpc3C9UgLi+DPzuiHziC2HN TwZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tkrptTBK3N3R404MIQhBTsLhuUAmLX1yl6cqYDzKwyA=; b=NBk29M/FsnFL+Ds7XE59wNlwUSi39Z+1VLeFUZvx5ezXUL5F8YObobibVmiOoe9VfP xNmT1vGE/8pckTh+UyJY9ZBuUYxjVHPiHrXH/lQk8G/woFiZzf4t0pfHU1KG/c2D86fW JfO9y9b3mgCGvLHGvPMTn6DRnE0eqON0bkgJNa3llbI5WjEQRjk4jSEPvEfWtGJBRrBO QoKXRakH4PyQl0S0Xe2pa98hZPNbRvmEp9qaKwzfn2nKLHLMLEXYhLTLvSi3+/qihPLJ s0NnxxSHtUs3mn0WKYXMmxW9ekapadOWKvjUfsylXpOFdP25OM8/JDFfrLSeK1uEEfuM Otog== X-Gm-Message-State: AOAM533qQQ1Z0IFifI4kuTTYqetCeAruVWl+EGF1M6F9DeC3vbYSEN8F KT8tvMi1beXy3RlyRrlVs36m/AG2c48= X-Google-Smtp-Source: ABdhPJwXMFg7+1aJTyLt9vDF5ckFJc96jqdhXgkz0Lsqm2Hnw2vjWguS2Ee3rjHPCC7agJskagZBmQ== X-Received: by 2002:a65:430b:: with SMTP id j11mr10240396pgq.130.1612052191427; Sat, 30 Jan 2021 16:16:31 -0800 (PST) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id e12sm13127365pga.13.2021.01.30.16.16.29 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sat, 30 Jan 2021 16:16:30 -0800 (PST) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Nadav Amit , Mel Gorman , Andrea Arcangeli , Andrew Morton , Andy Lutomirski , Dave Hansen , Peter Zijlstra , Thomas Gleixner , Will Deacon , Yu Zhao , x86@kernel.org Subject: [RFC 19/20] lib/cpumask: introduce cpumask_atomic_or() Date: Sat, 30 Jan 2021 16:11:31 -0800 Message-Id: <20210131001132.3368247-20-namit@vmware.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210131001132.3368247-1-namit@vmware.com> References: <20210131001132.3368247-1-namit@vmware.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit Introduce cpumask_atomic_or() and bitmask_atomic_or() to allow to perform atomic or operations atomically on cpumasks. This will be used by the next patch. To be more efficient, skip atomic operations when no changes are needed. Signed-off-by: Nadav Amit Cc: Mel Gorman Cc: Andrea Arcangeli Cc: Andrew Morton Cc: Andy Lutomirski Cc: Dave Hansen Cc: Peter Zijlstra Cc: Thomas Gleixner Cc: Will Deacon Cc: Yu Zhao Cc: x86@kernel.org --- include/linux/bitmap.h | 5 +++++ include/linux/cpumask.h | 12 ++++++++++++ lib/bitmap.c | 25 +++++++++++++++++++++++++ 3 files changed, 42 insertions(+) diff --git a/include/linux/bitmap.h b/include/linux/bitmap.h index 769b7a98e12f..c9a9b784b244 100644 --- a/include/linux/bitmap.h +++ b/include/linux/bitmap.h @@ -76,6 +76,7 @@ * bitmap_to_arr32(buf, src, nbits) Copy nbits from buf to u32[] dst * bitmap_get_value8(map, start) Get 8bit value from map at start * bitmap_set_value8(map, value, start) Set 8bit value to map at start + * bitmap_atomic_or(dst, src, nbits) *dst |= *src (atomically) * * Note, bitmap_zero() and bitmap_fill() operate over the region of * unsigned longs, that is, bits behind bitmap till the unsigned long @@ -577,6 +578,10 @@ static inline void bitmap_set_value8(unsigned long *map, unsigned long value, map[index] |= value << offset; } +extern void bitmap_atomic_or(volatile unsigned long *dst, + const volatile unsigned long *bitmap, unsigned int bits); + + #endif /* __ASSEMBLY__ */ #endif /* __LINUX_BITMAP_H */ diff --git a/include/linux/cpumask.h b/include/linux/cpumask.h index 3d7e418aa113..0567d73a0192 100644 --- a/include/linux/cpumask.h +++ b/include/linux/cpumask.h @@ -699,6 +699,18 @@ static inline unsigned int cpumask_size(void) return BITS_TO_LONGS(nr_cpumask_bits) * sizeof(long); } +/** + * cpumask_atomic_or - *dstp |= *srcp (*dstp is set atomically) + * @dstp: the cpumask result (and source which is or'd) + * @srcp: the source input + */ +static inline void cpumask_atomic_or(volatile struct cpumask *dstp, + const volatile struct cpumask *srcp) +{ + bitmap_atomic_or(cpumask_bits(dstp), cpumask_bits(srcp), + nr_cpumask_bits); +} + /* * cpumask_var_t: struct cpumask for stack usage. * diff --git a/lib/bitmap.c b/lib/bitmap.c index 6df7b13727d3..50f1842ff891 100644 --- a/lib/bitmap.c +++ b/lib/bitmap.c @@ -1310,3 +1310,28 @@ void bitmap_to_arr32(u32 *buf, const unsigned long *bitmap, unsigned int nbits) EXPORT_SYMBOL(bitmap_to_arr32); #endif + +void bitmap_atomic_or(volatile unsigned long *dst, + const volatile unsigned long *bitmap, unsigned int bits) +{ + unsigned int k; + unsigned int nr = BITS_TO_LONGS(bits); + + for (k = 0; k < nr; k++) { + unsigned long src = bitmap[k]; + + /* + * Skip atomic operations when no bits are changed. Do not use + * bitmap[k] directly to avoid redundant loads as bitmap + * variable is volatile. + */ + if (!(src & ~dst[k])) + continue; + + if (BITS_PER_LONG == 64) + atomic64_or(src, (atomic64_t*)&dst[k]); + else + atomic_or(src, (atomic_t*)&dst[k]); + } +} +EXPORT_SYMBOL(bitmap_atomic_or);