From patchwork Thu Oct 12 04:40:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 13418318 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B27B8CDB46E for ; Thu, 12 Oct 2023 04:40:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CEB8E8D0107; Thu, 12 Oct 2023 00:40:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C9C378D0002; Thu, 12 Oct 2023 00:40:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B63158D0107; Thu, 12 Oct 2023 00:40:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A6D418D0002 for ; Thu, 12 Oct 2023 00:40:15 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 6C095804E2 for ; Thu, 12 Oct 2023 04:40:15 +0000 (UTC) X-FDA: 81335557590.15.44A9C01 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) by imf09.hostedemail.com (Postfix) with ESMTP id B1FF114000D for ; Thu, 12 Oct 2023 04:40:13 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LD3e055e; spf=pass (imf09.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1697085613; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+l9a+mEwNtEsZNHCdrwSvDlM5U4OjWSgjs4wg5g9aAk=; b=iGw+tJfAo/39qsJb9Oun4oU+SfAuSx3SjojvNgoeXdg6ZocMmiBiP37Fza9kFnSfu+szny buRZ7LRLbUYV40i/7GmFOR9IOkn6qzYsjNGpkQKpNUotgC98jsXM2uIMrHiTTmxPZYEcxb L4s9zhKZwIiLQBRsBU0zYfCLrYtF5RY= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1697085613; a=rsa-sha256; cv=none; b=l7Ep96+6zo3hSCTzKtMqrMX8TaZUdYdUqa/llOBZDrT0zQ61LY8grNDSuw+F85BbAsxCSA pzy+/vUmLlAg9ye1DSKfeW0n582sxYU99xSuapCOt3Sd+Ci0yegAPulpbcsJem2I7XWlUC Ay4jCJX6J5l/8k5qBphU1U+0e27ZsVk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=LD3e055e; spf=pass (imf09.hostedemail.com: domain of hughd@google.com designates 209.85.128.175 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-59f82ad1e09so6949847b3.0 for ; Wed, 11 Oct 2023 21:40:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1697085613; x=1697690413; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=+l9a+mEwNtEsZNHCdrwSvDlM5U4OjWSgjs4wg5g9aAk=; b=LD3e055eTs4z2GC0E+n8Chv4JxcoJBgXODSY68OVQiGUtOeZyfjGW5ORAzaupuFl++ uDOO41MmoIiRpl/Ci/DjPv/E7QsPhfiQCRBRY1OpGl/i+MijAKo176SPcPSuppAJxJns WzyQGXQAiLmmyVNKYHay/ShSSXfy5PnlFR62Qq94KVLVrkOcFInaus1hcoAO+sY40n3b DCOe13djdKXdnUSf0HzkEeVs0uRi0Sdv06y8CVYB1M5NM/Ks7kwtzp1TnOKKfd9vZi/s dyOm1YpiCKOVc2MVKVcBzYbClmjU9XVdB+7N8AnLvy9M0I63XZMWf+LtKyFaKrjvZ848 SBSg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1697085613; x=1697690413; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+l9a+mEwNtEsZNHCdrwSvDlM5U4OjWSgjs4wg5g9aAk=; b=T+an2mEJ+INTV+USMGb18NryiLRDXHWcDuCkwkcPLQGqasODivKepG0NAe1FQPxtjq 2mn0lpOzDZcpYs9RWKwTgYotYtLCLcXv0V3nbJeJ1JeUZChw9YfYYZ3niwvOqcuNvgXT g+U9M9+UaAuSde7vy6nvkTiIOWR2pveJDZOBB7RJm9l4xtS1LpwQ1tllFWVKPmz7Rehk xYUkDupk0+Pm78IT80lR+LK7zNfIWCuzBg8EV2DCXKAJAWu7PRp2Ez+YSU7Xqn5RADDz actz+V+E0xT246KEFvJzibk82658kShCEiwEqbnRttS6cqeTkXgUyC2ARXUNx0xDFAPe AU0A== X-Gm-Message-State: AOJu0YwI6g8L1e1KyXk2fOfPhMvekLGogyoQP2vMfBjftUP0ZwCl4lUQ jrstLOyd0JkiCWdXr9+v4rvcNQ== X-Google-Smtp-Source: AGHT+IEy29AB2gZVMxb+IATr0dLpEGj4Y9wZwZWi4zoiPAFB4Rim0MuDfZV+1buXtmaVCIwWelmNMQ== X-Received: by 2002:a5b:285:0:b0:d7b:9d44:7574 with SMTP id x5-20020a5b0285000000b00d7b9d447574mr20855492ybl.64.1697085612762; Wed, 11 Oct 2023 21:40:12 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x142-20020a25ce94000000b00d89679f6d22sm1734993ybe.64.2023.10.11.21.40.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 11 Oct 2023 21:40:11 -0700 (PDT) Date: Wed, 11 Oct 2023 21:40:09 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.attlocal.net To: Andrew Morton cc: Dave Chinner , Tim Chen , Dave Chinner , "Darrick J. Wong" , Christian Brauner , Carlos Maiolino , Chuck Lever , Jan Kara , Matthew Wilcox , Johannes Weiner , Axel Rasmussen , Dennis Zhou , linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH 9/8] percpu_counter: extend _limited_add() to negative amounts In-Reply-To: Message-ID: <8f86083b-c452-95d4-365b-f16a2e4ebcd4@google.com> References: <2451f678-38b3-46c7-82fe-8eaf4d50a3a6@google.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: B1FF114000D X-Rspam-User: X-Stat-Signature: 7cj3zwwxnzuiddafukjfq8hdm9sn8rp6 X-Rspamd-Server: rspam03 X-HE-Tag: 1697085613-612527 X-HE-Meta: U2FsdGVkX19Q8tCU8U1yNUgZyzpbl/NL0zzzutvl8ZW02X/tere5JWoKX7mRdyQH7Whiz8dHY8Vh7Z7htYMnHrQ/RvL0B2d+MSq2Y/dioDdnKbixMaPrg3QNnyqV5Xt3yXC1E3PaoAGIqv8VYtJk1O1B4loHuk5dqcGHKxqAC+14l/60EjhL2bBT351X5wC8jeBHuCYoDSci+6bDedFyOIavYzi7HIc6obS+25oatX2CV74IJvRdEtnohlECJ5ptwZeIqzzGJjtBMYSfXgvB7XZ8I0bLSmklfREByGuvXVsvl5Wjnt4g3NeHViODQN3LAf3NOFswVTbDVF52AgRKl9FxLHsAV4d74Wf0OQ6O6lBQBUKj5RQzwWTqCOkWd1sdCbLmp945o3/SYXgoVo41ZKYP7gU6adJN3tkRPDJKcBTkJQFSaT4TVFXFFvdZhD+kC9+4uKjC5+0HqOefrOFB6mNfqYK+4QogRaPXbAe44/e3mE4WOua6NXyxB0pxcs3As4sX39iNs1a3hvYSOD1LtFxc1Bx7OHV7mVxcAwNUHqaVDPVGcd4XsNwrD2pr9UoJlhmSQ8pzLSysZOBODL/pBvYtrGM3476xBHZqMf1h5hzb9kCtVcpV609Tj3/1HFepD6wf/jetIAzPntWFH9fDmNEHdXCCS8Ssew9c1dwM0UfyPqeYYve/zbpYl8wVV51XgmEEDgWFNoIueVtd/nHFJbXUmAObcJ85o+LvhrTVGGtGtUvPnltN7uVYABEf3w3GxdMfJEXQyRmKLiZsLbf7OhVLkfHlvNN44fecgKHPfM7Ce1Ll/E6g5439RbzETwyyYhxcvFaZXjqH381BNpjr9ESEwrz2gvMQkfkaDt6KTxHqyWnWNLgNsaGOOpcTzNOCpYsUDim9Z+h66Gd9OgIpU+06vktFuWZmv7hKSsdDCJNCGEnc+WOJQU+u48OfaWlUvCmU86Gm92JVN3leDV1 2PEo/T7P OUs/d/USG9r62ZkFnIOWUfyC/egPvqjYaDG0DAwh0X51hp4/fiXGswhmXEwOB0lSX5zmn610ZPRDogZOAb3d29K5ZEIeksEQyH9is+OYuIhIaaIcSXV224JCwSiHh+ytYMkEy0D5+CrxPjCpLXyneujL4K1N7mbGyJXFOe4qw+PuoWx0zzvE2QRhTUBaTKxRrp+z+tZogsffVWSf2ot0A+EcBctP6PEwYk/VcW7+KTNbOd+takvZHCXlV9VtZKFDlrcjolT2meySNYLgLlYoFEuBA9pIHtDpHjafGlkTgZufzTNlEttr3IRjGsdSZoPZPDbOvzVaUQ4rBiARfD5yu/KsaS8fIhV9/Qj7iDkW8ro/s3mrGnobYZEElAMc+JFB4JrLptoWCN9gbBS6aMLhMCsk61WMA7+/2Iw5voOZr8efz2tQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Though tmpfs does not need it, percpu_counter_limited_add() can be twice as useful if it works sensibly with negative amounts (subs) - typically decrements towards a limit of 0 or nearby: as suggested by Dave Chinner. And in the course of that reworking, skip the percpu counter sum if it is already obvious that the limit would be passed: as suggested by Tim Chen. Extend the comment above __percpu_counter_limited_add(), defining the behaviour with positive and negative amounts, allowing negative limits, but not bothering about overflow beyond S64_MAX. Signed-off-by: Hugh Dickins --- include/linux/percpu_counter.h | 11 +++++-- lib/percpu_counter.c | 54 +++++++++++++++++++++++++--------- 2 files changed, 49 insertions(+), 16 deletions(-) diff --git a/include/linux/percpu_counter.h b/include/linux/percpu_counter.h index 8cb7c071bd5c..3a44dd1e33d2 100644 --- a/include/linux/percpu_counter.h +++ b/include/linux/percpu_counter.h @@ -198,14 +198,21 @@ static inline bool percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit, s64 amount) { unsigned long flags; + bool good = false; s64 count; + if (amount == 0) + return true; + local_irq_save(flags); count = fbc->count + amount; - if (count <= limit) + if ((amount > 0 && count <= limit) || + (amount < 0 && count >= limit)) { fbc->count = count; + good = true; + } local_irq_restore(flags); - return count <= limit; + return good; } /* non-SMP percpu_counter_add_local is the same with percpu_counter_add */ diff --git a/lib/percpu_counter.c b/lib/percpu_counter.c index 58a3392f471b..44dd133594d4 100644 --- a/lib/percpu_counter.c +++ b/lib/percpu_counter.c @@ -279,8 +279,16 @@ int __percpu_counter_compare(struct percpu_counter *fbc, s64 rhs, s32 batch) EXPORT_SYMBOL(__percpu_counter_compare); /* - * Compare counter, and add amount if the total is within limit. - * Return true if amount was added, false if it would exceed limit. + * Compare counter, and add amount if total is: less than or equal to limit if + * amount is positive, or greater than or equal to limit if amount is negative. + * Return true if amount is added, or false if total would be beyond the limit. + * + * Negative limit is allowed, but unusual. + * When negative amounts (subs) are given to percpu_counter_limited_add(), + * the limit would most naturally be 0 - but other limits are also allowed. + * + * Overflow beyond S64_MAX is not allowed for: counter, limit and amount + * are all assumed to be sane (far from S64_MIN and S64_MAX). */ bool __percpu_counter_limited_add(struct percpu_counter *fbc, s64 limit, s64 amount, s32 batch) @@ -288,10 +296,10 @@ bool __percpu_counter_limited_add(struct percpu_counter *fbc, s64 count; s64 unknown; unsigned long flags; - bool good; + bool good = false; - if (amount > limit) - return false; + if (amount == 0) + return true; local_irq_save(flags); unknown = batch * num_online_cpus(); @@ -299,7 +307,8 @@ bool __percpu_counter_limited_add(struct percpu_counter *fbc, /* Skip taking the lock when safe */ if (abs(count + amount) <= batch && - fbc->count + unknown <= limit) { + ((amount > 0 && fbc->count + unknown <= limit) || + (amount < 0 && fbc->count - unknown >= limit))) { this_cpu_add(*fbc->counters, amount); local_irq_restore(flags); return true; @@ -309,7 +318,19 @@ bool __percpu_counter_limited_add(struct percpu_counter *fbc, count = fbc->count + amount; /* Skip percpu_counter_sum() when safe */ - if (count + unknown > limit) { + if (amount > 0) { + if (count - unknown > limit) + goto out; + if (count + unknown <= limit) + good = true; + } else { + if (count + unknown < limit) + goto out; + if (count - unknown >= limit) + good = true; + } + + if (!good) { s32 *pcount; int cpu; @@ -317,15 +338,20 @@ bool __percpu_counter_limited_add(struct percpu_counter *fbc, pcount = per_cpu_ptr(fbc->counters, cpu); count += *pcount; } + if (amount > 0) { + if (count > limit) + goto out; + } else { + if (count < limit) + goto out; + } + good = true; } - good = count <= limit; - if (good) { - count = __this_cpu_read(*fbc->counters); - fbc->count += count + amount; - __this_cpu_sub(*fbc->counters, count); - } - + count = __this_cpu_read(*fbc->counters); + fbc->count += count + amount; + __this_cpu_sub(*fbc->counters, count); +out: raw_spin_unlock(&fbc->lock); local_irq_restore(flags); return good;