From patchwork Wed Feb 16 11:51:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Muchun Song X-Patchwork-Id: 12748463 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5B92C433EF for ; Wed, 16 Feb 2022 11:52:39 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3645F6B0080; Wed, 16 Feb 2022 06:52:39 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2ED256B0081; Wed, 16 Feb 2022 06:52:39 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 166BC6B0082; Wed, 16 Feb 2022 06:52:39 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 03F8F6B0080 for ; Wed, 16 Feb 2022 06:52:39 -0500 (EST) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B6E7F8249980 for ; Wed, 16 Feb 2022 11:52:38 +0000 (UTC) X-FDA: 79148480796.29.A80A018 Received: from mail-pj1-f50.google.com (mail-pj1-f50.google.com [209.85.216.50]) by imf24.hostedemail.com (Postfix) with ESMTP id 4A0C6180008 for ; Wed, 16 Feb 2022 11:52:38 +0000 (UTC) Received: by mail-pj1-f50.google.com with SMTP id y9so2193433pjf.1 for ; Wed, 16 Feb 2022 03:52:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=165dLhQ7hyh1L/DjdfIFCITpDk7lPjHyUo7YV5H1F1Y=; b=yS5Od63QJjgrf2sJM68+B+Eq/OWFlIDeRLQ9OZM5Dr7wRL1cF4uw+9WbmrG0CDmp4k HpKT78ZM9yqU0d1nofwrowKJGJUI99ewSbmK2iODzD3qk9DsIpkJXDnMAiJ+QnsgXj44 BFGAIxgqj0DpYxrj3e7+SLoTV3VvSMBxx0v2pBNZaEP4yxWAv6il6R2abz4ZU2f6Ek4b dX1cXNZjDFgo0TXtQiwZeTNNF2sJSRr1Z1Dy7+2pUui2jZxsX1F+XcRd5kjthUVup2jm oyIyXbTn5pyP2r/LxGjk6PRqXUCwx5c07AKz9lv2Oqmo1oV10fS/02XtXG1p6U7zLQmE N5Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=165dLhQ7hyh1L/DjdfIFCITpDk7lPjHyUo7YV5H1F1Y=; b=rxl6XEwW8QBFaBvtMcdn/raR1r6aV8UYtp38PKqXvVNaC28oD+NlbVAM4bqc/2SIyf LSU/wXgumZYxcX6d9U3O4d4WyeH/MbJ76JPBPAHW8CJQDKyoaRFirAlsYAEojOufqSmk kiw9koEf+KSvOFuYiMeNFWx8EvnqBSfjMzyADOavDzbKBNChD0u50AY67U/3kLsFagMN uw9pXX+ztX7wBiVTJaJa4dukFkj8WLLcKgC6JlfLNtipzQTL3SUDE9AflmqAxZCsXveW SXFjbTRZXsWEy2axrYSqFBIaqtK84sGnIvdx+wLtlW6y91I65avyA3ofjgQBAHrxFpoB LlnQ== X-Gm-Message-State: AOAM530trIDFu9BQVhAG6nrhOssibxNI/QdUmokYV443seIrNRq1l0+L KQOE/+IndlFxsIGkCi8j5Hh3oQ== X-Google-Smtp-Source: ABdhPJwAJOwLAtA5/2JhCA4A3zJUOykMJidUQdD198hj5IxFHa45Xeu8iRkX8Z8y9nx9M87a9h0OJQ== X-Received: by 2002:a17:902:ab52:b0:14d:7ce1:8d66 with SMTP id ij18-20020a170902ab5200b0014d7ce18d66mr2249555plb.88.1645012357348; Wed, 16 Feb 2022 03:52:37 -0800 (PST) Received: from FVFYT0MHHV2J.tiktokcdn.com ([139.177.225.249]) by smtp.gmail.com with ESMTPSA id m16sm14790221pfc.156.2022.02.16.03.52.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Feb 2022 03:52:37 -0800 (PST) From: Muchun Song To: guro@fb.com, hannes@cmpxchg.org, mhocko@kernel.org, akpm@linux-foundation.org, shakeelb@google.com, vdavydov.dev@gmail.com Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, fam.zheng@bytedance.com, bsingharora@gmail.com, shy828301@gmail.com, alexs@kernel.org, smuchun@gmail.com, zhengqi.arch@bytedance.com, Muchun Song Subject: [PATCH v3 06/12] mm: thp: make split queue lock safe when LRU pages are reparented Date: Wed, 16 Feb 2022 19:51:26 +0800 Message-Id: <20220216115132.52602-7-songmuchun@bytedance.com> X-Mailer: git-send-email 2.32.0 (Apple Git-132) In-Reply-To: <20220216115132.52602-1-songmuchun@bytedance.com> References: <20220216115132.52602-1-songmuchun@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 4A0C6180008 X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=yS5Od63Q; spf=pass (imf24.hostedemail.com: domain of songmuchun@bytedance.com designates 209.85.216.50 as permitted sender) smtp.mailfrom=songmuchun@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-Stat-Signature: nuzqxz6d4tswwgyb69d8dhtd8bpiruhi X-HE-Tag: 1645012358-366406 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Similar to the lruvec lock, we use the same approach to make the split queue lock safe when LRU pages are reparented. Signed-off-by: Muchun Song --- mm/huge_memory.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index a227731988b3..b8c6e766c91c 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -535,9 +535,22 @@ static struct deferred_split *folio_split_queue_lock(struct folio *folio) { struct deferred_split *queue; + rcu_read_lock(); +retry: queue = folio_split_queue(folio); spin_lock(&queue->split_queue_lock); + if (unlikely(split_queue_memcg(queue) != folio_memcg(folio))) { + spin_unlock(&queue->split_queue_lock); + goto retry; + } + + /* + * Preemption is disabled in the internal of spin_lock, which can serve + * as RCU read-side critical sections. + */ + rcu_read_unlock(); + return queue; } @@ -546,9 +559,19 @@ folio_split_queue_lock_irqsave(struct folio *folio, unsigned long *flags) { struct deferred_split *queue; + rcu_read_lock(); +retry: queue = folio_split_queue(folio); spin_lock_irqsave(&queue->split_queue_lock, *flags); + if (unlikely(split_queue_memcg(queue) != folio_memcg(folio))) { + spin_unlock_irqrestore(&queue->split_queue_lock, *flags); + goto retry; + } + + /* See the comments in folio_split_queue_lock(). */ + rcu_read_unlock(); + return queue; }