From patchwork Wed Oct 16 04:36:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: lizhe.67@bytedance.com X-Patchwork-Id: 13837715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 55E3FD2069D for ; Wed, 16 Oct 2024 04:36:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DEF7E6B0089; Wed, 16 Oct 2024 00:36:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DA0046B008A; Wed, 16 Oct 2024 00:36:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C68AC6B008C; Wed, 16 Oct 2024 00:36:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id AE8146B0089 for ; Wed, 16 Oct 2024 00:36:30 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id C63F21C728F for ; Wed, 16 Oct 2024 04:36:19 +0000 (UTC) X-FDA: 82678203678.20.C8071F9 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf12.hostedemail.com (Postfix) with ESMTP id 986F240012 for ; Wed, 16 Oct 2024 04:36:24 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZdmAi+az; spf=pass (imf12.hostedemail.com: domain of lizhe.67@bytedance.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=lizhe.67@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729053197; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Q7wGUcAWkY/xXNhwj5XB7vCbP6fvR8LLicMUY6R9k4E=; b=EfCbtu2ELmEsETrNfMNeh+ImhqWPbbDBjpHQQ/9SpPqGTaE5K1jwiW/N6BMo9YUWB8Advs Wgu4ibupvaacHhFIoHsCe1zTpIsdhz5QdSR8GR/EKG1J38kvbL8rBVUkbEe47uo52NwSZp WtN4Un4zVo4iKMy+UtNHKCsVqPSEdUE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=ZdmAi+az; spf=pass (imf12.hostedemail.com: domain of lizhe.67@bytedance.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=lizhe.67@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729053197; a=rsa-sha256; cv=none; b=KiorpY4HL3UuMoUj5OxfUCG2VjXJUKrYcNqNT7OXx2hBA5VEdY9Je788gcRHDt21XIlcy0 F/hVFLC8wGCbMweNUB2I+22pJyuYeWqo/6TE6DPuHv/k6xEDPoX+PVf4/8NEh0Cc0tcGul qRzYpkaA7GFxfso7c+4FoSafXq0YHhE= Received: by mail-pf1-f174.google.com with SMTP id d2e1a72fcca58-71e8235f0b6so242267b3a.3 for ; Tue, 15 Oct 2024 21:36:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1729053387; x=1729658187; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Q7wGUcAWkY/xXNhwj5XB7vCbP6fvR8LLicMUY6R9k4E=; b=ZdmAi+azZq0D+Qg6tZ4y9XCKu26fWO+mfI/B2NO2PaL16O4xEXQ9QTm7CrnI21gdLJ AcBnQ34sBdT/GBtOW0L+e6hmCBAcbzD2+cDBp1PZBKtleY80J5qmAKxAnl42TgkxR3nF kzBeGGtzJO2GoP5JsaA00r/3W3HnIpoAfv8ehYnk093IMmBoFAA5dnRV1sQ5SO0sUrow laOnewUVR7JvLcxUVA/rSC6GBY1XuHYz5HK3vJfGkeVzw3OIfsmaegd3MG/JF/gq8qJE 9Kn8SG7/IDDvs0ps4UWcMdHwetwV3m1qs/VDV8qYJx5H5iXQ7O8xZv/jE7KbFfQ/6esy N0JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1729053387; x=1729658187; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Q7wGUcAWkY/xXNhwj5XB7vCbP6fvR8LLicMUY6R9k4E=; b=SSPn5vAayHM98ycBk7ka7KXWN7Tmof9xXXoz81kMFRfJH7yi1DgxvJ3+ah9GmhUBnu X4X/nukAfaPF17Jj4TvNIsGlU0X7myTJR3mdayTNPlNqghNDksfIOBVKZwr1uRXNMQE3 nbzza143cf3B6D0h7ge6So9iWdQuJS7NBwSNdl+iCYS45zsv41SKmYScp686n/ZWfyYu 4GTHg3s+oPkZ1ryojLdkmLcDDYkf3SoYQi8b8NJWH0yfSkVqyyRshZMdrzWuMTlEKuOC AvisKBWg5MHy64DGQtKaNGoTR1UlYaskylTT/COD6dQA3NUb9fOKtHHr4QNs+G1wPdMe CsGg== X-Forwarded-Encrypted: i=1; AJvYcCW35LKFji28kxU0LF+o4TNy3NaqmrKIxsw+0o/oEoUBDMMDhMcr0FVLHgEvQAL7Xz88+/1kHcMEhQ==@kvack.org X-Gm-Message-State: AOJu0Yy92DRclXLXqXcr43KBhqPYWaeupFdcaVGH75LBqDGZXyv1dkMy tHlSVXcPYJUOIjKvrK31NGZ+CSBCuYbrrxJ/S0gjHA+lsi4z9fBSDrbQagS221tsShGosAyAw6t NfMU= X-Google-Smtp-Source: AGHT+IGsHi96TSHaDa9/5tTO7fMcmHr9rDiWd6O9B7Avgxv2U6sxlJiV4p4N3nYNqbJHGkIB2KwGlA== X-Received: by 2002:a05:6a00:2288:b0:71e:4dc5:259e with SMTP id d2e1a72fcca58-71e4dc5275dmr18417384b3a.17.1729053387124; Tue, 15 Oct 2024 21:36:27 -0700 (PDT) Received: from GQ6QX3JCW2.bytedance.net ([203.208.189.8]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-71e77518a76sm2189192b3a.220.2024.10.15.21.36.23 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Tue, 15 Oct 2024 21:36:26 -0700 (PDT) From: lizhe.67@bytedance.com To: peterz@infradead.org, mingo@redhat.com, will@kernel.org, longman@redhat.com, boqun.feng@gmail.com, akpm@linux-foundation.org Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, lizhe.67@bytedance.com Subject: [RFC 2/2] khugepaged: use upgrade_read() to optimize collapse_huge_page Date: Wed, 16 Oct 2024 12:36:00 +0800 Message-ID: <20241016043600.35139-3-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241016043600.35139-1-lizhe.67@bytedance.com> References: <20241016043600.35139-1-lizhe.67@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 986F240012 X-Stat-Signature: 4da6g4ux93ds9prxprr6acsyzemreh1b X-Rspam-User: X-HE-Tag: 1729053384-6404 X-HE-Meta: U2FsdGVkX1+AQqsnOxIUvgJGo8CjNIJyEeiEgEKDfHtm/IiGWPrHrtujoHOyqNts7cWwDJZx7eoYSmuGFDwFJPCBxGBsqoO+cSUhM0gW41O30aKwXmwKnw/vTxDHTnYQIuD7ejXaYoTOBqS1bC4WKsCR1UbcG3KFsKNsAHfDEtMGMXJvoVuoko1IJYy58x1N/YZmbZKNafbNXBXo2RPCvoBOcm5jpoNjksdP9tL6MAlZBS8mUQhC74cxW4E9AlUWEW+frvErEbrp+XT/Qp2ORCe3rO4knoCWJBIb1wDe4tssOOAGYwELNfryN0MShBwpOf6kvNyaBpM3g6Zxy1XAFk9nRUMpUnDETw6cuD1dRmfKQDS4tk8YerWoZWSo8MVXN+XJIMxwsghMpoODfsrN2EC/nWGzG2x33a6VlKj401LfXE0nijePcLLCylfjq2O2KTw4OnIoug1UjOCvYIo1AwvtLCqolG+g2Sf0LxrWVijk++swrQnMcgQTXhXOXiiBu/pzUPy+bbgoQL3C3bTNylquVKxQ2fnwVPClnstJZsNnSFd6fH8apnBvWhueaUo9zqsfy685JdeNAHKqyUKF9NQhmKP6Qwr0Lup9+exk4DIcKi0TGejvw24mPhwF98hraY3m60hnzP0F5uNbs4amisaf5AGa9n2rPt2i9SfaB6CzNf9Z0645au5uV9GmfFVrtj8iy/zNA4f207OYQn+fRa6gO0L20jifAz7oRNdSylsS1nYkdM6zyb+X4JVHHgIcwY91g+MciU6OZtQuGcr/I97/3/iCpXVzGMN/WH1qVCay4dTSy2zDWjvkYtP9ByU74lfPXVzoEbBhm7JGZ2SSy7XaC3D2vVFKWfmM6nuxERKyDI/jtDtZhd5Xk/+MIc8xMei6bCGdQ9DoeYa0ltwcqqLQXGtK6i4oCy+OF3vC2OR5OD8r9wjYBY+X92CDzC6KIU4++xOdcXzjdxcW1Lm G0oiEogX bzNB8oOu3Xch7E5SfGulK5JtQVQGW22nG5pyn/dWkYPw2AGAfKyls/8XxlC5TuCZ71BHkUchQfG1yXDU4tDfEaq44GjlbAaar7cVE5W6L6VccVzNNpENIYgOI+6/j8dGdETcZCLJL7U8v4TC1IOxoiDd2ISyU4osjYRiDiEcVmhARDi5DZ38FYg//FsiUgXkm3KOxE7/0o9xxHZsqyPwA4TA3wzasSU7vcRHLtzk7c8aNBzFdfZyHNN1K64Q+0U9I2IC8OH1h18otbWrGtNVN9KeMSijtXK7q4+bGrjYDNa1tAxxrxKQtmRqp9zGYC5KKQme/LcwH9K21kKkCqP1mTP0SBSOGqoM6i/POOxtMWbw+c4NxTkVvA5Vhp0lf+BbC4vx1+Q6XM5+H8GyDZmNMqICj1yBHdXYVSEZtiOTGWz2ZrMU= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Li Zhe In function collapse_huge_page(), we drop mmap read lock and get mmap write lock to prevent most accesses to pagetables. There is a small time window to allow other tasks to acquire the mmap lock. With the use of upgrade_read(), we don't need to check vma and pmd again in most cases. Signed-off-by: Li Zhe --- mm/khugepaged.c | 36 +++++++++++++++++++----------------- 1 file changed, 19 insertions(+), 17 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index f9c39898eaff..934051274f7a 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1142,23 +1142,25 @@ static int collapse_huge_page(struct mm_struct *mm, unsigned long address, goto out_nolock; } - mmap_read_unlock(mm); - /* - * Prevent all access to pagetables with the exception of - * gup_fast later handled by the ptep_clear_flush and the VM - * handled by the anon_vma lock + PG_lock. - * - * UFFDIO_MOVE is prevented to race as well thanks to the - * mmap_lock. - */ - mmap_write_lock(mm); - result = hugepage_vma_revalidate(mm, address, true, &vma, cc); - if (result != SCAN_SUCCEED) - goto out_up_write; - /* check if the pmd is still valid */ - result = check_pmd_still_valid(mm, address, pmd); - if (result != SCAN_SUCCEED) - goto out_up_write; + if (upgrade_read(&mm->mmap_lock)) { + mmap_read_unlock(mm); + /* + * Prevent all access to pagetables with the exception of + * gup_fast later handled by the ptep_clear_flush and the VM + * handled by the anon_vma lock + PG_lock. + * + * UFFDIO_MOVE is prevented to race as well thanks to the + * mmap_lock. + */ + mmap_write_lock(mm); + result = hugepage_vma_revalidate(mm, address, true, &vma, cc); + if (result != SCAN_SUCCEED) + goto out_up_write; + /* check if the pmd is still valid */ + result = check_pmd_still_valid(mm, address, pmd); + if (result != SCAN_SUCCEED) + goto out_up_write; + } vma_start_write(vma); anon_vma_lock_write(vma->anon_vma);