From patchwork Wed Feb 1 03:41:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: David Stevens X-Patchwork-Id: 13123726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 37DD9C38142 for ; Wed, 1 Feb 2023 03:42:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C310B6B0071; Tue, 31 Jan 2023 22:42:10 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BE0A26B0072; Tue, 31 Jan 2023 22:42:10 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AA8156B0074; Tue, 31 Jan 2023 22:42:10 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 9A96B6B0071 for ; Tue, 31 Jan 2023 22:42:10 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 5864480CEC for ; Wed, 1 Feb 2023 03:42:10 +0000 (UTC) X-FDA: 80417324820.27.3AEDDA3 Received: from mail-pl1-f179.google.com (mail-pl1-f179.google.com [209.85.214.179]) by imf19.hostedemail.com (Postfix) with ESMTP id 6AD621A0011 for ; Wed, 1 Feb 2023 03:42:08 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GKVuK+BQ; spf=pass (imf19.hostedemail.com: domain of stevensd@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=stevensd@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675222928; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=B9gFWLBbXJO+fTDUkJEhDE5AAFHtlij46AyAy+kzk8A=; b=nQIjtqqfYPZFDXbpnPxNATjQXVH5NlwsQW6K1zUERPWhBcGHu1SAvj2fNDv+5YBO4cpkcH d8FmBFV4VieFvXvd9f7hDsCjI+CwMItTvhjz85UeYMLKZs9VUOPgdWB/9GM4Du4CxlSxeB qGgXXjqFa/QG8/TOX61kfhRbrOfZYcM= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=chromium.org header.s=google header.b=GKVuK+BQ; spf=pass (imf19.hostedemail.com: domain of stevensd@chromium.org designates 209.85.214.179 as permitted sender) smtp.mailfrom=stevensd@chromium.org; dmarc=pass (policy=none) header.from=chromium.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675222928; a=rsa-sha256; cv=none; b=hliICSWoWZRLZ9DjIgWAs1nX1OPjHZt8i2aU/GifHY5vb/Tj7HFoAEjqkLiTYDUZ1VM0Q4 /8NlMW9oBT9D6OLnFJxr5/uINszYfd2teWdd/xOKE9EhJmRW4JaqGhXoBT+BzAVyM3jQxI xpxWCkjUNf93drHrduNxEbpJUdCabcY= Received: by mail-pl1-f179.google.com with SMTP id k13so17250320plg.0 for ; Tue, 31 Jan 2023 19:42:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chromium.org; s=google; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:from:to:cc:subject:date:message-id:reply-to; bh=B9gFWLBbXJO+fTDUkJEhDE5AAFHtlij46AyAy+kzk8A=; b=GKVuK+BQJLSYEymgkJUxh8Dw/ZsSmcYWiU+b1qmS0eN3ZqS0imn5FW6nWzGfxgR/H9 pHNj/GU92PlZQ+iwET4xzhDDJ0p86+k947xodsf/HUfh+yLx2IS5gQr22p7MHYPM5JmF D53waLbG7LDxKOFC/ZqzI4viIz+U6BeV+7rKo= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=B9gFWLBbXJO+fTDUkJEhDE5AAFHtlij46AyAy+kzk8A=; b=XJXDy8AYEKMi7Xq8R2Q+gFY6po34gYkpknKjhh4zql0ezh0wCMLkvU6ma233X+Ajos Pn6oMcmwltqYcdE/DuC/LwLZlq+Wbe+P4R1R1wFFz6FskPXyXyrt+vZIf6n8I+YyLLOG 4cw0y+XEfOnEn0qDEubfNsRPj6NfjmOWp4yBeEUSzLbhYVB2W7aVts7A8DYdh/glYfQV PC74POtotE2TbcVu5i3VZ3zPMpPA+qA9U9RPL16//BvIPHkT4OJ0Khmk3cCNPkD3MsLk sSgDkckxRmXvilLbs1kj/rlGPkj3uesPhZFh9u+K/jF7qGWj0M4/mp9NdaF2uaL2zZQ1 Q8+g== X-Gm-Message-State: AO0yUKUa/NRDG9Q76hsv8JE9Q7deJUmM2ih3ifffq0TsSNf9HaABdnX3 mQTOmQN9tE++hj7N4nztZxmf+xTSrRJggo7+ X-Google-Smtp-Source: AK7set//Rgle2d/HSuclPsF6r6tDzpB8dTbPaMn3vTeq1onPvJg8TCuez/OhN4vx4gDSXr6Y3YpXIw== X-Received: by 2002:a17:90a:30d:b0:22c:932:2870 with SMTP id 13-20020a17090a030d00b0022c09322870mr568622pje.33.1675222926348; Tue, 31 Jan 2023 19:42:06 -0800 (PST) Received: from localhost ([2401:fa00:8f:203:e9a3:1082:396d:c1c1]) by smtp.gmail.com with UTF8SMTPSA id l63-20020a639142000000b004d1d89ec63fsm9584677pge.6.2023.01.31.19.42.04 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 31 Jan 2023 19:42:05 -0800 (PST) From: David Stevens X-Google-Original-From: David Stevens To: linux-mm@kvack.org Cc: Andrew Morton , linux-kernel@vger.kernel.org, David Stevens Subject: [PATCH] mm/khugepaged: skip shmem with armed userfaultfd Date: Wed, 1 Feb 2023 12:41:37 +0900 Message-Id: <20230201034137.2463113-1-stevensd@google.com> X-Mailer: git-send-email 2.39.1.456.gfc5497dd1b-goog MIME-Version: 1.0 X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6AD621A0011 X-Stat-Signature: htzzme4mspp1u9omdhja6mtg9snepepf X-Rspam-User: X-HE-Tag: 1675222928-815016 X-HE-Meta: U2FsdGVkX1+KBjA0a/Xkcdw63NQ5VifSC89FX49lOl7OZztldL6pB9mb2geXzTymVFvneG1i4dlmdJwRTEAE6Y8Uuatz2J1eX/JfZoI+aG74lVYl4QmhUqn76CEYmfJxyOVYCCl5e8zxWwXSYsDlT77sz/JPpeG9LeeJp8sO41pBPCWYeN2/ifgsNFwsnwq64VfVU+ito+JetQegtMfC0IeWw7Bmfx+mOw1eVwZLkk1A6X9FMGAd03bXJ+oZUnLBJnmnODoiHbZAXoNwbprhpDDNLQk1D/OKgyrgozLInsR+4YvpM9B2V4+UO6Ey3u80ahhNpQRB0XZGhY+1WCJRMZ5V+ew0G/l1L6pVA42Neqi+J1Ypc/wbtrl/l73CT1GeSj5EDbkJLfJlCskXfCmixF8BOiU0BL1bOcKLoCFHSbeoOj7N2/1raj6B8yETz99f8LM8yj0t18Q5cwLMkvns185tO3bSBqBSehMia1o/3vEdlVgAfqAd/A8LW+q/r31HZTSGwVg26SVxFzLGU4izZ+TVZg05+FKQJE5XfKM00r1z63q04rjqWc8D+CQ6LhXe/qJfNUOWJDIF5trFIZ0x2rmm0XAe/8uYfcbRQQ2Q3Q0W90sN4ew2QSsOfh3jtBmEAibyK0282knnULLea1DNt7ygBYndHLKo9SbdHfVTSVwxlcf79d4N7qWPf5tGW3Toj6IFS41uDcHPPyJn56epH5sxz5RtE7Mw1+PYg6eHVnzBbu4GKW7PSPbO2pKsoxkcqm+e3ceYCWku+u0m8JdYILGsEYYqDn+XuN7qrnas5Gcmnr3nb3PqQwcvy2ftlm2Nskeec3ZX4Mpxu5i1sCNAP1cE/jsJHzDpX3r++0S8AeGXCGAntqqAbjln184WBvGXpqprDYk5jS/JUCt2dd+6YICYGorqtkSU4llJga9G5ml8IbAPvwZTBFbe8ItveaVKXSJb4Sq4FH/V9d3oTCM PbiSsxI3 ZbfBWolq51UDkYHDZexgBc9v3WOTF1vvlEGGda1zHWt/aiZO52laeW2pnvwQRK2V3/gTZ9LNN/TzA7h7wTxCuGfR3Dw6PxJ9S4ozPboRCBwNVTDqvI8AsUIv3LfSUL8DO3jmQDSO2eVufLfnfib6v0ppkl/FPdPD7AKEvxriQRWX7CkYNIGKWhEd7CV9cIax3NcFCknneUq/fIOPRmdpMf60fJ3xMuUFFINbcJKDazeQ88CTjvK6+b1Gw0U96fsplzJmuTqKYX9IkjNCJxLEzA5S1JvFAI4pKlzIYc+hsMR6+9e4zE1w/RkEbubsHkUMsgOTU X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: David Stevens Collapsing memory in a vma that has an armed userfaultfd results in zero-filling any missing pages, which breaks user-space paging for those filled pages. Avoid khugepage bypassing userfaultfd by not collapsing pages in shmem reached via scanning a vma with an armed userfaultfd if doing so would zero-fill any pages. Signed-off-by: David Stevens --- mm/khugepaged.c | 35 ++++++++++++++++++++++++----------- 1 file changed, 24 insertions(+), 11 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 79be13133322..48e944fb8972 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1736,8 +1736,8 @@ static int retract_page_tables(struct address_space *mapping, pgoff_t pgoff, * + restore gaps in the page cache; * + unlock and free huge page; */ -static int collapse_file(struct mm_struct *mm, unsigned long addr, - struct file *file, pgoff_t start, +static int collapse_file(struct mm_struct *mm, struct vm_area_struct *vma, + unsigned long addr, struct file *file, pgoff_t start, struct collapse_control *cc) { struct address_space *mapping = file->f_mapping; @@ -1784,6 +1784,9 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, * be able to map it or use it in another way until we unlock it. */ + if (is_shmem) + mmap_read_lock(mm); + xas_set(&xas, start); for (index = start; index < end; index++) { struct page *page = xas_next(&xas); @@ -1792,6 +1795,10 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, VM_BUG_ON(index != xas.xa_index); if (is_shmem) { if (!page) { + if (userfaultfd_armed(vma)) { + result = SCAN_EXCEED_NONE_PTE; + goto xa_locked; + } /* * Stop if extent has been truncated or * hole-punched, and is now completely @@ -2095,6 +2102,8 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, hpage->mapping = NULL; } + if (is_shmem) + mmap_read_unlock(mm); if (hpage) unlock_page(hpage); out: @@ -2108,8 +2117,8 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, return result; } -static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, - struct file *file, pgoff_t start, +static int hpage_collapse_scan_file(struct mm_struct *mm, struct vm_area_struct *vma, + unsigned long addr, struct file *file, pgoff_t start, struct collapse_control *cc) { struct page *page = NULL; @@ -2118,6 +2127,9 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, int present, swap; int node = NUMA_NO_NODE; int result = SCAN_SUCCEED; + bool uffd_was_armed = userfaultfd_armed(vma); + + mmap_read_unlock(mm); present = 0; swap = 0; @@ -2193,13 +2205,16 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, } rcu_read_unlock(); + if (uffd_was_armed && present < HPAGE_PMD_NR) + result = SCAN_EXCEED_SWAP_PTE; + if (result == SCAN_SUCCEED) { if (cc->is_khugepaged && present < HPAGE_PMD_NR - khugepaged_max_ptes_none) { result = SCAN_EXCEED_NONE_PTE; count_vm_event(THP_SCAN_EXCEED_NONE_PTE); } else { - result = collapse_file(mm, addr, file, start, cc); + result = collapse_file(mm, vma, addr, file, start, cc); } } @@ -2207,8 +2222,8 @@ static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, return result; } #else -static int hpage_collapse_scan_file(struct mm_struct *mm, unsigned long addr, - struct file *file, pgoff_t start, +static int hpage_collapse_scan_file(struct mm_struct *mm, struct vm_area_struct *vma, + unsigned long addr, struct file *file, pgoff_t start, struct collapse_control *cc) { BUILD_BUG(); @@ -2304,8 +2319,7 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, pgoff_t pgoff = linear_page_index(vma, khugepaged_scan.address); - mmap_read_unlock(mm); - *result = hpage_collapse_scan_file(mm, + *result = hpage_collapse_scan_file(mm, vma, khugepaged_scan.address, file, pgoff, cc); mmap_locked = false; @@ -2656,9 +2670,8 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, struct file *file = get_file(vma->vm_file); pgoff_t pgoff = linear_page_index(vma, addr); - mmap_read_unlock(mm); mmap_locked = false; - result = hpage_collapse_scan_file(mm, addr, file, pgoff, + result = hpage_collapse_scan_file(mm, vma, addr, file, pgoff, cc); fput(file); } else {