From patchwork Tue Jun 13 21:53:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13279234 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D4CA6EB64D8 for ; Tue, 13 Jun 2023 21:54:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 56FA06B0075; Tue, 13 Jun 2023 17:54:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 51FCC6B0078; Tue, 13 Jun 2023 17:54:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3E8476B007B; Tue, 13 Jun 2023 17:54:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 3241C6B0075 for ; Tue, 13 Jun 2023 17:54:00 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id E16FC140602 for ; Tue, 13 Jun 2023 21:53:59 +0000 (UTC) X-FDA: 80899077798.16.BB047D5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 9E13740020 for ; Tue, 13 Jun 2023 21:53:57 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gmXL4wyN; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686693237; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RMeXibFgr6WGH9TADmv6GKzvddxtLLfsVSaxyvBxKHs=; b=0FBQRRFEBAsJ3uHZUfHRAtfNQXJ3FwMtHNxlvuJsJK9NApNgFlxQqk+Say61qBy2adbbjX q200coOT3o2evaDQzlRQFcq3kzcZci4PAM2J875wqB9ibS2jDvJUCndx0hS52rxTLlSwcI VjKXjYGWnqbOd/OLXsune+TQksDpdyw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686693237; a=rsa-sha256; cv=none; b=K75XaBsN5xqokRLW6TiTq950ecBcGnzn9VOoaNBTpLwaSnjdFrYui+F30bn86mqX8Wjjxf OKT9j/ZKXH9v+ysisvw01WTPSX5pqKZbjVQ0Upsvp1D3K8wSvnYkN+AY/VYusRb/K2BYy4 hBw5v8EQgfqQN2/W6CWvNbocmeKznQs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gmXL4wyN; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686693236; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RMeXibFgr6WGH9TADmv6GKzvddxtLLfsVSaxyvBxKHs=; b=gmXL4wyNLjxv09p6dTqLBFX7a7fvOg85tVn+h3AnSMp36CQK5fLxD8n6DiFO7WfmyvX5GF aSRKC25MFBdprZT57LEruZWbKzUbbZLH1D1MwTSXTxCbUeCXZyKxJSYeUY4zXzVSe9taB9 Z2Ngu4pJzwpLejrToJUIOqpg9+j8XdQ= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-54-6zzRuUP0PeS6Hz7dIss30Q-1; Tue, 13 Jun 2023 17:53:54 -0400 X-MC-Unique: 6zzRuUP0PeS6Hz7dIss30Q-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-3f9eb7d5202so4776221cf.1 for ; Tue, 13 Jun 2023 14:53:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686693232; x=1689285232; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RMeXibFgr6WGH9TADmv6GKzvddxtLLfsVSaxyvBxKHs=; b=N+tCkGW0sjzWx4gPp7qEb7SM3LAN2s1MoDZntIudSvlc006nPWYzSuN0BH36I0UyRD wgcT7p8VrR8DxyaLtXditeGpYkOGaZ652/ws4ihNZQU9iF/1nMVtzVlc/yRpj7gQ3gvJ xKWDABO1XtRv7HP8VqqaJKkpeSrMDx86jsVuXvEpfIGAj3Y6x1kB7trXAaIkM0S2ZpZd X7rrYzRzT6z1Bk08i4wa6FMuISQ5OeDz4Zh60ImolJ4JtD33H6km7Izvh/88dEy93qpI R+Zd4LNPyXOMxUhEtWmDF6MDtWWCqGMOfxlIVi2PTKD0u+I+cHFcSbebHqacXyy0jT4N YsZQ== X-Gm-Message-State: AC+VfDxLzFNbPcbSvZzE8ux5TIjBYGYG6go7Q/l9uLBNCiFsWy+h7CHJ P4ZMPNfhF9HvympDsDvIDqzVLr0K6/5xXWp2+31aFYWxP6ndBczkJnoNpqyh5PVskcxxvSq1rBE kxWi4arI5Pq0= X-Received: by 2002:a05:622a:1887:b0:3f9:ab2c:8895 with SMTP id v7-20020a05622a188700b003f9ab2c8895mr18016472qtc.3.1686693232157; Tue, 13 Jun 2023 14:53:52 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ72MmVJ/KYdtXeaLxllJ/hfKOlDEtGiOvMuFZzCmuKll0yFpjBUzckwXKh9hVVGptkRtecEgQ== X-Received: by 2002:a05:622a:1887:b0:3f9:ab2c:8895 with SMTP id v7-20020a05622a188700b003f9ab2c8895mr18016457qtc.3.1686693231907; Tue, 13 Jun 2023 14:53:51 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id fz24-20020a05622a5a9800b003f9bccc3182sm4522330qtb.32.2023.06.13.14.53.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jun 2023 14:53:50 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Andrea Arcangeli , John Hubbard , Mike Rapoport , David Hildenbrand , Vlastimil Babka , peterx@redhat.com, "Kirill A . Shutemov" , Andrew Morton , Mike Kravetz , James Houghton , Hugh Dickins Subject: [PATCH 1/7] mm/hugetlb: Handle FOLL_DUMP well in follow_page_mask() Date: Tue, 13 Jun 2023 17:53:40 -0400 Message-Id: <20230613215346.1022773-2-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613215346.1022773-1-peterx@redhat.com> References: <20230613215346.1022773-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 9E13740020 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: d8wzessyoyqh9459gxsxcqk4hgzfmbzz X-HE-Tag: 1686693237-456230 X-HE-Meta: U2FsdGVkX1/X4dBNw9hGBV1SXEEpB+9PWQP0MNe4rGQhL1965nQ32en6f6+S9U+92dV18PFy1IbO9fM0fn6rYxyoXmI/06oKCIoyLDp5qshpuONRxKZV5GFBhFSdky9y4xPCTssq+uHkJcOVrH0E0dMZm5x3PQDQPFEEIX+cU06OcnpMu+lrAtMSGmG5bUCf2Lvb2PdpUuX7onO6m8t8dQg+qNRjUp90p1gJ8V+JwdjiYrdldOGh/0ZSKO0L6iQ6SxH3UEvHYUFxf9xdEpeGl2gofXuOK/c8pCDMxU1FOjq1HLhjlbBcfsD6C0v1t8CkpsZ4vYrm3LttHaU3XHtWElYdiJdZLT3VdGdT1JS3VOBln1ltIqblKoeI77wiel+tjKcnAurIWfDo7QAac/rw+1qncxCqSDx04ks+HYATDMf3hvJgSpTqHnp6QFhqqSXlzqOtLu848EsRZ/RyfnhwNa95ujuNSXLkKN4rgp49LYcwRPk4U5MTT0NJFoSXNxOZKqt0qx1TFWD8EMPCaC5eIYlJ64UzjW9NgD6cA36OK4mOvUEFYUTNpvUfY9D/E/D9QIj1th9aGaO7XSWq5mfmyavdumaEhlwmsCdlSDHK/Ybucm2lInkvG/7x97YWYJhnm7762gVHacDOvB9Vr8g0fJLcmPgxs0SRXYZLSo13pKiMJx7u8uabBm+syQvEzAtNviWZlNVRBr7DHmSueiUQfmJvu14CbKk9sQ+Fm3H6YBRar6mqDFBSKQHd/7/n/fCRrWhjnkHGftumyhpBwRSsm65PnJwtN6dsFeU6+aQFD966b8piW79xs8EnliXceWz0YbHhWmyVGI1DfzJP9qWvc6ERXbJ4AGaCSRpQ3lZZQRDPpTXDVJ/Fu9FVSfeC/qp3yaeEnzzarV5lxR8PYjPd8zp7RUtzQ07isQfe/oo0NiycKN56rg+GMFD8P6CeXYhi0I64W3g+LcWNj+8vxwE BVtrW/fz 8Y+06k8SB0rZpq9TZQtYcI/SkszMweuHJ+Mh5yjkhtVz6HIOrRMN16F6gbzpbAmw/dX9K96pl7MCbfSlROIjIbcI1ocwvyW1tzDtxgHmivr5MAEh7vY4kc33pmN0t3nAwRDo5b6/X/Re0yKg5KlirHWIvAo7aptBYsSBq5aIu0fKuU+CEhRdKePNp+1hWs83EgmmVYqLYSySC+2F9jMNACnyRFMd+r/8Z/gRHV5Mw0GPc/dQyYpYQeGpFc8g9cswwmR0VO25gPmCBO0/C2TMIDj9TnqxE9PvvBEwfA7uOXjkthcggBFNFi72r5GIwJxvsbypzPdrtPUjFlE1woywKv393ztg5OEZAkLW9akHFZLCNLHTJiKLICgbhHqSKm4DfCLyX/ya7jg7KT+P4RzZat6+fju3D1XfxrpArNRyVZu8t0goKlZ34t9gVnw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firstly, the no_page_table() is meaningless for hugetlb which is a no-op there, because a hugetlb page always satisfies: - vma_is_anonymous() == false - vma->vm_ops->fault != NULL So we can already safely remove it in hugetlb_follow_page_mask(), alongside with the page* variable. Meanwhile, what we do in follow_hugetlb_page() actually makes sense for a dump: we try to fault in the page only if the page cache is already allocated. Let's do the same here for follow_page_mask() on hugetlb. It should so far has zero effect on real dumps, because that still goes into follow_hugetlb_page(). But this may start to influence a bit on follow_page() users who mimics a "dump page" scenario, but hopefully in a good way. This also paves way for unifying the hugetlb gup-slow. Signed-off-by: Peter Xu Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand --- mm/gup.c | 9 ++------- mm/hugetlb.c | 9 +++++++++ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index dbe96d266670..aa0668505d61 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -781,7 +781,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, struct follow_page_context *ctx) { pgd_t *pgd; - struct page *page; struct mm_struct *mm = vma->vm_mm; ctx->page_mask = 0; @@ -794,12 +793,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * hugetlb_follow_page_mask is only for follow_page() handling here. * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ - if (is_vm_hugetlb_page(vma)) { - page = hugetlb_follow_page_mask(vma, address, flags); - if (!page) - page = no_page_table(vma, flags); - return page; - } + if (is_vm_hugetlb_page(vma)) + return hugetlb_follow_page_mask(vma, address, flags); pgd = pgd_offset(mm, address); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 270ec0ecd5a1..82dfdd96db4c 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6501,6 +6501,15 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, spin_unlock(ptl); out_unlock: hugetlb_vma_unlock_read(vma); + + /* + * Fixup retval for dump requests: if pagecache doesn't exist, + * don't try to allocate a new page but just skip it. + */ + if (!page && (flags & FOLL_DUMP) && + !hugetlbfs_pagecache_present(h, vma, address)) + page = ERR_PTR(-EFAULT); + return page; } From patchwork Tue Jun 13 21:53:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13279233 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DBB9FEB64D7 for ; Tue, 13 Jun 2023 21:54:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D86E06B0078; Tue, 13 Jun 2023 17:54:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D361B6B007B; Tue, 13 Jun 2023 17:54:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BD7238E0002; Tue, 13 Jun 2023 17:54:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id B09F56B0078 for ; Tue, 13 Jun 2023 17:54:00 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 8480A40207 for ; Tue, 13 Jun 2023 21:54:00 +0000 (UTC) X-FDA: 80899077840.14.AA03CC8 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 671CF160012 for ; Tue, 13 Jun 2023 21:53:58 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NwkvWmSt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686693238; a=rsa-sha256; cv=none; b=wE1Z7cgSNppI2W6C+H/DTWoDgKaiB3wjHWUwxCDXo7QOyIEKMZgyVnK0VXxKNKYWdeEmAQ fOSXJE4b51jDUC9yDuK+d6zMU9gi0YW0URk8b0jV139kEJhXdTGW41zdcdjpgM6bVAMded x6E96SinOEjpQXA7eA2LM5P23BKzZcE= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NwkvWmSt; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686693238; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=J0Si/SHg9D7WH2a+07u6wT7XlhbAs86SLIvfSHyM0qs=; b=QGCRafepJaY2mNY1XN15/rhiIZCwRZtzcZ2CILBJdyUmHAXXiT34mm6bmM1+mO1wSz+jfF R9jgziGWMS/5oNsonjTy3rHBEh7l1b9BBwchX2JiWyCE8gUxDeqcvMukOlxQL88kt3PN8H W/eKXEeqTUFH7/3DFNSrPoQl3+q4XKw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686693237; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=J0Si/SHg9D7WH2a+07u6wT7XlhbAs86SLIvfSHyM0qs=; b=NwkvWmStPzXjOE1c3eSf0Uy0SsB8gMjkT54CgKFhebI1/+EBDVLiRjMql09zPzRW8v5Tjg EvmgvwdpqCtUCjS9bbZfWeAbX0Ff7gTaUrKjL0gaMwsHeyht8uFMhvW9tVZNqVab5EsJWl XqXdfCiiAQx/ZcEw2ZwkibBBQ1n7kDU= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-416-hTNqCtOGNRSf8Z5zWwSbIg-1; Tue, 13 Jun 2023 17:53:55 -0400 X-MC-Unique: hTNqCtOGNRSf8Z5zWwSbIg-1 Received: by mail-qt1-f198.google.com with SMTP id d75a77b69052e-3f86a3ce946so13356631cf.1 for ; Tue, 13 Jun 2023 14:53:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686693234; x=1689285234; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=J0Si/SHg9D7WH2a+07u6wT7XlhbAs86SLIvfSHyM0qs=; b=fFb4alVz5WiW09Gspsx7kyZ+3UvPu2AUvCCKjDWkArbKoiokTvc3zhLkbOeVMbCY2o 4wztvdH6nsFuGW2MhBDFyglV8MENF5lLx/RDvYdIMTdqkVaz7Qn/6myk8Cm7yeL0/xcj Mu81xwtwU5oUSpXszDqo4YESjBeHgMTxOnTiEGHvZIcJ9srZv2/cNKt6nB5aJ/oquAIj rjzC1s8HJ1H1ybOpyl/OmZfHYhq0z5Y6KTpSGaO3zNySp58DyeomID/pW58NFYRl0qUZ Z3L81/zNYNa9w0u9x0J6Xc7CBs00ViYI2JjEGntfPntT/1xXoWWPkhPsykWILqOtdsnw cSfw== X-Gm-Message-State: AC+VfDx0Lb/s4WFVLX44WDK+H9nZSBc/gCtqnNR5NTGmQGROzfd5pzut vVUUpfoQrOZaaFuFO0WD9qmgbHDFmdAd47+fr6MINlXpE0JF28NiOGD3xJZQ5zafVBwHniA7Xm5 Y+Mpeol6XkjU= X-Received: by 2002:a05:622a:1981:b0:3f5:315f:5c1d with SMTP id u1-20020a05622a198100b003f5315f5c1dmr16598118qtc.4.1686693234461; Tue, 13 Jun 2023 14:53:54 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4tfbjVj7iW6/DtRd/s2e+Ti93d6WI6EV2BJqT02+lq+41oEnqcND6tzmbEH3x8gd+k6nduCQ== X-Received: by 2002:a05:622a:1981:b0:3f5:315f:5c1d with SMTP id u1-20020a05622a198100b003f5315f5c1dmr16598098qtc.4.1686693234224; Tue, 13 Jun 2023 14:53:54 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id fz24-20020a05622a5a9800b003f9bccc3182sm4522330qtb.32.2023.06.13.14.53.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jun 2023 14:53:53 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Andrea Arcangeli , John Hubbard , Mike Rapoport , David Hildenbrand , Vlastimil Babka , peterx@redhat.com, "Kirill A . Shutemov" , Andrew Morton , Mike Kravetz , James Houghton , Hugh Dickins Subject: [PATCH 2/7] mm/hugetlb: Fix hugetlb_follow_page_mask() on permission checks Date: Tue, 13 Jun 2023 17:53:41 -0400 Message-Id: <20230613215346.1022773-3-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613215346.1022773-1-peterx@redhat.com> References: <20230613215346.1022773-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 671CF160012 X-Stat-Signature: ntf6gfi5hhkpg3b17xujgiefz4kpwz8a X-HE-Tag: 1686693238-817898 X-HE-Meta: U2FsdGVkX1+l99wwiY1ENcSBpGoUqRqojtZ9AKMojRTRgmHnbKoBv1glpKpMx2VVb6XBJDO6ZvLMGJxlFIAuGdoVbQHUGgSLwzBtSWjyxCeIHPqJ7cZ6YV+smNzNIWqTwXBy2XC26HbDa6/A8uBUFDbfgA4+rd9azc+bhEvzh3WN/JWl24bcyb4a2r/zZdnnBnAyUiUqadcB2YLchrsI0FCC2C/hwpfw1elSyb8TJaFh3iit8yBlDotht3hw2/S1KUD5CDFX6WMgrlL4jYrMqRXCJCeskBrt3e1tyAtVx3BYPEk53m3Us6+13ibvOo51Jai4rA3DtNshetn5kb8Dd81+yAJ623L8UioSUl8FPEFhSMRwpAnz8UmYdv6BiG145Rm48WhgXPlXqHzMO2txRuNPtt/GaTL2eLXeKPa7B3hUEuu4CXSWf1brw11iTIQCe0QmvyACyzXRRH+StPEfsDjOUIJdDr9V7+kY4jjNkWNnM8SR2I4V5cEPbERriMbGzPt3S7V3Ih24VJrYFtOVqIjniEaRRZ15F2hLF0Wzs59g2s/SK7MloWIfi9k1NL8nNiZKthfj30tzVA6Rf0fTR3LsBecTm+ElKZmoYnBOiC1L0z+hi91SnKqJD35w/+WsqOHjBCCa/8fUde0PVUTHAOJve42MlLx4DECVXd8Rj2VVk3NcHYSb+ERRl6T6J6ERjyzvpx2J12zSbeG1UWxmOXW9MAjJwsQ9/dG1JjXXNoLYHJST/g/yqbbtyqV9IgIHnEjtjdU892bZEczS+LZjWWnNe8er9Npakzu/TCD3jelGzG/KJTOgW6YADvjz81I1HU23Fm8VVD/LLhdEq8IWVYLbSJBu4sKMZT1CxXzU5WIudCmkva5Zf8+p81YpJYZ8lu9sFjAjFEHoCj1v13xOHBR9SNgDloZ8CStp8H/p2qXnZvwi03rd/F14ErY/OPf3KD5Lv3grNzV6OzlGU3u IYfW3Wzs tlY3bkz+iRIpf1s7awyNDobTQx6Gfb7ShB0sLkF1bvyRrJLooKEks8qnAXXxd8Y8US47RsmsVW0NiO0g+nRW9YqilPLCCiFei5OdQG+DynnG0bkmITf71nI7vd5GyFfQW69Lc0pRI/BXlTcznCArNmYzu2yLE3aHgWwjT6jPlq7eH7Sn7h5Y0ZBPuovIl0w0XqJuQabYwo2pCGV3vJ5uHc1hqqsXy+c7OTcQIybigKtc5IZfJt8LC6GoAXF9+CWciyxOzkHGyMLYuAKaIFmMVXb75FG3j0d149fNglIB30Sh705norvjM3JyS4nmhwXtiJzPuf4XNw0ZWGcB8rvf83BAG7RwYprP6/ec7mx/ZaRJL4084Dw3kENTtm/lkFUjwb4/MI+NGrxkmTSxlQTMX+/inbyuVrYPXTD17Oytr1ZxRS7rpm7GqPaMl7Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It seems hugetlb_follow_page_mask() was missing permission checks. For example, one follow_page() can get the hugetlb page with FOLL_WRITE even if the page is read-only. And it wasn't there even in the old follow_page_mask(), where we can reference from before commit 57a196a58421 ("hugetlb: simplify hugetlb handling in follow_page_mask"). Let's add them, namely, either the need to CoW due to missing write bit, or proper CoR on !AnonExclusive pages over R/O pins to reject the follow page. That brings this function closer to follow_hugetlb_page(). I just doubt how many of us care for that, for FOLL_PIN follow_page doesn't really happen at all. But we'll care, and care more if we switch over slow-gup to use hugetlb_follow_page_mask(). We'll also care when to return -EMLINK then, as that's the gup internal api to mean "we should do CoR". When at it, switching the try_grab_page() to use WARN_ON_ONCE(), to be clear that it just should never fail. Signed-off-by: Peter Xu Reviewed-by: Mike Kravetz --- mm/hugetlb.c | 22 ++++++++++++++++------ 1 file changed, 16 insertions(+), 6 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 82dfdd96db4c..9c261921b2cf 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6481,8 +6481,21 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, ptl = huge_pte_lock(h, mm, pte); entry = huge_ptep_get(pte); if (pte_present(entry)) { - page = pte_page(entry) + - ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + page = pte_page(entry); + + if (gup_must_unshare(vma, flags, page)) { + /* Tell the caller to do Copy-On-Read */ + page = ERR_PTR(-EMLINK); + goto out; + } + + if ((flags & FOLL_WRITE) && !pte_write(entry)) { + page = NULL; + goto out; + } + + page += ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + /* * Note that page may be a sub-page, and with vmemmap * optimizations the page struct may be read only. @@ -6492,10 +6505,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, * try_grab_page() should always be able to get the page here, * because we hold the ptl lock and have verified pte_present(). */ - if (try_grab_page(page, flags)) { - page = NULL; - goto out; - } + WARN_ON_ONCE(try_grab_page(page, flags)); } out: spin_unlock(ptl); From patchwork Tue Jun 13 21:53:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13279235 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F580EB64DB for ; Tue, 13 Jun 2023 21:54:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6F6FC6B007B; Tue, 13 Jun 2023 17:54:02 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6CE4A6B007E; Tue, 13 Jun 2023 17:54:02 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 437B18E0002; Tue, 13 Jun 2023 17:54:02 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 2DE256B007B for ; Tue, 13 Jun 2023 17:54:02 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id EF1648063C for ; Tue, 13 Jun 2023 21:54:01 +0000 (UTC) X-FDA: 80899077882.02.9BEDAC1 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf03.hostedemail.com (Postfix) with ESMTP id DF37C20007 for ; Tue, 13 Jun 2023 21:53:59 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BhbO3fDe; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686693239; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=V6YucMw264yJynzunMB03zc5mw1fxLam6CUcenZWneI=; b=mzdqher6JuhEfyh6XDNrsoOLWiqUGAbGx73I7/ItxXbd1lwmmpi+kGOCj4vOHihMCvGDS9 tpRBPVPaOlm4Aa2HvSTehpljli6tDjwdw60pvF37vnv4JZNiSnOntKdDgwxXMgNV40kkg1 BDp+dn8PmlecW0eDN3jVZVUaO4IR9hs= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=BhbO3fDe; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686693239; a=rsa-sha256; cv=none; b=2t965fnaihg+u4j09TigbOtnloSrTLy7EJOMU3UpwttsCN/Nv4CpjPNGNRd8tdVjc3Gzel DBv4BUEic5YwNj1xAp0yHrTxM7CusXOiRyI+LxnavZYQEgAg49cxxakDD/YEvNdZ3lWBbs EummLP4NWIur7R1g+I4tny5S0nhTOq4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686693239; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=V6YucMw264yJynzunMB03zc5mw1fxLam6CUcenZWneI=; b=BhbO3fDeiukURbDWqGWVglUusSMEQ0TS3ULmRRcc3moX7gSWmNiyzRlU0AKmJHCaW3sPA8 T4gd/jivxAA9FLp1U7gbdJh9r+gQmmAZuFnH/tRJsg4ZkJTdUBbyMjtQLR7QkCkdiMbiq7 y1F9OciWzk1HnIsl8FQlmpt+3LOZ8HY= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-214-XOEF5OqrP5KROSte1TWouw-1; Tue, 13 Jun 2023 17:53:58 -0400 X-MC-Unique: XOEF5OqrP5KROSte1TWouw-1 Received: by mail-qt1-f200.google.com with SMTP id d75a77b69052e-3fb2e6ca6eeso2354231cf.1 for ; Tue, 13 Jun 2023 14:53:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686693237; x=1689285237; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=V6YucMw264yJynzunMB03zc5mw1fxLam6CUcenZWneI=; b=JvAI8qiOqYmwFZpXBGsiEp0z4Yw2bJZnjCGYyxjyN1+8G4JNbLjtu2KrGgzEPaz2mY 7LohlSGmmIcVLleQ3MEW8JUwTWgPzc83JtJP9zR1r3tHcAc5+3rlyWF6rGPUSSqPlAnF IGguwpXyLnWvH7wkE2LhNpNySPStbKw2rJ1koCGIqlXg85aTlNRz/CHiSthUX4+bMbb5 eUzb153bzXOz1tcX6dcu+TajpPjt8AkvEr5SqlPDVBiR6DK8mA5FqgxTYA8cSrtDtrAc eYBHQdClLN0feEzcaF0L7IEbc2kWZmLwupdtskRy6l65aLmlkGn0HmvoBR2jF/saN0DY Izmg== X-Gm-Message-State: AC+VfDwerghiKjnErrHHNKCOuOgYR+E3nj7F+/EZc3CSvLAWw0H+K388 N088PM02V+H+ufxkM8xCmmv+cqFR4zQcVpZk5SzfjCcDL17D+rwZYQP8eCtjtyNT+WlXSUq3znX JC44bVKSO1lg= X-Received: by 2002:ac8:5c4e:0:b0:3f6:b556:7c97 with SMTP id j14-20020ac85c4e000000b003f6b5567c97mr18596643qtj.4.1686693237071; Tue, 13 Jun 2023 14:53:57 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ47VYSlqHcJsfV8CfLHB7UhI/dxHot2PSaSbvtLl2psd47hStRHqC+I7+xcOOhhfmPV53DjgA== X-Received: by 2002:ac8:5c4e:0:b0:3f6:b556:7c97 with SMTP id j14-20020ac85c4e000000b003f6b5567c97mr18596618qtj.4.1686693236845; Tue, 13 Jun 2023 14:53:56 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id fz24-20020a05622a5a9800b003f9bccc3182sm4522330qtb.32.2023.06.13.14.53.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jun 2023 14:53:56 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Andrea Arcangeli , John Hubbard , Mike Rapoport , David Hildenbrand , Vlastimil Babka , peterx@redhat.com, "Kirill A . Shutemov" , Andrew Morton , Mike Kravetz , James Houghton , Hugh Dickins Subject: [PATCH 3/7] mm/hugetlb: Add page_mask for hugetlb_follow_page_mask() Date: Tue, 13 Jun 2023 17:53:42 -0400 Message-Id: <20230613215346.1022773-4-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613215346.1022773-1-peterx@redhat.com> References: <20230613215346.1022773-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: DF37C20007 X-Rspam-User: X-Stat-Signature: bjhtwmhxfge3eixi9en8d9ftfisecm5p X-Rspamd-Server: rspam01 X-HE-Tag: 1686693239-480944 X-HE-Meta: U2FsdGVkX18idRiffbnGmVwV2IhdRP2cdmlOTAeGQDfyzHMK9F72p9YJMq0CjKHLDOW3bS09IdP9mrBESWXz6+dCQs8+BOwYoqTD/xcSilY4PsArL6yB/9cztCf5EmlLwDN9zkfs9FEAr4f+dB84+eLEEtpgCvhD/KYPvZAkwnIrje0VUmq8QzbryPcHxKhO5Pr6UTY9JSl0tMRGwmRKKto+EYEXB3R8yoTnTZMqSdCTUN/7bkb75ps/VIxFU4qEyHTqXHdz3b72a0UBnhJoUI30WqEEty6EUlnIeesEdsHgnp8i6KmWBiahGwrJsPZ3qAGFjUAionMDaJTNo51mdH5ewvAyCAQ4Bx04u48dZFn4rhWpZX8/emTeiz/C2ZvD4kDTYtTii6KAY8EL/j0+O4EDTKDhhB4bpNN813GwBYHQakaIImUHghA6NAThpEIRP1cBbjRGEqAt8HHupP3eQSIYw7rO1FbRf0MaSMzkoImPdYxIcKTh4kIlSCNv5AnQ00LkyJVU/7PDz7QAyNf9t3ILRKnroywUVVXETfUw6zt0/ds/GiKjJqLNvG5eKPnY7C7ZVAMs8vP+XxM4T/v3hq/kKZpkUwJ5bf7mZM6UvPjZRMsQFTKAwso8wKsY2TDTFm1PxL2HcOlAwpiIKlqKa153E7aFIDPAVVd3onNkT5CDVIfRa8ZC39yDVS0/iT87VGbrZgtQAdVOuzpKS5k1l3gDnL8YwjKjhqCQDvXV7eaU3oD0/civAWEWEIwyx9/W8xzWfVFBngEmkyNWFw0rTVvjPauv8YKqyqerHJnaJrnGq57FtbUj3SgPoVzsjWKHuRqiNGU8lpsyqjtA6KB3mHIb78xtDXGY7AThF3Ov2gd++FZIfyPhrItqFunOSzpUVqT2EjLYJNQrz+4hvpTTUk0NuuIhDtDjGe9MXSQiifTBogtTLsi1rOCPC6g1WWDjx4TIpTMfrsfxEyNwg2w 0D71k/DS ucCzLMEtdvHd5eN4mTGLHgCLp+CIxYtNLdr8tX4pDt2CHRYHrlCy+k6zFlIC2NpV1zm9QV3oF91qs82HsLX8kfIAbL4TSUhbP00+81SEFKwITnj5ST+4wFS9hOdlgxfvpjyCPsM6AwTWVIJLKGpzCifyNmbGMHZk5H9VtTJ5ZV5lRkzLS7HE/WfxFfHMnaGg9igvhPpo5C9p1jlMMnmksdsoSWUoUZy1MdkPu2sfa9qfd+/UZlb0sdGqTVzQTpcK+Nz6WdBwiT/rpNM0iOWaxLS8xXcHgXjxtKGS66VFTvSgtfOF2UKGnkJrqxkp4W2nJOOqLzvwnfT04tEDbeq688Vu9afDb5BbG0+ZgJpLm8XxxvwOGC3N6zWA/+dHRxxbQdc1Qa8eUdxueAslqp31tHAESWmuAI6iblEgL03uRhDYgWiym/ttnZgJSDg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: follow_page() doesn't need it, but we'll start to need it when unifying gup for hugetlb. Signed-off-by: Peter Xu Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand --- include/linux/hugetlb.h | 8 +++++--- mm/gup.c | 3 ++- mm/hugetlb.c | 4 +++- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 21f942025fec..0d6f389d98de 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -131,7 +131,8 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_area_struct *, struct vm_area_struct *); struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags); + unsigned long address, unsigned int flags, + unsigned int *page_mask); long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, unsigned long *, unsigned long *, long, unsigned int, int *); @@ -297,8 +298,9 @@ static inline void adjust_range_if_pmd_sharing_possible( { } -static inline struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +static inline struct page *hugetlb_follow_page_mask( + struct vm_area_struct *vma, unsigned long address, unsigned int flags, + unsigned int *page_mask) { BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } diff --git a/mm/gup.c b/mm/gup.c index aa0668505d61..8d59ae4554e7 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -794,7 +794,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags); + return hugetlb_follow_page_mask(vma, address, flags, + &ctx->page_mask); pgd = pgd_offset(mm, address); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9c261921b2cf..f037eaf9d819 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6457,7 +6457,8 @@ static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, } struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) + unsigned long address, unsigned int flags, + unsigned int *page_mask) { struct hstate *h = hstate_vma(vma); struct mm_struct *mm = vma->vm_mm; @@ -6506,6 +6507,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, * because we hold the ptl lock and have verified pte_present(). */ WARN_ON_ONCE(try_grab_page(page, flags)); + *page_mask = huge_page_mask(h); } out: spin_unlock(ptl); From patchwork Tue Jun 13 21:53:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13279236 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2DE2EB64D7 for ; Tue, 13 Jun 2023 21:54:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BA746B007E; Tue, 13 Jun 2023 17:54:07 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 769FC6B0080; Tue, 13 Jun 2023 17:54:07 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 632648E0002; Tue, 13 Jun 2023 17:54:07 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 564606B007E for ; Tue, 13 Jun 2023 17:54:07 -0400 (EDT) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 20DF0C061B for ; Tue, 13 Jun 2023 21:54:07 +0000 (UTC) X-FDA: 80899078134.24.C950D1A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id 151B42000C for ; Tue, 13 Jun 2023 21:54:04 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IPNrbSgJ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686693245; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nVhPI7Qoz7ZqnzsG9+Pwmnk6AKrA3Y9yvdpVEt0mHXM=; b=nbEf/hR9RhCMCT4lkHJ71F8zF6xjW4JvVH98DFQ3r9+H/BDGNlAVntXZr2r/FgXU6S56OI Lbfvo/ZDWyymiZlWKBsIhCP8IsWdmUCKxsittCAiF9F//Pggex4Y6cTTxeXiQcotDKyeO8 SMaNGn+bWemSb9XjGtSCo9QPTIZ2k2s= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IPNrbSgJ; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686693245; a=rsa-sha256; cv=none; b=0NCwa2w+Asq8AzptyMfP8LVXh2q+gvrKpVMbWMF32xe17NhI8mo7pg1Oh8TXPLvYivnBgm 5V1LeTxegQmL1Oz815jyZ0JbBGgB540eWXlC8cn0/BROepkm0H0vxwL4h+aZU07dfZanGM FDqMCC1tKkly2LqBQa+yL6BkvHJLSQ4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686693244; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=nVhPI7Qoz7ZqnzsG9+Pwmnk6AKrA3Y9yvdpVEt0mHXM=; b=IPNrbSgJa9QT+ExJtse1wMncXd5Yr/yfgJiM5xoXpNMPkwfqCH9CO3DIcfXZCTq0ZQ+2WU RxbgR5yKNf9gsQJHnowZIeuLSZGY7Y2+jTU6RvwiIKmduHzCuBJt6KIZOkowV4ib7zNRsx dQXmiH4JCGhzZ/voU22g+GI60i/1nEs= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-557-JUWMdCrcOE6fy-Q3CGZuRQ-1; Tue, 13 Jun 2023 17:54:00 -0400 X-MC-Unique: JUWMdCrcOE6fy-Q3CGZuRQ-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-3fb2e6ca6eeso2354281cf.1 for ; Tue, 13 Jun 2023 14:54:00 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686693240; x=1689285240; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=nVhPI7Qoz7ZqnzsG9+Pwmnk6AKrA3Y9yvdpVEt0mHXM=; b=W3YSkrdZcRr5b4Uk3kT4uUXbfj/MtBzEpaef1qujQUEk2erBWm3LhhB1/KSthjE3/M CPWD0+FQyZbguhrPrNR8lBQ2JQrypmsFCzqwGIFqTLrYcdQoCEglb4y5L0nRvoRBUDlr KEC6XjYRaADkU4pCto+mrppxZmcybl5u1WxNb/DANpzboJoKC8Awt5I/XukFyKpYwCB+ VUnltHpU6W2P4+1CGWP0Ol5JLy5/bd7KrgSVthMcyUCirO6OnCgHp2NW+l6uYp8AktNy TpF5CSmsabLFwGyPQAc6a1ovx5pYyBdbpqj9zF3qCBbQoSAMx/UzJY0xVrJAa4++w52F nGSA== X-Gm-Message-State: AC+VfDzz8nCDMkdeCzNVTgZOHxZMAklPpEOZX0ylDuKDzsjPuaGjqYC8 JQ2NndaS6mhYTPy3AZTGh1U2pQSUsodoPCvsPSy2JBIKmyjL/f7Qp1o6w13oGC5sm216GMlNHgi 55Z1nQ4v0ErE= X-Received: by 2002:a05:622a:288:b0:3ef:3dc3:4a3e with SMTP id z8-20020a05622a028800b003ef3dc34a3emr17942030qtw.0.1686693239732; Tue, 13 Jun 2023 14:53:59 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ4MJSKcPdnj4lChUYnJRMcBfBWV2PeylliGTEYFlrviAqe21ltK55iRSzWqqMvZUCA8so0tsg== X-Received: by 2002:a05:622a:288:b0:3ef:3dc3:4a3e with SMTP id z8-20020a05622a028800b003ef3dc34a3emr17942021qtw.0.1686693239513; Tue, 13 Jun 2023 14:53:59 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id fz24-20020a05622a5a9800b003f9bccc3182sm4522330qtb.32.2023.06.13.14.53.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jun 2023 14:53:58 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Andrea Arcangeli , John Hubbard , Mike Rapoport , David Hildenbrand , Vlastimil Babka , peterx@redhat.com, "Kirill A . Shutemov" , Andrew Morton , Mike Kravetz , James Houghton , Hugh Dickins Subject: [PATCH 4/7] mm/hugetlb: Prepare hugetlb_follow_page_mask() for FOLL_PIN Date: Tue, 13 Jun 2023 17:53:43 -0400 Message-Id: <20230613215346.1022773-5-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613215346.1022773-1-peterx@redhat.com> References: <20230613215346.1022773-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Stat-Signature: n433wzqjcfd31ikhh59i39wz19gfuyzw X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 151B42000C X-HE-Tag: 1686693244-861430 X-HE-Meta: U2FsdGVkX1+BtBlPXLqpksCxgyzg9bFfs49dcnqwIB00EcbcDbbS6PJAqlEnh8KBAdurNfmCSri++CRQiVVl1P4VaFB3UdPpaUUqooOni46jmLfh5pLyOHV6XfkV9TAABnHChm+Q36jlhXGzSDdlkmFuhDp0SapW1j7lL6Exu6HcdLdmpTRxnmxkTxfWmyzqaOSsColUK/Q7qJ9R8BpYodKmdQoD5hVvMyeqhmC9ttCt5Y9ItDpPKwsDr/+2Z9fU1MrKFNu3HvCK+z1BD7b6gN+fDUOBLcW39EyI7yScwtFddXz7q9RJOqp9Nygs8HUk3V7lOSBhZRXfSUAipEP2UFYbbSJ2Wi93SCOXtPqcuvSb8GjlOAVbpEKd76YXV3DXNWSBKbz3FQtMD3kGexlIoBKxBFxk4TI4+zHafbnQ1G8s3ZJLPHTudo+VFc1fbWZjmLnk60cZtCq0Kx8c0aXtaO2FndWacvqoSFjHVqMQ4R++K+g0nG3L8HiiJYE9TYMNYRG5PpvAjmwkN9QuM1WZeyd/X/PKlpupjQJgknl54BqnkgPQDJ67I1MfqGQi/C5U5N74DI9J02DGUDMtow0DTWWGUZg4aWoeg3eNQoJonR/0oJ2dpyOl/1SqjNpJx3E5CxA+oYXQR7XMfDcfUMBDM9n3ABqv5Qi4WkGMgpWYW85tr6TGS16QrgPLGfPryiETB+MlgaIV0R2+pZWWvXghJxAYWg3GdItUqp9yO4k12URTKpYy4X89lGl9BzcrpYrnm48t+iYXVnNxrgXqG6Di8e9OHsYPPskDbn8XXGD7CAddWi6lgV65pNZvUB3s4nIuTSwmwOY5PMAXK407TxV3JUsBLW6aukmoQYiBaaBbfIxBbZUESC1vYAryGGExIy+W6aFWwumRFlpyQ+D1FT3xqlW2WCKlmBSi0Cy0kNPCynh2tqYWQ9sGc48BcJ3cmBaN2axbRyq3ZFMQkJKIDni dPTSi7uk rwirMkLHbV6DWbczEnPTedUTc7A77eaAGNEIl4m3atdxZU0XG8MRTNmY3EJXQuqKMH74E80QT85lweHs2PAcqWzWeikm0okrnvplb2OM/ZKvOCqKy140YhrTJvgC25jwiN9gyTQTATNicp5DT+QSAndpuIzFBCJVnrxKTCYj0jHdBBXvL05kBVspZayc6XBw575lg68EIW5b6gcnMZeJWjcw5ochWk608EffBVoJe2ma1H/e1uSjwVQiC159hXI8aX4/4zME2/x1bwwDh34wIgpOIH/AKB9dW9jp6k1oHvmFLn56xRY1CwSgp6JyjP+BKiP1bBfvlLAQEu+S87DjXAsvFw5k7rldLAy9sWe3qbUd3EHXS83iqIIohYJqXW7kanlUqoPFk+fKCZemFLhiSauxeNfVeHD/slvYHV40L2ZrXgAWs6cTdFqtQDA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It's coming, not yet, but soon. Loose the restriction. Signed-off-by: Peter Xu --- mm/hugetlb.c | 7 ------- 1 file changed, 7 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f037eaf9d819..31d8f18bc2e4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6467,13 +6467,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, spinlock_t *ptl; pte_t *pte, entry; - /* - * FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via - * follow_hugetlb_page(). - */ - if (WARN_ON_ONCE(flags & FOLL_PIN)) - return NULL; - hugetlb_vma_lock_read(vma); pte = hugetlb_walk(vma, haddr, huge_page_size(h)); if (!pte) From patchwork Tue Jun 13 21:53:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13279237 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id EB25FEB64D7 for ; Tue, 13 Jun 2023 21:54:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4FA656B0080; Tue, 13 Jun 2023 17:54:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4AACE8E0002; Tue, 13 Jun 2023 17:54:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 34BF56B0082; Tue, 13 Jun 2023 17:54:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 27FBA6B0080 for ; Tue, 13 Jun 2023 17:54:10 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id ED67C1C81FE for ; Tue, 13 Jun 2023 21:54:09 +0000 (UTC) X-FDA: 80899078218.07.60E1D26 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf11.hostedemail.com (Postfix) with ESMTP id CD9964001C for ; Tue, 13 Jun 2023 21:54:07 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CtL1Lr3u; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686693247; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=P+etnUAQLV4NR7aCVrXKWbeziVnwWOuYfO0O45I26FE=; b=Op2lmHeSO/EZsnAUdh6A9/tf2AqOPXTsiitcf21m9wsv89QBv58xyC9nN+yFOvW6Z/dotS UcocOY7m90mDTw4tOwMvZ8Ftr+FpaEjdVkDr4XFk/PZviVO24qNRYMNdk0ZJ523xepWhcI pFxTShClttmQ1/J1OwY08FX7xcyIFqo= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=CtL1Lr3u; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf11.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686693247; a=rsa-sha256; cv=none; b=k+Su2EJ45fbZ/5tZLkY78xSvFEu0OfblMoFSbldp7edfc8rO1qqEYELDVe4U8aee11Jml4 gAFOV8NYUDnvqWEYE3bGWis0amwmVH8gE79ZtIlJvMiyXMoEUAFxs9/8Y/c06wLvwOpB2k Syxaf5uSlmnKgdnpkmdkVjpN0toT0Cg= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686693247; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=P+etnUAQLV4NR7aCVrXKWbeziVnwWOuYfO0O45I26FE=; b=CtL1Lr3us30LUzvnAIEiiyHQrjEYxCevmJ3kfMhuQDDGoaND/YK4b6kXU68MC+07Oh8HOU mxcNT+nD4WqYSx5qKn8ILFNUrzaOhve5SJgJYxxWjYVZYDHDpeFurxamfJbFt4+EdFZ3V6 7vdxy5TbaHYjbv+2z1KnXbx2gHBPcz4= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-589-PcVpfU0WNSiyLCF0cUkE1g-1; Tue, 13 Jun 2023 17:54:04 -0400 X-MC-Unique: PcVpfU0WNSiyLCF0cUkE1g-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-3f8283a3a7aso13777241cf.1 for ; Tue, 13 Jun 2023 14:54:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686693242; x=1689285242; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=P+etnUAQLV4NR7aCVrXKWbeziVnwWOuYfO0O45I26FE=; b=LKHvn2l8bOYep7kYd0NRm4FAvsjAbzdrCz5isxxQwYWNYgs10Pk5fbZRrA5cXGqXQi jc6UqjhZkXbphQinKHXfDUH6r48hP6K3hThBsw7XwJq2wQXeR0gPU9+bwuFVM6E6lYUD ugihVWlAykxN6ujRC6Xrl5nIlCdsuSQRvaxTPofoIl+hFzuKb1Wg7H/XD/b8qLmpZGn6 DavxEh2sFt8cfCyF7MYq/FLPUMnNN+2LIBbH/JhqMqJCR6UgXqoTKHgNePmonm53/SGJ vZQ7j6i0m1OIYIjjMnTx/t169s48inKSUGI+rADniyhXBIrdLbZEU7Hjsa0xwkgqT4/m aAPw== X-Gm-Message-State: AC+VfDyy/LwjKrl65rRpFwn5JPwhPPSBQeRpTW81/Dm6z8X86jDg0MBR VCFCUbIpp4Bn5NKomUDdO6lW/JUx3IhcABlKz9kiL8qHTxRAaFv5Co/fs7Fuy9DlcQ0N0DFRuxQ 3wztefrEvebo= X-Received: by 2002:a05:622a:288:b0:3e3:c889:ecf9 with SMTP id z8-20020a05622a028800b003e3c889ecf9mr18140058qtw.1.1686693242157; Tue, 13 Jun 2023 14:54:02 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ63MJGi+Hqlnli2JVzskYtPLZcbne7s+MQZPsj8TgX9mGHr+EIA5dm91k8Xq113ELiGPseUlA== X-Received: by 2002:a05:622a:288:b0:3e3:c889:ecf9 with SMTP id z8-20020a05622a028800b003e3c889ecf9mr18140043qtw.1.1686693241922; Tue, 13 Jun 2023 14:54:01 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id fz24-20020a05622a5a9800b003f9bccc3182sm4522330qtb.32.2023.06.13.14.53.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jun 2023 14:54:01 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Andrea Arcangeli , John Hubbard , Mike Rapoport , David Hildenbrand , Vlastimil Babka , peterx@redhat.com, "Kirill A . Shutemov" , Andrew Morton , Mike Kravetz , James Houghton , Hugh Dickins Subject: [PATCH 5/7] mm/gup: Cleanup next_page handling Date: Tue, 13 Jun 2023 17:53:44 -0400 Message-Id: <20230613215346.1022773-6-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613215346.1022773-1-peterx@redhat.com> References: <20230613215346.1022773-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: CD9964001C X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: jskjfo7e4hkazgo4y3w38yznq6w8ioam X-HE-Tag: 1686693247-347603 X-HE-Meta: U2FsdGVkX1/cA+gAWQApCf8Px8co6fvKJr28U7V04TPWZWenYFUZ48tirCdl1IyyFCcQFvVQGVlpF1v+/GMduQP8ttp2fhycmIGnxIb7d/zGOlJN5iZ4mdy0xPvTMPC1Cyrf5848QmkyLkKh04U9JBvbeQiRFMPjiu7nprZRrVOoiypyuBK0H1iQrJlBJY/QLv4QwirvHzaoCKOp5y6zo3I/JVPd6wikEXBREWyWpjNBOeSEMSJwdDDijcxQpOpETl1P2fT3j3T8dq05rC6ek3Ce+vfMY08yOmyQwmxzMyMcLXsZDTOUrPX91NQN5vz7OM8EpSjaT8MhReyEmquXcsfbjcew/4ii+JbCxiY3VoGkAK5GE1513z327HW97B0xblmiy+CHBtf+KHz4RpzKTYB5Am+3lsO/t87hvqyjrbi/muRs4jN5vnHenBovXsZbnSCjGDLg5zzgd3De3ybEW8PmSBtwSKYTvxCuwnut1f4RCiIh175gNP6OOKQRrIHuocsgmfl5sEvM/rM5M8PQdviwdpVV3ZMGP36IJvS5W8kFnBLIosjuRxMB6EprZY8+uuYrPlK4cRqvZBEt0vFsAyqqVu6kMLM6J//9E+TEKjVhL6uFVqaTVv8Z13PBG0C+gfNgpQX4jX+ESlJORr5WvZM4qLDtLfmdfRjfojV76c5i1bb13NGFtKohufBeTTJJreQ2QIMTRVjcWTF/ay+kNUkwm1DcX4kkftJ3vQf90S0lg2Ogr099Wm4IXT3ja5xJ9XRy0vS4+b6MTI5WvSCORuMET15B+UPkQzaiYhcAjfaSQCRljWpynty2GeBpOiy09ZWOq9B6f/TnnG07QOK/NaqTXOVMvXPPukAGfN97QBGMZ4APafFeNsF6DkKrV2SeFx+IfhHGpKLlvMmrozncXmhzmzK8KN3R7wmqWDIDPw5KIpou3d2K6GqzePgzc+h+8EIfRz9Bao+6GiuTUQu 4+bpjGng RvDj+1ic1N3e1yFaeudrujJ5Az1AzGgdaePt6SlmoX88SPk89dQaFPvMpl8gIkmV9c7ad2V3jp/Or84D+3p3Ha/NAWShYPTXtvddMUnGdezTPim/HdP0lakckNw1+WVzTlMJm5A6uskafVWWSBYVZaloM+Koxh1KB3E34O3grCVC8MOkx0rPWcwId8JkLXCnMb9K8G2KmOwVo9pMyb0HXzyTkEAYK3+7LOUttR9Osc8YQEEGqZyV2FBK0+lYHU/jBIpudx/ugRKVqoSwp7wjYlFc+QOdN7UF8/LyQjHLNqHWhR9WdSa8Jw9mJS0bc36CTPznVMwUEmreWXXYU7z/7Me6Bx67ayjUrz3Mo33Cps2NUaFwnbppxAkdecMn+/KkZHzOmdQG+mZ8WHkOlF21o4T/yReRMNMoC6J0mdQw7Akz4jfwQfv+Q2JyFXg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The only path that doesn't use generic "**pages" handling is the gate vma. Make it use the same path, meanwhile tune the next_page label upper to cover "**pages" handling. This prepares for THP handling for "**pages". Signed-off-by: Peter Xu Reviewed-by: Lorenzo Stoakes --- mm/gup.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 8d59ae4554e7..a2d1b3c4b104 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1135,7 +1135,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!vma && in_gate_area(mm, start)) { ret = get_gate_page(mm, start & PAGE_MASK, gup_flags, &vma, - pages ? &pages[i] : NULL); + pages ? &page : NULL); if (ret) goto out; ctx.page_mask = 0; @@ -1205,19 +1205,18 @@ static long __get_user_pages(struct mm_struct *mm, ret = PTR_ERR(page); goto out; } - - goto next_page; } else if (IS_ERR(page)) { ret = PTR_ERR(page); goto out; } +next_page: if (pages) { pages[i] = page; flush_anon_page(vma, page, start); flush_dcache_page(page); ctx.page_mask = 0; } -next_page: + page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm = nr_pages; From patchwork Tue Jun 13 21:53:45 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13279238 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 286A0EB64DB for ; Tue, 13 Jun 2023 21:54:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 745706B0081; Tue, 13 Jun 2023 17:54:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6F5E86B0082; Tue, 13 Jun 2023 17:54:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 597966B0083; Tue, 13 Jun 2023 17:54:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4BEEE6B0081 for ; Tue, 13 Jun 2023 17:54:11 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 23BA3140369 for ; Tue, 13 Jun 2023 21:54:11 +0000 (UTC) X-FDA: 80899078302.23.B846919 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf08.hostedemail.com (Postfix) with ESMTP id 19C8916001E for ; Tue, 13 Jun 2023 21:54:08 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XzCv58Ck; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686693249; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=HwdaHYLXZAtZykSCGumfFt7VLYdtQYhxN3KKFt+1ERs=; b=TEJgJYbyUDPpx8NiY68YfBLVzZbOdJ6AmRUhkf+xrE3F1yPyIUcCmgSZbH+JhS6nv115+8 QEqlyCceSeUFB+pi0n5W67IkAqSC+d6g7il/sWMlwAk305Gthv6JBIB/0LNE4KOcw3vMJ2 2oWFEbWqobaxFhuKdgkwrstzBC5H3Ak= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=XzCv58Ck; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686693249; a=rsa-sha256; cv=none; b=KKxAiNDvPeQPWVV0mXNzDro7lPJjk2vB4VIXFM/JiIHJspqGQaq1FRhX+STKimhH5lqD/9 LTatabM/5VJI+Ylw9ea6wJTOpWulJmwL1UyzB4ZLx4PfM4hpWzfffapno4vopH3efiXd0A YWelobjAsGef03AdaNN1PfpZkOL4FBM= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686693248; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HwdaHYLXZAtZykSCGumfFt7VLYdtQYhxN3KKFt+1ERs=; b=XzCv58CkTaBQ956sxi7DDhvu3KpiIyryJLgZsGWJbtd/nNIBtkV/h20i67xHrXhomfXaoS VQ/2VVdQskEGZqQTw3X2N1A9n1shPlzXZ5iXKFxKUaVB4IKfVnqffnaPvVbfNzB+ylyXRm 3oRDujfh3RaRqZnha4CTlO6Kpsanxnc= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-490-bdPDXKZDMseqwO2MOkNl5Q-1; Tue, 13 Jun 2023 17:54:06 -0400 X-MC-Unique: bdPDXKZDMseqwO2MOkNl5Q-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-3f8b055287fso11088951cf.0 for ; Tue, 13 Jun 2023 14:54:05 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686693245; x=1689285245; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=HwdaHYLXZAtZykSCGumfFt7VLYdtQYhxN3KKFt+1ERs=; b=iaLb2ijftDWp0nLjYUsZleQQjuLKrp2jbwIKvPZ7yVlvHQkBPBc9kwWpMj+fDtAT2t 0chsmVfRC8RYVHDdqjqf4XcUMjG1H6bcbNI3R1e2e904aMKFROUoFJWyRAt0vvJn0Grc l+vn/SVdOFCmGu/ThnOg9NDFVREIGuZQMiGausgg4kHnX39pB0A4eFFMiFdAaiZRigGj /EaHYVoQd+uO9RImEkln20pLGkF+VRrDsuY3RwtZNTwi2xTSYUpK7NbT6IHt9tMz94Ki JaKtVte9ZvEQWzWUYOGJoeHbQRZqcBl9LDmNeRfBixwpDoVqYK6KwRXBDCJSmNaKVBPM 1/Kw== X-Gm-Message-State: AC+VfDxxlbE2GKofQVakvTdzkjsgQjve4ck+kS+c7lxdCuxJBoydDyU2 W9+9TJTwn0Dxs+JNVJK63lOUFlp+Pp0OgaqgaSEbUoEMx/GPDLas+D1wNkQzbaRDDogaeFKYBO9 aWAoyJn9k10M= X-Received: by 2002:ac8:5c4e:0:b0:3f6:b556:7c97 with SMTP id j14-20020ac85c4e000000b003f6b5567c97mr18597053qtj.4.1686693245182; Tue, 13 Jun 2023 14:54:05 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5oqukQ8w9LAlWsT5JOrpxotagWUzu2HoxH3Yt91lRCBju32vz1LdI431t9L/5PA4lN4Cy+gA== X-Received: by 2002:ac8:5c4e:0:b0:3f6:b556:7c97 with SMTP id j14-20020ac85c4e000000b003f6b5567c97mr18597030qtj.4.1686693244966; Tue, 13 Jun 2023 14:54:04 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id fz24-20020a05622a5a9800b003f9bccc3182sm4522330qtb.32.2023.06.13.14.54.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jun 2023 14:54:04 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Andrea Arcangeli , John Hubbard , Mike Rapoport , David Hildenbrand , Vlastimil Babka , peterx@redhat.com, "Kirill A . Shutemov" , Andrew Morton , Mike Kravetz , James Houghton , Hugh Dickins Subject: [PATCH 6/7] mm/gup: Accelerate thp gup even for "pages != NULL" Date: Tue, 13 Jun 2023 17:53:45 -0400 Message-Id: <20230613215346.1022773-7-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613215346.1022773-1-peterx@redhat.com> References: <20230613215346.1022773-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 19C8916001E X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: x4ifc1zz8ikwcp5sryjwnu5dc79m19qe X-HE-Tag: 1686693248-962996 X-HE-Meta: U2FsdGVkX19/dedTKdFyAM+BNXiqzVbfh1Zj4M/+Hz/O7Ix8QCtQ/GHRYix3wFljalG0EsY28d95fKhpiNtrRGo1+CMbuDvNJaPNDA93l/CAsjfGgzpgbEPXvlcA1NRX0uQTsn+af5nF8h7FFHn7nDrkPXzzcYs8+lcnqm/hMYouXlp47+OqXVb2BvHFJeiU2GNYGcOOJ51C5zadmUvLCGX9KttT2KSVGXa2hHOmLoq5YCYvaTdGOJn8r7A4UJw+SbYEy9+DBEGdX2kf51E36+uk3ULBsyjwKW/tvcU/JqcxUX0+ryAgcfvb010oSFYi/cbnQsIdixe4w2uASK9VSrWlB9JwCgk35X1T6QXn8NNuOeNVxE7VzM1OuDDzSI6r0fCq4D+2Tie2muzvXb2/FqNQebCxXxS7XH2qThcZRTgJEasqW7zJPILxeXXOrpNTebrXH7MhMVnuBjC0upOa+dWA7HvVkm073FvwpHItvbAMqiVVfQqMyEKQH1xpbY40NUPqQT/y4RgYsnq/9jXqVBHdKAiebxiKfh3ZVfhfRak2erK1TWW0bveYWxLba4pYw+ssiv3NS+V4PwVno3cbAFqr1Lg9/tcgF0haN5kSxzqJqpxpXPRcluE4cOo+l3R4uymDvZNIexotLu5Q/C/vWBjo3b9Ho0oe3fAoQWqLJztWc0U6jd/l1mAp9TTF4pRGpl3uQtbci2T+7xCM3TDlS3x5e5KYe9PAofPPx7Jsjj3xFJ8oj0fw37LS+9KtyBdwzqni1JccqEFoWlnhxiHvorbNBov7xeIfM0VE0CYeLcQdVnLBFVJrsSEOmMHF3wIXPxPsTplqbBW8icwiYC4DQMvuBowC/SwREcx3ao74HBJvIeyc5uz8eUk63C98BdaZiq8Idk8VI3wZD5rUp74AIO6QeaWYd8J9IQrYn2CkAQzWGr59EuiwLpkUSsstVTaXuvEiuSBQ/pZ7u4Idopo sHCIAKIy bq4A6Pev8Zbw/rR6uRWykkr3Izma5IEbHdYPlX0sja8FTD7Pm2+t5vxUJejfetkhfND12IM2abVMxTOdcWvIq/AlUCixaRI6Pvu5Bqqtyd4R7vaYZsyQ9LaNPgzNQ2pxrzXDIUmxmqcb44AvxHSX6vFxWGjTFG3iMscYDAJRaH+UsOffneT8JDzit/uoJKQdmHpzrudvXdZowHXtG10rhSMxm11AspvETlMNBOPIQUd/RU468hGK3EHyqZuTNmPt1sEbUjn6FRiaZx8RXreVyi5HtGuNJ6kyNbP51b1p+a/7vYVlgwjenSLXnUqaV2y9fQvmuWhrLQB0YY8ZMT8OliAD0K3lT0rEPY7Cy26hFvOOesW+q4vfgWjlmy2NScbN/Vg5l8oqC2HvZqp8ohFkJ20MwyUr5Ee2Byj+ekJc+BZIQCrFPhfCNZaHx1Q== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The acceleration of THP was done with ctx.page_mask, however it'll be ignored if **pages is non-NULL. The old optimization was introduced in 2013 in 240aadeedc4a ("mm: accelerate mm_populate() treatment of THP pages"). It didn't explain why we can't optimize the **pages non-NULL case. It's possible that at that time the major goal was for mm_populate() which should be enough back then. Optimize thp for all cases, by properly looping over each subpage, doing cache flushes, and boost refcounts / pincounts where needed in one go. This can be verified using gup_test below: # chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10 Before: 13992.50 ( +-8.75%) After: 378.50 (+-69.62%) Signed-off-by: Peter Xu --- mm/gup.c | 36 +++++++++++++++++++++++++++++------- 1 file changed, 29 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index a2d1b3c4b104..cdabc8ea783b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1210,16 +1210,38 @@ static long __get_user_pages(struct mm_struct *mm, goto out; } next_page: - if (pages) { - pages[i] = page; - flush_anon_page(vma, page, start); - flush_dcache_page(page); - ctx.page_mask = 0; - } - page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm = nr_pages; + + if (pages) { + struct page *subpage; + unsigned int j; + + /* + * This must be a large folio (and doesn't need to + * be the whole folio; it can be part of it), do + * the refcount work for all the subpages too. + * Since we already hold refcount on the head page, + * it should never fail. + * + * NOTE: here the page may not be the head page + * e.g. when start addr is not thp-size aligned. + */ + if (page_increm > 1) + WARN_ON_ONCE( + try_grab_folio(compound_head(page), + page_increm - 1, + foll_flags) == NULL); + + for (j = 0; j < page_increm; j++) { + subpage = nth_page(page, j); + pages[i+j] = subpage; + flush_anon_page(vma, subpage, start + j * PAGE_SIZE); + flush_dcache_page(subpage); + } + } + i += page_increm; start += page_increm * PAGE_SIZE; nr_pages -= page_increm; From patchwork Tue Jun 13 21:53:46 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13279239 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E542EB64DA for ; Tue, 13 Jun 2023 21:54:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id F0DB68E0003; Tue, 13 Jun 2023 17:54:15 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id EBCE68E0002; Tue, 13 Jun 2023 17:54:15 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D0FAE8E0003; Tue, 13 Jun 2023 17:54:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id C0BC48E0002 for ; Tue, 13 Jun 2023 17:54:15 -0400 (EDT) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 6F914AFC62 for ; Tue, 13 Jun 2023 21:54:15 +0000 (UTC) X-FDA: 80899078470.26.861F9F4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf21.hostedemail.com (Postfix) with ESMTP id 51CEF1C0003 for ; Tue, 13 Jun 2023 21:54:13 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EknLURo9; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686693253; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/X3bGqn1q+zHUVNMkc4k9dYCbdcixuWgARgx8VaUnYY=; b=YPwF4zhToqrpqM0CFirKRxaSAFxjsLDkRJW+GbRhSIWoEBKnP16meA/ppvJxdg62ckHkhU Ro4xIP+A3KygI/Q7euU3ndaZy11/zbQYlaeN8kpiEuaKaXTmiYJgypTJbKVBSniPIbFYpT 903+w88LucpawnxwxyoGOJ6xvKl6CQ0= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EknLURo9; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf21.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686693253; a=rsa-sha256; cv=none; b=Es1S9SEjJGJeWiGdwhvxkvmYmzHFvoH7Fe+LMFnasFX5gsYqzKHpQnuAVwgyeP3SCKPquP Y6gFfXjdyog2L5JdDJRHkEkrmQBg3WBMkUveZjm6HkmUWJ8yTN/P257SDbSGG50uyVRJy4 D/mvKwpIA/PQmyZyAO9FaHdSjkeIVNQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1686693252; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/X3bGqn1q+zHUVNMkc4k9dYCbdcixuWgARgx8VaUnYY=; b=EknLURo9ydjhZF642WII0DY2Ihy2Q3AXsQhj8HvTmkf05hKcEUmF6ZrQZC3D5nzxYzndR+ EE1Z+VKgItSxyS4YxxEAu0FYb7V+9Ya3yRwyOAJO18V4BsmB6Ykvn8CmZe0aLOGqPrXMJn QHPPVALLKPmSLItoQh9dDwiWALeqdi4= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-605-62CorX6JOSCl0AtysufExA-1; Tue, 13 Jun 2023 17:54:11 -0400 X-MC-Unique: 62CorX6JOSCl0AtysufExA-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-3f9eb7d5202so4777521cf.1 for ; Tue, 13 Jun 2023 14:54:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1686693247; x=1689285247; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/X3bGqn1q+zHUVNMkc4k9dYCbdcixuWgARgx8VaUnYY=; b=NFGdEaWrX6D2/ib/fhLUQoVeyD5po8H7D/9bQ47eqidoHsL/G0Kb49xAed9iNujO6s YDs1LwIXJ6jFCUMLZOOHA2TCdB/6SdpbBIQVzznO1ltA72f9AMQs879Kh/TIbOkIBEvz mbh0e3u49csLp5+kMNd8wblwI+yhVBiyd64GwnoZRUPOwpCaLtQchwFd2XTSLdr49Kla mXWuMaMVAsZyD1/+t9C26pLrRVcU/cCv2189721LqYvbQkAYaEWuLpA149dLK1TzstWc gb/4d5NTaaHrMFVlDstvq4znbJ26LhJDEbwLUHfTY/j7egX/WZPC0MoBzCbNN3oaSTKs rJBA== X-Gm-Message-State: AC+VfDyKjMC8Vj35+c5jVgXMI9IJcjygZS+SAFwVxhSXJkzCTa1qJENr mh1C7llgh1PdQOYgXMQflJvv/Cv46cvC7LXgzVGWLGaoEu5ge/nyulUBruNdg33Ed+df0IVexkn XeLJa2KP/6zw= X-Received: by 2002:a05:622a:1a02:b0:3f6:ab9a:3d8e with SMTP id f2-20020a05622a1a0200b003f6ab9a3d8emr17758119qtb.4.1686693247539; Tue, 13 Jun 2023 14:54:07 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5w87HGnELUi5EkIzhKvNW57lYdNaWbCEMcUpk8sbBZqYIrssusFuVowDO7JEjTYM2qoSnZFw== X-Received: by 2002:a05:622a:1a02:b0:3f6:ab9a:3d8e with SMTP id f2-20020a05622a1a0200b003f6ab9a3d8emr17758109qtb.4.1686693247132; Tue, 13 Jun 2023 14:54:07 -0700 (PDT) Received: from x1n.redhat.com (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id fz24-20020a05622a5a9800b003f9bccc3182sm4522330qtb.32.2023.06.13.14.54.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 13 Jun 2023 14:54:06 -0700 (PDT) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Matthew Wilcox , Andrea Arcangeli , John Hubbard , Mike Rapoport , David Hildenbrand , Vlastimil Babka , peterx@redhat.com, "Kirill A . Shutemov" , Andrew Morton , Mike Kravetz , James Houghton , Hugh Dickins Subject: [PATCH 7/7] mm/gup: Retire follow_hugetlb_page() Date: Tue, 13 Jun 2023 17:53:46 -0400 Message-Id: <20230613215346.1022773-8-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230613215346.1022773-1-peterx@redhat.com> References: <20230613215346.1022773-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 51CEF1C0003 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 8fo5rw1fuhojr47iggqystxy9o4m6tyu X-HE-Tag: 1686693253-257658 X-HE-Meta: U2FsdGVkX1/FT6/kAnknXZ2dQONRc40yh4uORSe7tReO3UHT4sq5atyAEjoioW15EVS7MWvydzeuTP+45ZJZWSsVkGk2v+aRDy/nTnCHwFjII53dJVcnNk6Y3ZbSTZ+EHJurRR3Z3Mm4y7J9Rm64ZxTRTY+fkUgOKWTG0GVoyKvIv4NPkf/A/91o7G4BxudRC4DRkY0MZ/4gslXMMmTPL//IMtoNT47iQXm7jmdMJCSK1HpZsuIHwr9GxG9tHYEfxbKNPRvXsTHnOTZNt2clhZPmtNYDckRHbPUrmbkyh7/Tm+QlRkLGFpLue8GVVqY17THr5mHjVc3iM/yYqv4goRPRJGKhbp3aDzP0j3m7NNL0GhYMTFQ2mUy3Ow+AxioyOyi+CJfvDKUCgJQ2QMMXFsoCdtdf1it7Ge7h3n5Lyue4twvzdoZf0LowTTIjY6WF3gP6Fm2kAjCWOXilp0li9mVVb6fe9UxU7NwwHwtLH7qSFhP/6MESXKDmilS2SGoT7aFl6oEyvvfSy3kkx8vTKfuFWX/KkW6WRkpDk0DF3D39bbVqGua9AcEjeLLRalVqJKACB33HDJiS96WrUmgrDb9yfvJFFvyZpPYEx0Buo6rD14JvN/NxEluKppjye8uLEUU/whARv+ksnJs0qB5AwMjeRlHDBcc/TSwImJZ76/8zvE+on8S5B7y219bSw55Z94zec9m5ptI13ourkorN7ufQw26xdHPHEBJJY/C5KKScp4wK3z4okBnnp0Cx6pbbj1ejHggI1Vtf/UfO90jZHeXaBoL4fw4ox3fA6F6DUBmfPWpiCG7DncQiPp5UCeNKb6zyPp2M+b7nb0rL21bppiNWFOu/wPvOzmlW1b/jxFHZ9CTnTnYWO7O56+diIR/xAA3G6hw8fw1tGDo1pK7FI4M0x+Ls+YOzl9ejK1K5RniJ+uJLeH+8Y+77FhdVshA+pmJz1Ug/bLRNPiEPQmJ No3cdEfz ZAHMVxkah5iXZLHfNFLK8Pd86K7eplByo4tcO/s/tKcbF+fERTWEm6RWwWSdiqWqdRSnXCCF6c4OU5u1s/W1WokJrz+ifKslrbH/Tv5qFSEI761kYG329wE5+BAfTJMMI1QFa8bkz3KGhyVXuSnwlUCuSFs7Zw+p18l90yelxu2pCbEWwfFWBWn8Zc3iC+UJAgu6RrNmKr4A9DGvvx9ZU85cptkjOaTckaqa68ZA1kKt29huM/9FrBLgfT7RMuhbUupwVMCuOTs6LR5WCO00SsktQrM6/fXlDzrR4Ksk/6L8xPVE+yXmhxuA2+jTkTA2euKAed3j3X42jKPno1JEF9HjhowSIYb5l6W3eEGK9okamMoRPfBi0CcTLboeUYMWkyK/8BjsqyfGBb3ZSQdLT/9ljMMjcJ/OSajLcGAaVaa7Mnt897gL/BH/e1A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now __get_user_pages() should be well prepared to handle thp completely, as long as hugetlb gup requests even without the hugetlb's special path. Time to retire follow_hugetlb_page(). Tweak the comments in follow_page_mask() to reflect reality, by dropping the "follow_page()" description. Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 12 --- mm/gup.c | 19 ---- mm/hugetlb.c | 223 ---------------------------------------- 3 files changed, 254 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 0d6f389d98de..44e5836eed15 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -133,9 +133,6 @@ int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask); -long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, - struct page **, unsigned long *, unsigned long *, - long, unsigned int, int *); void unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long, struct page *, zap_flags_t); @@ -305,15 +302,6 @@ static inline struct page *hugetlb_follow_page_mask( BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } -static inline long follow_hugetlb_page(struct mm_struct *mm, - struct vm_area_struct *vma, struct page **pages, - unsigned long *position, unsigned long *nr_pages, - long i, unsigned int flags, int *nonblocking) -{ - BUG(); - return 0; -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index cdabc8ea783b..a65b80953b7a 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -789,9 +789,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * Call hugetlb_follow_page_mask for hugetlb vmas as it will use * special hugetlb page table walking code. This eliminates the * need to check for hugetlb entries in the general walking code. - * - * hugetlb_follow_page_mask is only for follow_page() handling here. - * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) return hugetlb_follow_page_mask(vma, address, flags, @@ -1149,22 +1146,6 @@ static long __get_user_pages(struct mm_struct *mm, ret = check_vma_flags(vma, gup_flags); if (ret) goto out; - - if (is_vm_hugetlb_page(vma)) { - i = follow_hugetlb_page(mm, vma, pages, - &start, &nr_pages, i, - gup_flags, locked); - if (!*locked) { - /* - * We've got a VM_FAULT_RETRY - * and we've lost mmap_lock. - * We must stop here. - */ - BUG_ON(gup_flags & FOLL_NOWAIT); - goto out; - } - continue; - } } retry: /* diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 31d8f18bc2e4..b7ff413ff68b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6425,37 +6425,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -static void record_subpages(struct page *page, struct vm_area_struct *vma, - int refs, struct page **pages) -{ - int nr; - - for (nr = 0; nr < refs; nr++) { - if (likely(pages)) - pages[nr] = nth_page(page, nr); - } -} - -static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, - unsigned int flags, pte_t *pte, - bool *unshare) -{ - pte_t pteval = huge_ptep_get(pte); - - *unshare = false; - if (is_swap_pte(pteval)) - return true; - if (huge_pte_write(pteval)) - return false; - if (flags & FOLL_WRITE) - return true; - if (gup_must_unshare(vma, flags, pte_page(pteval))) { - *unshare = true; - return true; - } - return false; -} - struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask) @@ -6518,198 +6487,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, return page; } -long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, - struct page **pages, unsigned long *position, - unsigned long *nr_pages, long i, unsigned int flags, - int *locked) -{ - unsigned long pfn_offset; - unsigned long vaddr = *position; - unsigned long remainder = *nr_pages; - struct hstate *h = hstate_vma(vma); - int err = -EFAULT, refs; - - while (vaddr < vma->vm_end && remainder) { - pte_t *pte; - spinlock_t *ptl = NULL; - bool unshare = false; - int absent; - struct page *page; - - /* - * If we have a pending SIGKILL, don't keep faulting pages and - * potentially allocating memory. - */ - if (fatal_signal_pending(current)) { - remainder = 0; - break; - } - - hugetlb_vma_lock_read(vma); - /* - * Some archs (sparc64, sh*) have multiple pte_ts to - * each hugepage. We have to make sure we get the - * first, for the page indexing below to work. - * - * Note that page table lock is not held when pte is null. - */ - pte = hugetlb_walk(vma, vaddr & huge_page_mask(h), - huge_page_size(h)); - if (pte) - ptl = huge_pte_lock(h, mm, pte); - absent = !pte || huge_pte_none(huge_ptep_get(pte)); - - /* - * When coredumping, it suits get_dump_page if we just return - * an error where there's an empty slot with no huge pagecache - * to back it. This way, we avoid allocating a hugepage, and - * the sparse dumpfile avoids allocating disk blocks, but its - * huge holes still show up with zeroes where they need to be. - */ - if (absent && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, vaddr)) { - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder = 0; - break; - } - - /* - * We need call hugetlb_fault for both hugepages under migration - * (in which case hugetlb_fault waits for the migration,) and - * hwpoisoned hugepages (in which case we need to prevent the - * caller from accessing to them.) In order to do this, we use - * here is_swap_pte instead of is_hugetlb_entry_migration and - * is_hugetlb_entry_hwpoisoned. This is because it simply covers - * both cases, and because we can't follow correct pages - * directly from any kind of swap entries. - */ - if (absent || - __follow_hugetlb_must_fault(vma, flags, pte, &unshare)) { - vm_fault_t ret; - unsigned int fault_flags = 0; - - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - - if (flags & FOLL_WRITE) - fault_flags |= FAULT_FLAG_WRITE; - else if (unshare) - fault_flags |= FAULT_FLAG_UNSHARE; - if (locked) { - fault_flags |= FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_KILLABLE; - if (flags & FOLL_INTERRUPTIBLE) - fault_flags |= FAULT_FLAG_INTERRUPTIBLE; - } - if (flags & FOLL_NOWAIT) - fault_flags |= FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_RETRY_NOWAIT; - if (flags & FOLL_TRIED) { - /* - * Note: FAULT_FLAG_ALLOW_RETRY and - * FAULT_FLAG_TRIED can co-exist - */ - fault_flags |= FAULT_FLAG_TRIED; - } - ret = hugetlb_fault(mm, vma, vaddr, fault_flags); - if (ret & VM_FAULT_ERROR) { - err = vm_fault_to_errno(ret, flags); - remainder = 0; - break; - } - if (ret & VM_FAULT_RETRY) { - if (locked && - !(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) - *locked = 0; - *nr_pages = 0; - /* - * VM_FAULT_RETRY must not return an - * error, it will return zero - * instead. - * - * No need to update "position" as the - * caller will not check it after - * *nr_pages is set to 0. - */ - return i; - } - continue; - } - - pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT; - page = pte_page(huge_ptep_get(pte)); - - VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && - !PageAnonExclusive(page), page); - - /* - * If subpage information not requested, update counters - * and skip the same_page loop below. - */ - if (!pages && !pfn_offset && - (vaddr + huge_page_size(h) < vma->vm_end) && - (remainder >= pages_per_huge_page(h))) { - vaddr += huge_page_size(h); - remainder -= pages_per_huge_page(h); - i += pages_per_huge_page(h); - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - continue; - } - - /* vaddr may not be aligned to PAGE_SIZE */ - refs = min3(pages_per_huge_page(h) - pfn_offset, remainder, - (vma->vm_end - ALIGN_DOWN(vaddr, PAGE_SIZE)) >> PAGE_SHIFT); - - if (pages) - record_subpages(nth_page(page, pfn_offset), - vma, refs, - likely(pages) ? pages + i : NULL); - - if (pages) { - /* - * try_grab_folio() should always succeed here, - * because: a) we hold the ptl lock, and b) we've just - * checked that the huge page is present in the page - * tables. If the huge page is present, then the tail - * pages must also be present. The ptl prevents the - * head page and tail pages from being rearranged in - * any way. As this is hugetlb, the pages will never - * be p2pdma or not longterm pinable. So this page - * must be available at this point, unless the page - * refcount overflowed: - */ - if (WARN_ON_ONCE(!try_grab_folio(pages[i], refs, - flags))) { - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder = 0; - err = -ENOMEM; - break; - } - } - - vaddr += (refs << PAGE_SHIFT); - remainder -= refs; - i += refs; - - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - } - *nr_pages = remainder; - /* - * setting position is actually required only if remainder is - * not zero but it's faster not to add a "if (remainder)" - * branch. - */ - *position = vaddr; - - return i ? i : err; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags)