From patchwork Mon Jun 19 23:10:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13285008 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8303BEB64DA for ; Mon, 19 Jun 2023 23:10:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1ADA68D0003; Mon, 19 Jun 2023 19:10:54 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 15C488D0001; Mon, 19 Jun 2023 19:10:54 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F19118D0003; Mon, 19 Jun 2023 19:10:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id DEC178D0001 for ; Mon, 19 Jun 2023 19:10:53 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A74031C80ED for ; Mon, 19 Jun 2023 23:10:53 +0000 (UTC) X-FDA: 80921044386.01.308292C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 8C6E04000C for ; Mon, 19 Jun 2023 23:10:50 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=i+x7nZUP; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687216250; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KBG5VmVtD6hD7uqVIm/KwEJm6kTIhRuBKiuOHix9n+4=; b=P9T/rzS6pLC7FGy/IlPpVfZ04OKXZj56Kw+HTpJ991/exSQ5poZMIm90Rv3ge20/k2tClw 4fNdQu+JalTmFcHhjShDp0knzXlF+Bri7ve9UOgJEkLjf0Iy1K/VAjuDuQbwEjknN00huM vMwpKcZvPSKOWhM1WJcsJpUQMV5yKIE= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=i+x7nZUP; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687216250; a=rsa-sha256; cv=none; b=gQJGMTkymsVzxoR0ApYgxwdYRCIOx1s9GJ+lxDDCf1avUhZCrBN0lv8QCQxHmIbymn9gUp lF9GdDgO/3Ey0ipgv7DUGMC/PsjUW8Lgpc7HsOQIx3NMxg26Yw4Vfy0QNmmaiVtDl/4gf6 Jhza1kiwafgRT0xKyugC1wPO8IeIpPk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687216249; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KBG5VmVtD6hD7uqVIm/KwEJm6kTIhRuBKiuOHix9n+4=; b=i+x7nZUPGwfTQMhk+Ap2Q1Zv4G1I+IYFIlUkZJTqV6Qbk5CsnsUZGXFKw0EDULz4vNN+GS aGd7+CdEeQGxZjITfGUz74xXG1LXQ4dgeYkIDXkTQFTFeNcVR11uE0L7+q54bBTKmR5Rl8 2BN1vhgRuDTC0WqZqaLOrcMMTN4IR9g= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-449-2xm7KTEFODWIjzFM414adQ-1; Mon, 19 Jun 2023 19:10:48 -0400 X-MC-Unique: 2xm7KTEFODWIjzFM414adQ-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7625940fbf3so29805385a.0 for ; Mon, 19 Jun 2023 16:10:48 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687216248; x=1689808248; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KBG5VmVtD6hD7uqVIm/KwEJm6kTIhRuBKiuOHix9n+4=; b=VQCt7JwxsMD6U/eewXnCrbJbiF9TycIGfpnlVrpkz6lU9NSaezy/UVGKops1v0mxKb d5lsiqIK7Cg3c1E7dt8fUWVretSVLUr5ePWviy7kRUBjInLpuZlBdsjMZ0a1nIOAEPdL v/E0aixa+cOlRC8vvNzGOy/fJirlJfB6NnXGDDzSaPICYXvlc0+qlqqFWDYCm1mJ8oCa 5nJcQlAEHRKRQBajeH4VEI9vQ8aQcVh+u97b/SqFIRZzkSEB4cKHr6hK5c723/uxBd1k ergDt/XpHqfjaZwZlUt7PdNFqWq5IhxUHeU6Ut/j+5Qw0hEEELr1t+5dfmTJ4nHTKUMT cdDA== X-Gm-Message-State: AC+VfDwPYCJvqyzDXSGBbWCfSv7JYswAYK6JGoVOU3XCRfu8YrL/vQbX kvd0KFaoCEbR+OnW24uAMEzVgO+Hhg7y8xNiCYPRo7gTpXsI9G2J5vZ/iPJ31o7hKhL0NWA1WQF 6M6hqVXHiO4v5QDSTplXebxYz5xZQN0bZ2Mfw3/JOYaof16ViN9FCdhmzdJcMvFuS7qVw X-Received: by 2002:a05:620a:2889:b0:75b:23a1:82a4 with SMTP id j9-20020a05620a288900b0075b23a182a4mr13246178qkp.5.1687216248113; Mon, 19 Jun 2023 16:10:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5uYNYA6ud1mXk5kCO8SQ709g56KLRkH0S51iN3gvNVcm96teEhHrAWMqahC/SXJPrCrLeZqA== X-Received: by 2002:a05:620a:2889:b0:75b:23a1:82a4 with SMTP id j9-20020a05620a288900b0075b23a182a4mr13246155qkp.5.1687216247779; Mon, 19 Jun 2023 16:10:47 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id t15-20020a05620a034f00b007592f2016f4sm405864qkm.110.2023.06.19.16.10.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Jun 2023 16:10:47 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Mike Rapoport , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , John Hubbard , "Kirill A . Shutemov" , James Houghton , Andrew Morton , Lorenzo Stoakes , Hugh Dickins , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe Subject: [PATCH v2 1/8] mm/hugetlb: Handle FOLL_DUMP well in follow_page_mask() Date: Mon, 19 Jun 2023 19:10:37 -0400 Message-Id: <20230619231044.112894-2-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230619231044.112894-1-peterx@redhat.com> References: <20230619231044.112894-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 8C6E04000C X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: dy4szqq99s3sd5bfhg51ezjxs1hdwdwh X-HE-Tag: 1687216250-275305 X-HE-Meta: U2FsdGVkX19aNmqAA+wOzXSRjLRNz1ubz1CailU4rXF3pC/5B2aOxnbVqtecQ4L0mXc5+gyX+JGU76Jt27NT2EiDFALBnq2vWyB8rDyb/FbQEPrcfyVRAgbQNL8O+SaYw6E8s4SZLUTDiVFhjUAIyC7PwfgzgsjNQlXv5AjXeaSLWWN2OKhB586cVnA5esvS0LkewQFMrYiN1VqxRiMoI/Xm5sf+dRNLy11cwSQUtkOUdPWpAjP5cAfSm9xyheCIkRB0Vni2lZYHMNI+7C+KlyiFc9ZKxTNVgd1na+qb6t6TM1SA6LIZdvwS/SlhtCxW2eVTIsvBJlSGAFlAADbqvsj5URcyW04QqIlJD3FRD1IkiuhJQbDFTyS1Cbe4WMKk5SglZb9z69mPme9ASRKuUXZhZStBkJzxh9ccMaE+6tpDW/ZJAFPZZqwRL5ZA9Lt0KIDPqMhBu+43JfAusXM9s/SjIZCaEQupvveC4zRCBg264r8KP5Qn45khedwR4l7B0Iu8SdREjRLF7AGTrCRP/kkP94c/0oBNBLi936fhPz3UORCeROhlksOksrVweMUaeARscF5T7EWbDpThF4HkCX+/xTYrOK4k3om0j7MD5Ae961Gx3V4QzV1IRsvS15yK1GiA12X9MNNjRfi49bYKmPmIJ0zTCLRAdymm9KlPX8kH3L+/Py8pDi6AXqfX4TeaH4A4f+Au7Up6jxZPKdotGvU1ScMtLEmHFX3LBlM+ApUs+tUEu3YXnx28FM9A3S7XwRIt3CypP7ZCFRgWCcPhfwLR9WFYEqOKxuxEhreDJtgu4D7kxXNnaF4XW/DYecGsdRue8ybCKw/MBO/KE3SiQt2k313cd2JjZ3E38FaNmtUuQIfJyBQPbV4m7stRnw4CJmXI4ovCM1ieaCU8oTciw8YIyiDQPZTAR9GWnEVhdsAQYG0ttuStivk1ZXfGd3loyAHKFOom+Y54fhPbAzV zJV4hfGo ajuvPDFtdL6xx495Xo3zvnzMmWaTyjoJLuqM8gGhjTGk2zrS/NoxzO3B78AudP1lVCkjah5OfvPY2SDEREl12bcXDNmSMVhE+eDboTk9Cfbk/Cl+C6a7PmSf1/Nqhe0UxfLyryps7GC5mf268Bh6z3Z3/Nt4at1skEIs1kG4bywRHAwYBqGzstGGJr+ojy69oWPjRv0Up80ofdOXVz5lCVewkdbtcRwKyDILgPFmBi0b0C2mfAiNMXHS0I3YR+IB5n5jxh5cPfkaxYjRS2Ca7G5xmG58VLvbRnffgwol8m4OVffXRBWgEAbw2zWUp6MIvo7M9BdfUHuNvQqp6nqt2ZR+UcAqAtj4AE2mJc9fqz0ZEDFZf4G8Pwi1dgctMitAaNLAyeUOdOEGLizQ11VgYH7I3rMokXgHmokUZkZJOAk5YRN0+hVL9IGNKALUGfCOD0113ZCi6iaITnHA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Firstly, the no_page_table() is meaningless for hugetlb which is a no-op there, because a hugetlb page always satisfies: - vma_is_anonymous() == false - vma->vm_ops->fault != NULL So we can already safely remove it in hugetlb_follow_page_mask(), alongside with the page* variable. Meanwhile, what we do in follow_hugetlb_page() actually makes sense for a dump: we try to fault in the page only if the page cache is already allocated. Let's do the same here for follow_page_mask() on hugetlb. It should so far has zero effect on real dumps, because that still goes into follow_hugetlb_page(). But this may start to influence a bit on follow_page() users who mimics a "dump page" scenario, but hopefully in a good way. This also paves way for unifying the hugetlb gup-slow. Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- mm/gup.c | 9 ++------- mm/hugetlb.c | 9 +++++++++ 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index ce14d4d28503..abcd841d94b7 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -767,7 +767,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, struct follow_page_context *ctx) { pgd_t *pgd; - struct page *page; struct mm_struct *mm = vma->vm_mm; ctx->page_mask = 0; @@ -780,12 +779,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * hugetlb_follow_page_mask is only for follow_page() handling here. * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ - if (is_vm_hugetlb_page(vma)) { - page = hugetlb_follow_page_mask(vma, address, flags); - if (!page) - page = no_page_table(vma, flags); - return page; - } + if (is_vm_hugetlb_page(vma)) + return hugetlb_follow_page_mask(vma, address, flags); pgd = pgd_offset(mm, address); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index d76574425da3..f75f5e78ff0b 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6498,6 +6498,15 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, spin_unlock(ptl); out_unlock: hugetlb_vma_unlock_read(vma); + + /* + * Fixup retval for dump requests: if pagecache doesn't exist, + * don't try to allocate a new page but just skip it. + */ + if (!page && (flags & FOLL_DUMP) && + !hugetlbfs_pagecache_present(h, vma, address)) + page = ERR_PTR(-EFAULT); + return page; } From patchwork Mon Jun 19 23:10:38 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13285009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8AD0BEB64DB for ; Mon, 19 Jun 2023 23:10:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0C1A88D0005; Mon, 19 Jun 2023 19:10:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 073DF8D0001; Mon, 19 Jun 2023 19:10:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2D618D0005; Mon, 19 Jun 2023 19:10:54 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id D63138D0001 for ; Mon, 19 Jun 2023 19:10:54 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id A46EB4047B for ; Mon, 19 Jun 2023 23:10:54 +0000 (UTC) X-FDA: 80921044428.23.19EDF3F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 7B8BC20006 for ; Mon, 19 Jun 2023 23:10:52 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DRTFrl8V; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687216252; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=AeVTfhKtj8N8KTpZF3pjKuqjoCywpEvllD4d3FaXf9c=; b=0qri0enfVBJWxuGGM/bx3DHoMSJkKASAaNg507vwoJJ832f6psao43l7PziVxvyUYVFJ4R VJ7yy9OE1lq6Aa6u1hkcPUnr11rCmwkb4431mhPr2ENEEsh5BHf3V09/CKpcF1U9o8/3iS hNz2EBJtV/xIxzRUu/yBbMJMTRAenvM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DRTFrl8V; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687216252; a=rsa-sha256; cv=none; b=s5Nd5XFINfMwbMtrBIyWH1AInzFUGi6+FpDjZOYiJXZRUSinLvujboXO10pKjddJMqoGlo PlcUcJfmhHggXv/PMoAVkFtNCQqfgOL1pzIaRJVfUs6V/Fvbz5WhxuX84qm6UpCIW9/jOs +esoTRE51Kdl9YGHqTzlVbzX7qCAZk0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687216251; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=AeVTfhKtj8N8KTpZF3pjKuqjoCywpEvllD4d3FaXf9c=; b=DRTFrl8VDRY4j2hMgJ2dkEp64rZmU6jCP1Y3spWcZvS2WycmuhEaIduwfJm2VYbd7t2UQy RRrM1lQPSDmE0R3YTibu1S532jOKGJRZnGJ4XMrHvP9h1oOPrNaDzCfmDRL6+ck5zm1ay3 6N7xjcZVfxm/abqpvCHs+gjMYwkTMPw= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-22-tVQDZ8NLPkyJj5vft-H-Rg-1; Mon, 19 Jun 2023 19:10:50 -0400 X-MC-Unique: tVQDZ8NLPkyJj5vft-H-Rg-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7625940fbf3so29806485a.0 for ; Mon, 19 Jun 2023 16:10:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687216250; x=1689808250; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=AeVTfhKtj8N8KTpZF3pjKuqjoCywpEvllD4d3FaXf9c=; b=eSNx9yDp67xMQfoJ9bBl9Q5a2MxeRjBJTLjo5K0/Dj/EgDjF6AyAjJoB+JDmnL55rD Ack4Fg8m6VptYh8zO/kOKYBtz7luqYvPnZpHhronWZybSy9qKAfquFxNS/hCUKygtY5G xA2MtyDfin89PzAnAIP2xRFCoefuFy2zWhKt0dSS4pbMUULekGUL92iYiyx+iC/urtBz opgs5vnq5WPk4DEBscQIrKDJYdQ/tXm0hIR+QzUspuUgaes1Smd3m99/IcCLuFuFz/rg wDl6bLQ8XfjaqPzj3l7ZFXWO5AwHrRECp7FyjoqKqn1gY13UsZbjp0UjuumSkgk/vYep BTJA== X-Gm-Message-State: AC+VfDywy6TgCwsYqaqbKJyLVoXaNPlJAIq1d2YPuSpNxrTTp1huti8H M/t+cu7Gp53PNp/PfY3KU0RJyHYWGZmJrb1DgFHV6jjWmt+LLs7ahQhETYHGo9mNL0o27s06QoM EZpc6kwVaPfH8OBoGx3iHsdOONrmPtX6yhw3lL2bO1fM6n8XKGDsCAT/Fvts1+ePppXEs X-Received: by 2002:a05:620a:2889:b0:75b:23a1:82a4 with SMTP id j9-20020a05620a288900b0075b23a182a4mr13246253qkp.5.1687216249888; Mon, 19 Jun 2023 16:10:49 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5+hV0zbUQ9aN/HCxdwdLyhYSlUogcbP8i7JCZJTHjyygAvLUH/tHfcgC5j3ZztZhUlzdg3lA== X-Received: by 2002:a05:620a:2889:b0:75b:23a1:82a4 with SMTP id j9-20020a05620a288900b0075b23a182a4mr13246224qkp.5.1687216249526; Mon, 19 Jun 2023 16:10:49 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id t15-20020a05620a034f00b007592f2016f4sm405864qkm.110.2023.06.19.16.10.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Jun 2023 16:10:49 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Mike Rapoport , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , John Hubbard , "Kirill A . Shutemov" , James Houghton , Andrew Morton , Lorenzo Stoakes , Hugh Dickins , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe Subject: [PATCH v2 2/8] mm/hugetlb: Prepare hugetlb_follow_page_mask() for FOLL_PIN Date: Mon, 19 Jun 2023 19:10:38 -0400 Message-Id: <20230619231044.112894-3-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230619231044.112894-1-peterx@redhat.com> References: <20230619231044.112894-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 7B8BC20006 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: pxasjpgu419s5ott9356rqbewbtu49dw X-HE-Tag: 1687216252-270317 X-HE-Meta: U2FsdGVkX1/gv3VX4/dlfxJZ8/j/k0p+MR93nSniSwqvE6eVMDANypTD0MTSIPN3wfpVlzWeAfq+C/tCieO2sUZUaFngQf76bJNnG5u4xHEwRNUsuSf/KPZeOGldoC10ACLyubJb3i0kpu2q/RSCM3Q6GpaSedqEXYQJvUn1Fc/nKPb+G82xPYKg08vwBHk0FXhIo4nmGhlmuSbPj56iL+zPUEJQn98beNEE48iIac1hwdme1bjx79GRhAnBLSefMGYXlrErakdE2EPjkRS5zOzrsEdwF3VdvR38CgK04hge1WBNxBJrNShET1EzykE6zQWHs6NpWYzjJs5bDyrla+AI0B3Gf0HGaPdbjGJtawjXYTWi7kWtWspbZPotGhZiP1UA3b2hUk/Wsg9fKrUyendg1qi+Ais4eexw8mjSO5RKaCc8bSuTI2R8Rneyr6zrbiC67WcYDtfCT6m57GFmDUMV1oS14cBzVv8o7oLJTRzz4ERStm6HkvTC6yhyUYDYMlMqT1NqXYBJWCslUuR3h3afjtm/F+7Uy2C5qcQYrjUEIyX15jjfT1Ze+VytEyKAN6QXla5ul7WuC2UUZNM0lzHwJ7mXzAdy31clvvwWqXj8HqtpuYTJYT4qjI6Eas0Nt4ARcVZWIam7/M3FEiLk+VQUywijrmMVFU7vSgx56dPNwoOiWn/j7o3CBomLosRx18xPxsMdaGTud5jtWaX7jj0i1AvWcLkqimAP1a1cjdIV8UeyUD2JD0i+9SqPdtVRLHIzI/eGFp50W+yapNtTtCWpORKUSxAsGx3SCDlUWP0JlWQm6oxfOUAixXkOgxYKSnfKOGM1SSr/hxbMRPDMtitf4L4xFMzkh9GaHIMdvR62LfWNr3PtWnF/mWWB9oynf5uILH0FFDsXkl7Cq9gu17QusLZHhLYBJyjH6FSs7TbxAHmgaXWWE7bLKMwCnpe2GnXO8r2a36Cwm8WIf9H YctgR5oj Lt4UDZ8hh62cb4Lnu+QkogynbhBsaTUDGivrBEKTU+eFJytA887IzmFXP9nCOt0yMTrW+i0eDhfJ4qhM+ZKYgSmBWWKScujxZR2Mb/xsg4dTQ5O7kwvUVeoddR5Z5ZTCp2KGQNlbfRMsL+LIdW9qgDloA7mt6eAH9b8HXmfgKRzRqV01RevnRjcc0H/8fc2SO9M97ihPtWsq6lTeANXC8Y8DEtBB73qKzT5NfmOJIcv23QAHKh8fEN1adWOBtdxnAtD6m9Vv8yJ0oELe6dG02v2C35XrCZpY29xjDmFLy/+3KuSpAIWbTSi6iOkHKZFZEhpPDlFQ3pu6fnqVi9OqsRgFT0wSdECyP0Kg1UOmp2Hm4oy8MvrTPgZPsNrEIwQkWJD/q7vzqt57FlEOqd9LK53lAU7UJrom8Lln8IsnIxfn5Cw+CyXppRd0rbXfT9clh1/5qISEdxxzL18I= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: follow_page() doesn't use FOLL_PIN, meanwhile hugetlb seems to not be the target of FOLL_WRITE either. However add the checks. Namely, either the need to CoW due to missing write bit, or proper CoR on !AnonExclusive pages over R/O pins to reject the follow page. That brings this function closer to follow_hugetlb_page(). So we don't care before, and also for now. But we'll care if we switch over slow-gup to use hugetlb_follow_page_mask(). We'll also care when to return -EMLINK properly, as that's the gup internal api to mean "we should do CoR". Not really needed for follow page path, though. When at it, switching the try_grab_page() to use WARN_ON_ONCE(), to be clear that it just should never fail. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- mm/hugetlb.c | 24 +++++++++++++++--------- 1 file changed, 15 insertions(+), 9 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f75f5e78ff0b..9a6918c4250a 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6463,13 +6463,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, spinlock_t *ptl; pte_t *pte, entry; - /* - * FOLL_PIN is not supported for follow_page(). Ordinary GUP goes via - * follow_hugetlb_page(). - */ - if (WARN_ON_ONCE(flags & FOLL_PIN)) - return NULL; - hugetlb_vma_lock_read(vma); pte = hugetlb_walk(vma, haddr, huge_page_size(h)); if (!pte) @@ -6478,8 +6471,21 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, ptl = huge_pte_lock(h, mm, pte); entry = huge_ptep_get(pte); if (pte_present(entry)) { - page = pte_page(entry) + - ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + page = pte_page(entry); + + if (gup_must_unshare(vma, flags, page)) { + /* Tell the caller to do Copy-On-Read */ + page = ERR_PTR(-EMLINK); + goto out; + } + + if ((flags & FOLL_WRITE) && !pte_write(entry)) { + page = NULL; + goto out; + } + + page += ((address & ~huge_page_mask(h)) >> PAGE_SHIFT); + /* * Note that page may be a sub-page, and with vmemmap * optimizations the page struct may be read only. From patchwork Mon Jun 19 23:10:39 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13285010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94D4DEB64D9 for ; Mon, 19 Jun 2023 23:10:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8B0818D0006; Mon, 19 Jun 2023 19:10:56 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8628B8D0001; Mon, 19 Jun 2023 19:10:56 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6B4DA8D0006; Mon, 19 Jun 2023 19:10:56 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 5DC0F8D0001 for ; Mon, 19 Jun 2023 19:10:56 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 376B01C80C9 for ; Mon, 19 Jun 2023 23:10:56 +0000 (UTC) X-FDA: 80921044512.13.9315DA5 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 19EB6100005 for ; Mon, 19 Jun 2023 23:10:53 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WkZtwASx; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687216254; a=rsa-sha256; cv=none; b=SWWKkYF/WKztYlFHSyKKrpmykdwnFkB9pRpFFAkP16hncdy+dLloghNivuEi+1ZCAE0YND Kn27Zno6PKB7yKQUI6K9TS4/PzPRsGFv2Zb+kR4kRXuxH96Y3gUWN8Y5ynNvGx5zsi3YDv ZedPH1G75dlPjjt/8zjovWE2ku/XTHo= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WkZtwASx; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687216254; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=6bDfMbeRbNTGIwHWXqT4W/WFCKBAZr5szTZ9WWVoPyY=; b=kZqjBlNmEP6Hc+LljNUpAfn+EWXN1NqvMY5cCj1H1tL2bvHveFSOMhKC7lPfobu1BmV3n9 BrPCycJroRt96lNRgGvE4n1ECRqok2nWp33QfRa1KWQRD54TnIs3rFEnvQp1unJrbI0qLU kb5iU46uhnvSpKtcHQnEr5TwF3SgSnA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687216253; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6bDfMbeRbNTGIwHWXqT4W/WFCKBAZr5szTZ9WWVoPyY=; b=WkZtwASxGvwt+FQcGMYIuuSxxFRhEA71QP0syy8r2rkv0FhOdjQ0yv8oE0sO0YEnTRgBax rlBxMYLzM4TzKljJiyrHKReBgwveJPhtwREBHG+xcMVYztTzMOBeBFmZB7tIvYO01oU7G6 AJduXWJWQkRumX0BVoZzzgz3FuAfEbs= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-590-gwrobBKnNWGYPj0_mgfr_g-1; Mon, 19 Jun 2023 19:10:52 -0400 X-MC-Unique: gwrobBKnNWGYPj0_mgfr_g-1 Received: by mail-qk1-f199.google.com with SMTP id af79cd13be357-7606a134623so64595385a.1 for ; Mon, 19 Jun 2023 16:10:51 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687216251; x=1689808251; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=6bDfMbeRbNTGIwHWXqT4W/WFCKBAZr5szTZ9WWVoPyY=; b=mETjT0EeGtxqcZ8uGJoggzUApCuZDSJ92+d0b171dtZxP7KBFarlLVpb9YpsJ7rtte 1nX1PXScArucpI3hu78rV/QVdpVKzrf07p5fmpoGIwPSR+ccbMPMU3WHmw8H1eTjwls1 eDqrYh/71kTnNGJcQJj0Px+5/S1w3r+4zv+Dou5VRerISkeGYWdyKbOMgA71bhhbBXDw sOavDHeKgwOs3ioJNaCe8AB+QRwAszsImqlfFM6aDu72PzAPXzIcuGLR+GkHU8oGsQ/t qmeAjoAgrzfcSgGJhwzk/JhiCteSIR1J1U2HGXqsYI/hbfwxgXwl+6KI3e937i1ZoAkN 8vIA== X-Gm-Message-State: AC+VfDzZ/3CDP7fDaQt9OdUwNzSN7PzMSy+BRzP3kWha9wEX1N1jyltH /rZZTVspfX+iip4SOkGQe6nUXCZmA4IU8YdawzZvo0iv5G5sL0vUikBbveHdj2CrKOIHUzqX24b +gExUj+RZZSHbui5qILMZNYrMtgys9dDNCioesns4VSmYvSZiyifW6XskarpGr0x+r9Rz X-Received: by 2002:a05:620a:171f:b0:75d:e31a:a015 with SMTP id az31-20020a05620a171f00b0075de31aa015mr12743219qkb.2.1687216251249; Mon, 19 Jun 2023 16:10:51 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6l3RVvQSr/D1LXHPe5NTuMnLt2zR7kaZEIqF+D9ThQr41twmjAeA1Pct5PrS5DNmUPlNg8Gg== X-Received: by 2002:a05:620a:171f:b0:75d:e31a:a015 with SMTP id az31-20020a05620a171f00b0075de31aa015mr12743182qkb.2.1687216250831; Mon, 19 Jun 2023 16:10:50 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id t15-20020a05620a034f00b007592f2016f4sm405864qkm.110.2023.06.19.16.10.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Jun 2023 16:10:50 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Mike Rapoport , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , John Hubbard , "Kirill A . Shutemov" , James Houghton , Andrew Morton , Lorenzo Stoakes , Hugh Dickins , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe Subject: [PATCH v2 3/8] mm/hugetlb: Add page_mask for hugetlb_follow_page_mask() Date: Mon, 19 Jun 2023 19:10:39 -0400 Message-Id: <20230619231044.112894-4-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230619231044.112894-1-peterx@redhat.com> References: <20230619231044.112894-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 19EB6100005 X-Stat-Signature: j3tjn6hzum1yrsnnf1gn3fki6zmhae57 X-HE-Tag: 1687216253-438355 X-HE-Meta: U2FsdGVkX1/geUQnS96ALUDzrvRJq1A8qwAPC6ZHXMdu2vieZzQYkDM504CX28Px8DpTMoxsGSeyBdJ59p840ZZcR8EFVKZ8+bOK28XczfnPoFVChn5r4589bZwafqNVUxRA1z/51k1WSZmtHD2F7SolOtuS4VeqoHHoxiT6JbNjzkcuqNIb+ZI5aOUveJDgZFOyA05+BEH/afvAdkZWT2WbTQ5fdYHoLBlqE0mB0nKie2Wm2BNr0WE++HjJt5aMRq3ahb1Zf+aYL+q7UXs2Mfhy2lK2RxzFkAtmydzFRvJq8JRwti1brwjos1BWKN9J2xaob2Z9ezjwWjkjtV6hoDE7yp9pehPFb5CArMk9a96HP/2vacl6xtlC7RQzMZ/fqwkGuE6q3vfDOGd7aMPIe50laXBn3xwdAgRiJs9puqjfZNw/eOPYY+smPGUKwPkkC9Vg1ZHeWSQxtba1IRLpDT+hofjLhDdvRqE3K8f6rbLd9nNH85I3j0etDHtvYhrKqKaHjSiBuAwnT3NcgSSNl8wojXz3TVAd4tv0ptJqLiew4M6PB03cM1jIMaTXinh15c7VjPnzp8dA08xyVy4tP5kGQH2cMx9TNa8lOjVrw9AmetgrFKAbPIZvwf9N/+6ErQsrQY/qS0YgTi3sZ/VkQwnTqh1BUP0KeTgLyZEUkg6Jpbup3+7vpsMG2FNTFqSAPsGvGCiILGr6gKVep3tbolQ3QvKqa09QghE26+wl8lWWMcG6BWTG9pqhqgkWHUQn9DBAMfSb8Lpg/0Rsz3Ahik8Swp6kZuqR7905t5d8edqy0gZnYbNXPGhLirI05NPausThWhUxCX6GFj82BWfDrybEkLgvtZzJU9rYfbaOM9qos9W7A9XFMTRKFLQ17KxcfR6+r+gFuQ98mCwuRHp8vH2XgGsw7aFZgvFUlchIASkSj/nGgWvZGhDTGNfK/39lbse1iDYg7CXQCrCbWL/ 2yUxPEos aqldacS3r419qOP0xL8h3d8AaoTB2+Lwgx8zmKZKFqXq1+K+Fv+awVD50wC91aG2sCNUIomiSsDeaV4RGkv0CyBml8IOvIpbYkw8+YNweOs1hPIoIvvAL3Pd9SI8BlTBCuZdhf8xvW91fIfQLFajSeHdy4hmGnpFbadQ1MHeg39R/SjGj4P9dd8NYGw7XXMNGlcz3jHxB0I/FbQfYSt0kCN8wQu9tiOxxRlP6H6ljo/D2uXBk1d8cLV7GcZOnSRT8RWvE1RuFTLQMRV9EcQl0Z9ZoKkuMndBVQg4H4ld8ygRVIAH/w/AmpVYY+3sXnL9PYoUqjFUfEa8Zbhz6Tn4p/+i6XWGjHoXzLglmEeUUYL012KMp78dbldb5OlMRMYfFOOBKtdsR8CcTovYTzbxv4OOQI7hD8r3Bsp7zZZU1gH+5WUxKt3fOjJUjJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: follow_page() doesn't need it, but we'll start to need it when unifying gup for hugetlb. Signed-off-by: Peter Xu Reviewed-by: David Hildenbrand --- include/linux/hugetlb.h | 8 +++++--- mm/gup.c | 3 ++- mm/hugetlb.c | 5 ++++- 3 files changed, 11 insertions(+), 5 deletions(-) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index beb7c63d2871..2e2d89e79d6c 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -131,7 +131,8 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct vm_area_struct *, struct vm_area_struct *); struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags); + unsigned long address, unsigned int flags, + unsigned int *page_mask); long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, struct page **, unsigned long *, unsigned long *, long, unsigned int, int *); @@ -297,8 +298,9 @@ static inline void adjust_range_if_pmd_sharing_possible( { } -static inline struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) +static inline struct page *hugetlb_follow_page_mask( + struct vm_area_struct *vma, unsigned long address, unsigned int flags, + unsigned int *page_mask) { BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } diff --git a/mm/gup.c b/mm/gup.c index abcd841d94b7..9fc9271cba8d 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -780,7 +780,8 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) - return hugetlb_follow_page_mask(vma, address, flags); + return hugetlb_follow_page_mask(vma, address, flags, + &ctx->page_mask); pgd = pgd_offset(mm, address); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 9a6918c4250a..fbf6a09c0ec4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6454,7 +6454,8 @@ static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, } struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, - unsigned long address, unsigned int flags) + unsigned long address, unsigned int flags, + unsigned int *page_mask) { struct hstate *h = hstate_vma(vma); struct mm_struct *mm = vma->vm_mm; @@ -6499,6 +6500,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, page = NULL; goto out; } + + *page_mask = ~huge_page_mask(h) >> PAGE_SHIFT; } out: spin_unlock(ptl); From patchwork Mon Jun 19 23:10:40 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13285011 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE495EB64DA for ; Mon, 19 Jun 2023 23:10:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 68EF18D0007; Mon, 19 Jun 2023 19:10:57 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6174C8D0001; Mon, 19 Jun 2023 19:10:57 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 491A68D0007; Mon, 19 Jun 2023 19:10:57 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 37B528D0001 for ; Mon, 19 Jun 2023 19:10:57 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 084E51A0498 for ; Mon, 19 Jun 2023 23:10:57 +0000 (UTC) X-FDA: 80921044554.03.E93B8B4 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf03.hostedemail.com (Postfix) with ESMTP id ED9492000D for ; Mon, 19 Jun 2023 23:10:54 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dHz43Dvs; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687216255; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=G5PvpMSc9PWeUNzovzE5WASqMyhgs2MBMMaZX0mlkro=; b=1VCrKra4fNzQ4w50UCZBR877Er4ZyXH80xZAD44+typbKFrcTeK65YRfZp83ZSBDZkujJ3 XNOPmgrrLfx7K7sLtnRC+cJFNcIJiXSkEzxCFf/3FMEW94Jyk46s2+pfG5mjwjNZTPoKUl LhDZR9JmIqtwsPsbDhKwtxS2OUPQ8fk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687216255; a=rsa-sha256; cv=none; b=if44t2r61LW0BhdxyJOUAmQHbD5R/Nn0wCekR3fAcoU6D9cHgR4JN2biSwJ8+IkpF52lO3 wD94zrdDK1RY137zH/XwDL32F4DakOVzysIPup9L8pEhmEKODbJMjkKt6WA8Oxd/WDJihw N27ygjc1JOOH1JzHF+z0ZCuf1zGsFCY= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=dHz43Dvs; spf=pass (imf03.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687216254; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=G5PvpMSc9PWeUNzovzE5WASqMyhgs2MBMMaZX0mlkro=; b=dHz43Dvsm4fBhwFd/2fhq2lpYbl+k9ATohWF3W8ZX6EEm6H6XbBp9+GIHdtb3Oy12bslfm pApvNUmrTsBCMWY0jtvm+iyGiF0sZMvHJrfvdo//H4euTi3WqxKHWtdIYg4z37A3DqgClf yvXMJj8995aobP/xeXWQZneS1XkyHMQ= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-385-Ap4qN-SHO-eY3drogcSGzw-1; Mon, 19 Jun 2023 19:10:53 -0400 X-MC-Unique: Ap4qN-SHO-eY3drogcSGzw-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-76077669a5aso105210885a.0 for ; Mon, 19 Jun 2023 16:10:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687216252; x=1689808252; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G5PvpMSc9PWeUNzovzE5WASqMyhgs2MBMMaZX0mlkro=; b=euoBt+g7/uO9Tax/Hlcrx8WTPlbu1TbbtJszUdaL681eYVjL1msiqsp+RHC/CxoX+P kf9CbpfZe2WO6POGlvNFbSG1soimyE1sf7o+h+Wlf6SLHosZKCp4hN4ZsWEPtiohLIwO MnVqGw5vaNlLVJh8IfQk2thQddNKvGjYN4VTwTbqaqoWi/8qgNSB3bpxhQ0tBM7s/nm7 FUYNKJnphuoFaZ2/gcL7HaTWNY4AufczFShlD8UKzXaD0CtyQPpxNMQcfv8G9GIzD10W zmVpCWZnHLGqfbLvz9cxKZww4nFQ7PCQA/p0jHeERuuomAQFuzrYH4fiCfK/xXk/dDBD 6MdA== X-Gm-Message-State: AC+VfDyTgcDpyZOPKGGZLxfQgyP04h7KTQuReNJZ39DI/62eRBgiXJHe ArYpBh6yaPejAxXDP33qVv9A8SVcwdRO2STEZmVHP7iGc1cc8PuYHvLljH5JyqpawZ2GyWp2bEm Zr0lqXc15zd0QfqlBcZ1ghPfnKYH/16Otk8tKzPlm8zBT9nQoclY87S5bXh8CzxfIqUx9 X-Received: by 2002:a05:620a:800e:b0:763:b025:9f40 with SMTP id ee14-20020a05620a800e00b00763b0259f40mr146902qkb.7.1687216252621; Mon, 19 Jun 2023 16:10:52 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5gUaxPapWXj9fyTcJWWnL1Kgv2ncFDRbyXTEYoD6E6DhEtoWY/2AMdIngB3HsMpWlNRnZ+zw== X-Received: by 2002:a05:620a:800e:b0:763:b025:9f40 with SMTP id ee14-20020a05620a800e00b00763b0259f40mr146867qkb.7.1687216252208; Mon, 19 Jun 2023 16:10:52 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id t15-20020a05620a034f00b007592f2016f4sm405864qkm.110.2023.06.19.16.10.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Jun 2023 16:10:51 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Mike Rapoport , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , John Hubbard , "Kirill A . Shutemov" , James Houghton , Andrew Morton , Lorenzo Stoakes , Hugh Dickins , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe Subject: [PATCH v2 4/8] mm/gup: Cleanup next_page handling Date: Mon, 19 Jun 2023 19:10:40 -0400 Message-Id: <20230619231044.112894-5-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230619231044.112894-1-peterx@redhat.com> References: <20230619231044.112894-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: prioged9poxmyf571h68k5dx8n34yjos X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: ED9492000D X-Rspam-User: X-HE-Tag: 1687216254-508412 X-HE-Meta: U2FsdGVkX1/hu2tEcfgSwogx1fGDW+L5d2zM4+rDqMXn0rAaA85PHCxnE0dena+T7HFlmPsH4z1Sc8Z6JXLAMLqQ5F3/QFI3PSaJHapH434ZmemcsbN+S37wIXtemgpyQhS09f0Eu3AXOXMxjOv1OsPPh3nOez+6YUtv141idsTyVZ4uPUqBu5hzOsNQ4uEC23eZpMJbImKGczaFo6Oqm0ht584vubVBSMnri2Tdxl15TcBahE3ke77yZqhVrAg4InfeMjuSpsM1HKvzzcTxQEt/xJbB7TlwNn6V9ZTvasCnpoPAmOGfkuNnyc0mJHu5rz4tXnSgpaox8PEDvNrAQLZOqDnkbCaupoG1TV5NS0ePEUBl1Z6AOEkYbg3LIOyUykYKkBCEm0IliFyCZwTpmF19SxhOrGW8SApRyN+Mvx6QPpQYvneeSIw4PZSshvYLKZfD3lTjmXtZVMH85ZpKYoGbulq3n+D8FDHD68YyBO9nLzGDa9r+z6bevBZQq6VCrqYWdu5IszTY9zu7JT9+qrni75TuHWkuHeMwkS3A9Eh1gGp++7bhifptQjcUTZUAvPRyRzvOKe6vD2DGBW4dRY4Kzahxu0oxoQQHYaC5KnaoZqKZLC+6HmDsZhh/KmEG75Sm4HvYEh7Ex5Utp6N+DQdlEHG4ppcumn9cyyqXy65ne3HlRYbNo8pGXI5oMCDZTzpJO8l/VOgp11vb9p6NzQ0rZuaOOZNI4drRCpJyEGfaTTi3DYBVu6tlO2dSD9mfNCv6SEn0qllejStM+NYQmE/4qHDubmCes2ppg7NRDXYoKB7GTNfS1QEp3XSoO2w26SUVCaYX1NpExphlk+Q5SWyalhlIspeH2uuo6lsc/rbflZ+/5468N/CJFGbdAIcdje4QBwBuiaPB6LFV/aaHkjaxNQwooKSxBdsRdoItaZyd8U4XEhsZMKnOD8rJc3R8kycs2pn4Gz5/ka4p5/+ g31yY7La NapVqawCLTtRH9VjRSHys5P5Q8SFtp6whqRbzVpkgIykS/ajhAAtJIaJOhoBDQwXr3g9UHlNykFeHWvFIj4Qf1ohb5SLiKny/pcnDekEXLqTf6WVR5jjn3FpXxKbgHs38/JUZpFoW6/fhbM3RtKTLYBI8f8nhuIlGbWj6XdN6zI8k4ytaki+9Dvstzs8audqEb1Yijx21APDSQqEE1/bB4NtAl4SI/s4IULcU0u+tQ02jy7OE9yWA2TzKfoityKbt6E2ociWa/UpeZ7DG23fXSMLrV0IJdWdJ0zQ9LsZ4iGDaopnoIAv+PrFHcL5FBttgsEvIk7oLxzObD4fyAWSvfwrxP5JhKQ8LJEtcBYDNkm9PbrFLFTfvvsQKxUpGA5NbIoogzVHvxvQVawwzGiAdWtgVnW32SAmI705ULUCptO05sF49t3ehlRfKZrnaXFQbrJbLIrxUFnbOX4A= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The only path that doesn't use generic "**pages" handling is the gate vma. Make it use the same path, meanwhile tune the next_page label upper to cover "**pages" handling. This prepares for THP handling for "**pages". Reviewed-by: Lorenzo Stoakes Signed-off-by: Peter Xu Acked-by: David Hildenbrand --- mm/gup.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 9fc9271cba8d..4a00d609033e 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1124,7 +1124,7 @@ static long __get_user_pages(struct mm_struct *mm, if (!vma && in_gate_area(mm, start)) { ret = get_gate_page(mm, start & PAGE_MASK, gup_flags, &vma, - pages ? &pages[i] : NULL); + pages ? &page : NULL); if (ret) goto out; ctx.page_mask = 0; @@ -1194,19 +1194,18 @@ static long __get_user_pages(struct mm_struct *mm, ret = PTR_ERR(page); goto out; } - - goto next_page; } else if (IS_ERR(page)) { ret = PTR_ERR(page); goto out; } +next_page: if (pages) { pages[i] = page; flush_anon_page(vma, page, start); flush_dcache_page(page); ctx.page_mask = 0; } -next_page: + page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm = nr_pages; From patchwork Mon Jun 19 23:10:41 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13285012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD91CEB64D9 for ; Mon, 19 Jun 2023 23:11:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0E9788D0008; Mon, 19 Jun 2023 19:10:59 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 076968D0001; Mon, 19 Jun 2023 19:10:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DBAFF8D0008; Mon, 19 Jun 2023 19:10:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id C9DB68D0001 for ; Mon, 19 Jun 2023 19:10:58 -0400 (EDT) Received: from smtpin23.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id A10698045A for ; Mon, 19 Jun 2023 23:10:58 +0000 (UTC) X-FDA: 80921044596.23.63EF08E Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 6A012C0008 for ; Mon, 19 Jun 2023 23:10:56 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MjpfCDht; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687216256; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=00BT5P4M9oBALYuFaAi0Lj2RlKWYXU3sXFeAAd1zvs8=; b=MqMpobrD8rCXfisWMm/HJdJGPF7adhZH47zXayT2veMQwGXCMMNF6D3a+gh/vpvE8m7Cgv AD31E3kVmTyO30zFjJnQLUPSajWG5nGKDzS6mQlzPnvMypq+mWVENYXiNPVAIe/n+65VgD M/0UeVwDxw/gx16Mu97IhSbPYNiyyyc= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=MjpfCDht; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687216256; a=rsa-sha256; cv=none; b=oKU9iy0yErXIHZuzbh5n5T9dV1+kQlR15QK+EoYs0s1UHRzfZyNgFy0itOE/JJaLU4rqx1 wRHy016+3TGRatngB/vA/EIk4/1xeNYjlEi5NjxmlvWm9zbmgWJr7LWIzRFs1RJxJiMcjy DkPDe+cILW+9YJ/VrXnN6dsAteIlGZk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687216255; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=00BT5P4M9oBALYuFaAi0Lj2RlKWYXU3sXFeAAd1zvs8=; b=MjpfCDhtiFFmQK6WQS137pnmLSK5X+JNWAjV+1tqlRmqVyVaCgZxYaUIJtAu2B3yKoVU8A wXpmHtsmR5UggTFGWqAqGRJ5DdtTzfiYv/gIqdNckeWDkA+RcxfQy49PPJQP93eOr/RKFT HI5Espd9mJm31pgZDUfw41MDX+BlPS4= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-619-jtDlg1dTPTWG5AxD-sTk3A-1; Mon, 19 Jun 2023 19:10:54 -0400 X-MC-Unique: jtDlg1dTPTWG5AxD-sTk3A-1 Received: by mail-qk1-f197.google.com with SMTP id af79cd13be357-7625940fbf3so29807885a.0 for ; Mon, 19 Jun 2023 16:10:54 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687216254; x=1689808254; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=00BT5P4M9oBALYuFaAi0Lj2RlKWYXU3sXFeAAd1zvs8=; b=CWrIRt449jLYN7bMrNHSziBs7kNQyTHZLINOdIgdCKzy0Vkw8q6xUxMwhM0iWSXsZd FIlftFdNY20I+qgNdfVG0l0A83eUXRMSuit16pBkrm5iDr7RYj+6q/yV6Lt+gk1I2kWO VsmlxLszdXtmRWhmrF+844tKQsexH9znR2L3yWAlJd7y5Un7QKlavJ8XnNKw/N7/BRTV tb5W0n0iRr2xeVWlMshO5h+q75OXo4TbZXkla/EgPpBiGyOIDqRBNSXhmfd69PiJyw2P W41EN6hMU3qyLp/UPPkf/8C8TaVkv7yMVTNLnu1QxYCwdKtHZzGADHcI+877lp4sq4wx eiqg== X-Gm-Message-State: AC+VfDwdin8TRf5cVqEpNCppJfWJ7dSloE5UmXbJ6ryWdYmGQTSDqP/d 1+iT5kfeDCRVcnPQrlyK24tKoMzc2nQLTtVASNW39Qbjfte7MhfSDilmxyqQqk7+Nn448my9SDa cWRGBcLWJPR27Feo7jyX/vHc79rm45AmbPq0RGkSY2/MJi2A/xr6Ly7eUelcFf/y3Hwpz X-Received: by 2002:a05:620a:2889:b0:75b:23a1:82a4 with SMTP id j9-20020a05620a288900b0075b23a182a4mr13246421qkp.5.1687216253947; Mon, 19 Jun 2023 16:10:53 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6UOYeqlkYR0J6Chpj2TyYjEYwQWzUGGOpxx8Hlzqu19xtpfM0scU4fcVU/UZq38b/9mzJjIA== X-Received: by 2002:a05:620a:2889:b0:75b:23a1:82a4 with SMTP id j9-20020a05620a288900b0075b23a182a4mr13246392qkp.5.1687216253602; Mon, 19 Jun 2023 16:10:53 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id t15-20020a05620a034f00b007592f2016f4sm405864qkm.110.2023.06.19.16.10.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Jun 2023 16:10:53 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Mike Rapoport , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , John Hubbard , "Kirill A . Shutemov" , James Houghton , Andrew Morton , Lorenzo Stoakes , Hugh Dickins , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe Subject: [PATCH v2 5/8] mm/gup: Accelerate thp gup even for "pages != NULL" Date: Mon, 19 Jun 2023 19:10:41 -0400 Message-Id: <20230619231044.112894-6-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230619231044.112894-1-peterx@redhat.com> References: <20230619231044.112894-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 6A012C0008 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: mygcrwfqn5165c8qxrtrz1hyju17z7hg X-HE-Tag: 1687216256-705216 X-HE-Meta: U2FsdGVkX1+pev6fZh6LH9DYt3OvvQoTUD1n3z/Bhhi6CWxjZXeb880y0ugmcocDqjC6YexedogkDJQJZ5V9RAsk3SrON3v7epx0DuU+72k5UvLKk/5vfgJ8ArlhOkj9gPNxM5uq590LltscrVI1cOVm4m+MJO1b0JALEmyg/s0RtXMdaG6vcNzyTyHT0ibSFVcolRYofwBY/9giFNH+LcIIn8FHgqoLGICQo0zQ0XtOx6vlLiJhra3GRFNc1xXaBBBG4vS3AfuYMfIcDfONIgYvoi9OCqvQWEP68Ch4ENOXG7FlNb3syrK6SN5q6tFdYVZWMhYGh7oZzTXBqUjVa1KNlKyhYnuGarmo4wm+1fSjC+OXehifP2zWKYKi873n8plEza6JQOTTLIn+YgmlvJ0f8srTm3J4peottDTYEeBUQG0RGqlSKOZcgtwgeLBh3hwtV8G5myw5kV6JqkUrgZf2ysjd5JF0sAJZj/IqcvI4pa20Ih1s09zECvn+c1H1TKMVdlOe+E2RaUYddbTLbfsvMdXynr+NH0J7g7QrcEnA9xT7ufQ0/0BcPTqcMxBek8ItsinTU7aXfenZ9sCum7cEnBMxbwpriH7p6duHWvbTzpTExibd7TZxceq16p0iCvDTDywvlbVClCPaBfsFFHlWhCpR5q95tViQMjCnGEvUZzi136OnMs85fVIuJvA766h/bGpMdrCafO80d2cuxQxcdtTMJd4vPWK43BbDiDQBRw8YuUWY/dmS4oKSSizLVjXqpA+uLxtE9JQBwADU5pYj7AQl/vggpCNTOwavdPJfqh/HGXezNMrhoZxWPe1mjpsh7fMNXq8OK1J0J9BMrUtKkh2MH8WDKXeLUqxZqEAbVpGh9WirO5Pqph0AqDqRUSXB79Ho3GRl6sdzcGJvD2c7MQrsUSROhsPtbzCNxLG6Lr44eiFCD9aMUX7roRK4nIkceFKccm0fmFGXzYS Shpiv4Ek W+k/RCL/t8EgtSioRkkStgvVuL8tV9Mk52zmo9e5+/doBkQQOxhFUTZ89sWA/AtDgKYMVa6IdNf752vxUcjwpyum2ZiR4cmvi6GOcLW9uIOR/i/KRuRIa0IxtqkBDXaqrGMZFqCrUHA0vdL7ogeiHMRsLKAmxc4mF99lTEgaOrqtXpCySzjX4RpXRTk1a15J6BCirUU6u4U21sP/x3EfnPxHzekgENrgjNodZHggA536zFnrc7ufxdb4l2cB7vKnk65GAI/4NSP4xqMUxJLssAu6iWm3xmCzhrVK/l1Y0CBGrJ0CHa6EptUpKOVHiDvetG7jgw5H88ZA3QILmqI/gi3hP25BiMoLUjqlXMBHqogOac4cu9zduq2Y7yPBQfPv6MPK35ggom7XY4z/YYDYiWRBIVHJxokaSS4bye7dR+4XOTsMWIGTmJA0cXg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The acceleration of THP was done with ctx.page_mask, however it'll be ignored if **pages is non-NULL. The old optimization was introduced in 2013 in 240aadeedc4a ("mm: accelerate mm_populate() treatment of THP pages"). It didn't explain why we can't optimize the **pages non-NULL case. It's possible that at that time the major goal was for mm_populate() which should be enough back then. Optimize thp for all cases, by properly looping over each subpage, doing cache flushes, and boost refcounts / pincounts where needed in one go. This can be verified using gup_test below: # chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10 Before: 13992.50 ( +-8.75%) After: 378.50 (+-69.62%) Signed-off-by: Peter Xu Reviewed-by: Lorenzo Stoakes --- mm/gup.c | 51 ++++++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 44 insertions(+), 7 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 4a00d609033e..b50272012e49 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -1199,16 +1199,53 @@ static long __get_user_pages(struct mm_struct *mm, goto out; } next_page: - if (pages) { - pages[i] = page; - flush_anon_page(vma, page, start); - flush_dcache_page(page); - ctx.page_mask = 0; - } - page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); if (page_increm > nr_pages) page_increm = nr_pages; + + if (pages) { + struct page *subpage; + unsigned int j; + + /* + * This must be a large folio (and doesn't need to + * be the whole folio; it can be part of it), do + * the refcount work for all the subpages too. + * + * NOTE: here the page may not be the head page + * e.g. when start addr is not thp-size aligned. + * try_grab_folio() should have taken care of tail + * pages. + */ + if (page_increm > 1) { + struct folio *folio; + + /* + * Since we already hold refcount on the + * large folio, this should never fail. + */ + folio = try_grab_folio(page, page_increm - 1, + foll_flags); + if (WARN_ON_ONCE(!folio)) { + /* + * Release the 1st page ref if the + * folio is problematic, fail hard. + */ + gup_put_folio(page_folio(page), 1, + foll_flags); + ret = -EFAULT; + goto out; + } + } + + for (j = 0; j < page_increm; j++) { + subpage = nth_page(page, j); + pages[i+j] = subpage; + flush_anon_page(vma, subpage, start + j * PAGE_SIZE); + flush_dcache_page(subpage); + } + } + i += page_increm; start += page_increm * PAGE_SIZE; nr_pages -= page_increm; From patchwork Mon Jun 19 23:10:42 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13285013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 94E98EB64DA for ; Mon, 19 Jun 2023 23:11:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4BA458D0009; Mon, 19 Jun 2023 19:11:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 41C7C8D0001; Mon, 19 Jun 2023 19:11:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 246BC8D0009; Mon, 19 Jun 2023 19:11:00 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 16ACC8D0001 for ; Mon, 19 Jun 2023 19:11:00 -0400 (EDT) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id DD57EA0466 for ; Mon, 19 Jun 2023 23:10:59 +0000 (UTC) X-FDA: 80921044638.12.9AD7F60 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id B7E4B140014 for ; Mon, 19 Jun 2023 23:10:57 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RaJ0Xe7C; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687216257; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=LfImLhKhuRzr1PTA4t2FBf8hxw7M1QNNKWA4dIYrJXo=; b=g3AhVMMUTGx6gEF0vVRVvs7fC9zChnts0SigkCrhI33NRFb6vtdJL2Ch/6EdkgBdlp9ODB JAQKriOredOUry4zy9bIKYLjeDwdb0AwBDD3qG7OIJ6PNw1QM+5bDf8itw1gWE3VQ1iT/d MPoiVmW/ZcbsbWgrNHtwhx3t7WT2lEM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687216257; a=rsa-sha256; cv=none; b=aiupetN3uYqqs4dpjlDJUwjNX/2gKjMc7kGGu26d0r82f5MA5rbzQdTvXzqWzUQ/AwIM/x VbS+S81AQGhxxdg4AK66ClQrsDMtPqYLYs7k/lQ7iwGIg83PYauqt+ksn94FE2U+kfWGS/ T6R/zjmt4Y8ekrhtoPRhPrd26+4UNgE= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=RaJ0Xe7C; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687216257; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=LfImLhKhuRzr1PTA4t2FBf8hxw7M1QNNKWA4dIYrJXo=; b=RaJ0Xe7CNOia5af/2S+ONL909bsIBujNIjyRw7OYSiYTpGgbgUjJYDYow2pKEJExYWe66P iXMH92IdfLUmROZ2q7Lj+lm++SZwx13p/WzeDmOO5vOd4OW67kGKFmd2wAP/oW2D73x8iI uTdMenFb++wkidKSmVQyeh0IXBIXcpk= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-211-kphogRZjNqC1hKGSmMkN7A-1; Mon, 19 Jun 2023 19:10:56 -0400 X-MC-Unique: kphogRZjNqC1hKGSmMkN7A-1 Received: by mail-qk1-f198.google.com with SMTP id af79cd13be357-7623c0f2856so48480185a.0 for ; Mon, 19 Jun 2023 16:10:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687216255; x=1689808255; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LfImLhKhuRzr1PTA4t2FBf8hxw7M1QNNKWA4dIYrJXo=; b=PapnkrX2e5lynoYYiRdlO1mzhLFqkN/wNbuQWU28mTgeZRpcl5XLXUR1E2S5HNYlnc QR0HusRf3+08oJ5NsHP4bRqlBvvifQbA7853Zm8E1JUB91rzFfanvMIengfSz3fa6Nzn l279oX5/atvVZJFj3iio6j13SDTvFWBx+xpTcB3aOB9kBp4dOC6Zi/zRxEytThRyXLX+ jTq8n7GXl9y+QEYGcV+7nrA/YhXXiP3iNR26jq0fMtu+gRPzXCItpdqOINyTRFg38Do3 eAu0Cgt9e7X11c6aCG/jDVf3uy/eG7uAo/3XUCUxcdBwVpa48feziDNF7Byl/6B/r6T3 zCnw== X-Gm-Message-State: AC+VfDxOsNdJuNRPKzE4s+h6TgSZZXnPBntqyNv8Co1TohF1lHBj6bwh QYZKte8xsTU2bmtgd01KYlwGBTUTfU2dtcffZ0a+5B7xDh1P3wJbPEhYVeBL9czLkywY0Jq4/iq N2TJ2Ju7yQsPXpeEb8SKSrVUFLtrpr5BZQ0ARC8jKPg8mukbru0zLQOdlUroa8DF3rTxD X-Received: by 2002:a05:620a:171f:b0:75d:e31a:a015 with SMTP id az31-20020a05620a171f00b0075de31aa015mr12743429qkb.2.1687216255476; Mon, 19 Jun 2023 16:10:55 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6vad7WBGU4fEHA9ljF1jRAsPsdjqibUtSIB8Yd1Vctrt8m5QF0dustN7B6B8bg95U/g6D0kg== X-Received: by 2002:a05:620a:171f:b0:75d:e31a:a015 with SMTP id az31-20020a05620a171f00b0075de31aa015mr12743392qkb.2.1687216254978; Mon, 19 Jun 2023 16:10:54 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id t15-20020a05620a034f00b007592f2016f4sm405864qkm.110.2023.06.19.16.10.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Jun 2023 16:10:54 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Mike Rapoport , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , John Hubbard , "Kirill A . Shutemov" , James Houghton , Andrew Morton , Lorenzo Stoakes , Hugh Dickins , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe Subject: [PATCH v2 6/8] mm/gup: Retire follow_hugetlb_page() Date: Mon, 19 Jun 2023 19:10:42 -0400 Message-Id: <20230619231044.112894-7-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230619231044.112894-1-peterx@redhat.com> References: <20230619231044.112894-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: opqetjdqmd4fmuutanjrkfqjs1c6bnco X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: B7E4B140014 X-Rspam-User: X-HE-Tag: 1687216257-285119 X-HE-Meta: U2FsdGVkX19AojbIMceW2IUmUKF1odoqwJiuESjBAUB37EoYNUAHBgysbArUDZTtqxqLJzto1wJ42pB5Q5CgGGzEYEoYOEMk/xCG83hSqbNtQbg8zX3vr6Cg16CiyncdgDXUVVRB1dDYK8rkHBIgh27zlIMPN1422ez6fHm+H5wVkc+0Z15mzWPG1Fck7x+xoPDxhc+wFk+0bEC3U37a++BqNSDOX2dKO5a5EzcgF9DIleB9M4v3XioY8tKy4/3Ar0ql9gPicnCa3LGXQUIdY4w9wZWVNSp2G6hKYOXoJSEEklx+rS0TLFpm2H2q8gQe6blKyw5FPPbP5lMr6ycOlz8ukgenD8s9rqbsBWsrqW/Inj3XrJLacDL2qG+Q2XDOAU4Z8b8wT7ITr8ek7fcAJ0AEsBTweIIKQzsYvHYJMBIGfv8dvozaqy2NVX0p1Tt9uCdwRjEO6ZU12RbXqMw/08ff8yc8yDZmKXysYlbFuXxtWyuaLJLOst3D2NjWGEAZ9T5OEobmU7pS3mYC10rvjyNu45kQuTs0/lEe7n7vOgen0qG1qISWN2FVz5Y/sGc4ya3egosylvVb70VWkSGNE0wUs5fuARHK6FFx4ZF+FqZjom6iWusUQwEpsONQgfjVDZn3SX9gHZ7oxTn8+9nuWDtIv0rbi1sKcIiPype7gZ3H6LmJxAlG4C9DSR9Zm3bTWIKjJQRbLSB5+9MiGrvt4FEuawPxMA2oVcAfRZcWD27GQfVeHezDdCCQHVA2plqNVZnqQ7aEAxSK+5MHlTShNmg/Mq2KBmMSu6kEFI4zjnzjTnLt72i29WF7KDfYBMr8HiSfFNP4T2IZra0C/2Tu96YOVI/OmB6KTQIc4LE/zfMgVQI4xmhHT2yf2G0MNuFsN60PDJZ7lSc/veBMtH4Uzl8hJrxE4Hq0gtz8Lsf+OP1FmmiFlMygRGg/6+4p0tFGmBXCr68nUrYKqRMYZpz nD5qPOZL HpwvyIc9S8PXkZ+IHTCcz23ib8zKvdvq+yqiIcldZcDJozk/TKrLZZYC6lch+MKLiOSSJCZtF3FN9rQFoIQ2fU11ansVRfHPQTGhYZFRLEiJyOjxc+FKJUKWNVvwI7UEMDjUjnZi1JzaCAoZS6YTJuk0UZB3mGjCWWQ6ylfRDmxKeeJDL2xzNwby5rc2/fkz7FrNLmaBfsDPT1fCmYQAdCXnioa1YTNlOtEANO3skx37eGkh2KqTMR/0TT5mfmNDG7mCaUjDKDDGNpudrugg+TiIwD80l7wNZO1TM9EXoFdLPKOJi+PKM9uPoYDrn/rbWmG3DieMaOCi3S1sZG/fVc5aqR/X8a8HilqiMqI41Mj0Dk5UYhQq+njiVV/ViIzqQmqCQa70ZDFw/EivgU0ebjrJ/fdpZR14LHL+MadkmiP9WTOSqGU++PJ+jRw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Now __get_user_pages() should be well prepared to handle thp completely, as long as hugetlb gup requests even without the hugetlb's special path. Time to retire follow_hugetlb_page(). Tweak misc comments to reflect reality of follow_hugetlb_page()'s removal. Signed-off-by: Peter Xu --- fs/userfaultfd.c | 2 +- include/linux/hugetlb.h | 12 --- mm/gup.c | 19 ---- mm/hugetlb.c | 224 ---------------------------------------- 4 files changed, 1 insertion(+), 256 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 7cecd49e078b..ae711f1d7a83 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -427,7 +427,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) * * We also don't do userfault handling during * coredumping. hugetlbfs has the special - * follow_hugetlb_page() to skip missing pages in the + * hugetlb_follow_page_mask() to skip missing pages in the * FOLL_DUMP case, anon memory also checks for FOLL_DUMP with * the no_page_table() helper in follow_page_mask(), but the * shmem_vm_ops->fault method is invoked even during diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 2e2d89e79d6c..bb5024718fc1 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -133,9 +133,6 @@ int copy_hugetlb_page_range(struct mm_struct *, struct mm_struct *, struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask); -long follow_hugetlb_page(struct mm_struct *, struct vm_area_struct *, - struct page **, unsigned long *, unsigned long *, - long, unsigned int, int *); void unmap_hugepage_range(struct vm_area_struct *, unsigned long, unsigned long, struct page *, zap_flags_t); @@ -305,15 +302,6 @@ static inline struct page *hugetlb_follow_page_mask( BUILD_BUG(); /* should never be compiled in if !CONFIG_HUGETLB_PAGE*/ } -static inline long follow_hugetlb_page(struct mm_struct *mm, - struct vm_area_struct *vma, struct page **pages, - unsigned long *position, unsigned long *nr_pages, - long i, unsigned int flags, int *nonblocking) -{ - BUG(); - return 0; -} - static inline int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, struct vm_area_struct *dst_vma, diff --git a/mm/gup.c b/mm/gup.c index b50272012e49..e6c1e524bd6b 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -775,9 +775,6 @@ static struct page *follow_page_mask(struct vm_area_struct *vma, * Call hugetlb_follow_page_mask for hugetlb vmas as it will use * special hugetlb page table walking code. This eliminates the * need to check for hugetlb entries in the general walking code. - * - * hugetlb_follow_page_mask is only for follow_page() handling here. - * Ordinary GUP uses follow_hugetlb_page for hugetlb processing. */ if (is_vm_hugetlb_page(vma)) return hugetlb_follow_page_mask(vma, address, flags, @@ -1138,22 +1135,6 @@ static long __get_user_pages(struct mm_struct *mm, ret = check_vma_flags(vma, gup_flags); if (ret) goto out; - - if (is_vm_hugetlb_page(vma)) { - i = follow_hugetlb_page(mm, vma, pages, - &start, &nr_pages, i, - gup_flags, locked); - if (!*locked) { - /* - * We've got a VM_FAULT_RETRY - * and we've lost mmap_lock. - * We must stop here. - */ - BUG_ON(gup_flags & FOLL_NOWAIT); - goto out; - } - continue; - } } retry: /* diff --git a/mm/hugetlb.c b/mm/hugetlb.c index fbf6a09c0ec4..da4c76bee01f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5721,7 +5721,6 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, /* * Return whether there is a pagecache page to back given address within VMA. - * Caller follow_hugetlb_page() holds page_table_lock so we cannot lock_page. */ static bool hugetlbfs_pagecache_present(struct hstate *h, struct vm_area_struct *vma, unsigned long address) @@ -6422,37 +6421,6 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, } #endif /* CONFIG_USERFAULTFD */ -static void record_subpages(struct page *page, struct vm_area_struct *vma, - int refs, struct page **pages) -{ - int nr; - - for (nr = 0; nr < refs; nr++) { - if (likely(pages)) - pages[nr] = nth_page(page, nr); - } -} - -static inline bool __follow_hugetlb_must_fault(struct vm_area_struct *vma, - unsigned int flags, pte_t *pte, - bool *unshare) -{ - pte_t pteval = huge_ptep_get(pte); - - *unshare = false; - if (is_swap_pte(pteval)) - return true; - if (huge_pte_write(pteval)) - return false; - if (flags & FOLL_WRITE) - return true; - if (gup_must_unshare(vma, flags, pte_page(pteval))) { - *unshare = true; - return true; - } - return false; -} - struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, unsigned long address, unsigned int flags, unsigned int *page_mask) @@ -6519,198 +6487,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, return page; } -long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, - struct page **pages, unsigned long *position, - unsigned long *nr_pages, long i, unsigned int flags, - int *locked) -{ - unsigned long pfn_offset; - unsigned long vaddr = *position; - unsigned long remainder = *nr_pages; - struct hstate *h = hstate_vma(vma); - int err = -EFAULT, refs; - - while (vaddr < vma->vm_end && remainder) { - pte_t *pte; - spinlock_t *ptl = NULL; - bool unshare = false; - int absent; - struct page *page; - - /* - * If we have a pending SIGKILL, don't keep faulting pages and - * potentially allocating memory. - */ - if (fatal_signal_pending(current)) { - remainder = 0; - break; - } - - hugetlb_vma_lock_read(vma); - /* - * Some archs (sparc64, sh*) have multiple pte_ts to - * each hugepage. We have to make sure we get the - * first, for the page indexing below to work. - * - * Note that page table lock is not held when pte is null. - */ - pte = hugetlb_walk(vma, vaddr & huge_page_mask(h), - huge_page_size(h)); - if (pte) - ptl = huge_pte_lock(h, mm, pte); - absent = !pte || huge_pte_none(huge_ptep_get(pte)); - - /* - * When coredumping, it suits get_dump_page if we just return - * an error where there's an empty slot with no huge pagecache - * to back it. This way, we avoid allocating a hugepage, and - * the sparse dumpfile avoids allocating disk blocks, but its - * huge holes still show up with zeroes where they need to be. - */ - if (absent && (flags & FOLL_DUMP) && - !hugetlbfs_pagecache_present(h, vma, vaddr)) { - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder = 0; - break; - } - - /* - * We need call hugetlb_fault for both hugepages under migration - * (in which case hugetlb_fault waits for the migration,) and - * hwpoisoned hugepages (in which case we need to prevent the - * caller from accessing to them.) In order to do this, we use - * here is_swap_pte instead of is_hugetlb_entry_migration and - * is_hugetlb_entry_hwpoisoned. This is because it simply covers - * both cases, and because we can't follow correct pages - * directly from any kind of swap entries. - */ - if (absent || - __follow_hugetlb_must_fault(vma, flags, pte, &unshare)) { - vm_fault_t ret; - unsigned int fault_flags = 0; - - if (pte) - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - - if (flags & FOLL_WRITE) - fault_flags |= FAULT_FLAG_WRITE; - else if (unshare) - fault_flags |= FAULT_FLAG_UNSHARE; - if (locked) { - fault_flags |= FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_KILLABLE; - if (flags & FOLL_INTERRUPTIBLE) - fault_flags |= FAULT_FLAG_INTERRUPTIBLE; - } - if (flags & FOLL_NOWAIT) - fault_flags |= FAULT_FLAG_ALLOW_RETRY | - FAULT_FLAG_RETRY_NOWAIT; - if (flags & FOLL_TRIED) { - /* - * Note: FAULT_FLAG_ALLOW_RETRY and - * FAULT_FLAG_TRIED can co-exist - */ - fault_flags |= FAULT_FLAG_TRIED; - } - ret = hugetlb_fault(mm, vma, vaddr, fault_flags); - if (ret & VM_FAULT_ERROR) { - err = vm_fault_to_errno(ret, flags); - remainder = 0; - break; - } - if (ret & VM_FAULT_RETRY) { - if (locked && - !(fault_flags & FAULT_FLAG_RETRY_NOWAIT)) - *locked = 0; - *nr_pages = 0; - /* - * VM_FAULT_RETRY must not return an - * error, it will return zero - * instead. - * - * No need to update "position" as the - * caller will not check it after - * *nr_pages is set to 0. - */ - return i; - } - continue; - } - - pfn_offset = (vaddr & ~huge_page_mask(h)) >> PAGE_SHIFT; - page = pte_page(huge_ptep_get(pte)); - - VM_BUG_ON_PAGE((flags & FOLL_PIN) && PageAnon(page) && - !PageAnonExclusive(page), page); - - /* - * If subpage information not requested, update counters - * and skip the same_page loop below. - */ - if (!pages && !pfn_offset && - (vaddr + huge_page_size(h) < vma->vm_end) && - (remainder >= pages_per_huge_page(h))) { - vaddr += huge_page_size(h); - remainder -= pages_per_huge_page(h); - i += pages_per_huge_page(h); - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - continue; - } - - /* vaddr may not be aligned to PAGE_SIZE */ - refs = min3(pages_per_huge_page(h) - pfn_offset, remainder, - (vma->vm_end - ALIGN_DOWN(vaddr, PAGE_SIZE)) >> PAGE_SHIFT); - - if (pages) - record_subpages(nth_page(page, pfn_offset), - vma, refs, - likely(pages) ? pages + i : NULL); - - if (pages) { - /* - * try_grab_folio() should always succeed here, - * because: a) we hold the ptl lock, and b) we've just - * checked that the huge page is present in the page - * tables. If the huge page is present, then the tail - * pages must also be present. The ptl prevents the - * head page and tail pages from being rearranged in - * any way. As this is hugetlb, the pages will never - * be p2pdma or not longterm pinable. So this page - * must be available at this point, unless the page - * refcount overflowed: - */ - if (WARN_ON_ONCE(!try_grab_folio(pages[i], refs, - flags))) { - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - remainder = 0; - err = -ENOMEM; - break; - } - } - - vaddr += (refs << PAGE_SHIFT); - remainder -= refs; - i += refs; - - spin_unlock(ptl); - hugetlb_vma_unlock_read(vma); - } - *nr_pages = remainder; - /* - * setting position is actually required only if remainder is - * not zero but it's faster not to add a "if (remainder)" - * branch. - */ - *position = vaddr; - - return i ? i : err; -} - long hugetlb_change_protection(struct vm_area_struct *vma, unsigned long address, unsigned long end, pgprot_t newprot, unsigned long cp_flags) From patchwork Mon Jun 19 23:10:43 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13285014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5AC31EB64DB for ; Mon, 19 Jun 2023 23:11:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A5FB28D000A; Mon, 19 Jun 2023 19:11:01 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C2858D0001; Mon, 19 Jun 2023 19:11:01 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 814D48D000A; Mon, 19 Jun 2023 19:11:01 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 72ADE8D0001 for ; Mon, 19 Jun 2023 19:11:01 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 4B3071C80A2 for ; Mon, 19 Jun 2023 23:11:01 +0000 (UTC) X-FDA: 80921044722.01.F9F5B5D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 41AF91C0004 for ; Mon, 19 Jun 2023 23:10:59 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=T9dFRBjY; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687216259; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=3EQ9G/WyCR5BzbeYKEOc/c4ex09bzeUnnwnMR75AIjI=; b=EHgxq3FKGPnzfHZrLnRNCCc/fbqHrCu8geHEQ9GcsludLMKQ0vDJj01EEUsK5gMJBPtO86 ROujGW9Ja34R6NHE8bUGq9TXo+uKNUtZsdgjKWaMdhKyvogzOY3QlsZjtwLqMg2TxFsKbs 8PkIv1KHjoh8xmOGtAgr4ZHWSw99FPI= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=T9dFRBjY; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687216259; a=rsa-sha256; cv=none; b=Kd4L1AwP9bHnhZh8I7RWpowZ8jOGEXnXqcS2WMKBKy96xlICK/zanqR/eU8j5tyjg8bbk1 2/UC9HgKri/uaoTTH2fWJulCwht+4tiXKF7d1+DFtXBH1XlRC0QxztkPIyDiqJdnNdvr+9 Uf/PZ2qyXR5iEtXVw5dRhxKRL85vtqY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687216258; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=3EQ9G/WyCR5BzbeYKEOc/c4ex09bzeUnnwnMR75AIjI=; b=T9dFRBjYGTeMn3UW8PlByTdCkNcICD4LTiwroGjUr0roGUo4OIwALMmmj2Eg/66K61rnpr aikB+pMKTZ+O4yzMVGGbAA2u9jEWDZ3SvuPSmFAVy135RqWMh0oheOSi47+yRJRNzytQLp vuwbl/V0kIJDH+wjt30vZzCOxYOj5fY= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-654-LPJrre6wOOGjplTKvSh2lQ-1; Mon, 19 Jun 2023 19:10:57 -0400 X-MC-Unique: LPJrre6wOOGjplTKvSh2lQ-1 Received: by mail-qt1-f197.google.com with SMTP id d75a77b69052e-3fde794cd50so5980741cf.1 for ; Mon, 19 Jun 2023 16:10:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687216257; x=1689808257; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3EQ9G/WyCR5BzbeYKEOc/c4ex09bzeUnnwnMR75AIjI=; b=a2BNaK2O1jm52ULGgFaEOPOau1cOYs8BEBwkcFGX39pFg41Gu3DFJBQwimYOcG9Ijh xNx0fZdnoWLvtMSZ7aFQlzkvA0/V1bWwgKcA7clOSZbvhaRowHvbHUMZXQYkJjT+hshT 2qF1fh6ghew4lue+gySP5TAll27HfgP+2aLUCGzf6vAvnbzldVJPkiIuCybwyykmCuHu 5JyyEacZ023b01A6FnH1/T5gdy7xB9345kA2V4M5281pdoCmaVyoqxei32vbazkq/FzJ yYr8dw6N61midVg22nlj8WuoqFsiRsshoXy5FgE2l9A3zktTBD8gTIcgs7LF3ey08WjS 3NRw== X-Gm-Message-State: AC+VfDwe58CHSveQmlCebRjWTxe6SBFJeeGlcUsVSNRizNlPEXlZJmTb t1pzbaGZvIc9h10XFkZKJ71AuWmP8/vYqR9HYagigtc1y5uS6j0CdClNG+8PMSFXnLNUg72+BgW DjwWiW3avT+NHwr6eS++Y1GDWST+TOLKjFiuuoEqWga7/GMgQJzgKWVWXBuI9Hhr9feEz X-Received: by 2002:a05:6214:411c:b0:62d:f62b:907 with SMTP id kc28-20020a056214411c00b0062df62b0907mr14179237qvb.0.1687216256949; Mon, 19 Jun 2023 16:10:56 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ5QlF4GkYWNboxTnBtKcrBbY45j5tqglo1WSMcbfSsLOwxwdtYl1NbYzYEaZwqHBtQcKataQQ== X-Received: by 2002:a05:6214:411c:b0:62d:f62b:907 with SMTP id kc28-20020a056214411c00b0062df62b0907mr14179194qvb.0.1687216256500; Mon, 19 Jun 2023 16:10:56 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id t15-20020a05620a034f00b007592f2016f4sm405864qkm.110.2023.06.19.16.10.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Jun 2023 16:10:56 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Mike Rapoport , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , John Hubbard , "Kirill A . Shutemov" , James Houghton , Andrew Morton , Lorenzo Stoakes , Hugh Dickins , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe Subject: [PATCH v2 7/8] selftests/mm: Add -a to run_vmtests.sh Date: Mon, 19 Jun 2023 19:10:43 -0400 Message-Id: <20230619231044.112894-8-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230619231044.112894-1-peterx@redhat.com> References: <20230619231044.112894-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 41AF91C0004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: uqportte8q7yheinqfk9wfdwasintfaw X-HE-Tag: 1687216259-258527 X-HE-Meta: U2FsdGVkX18knNlDIFyT4Dqqn2rl4yXok8teWT7U9FbX99ZSA3ojWaFbw5Hgv7PO7omiSDEkIStDI0RqKjZiCumNMEiRcfjU3LNZJ/HU2KE04ViP0qs1ACAshrCd6sQc4ZW4zxq0+sfvZA6ydgzNDH464dCHfpkHhJHHELSVmKXH9uWame5fTzFwDAhfAcZv0FyrjaI6PllR7k9tr7XhQP1A3P3+KbyEyVf6BoWI6a3yzff/FH9ztNwHQnpG6Xrz+xp4MU/4BnELAocaOODQ7p1Q603Zhnu2JkSEm2GrLoWBCy6JXDnEPRvTGCZ7WOJfyzXOWY8BHx4aCg/eA7jqiVwHRTnbSIVsUeijVfJ5RgcCmAPNZWwXtxM5joXmNIrqRheyTjrYO3No16KNXqTdB407qrQyWOsihBsqK2pLUHn7O9w8Q+NQzpeNJw6GODcLzjfofHK2LbQVIhPZoE/WUuOhXaTl1bdMOS00JObOzc13fd0RxGMyvxLfU1/ZDA2EOvotNAROW76TFcotvtgooBkokLrB6DH8hx96iuTztZMokL4n3RZweGCYN3WyRKmuRg6Wiei0/Bpz/Ys3Mqkam4dnSb1gvvPn4as/Y53aqLjZJm9KcsoAueRr/tqGHU9545aignmqCz+3oxybe1IX7PpklPF1C9Hm8KmDX2hIH8Bdyj+ie3o/CuGXRaUPqmW+qfDrcyZsTdKTZN2E2ZQZhZ12rIPjXGlKak62SPJvbDapO6S9+qRrqFhjBPa4q1rETPzNhWIhLI2DewfDBX8EEt69HN+R+w8EbsD9g56qFwf8bvblTTES6DHDMLJdrROlYlK90uNrYnPl8DLiVUFIhvEbJwemMmgGf53mRRpu7dQfdmPsOx8XPs63IHW1vMGZ+0UUlyQWX3++KE2nSTYNkHFk/1n3YQH/rlgod6ujmwRivBYXSlw9oQxKdExWMr1bvlt9yZUxKVphAN+MKPx t0RtcETx VUoe8347mBARDwFBgqjn7XBSFqJr9iDJ2r0d+aOZtqACG/5UWjyXjGJhLwkj6teXJFjCAPGXF/2o4hXQajT/1IwBqh4EXJdeReLSja7bibTmrtEK6hEWZ8a1GjzBDfISty8d7dsoqplQW1JrM/pmJGpRkv9COJgKFnbtyUwpHfyTUzDrzS8yLJDerhiqkSd7x12dNF/zyienSeEa8aJ/hXiyts4Z+FJyWKYBtmin0ewQOwghcub6iqnbNE8LtrqTgVci4muEE5gm5LKmP/TWvRVUPnWKE+EeSe93RYIYK1sOAFQclgri6bC/X4AqQq3zSixeXJm2dAH/1IIXcp12MdXMYpKNnj9KM9cx8MaqgYStDjevFkHxQv9IrPUnOUA8O6EcCctNGfSGGECmFNfvBSG2LVgFiQGrMwmVjJfltcJMRnmpQFLaQCmSisg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Allows to specify optional tests in run_vmtests.sh, where we can run time consuming test matrix only when user specified "-a". Signed-off-by: Peter Xu --- tools/testing/selftests/mm/run_vmtests.sh | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 3f26f6e15b2a..824e651f62f4 100644 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -12,11 +12,14 @@ exitcode=0 usage() { cat <"] +usage: ${BASH_SOURCE[0]:-$0} [ options ] + + -a: run all tests, including extra ones -t: specify specific categories to tests to run -h: display this message -The default behavior is to run all tests. +The default behavior is to run required tests only. If -a is specified, +will run all tests. Alternatively, specific groups tests can be run by passing a string to the -t argument containing one or more of the following categories @@ -60,9 +63,11 @@ EOF exit 0 } +RUN_ALL=false -while getopts "ht:" OPT; do +while getopts "aht:" OPT; do case ${OPT} in + "a") RUN_ALL=true ;; "h") usage ;; "t") VM_SELFTEST_ITEMS=${OPTARG} ;; esac From patchwork Mon Jun 19 23:10:44 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13285015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1511FEB64DA for ; Mon, 19 Jun 2023 23:11:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B21BA8D000B; Mon, 19 Jun 2023 19:11:04 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD28E8D0001; Mon, 19 Jun 2023 19:11:04 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94BC28D000B; Mon, 19 Jun 2023 19:11:04 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 828D98D0001 for ; Mon, 19 Jun 2023 19:11:04 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 392FF140343 for ; Mon, 19 Jun 2023 23:11:03 +0000 (UTC) X-FDA: 80921044806.18.31ACE76 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf20.hostedemail.com (Postfix) with ESMTP id 18DFA1C0009 for ; Mon, 19 Jun 2023 23:11:00 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=J7pkeBTk; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687216261; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=lJajW3CLipjyQ640k+WW23diII/jzqCXLm/r+yoabGU=; b=Q9/12tNDzudhH2D96BB+do8pSbCXovnrQ0+foYJ/fZ2cbwLVnhTd2bSKhwxw6j3K27pj8d r6gGKKDfrT9+Wy3eQBATc8kIUkojst20/3SksyPjhX82BzKU4a0sNnKK0mo4IdI+/5xOeK teV00GRxWbUOFLsjdf0VsBaTAU12oow= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=J7pkeBTk; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf20.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687216261; a=rsa-sha256; cv=none; b=WpbNjIn3vkQbJLvCrrDYWwF9osUDd9INrjHU7j9YNvj/G1smVNbd8WlTPNmjhNZxsPDwQq fSZYJjMS9kX0eRrZXTdYFB9qau/8MHMQYrTQiMk9ezl0Lk00WvhRkvgJV9iH6iETi7bClE iZuL5SdBk9jzpSbNwcyhvwz9BUtJd4k= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687216260; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=lJajW3CLipjyQ640k+WW23diII/jzqCXLm/r+yoabGU=; b=J7pkeBTklplcTv8Y+ughcu9h4NNKM/E9ufPJ7QTXVfwQpjm6C+UtnqksTZvBRuHUpAJMgh zL9OyZu+TqkPohaeNh40VIURjRbuzBZDOlrBWXzDENj5/lbMXqAw6nanA6klaj56o4rwg3 5AQ3bpD5guy4Po0pMt4gPljaaJIzaM8= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-523-2u7MNBRJMsSWK3IzRA40Kg-1; Mon, 19 Jun 2023 19:10:58 -0400 X-MC-Unique: 2u7MNBRJMsSWK3IzRA40Kg-1 Received: by mail-qk1-f200.google.com with SMTP id af79cd13be357-7625940fbf3so29809285a.0 for ; Mon, 19 Jun 2023 16:10:58 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687216258; x=1689808258; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=lJajW3CLipjyQ640k+WW23diII/jzqCXLm/r+yoabGU=; b=AcvlZ3cM3RSk51XVSv5rxozi/2WGG3VN3xFEt5EHt6ejDtnWWFkQuN/B7YvzFDNC/J HwIPIUBIzOq0D6jzgJh0nfgBpKywDeuj1q5WxbHdcIj/IsRArGmTdlEQGg6e7m++UfXJ Gtt3OMIOrxpFJfNcRmKxjA3BN3s876Qvo7aJC5YVst09QCTvZFSAoLUVHvfhTLPUekAf peC2Gn44AbBPQSwrsmkhADpmlzSgniP/qAa6t43rrlFPKgtaVXe0x4AglRgqnjkms/h5 vaCwFrX2DK85Uqd0exsA8kOwGSLzbc1jca6KPNEl7Kzcxz5XJeKKMwtvQjYjfv3WlVDG KQUw== X-Gm-Message-State: AC+VfDyaBAgVwCm/GwN2A/FPA8abq4+zeb5ZRzU+wsK6Uw/MSRJmwsdx nyuh4eEHagbRYXTVz1W47ZnPOXmTmbLddnTXjKHcF/r7oqk+Ic1Mk9+IVQUrHZAMY4Cmk8l0Kb4 VAeUbesCb7Eg3BewsIpMRzv7UNbrcI855Xnf+BDuov7zX9iiY6XEZ2RwyW3kNvfgqjVNW X-Received: by 2002:a05:620a:4104:b0:762:63b:e10b with SMTP id j4-20020a05620a410400b00762063be10bmr14345389qko.1.1687216258336; Mon, 19 Jun 2023 16:10:58 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7WYX27wSLfWLAP6ptDSxeQjeiOuxTJdqk6j7utUmQ2LCWuj5byVhLohQPj0ZsEQC2nxlnrpw== X-Received: by 2002:a05:620a:4104:b0:762:63b:e10b with SMTP id j4-20020a05620a410400b00762063be10bmr14345358qko.1.1687216257943; Mon, 19 Jun 2023 16:10:57 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id t15-20020a05620a034f00b007592f2016f4sm405864qkm.110.2023.06.19.16.10.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 19 Jun 2023 16:10:57 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Andrea Arcangeli , Mike Rapoport , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , John Hubbard , "Kirill A . Shutemov" , James Houghton , Andrew Morton , Lorenzo Stoakes , Hugh Dickins , Mike Kravetz , peterx@redhat.com, Jason Gunthorpe Subject: [PATCH v2 8/8] selftests/mm: Add gup test matrix in run_vmtests.sh Date: Mon, 19 Jun 2023 19:10:44 -0400 Message-Id: <20230619231044.112894-9-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 In-Reply-To: <20230619231044.112894-1-peterx@redhat.com> References: <20230619231044.112894-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 18DFA1C0009 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 5i4rw3hnri6fmq83a1fxuryrs3ga7eid X-HE-Tag: 1687216260-827878 X-HE-Meta: U2FsdGVkX19UzSazUhch/hP5hbCnQO2e+1ejcRCrOYPNVN0gxacM9twtTG+KxhKblM9pY/WsdoxsYy//QOEyark7Gd8VnqA7GlmmXIwNncW6IijOyPOXGPOWQqbQLDdDqPmtfinqVDa0DcpnzIBYN12bi4ygTfw9QXojmTd750SmKnjrW6DhHCp1fI34AfQoV4JDM5npfqO27hRMyPE6ZB90d8Ovm63w0otnBZ0iLdBBooKFtoEEbvaJFw0glZ/GM5DuR5S5fAABtDSEV81yPAtO+DqYktMXKwVCqJNac7ms4MCfdVh1oU1wpBUDt+uB6q4jPF0mSgUWaC3+upcdJH3aSGYmSjTd5i0Qj5JYQ3/o91UL+u5UebEBP8JSwRWalc5CRlv5Bjv/J/Pl6/5+r9hWA/4YVzi0gL1vWOK5snXj8YeT6RiSyarMVRKmqo0ni1Tucgi70pJ907lMrHrmPgUn3E3f6GncJ38UMtdGEe7XchDidXe8L5uUJfNft1NZSuZV/qSQIHUooFD+tQKpz07OXG4klwpBcmXKrTXTLazjv+J8JEmG0PSUslCITAG1nHEKh4lileVSFoleBmTHXgXkbud6pHjGJTLEvVxGy5bnH7eokEmHZ/cxuqMISF4LeADYK7jy4Sz0YHw8lij/uaF+rpQ0YADB0ywlcpzfqUiXPXCw9cI3vDFSXYWSVsm2xfQAUBk8+e/QLqqVzINBUjo0i2qKHZp+4AHZIBE6Ydm+PNC1/5ZDlG+xzaiCFk830ivMsK133J+YvLkzUbVg9N4D8RIghAOnxDA4Nur8M/pMPmSvXTGuf/ttu+hpYY0lfhrCy6/n/ZKSpU8KMjo/LJ3JFTXfK7JMimn9VN/jkCeW9BPxaFS8oSY0U5j3sYTtreL64aNhVpSf5y3bPZbGd5GuQ/uAyy+IHvJhyD7C7UoRzlrUXuQMRIc7ELUlzYXJEe9+Sqdz/OXbCGh0Unn qTnbAEnN ZrwYSiKQZsrDfFDAyk5mqTpn08ukpXo3Dg4NK1VrOT8Jat3R8UM8UcCM7/f8aUMx6SsAtV4aeCL+52mm/Fe5uyAHBjP2y6gkW7yqeAghGGoswHI4Cir1z9g1tB4GtLzhtxmfASR09PUJuMdJ0+f7HEgDu3i4jF/rby6L8PW0hEcy0pF4eVc3aWpUIDoUJ0S6IyUw5bRXSR48fyGR2G4WJMmuWp6MHQpz5VJzs9uHxe7slSMaRtEnwa5q/PB52h4tsrOZOgclW2QyavoHvbQqNSaOgxXIsckE//SXOfPbSGtT4cEbjHbxkpxsmCce9DZU82iDp4glryPHPSXZj3fQzQ+9WUfV9rwZtjEXgGlzkASCpEAMFU+CPgCgYbi/wD4TBXgBpKx9cWy7e5MZpmpIhOXt58DbPxhoqwbm1jWraRazycjGWdo3+wLKZgg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add a matrix for testing gup based on the current gup_test. Only run the matrix when -a is specified because it's a bit slow. It covers: - Different types of huge pages: thp, hugetlb, or no huge page - Permissions: Write / Read-only - Fast-gup, with/without - Types of the GUP: pin / gup / longterm pins - Shared / Private memories - GUP size: 1 / 512 / random page sizes Signed-off-by: Peter Xu --- tools/testing/selftests/mm/run_vmtests.sh | 37 ++++++++++++++++++++--- 1 file changed, 32 insertions(+), 5 deletions(-) diff --git a/tools/testing/selftests/mm/run_vmtests.sh b/tools/testing/selftests/mm/run_vmtests.sh index 824e651f62f4..9666c0c171ab 100644 --- a/tools/testing/selftests/mm/run_vmtests.sh +++ b/tools/testing/selftests/mm/run_vmtests.sh @@ -90,6 +90,30 @@ test_selected() { fi } +run_gup_matrix() { + # -t: thp=on, -T: thp=off, -H: hugetlb=on + local hugetlb_mb=$(( needmem_KB / 1024 )) + + for huge in -t -T "-H -m $hugetlb_mb"; do + # -u: gup-fast, -U: gup-basic, -a: pin-fast, -b: pin-basic, -L: pin-longterm + for test_cmd in -u -U -a -b -L; do + # -w: write=1, -W: write=0 + for write in -w -W; do + # -S: shared + for share in -S " "; do + # -n: How many pages to fetch together? 512 is special + # because it's default thp size (or 2M on x86), 123 to + # just test partial gup when hit a huge in whatever form + for num in "-n 1" "-n 512" "-n 123"; do + CATEGORY="gup_test" run_test ./gup_test \ + $huge $test_cmd $write $share $num + done + done + done + done + done +} + # get huge pagesize and freepages from /proc/meminfo while read -r name size unit; do if [ "$name" = "HugePages_Free:" ]; then @@ -194,13 +218,16 @@ fi CATEGORY="mmap" run_test ./map_fixed_noreplace -# get_user_pages_fast() benchmark -CATEGORY="gup_test" run_test ./gup_test -u -# pin_user_pages_fast() benchmark -CATEGORY="gup_test" run_test ./gup_test -a +if $RUN_ALL; then + run_gup_matrix +else + # get_user_pages_fast() benchmark + CATEGORY="gup_test" run_test ./gup_test -u + # pin_user_pages_fast() benchmark + CATEGORY="gup_test" run_test ./gup_test -a +fi # Dump pages 0, 19, and 4096, using pin_user_pages: CATEGORY="gup_test" run_test ./gup_test -ct -F 0x1 0 19 0x1000 - CATEGORY="gup_test" run_test ./gup_longterm CATEGORY="userfaultfd" run_test ./uffd-unit-tests