From patchwork Fri Jun 23 14:29:28 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13290733 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5F31DC0015E for ; Fri, 23 Jun 2023 14:29:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2D7928D0003; Fri, 23 Jun 2023 10:29:48 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 239678D0001; Fri, 23 Jun 2023 10:29:48 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 08C038D0003; Fri, 23 Jun 2023 10:29:47 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E765E8D0001 for ; Fri, 23 Jun 2023 10:29:47 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 9BC3E120C28 for ; Fri, 23 Jun 2023 14:29:46 +0000 (UTC) X-FDA: 80934246372.09.58A5FDF Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf23.hostedemail.com (Postfix) with ESMTP id 6B5DE14001A for ; Fri, 23 Jun 2023 14:29:44 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="JY/g+LJ3"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687530584; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=k1Z0+FcpBLSfoRatQPF2CylpoPyQCv2lrvtlx7M/beM=; b=JPyuhqM5GK2657zRdC8QCYzu4+tJKM2hIhpkwOysajLZYgh/aGsDpAi/uDl9dSdk758Qcp 6JRnHXaZvQ9fLLYSMiSLIRMe7n9lVglOusMDoBlR9N8VywXY27SCCb3jHBfOfG2Fg1ZFRp 6Px1kYQE2D7OL/5G9HguXmc7YFtu/PA= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="JY/g+LJ3"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf23.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687530584; a=rsa-sha256; cv=none; b=UJjHRQtsM9c/nV3pINDvlMVqva6HLj6AVer5vdKMyZ9Qnyd6xJmwXjfPwAK2lZsCmFAD10 0nWjaHEjd6bYygxKG3jnEp3i6i4ufyemV9aF2ncfNeVSAQLI9/+Pd/p8YnEkjIPOWsnplS kaMjX91KGlqQAUmwlPPYPxTZKd9Dw+g= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1687530583; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=k1Z0+FcpBLSfoRatQPF2CylpoPyQCv2lrvtlx7M/beM=; b=JY/g+LJ34xSK+gEkEyDHTFf+TTpHqXhjTLbONATx1qlk3L9cGGb0w17M21eGMFQv3kCVu/ 5P8fnf3iMa4R3HzYB+AQ29CwpAxPt2oslfD/KcrBtKCnCrp/WLVj/n+eCxpAeuNMYKhSk6 55hjiGd39PQzkdrxsf7uyXhQpja80bE= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-209-RNrMR1lyNgaJ78evbB-A_w-1; Fri, 23 Jun 2023 10:29:40 -0400 X-MC-Unique: RNrMR1lyNgaJ78evbB-A_w-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-62dd79f63e0so1400506d6.0 for ; Fri, 23 Jun 2023 07:29:40 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687530580; x=1690122580; h=content-transfer-encoding:mime-version:message-id:date:subject:cc :to:from:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=k1Z0+FcpBLSfoRatQPF2CylpoPyQCv2lrvtlx7M/beM=; b=SMJ7a9XdM+uRHNNMWVArUviscAIpajw+pqZGbj0o8hYpMM2b0rGgbky5Nude4WZjr0 of6Ko9vhMNnWqW1qKHTHv40gIZF8Dmc9uIfjtDH/GAAy89bMkLCsIfK/SkVFCA1mjjZG aROda+hppnB5im+QMUCXsSiwAiuBmBMSVEaS1G9SB3beO1e9nM946qJXHMsegFs2FBXT tlDsrgjf1xPTP5d3Naeslpa0++rUFmQ3FvJaFEDozwQiNiR9v1crpICjWwxBqsDJ/uqy JdGYDPpOiaxGWXc2NAcEmdeNfpTwG4wwyBaXxKpugtRlh/9D7nAQzaxI7wQw1v3LWqtw ISmQ== X-Gm-Message-State: AC+VfDx4GVMhLLEvnqsxhhxWCBX9R0iOwymIfDJkFqVApABxrsHA06Lu D2U5HSVN6Tac/VasN3uCO4ci48twr6OX3gxV1rt1n7cVWeBDyq55RLE37Pn9mnyiZnh4MkC0ewr Akar5YYgl0WlUh6EOhiJBsb/Y7mA83Bg+cfPra6C4+TlIyH39gxSqcHram6FrlnqPHu4y X-Received: by 2002:a05:6214:5186:b0:62d:eceb:f7ce with SMTP id kl6-20020a056214518600b0062decebf7cemr8898221qvb.1.1687530580041; Fri, 23 Jun 2023 07:29:40 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ61iOgKq4TGE1+eFXklI8JtA4+EKdWL0nSblu+aHPTTyXIqpedJA43pT3tKCW+xiQEjFC+xqA== X-Received: by 2002:a05:6214:5186:b0:62d:eceb:f7ce with SMTP id kl6-20020a056214518600b0062decebf7cemr8898182qvb.1.1687530579566; Fri, 23 Jun 2023 07:29:39 -0700 (PDT) Received: from x1n.. (cpe5c7695f3aee0-cm5c7695f3aede.cpe.net.cable.rogers.com. [99.254.144.39]) by smtp.gmail.com with ESMTPSA id b9-20020a0cc989000000b0062821057ac7sm5104827qvk.39.2023.06.23.07.29.37 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Jun 2023 07:29:39 -0700 (PDT) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Lorenzo Stoakes , John Hubbard , Andrew Morton , Mike Rapoport , David Hildenbrand , peterx@redhat.com, Yang Shi , Andrea Arcangeli , Vlastimil Babka , "Kirill A . Shutemov" , James Houghton , Matthew Wilcox , Mike Kravetz , Hugh Dickins , Jason Gunthorpe Subject: [PATCH v3 0/8] mm/gup: Unify hugetlb, speed up thp Date: Fri, 23 Jun 2023 10:29:28 -0400 Message-Id: <20230623142936.268456-1-peterx@redhat.com> X-Mailer: git-send-email 2.40.1 MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 6B5DE14001A X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: k16zb6pjtcqp6h31jd15jgr5zkh44xcj X-HE-Tag: 1687530584-471956 X-HE-Meta: U2FsdGVkX19GEoaKqG05AGVDdagj/Fc/guzE/bbcL2ozExCjDqC4R5+m0Gp6NPCIotv8Yqm6cPdf+1ISPQ4u5Q8nvxRePnJSN/tIfUtBA0c6FH1zyUcOp7pQ5LUyASasEg5aVMb22r0RO/08qN1IjjVADoTqMvmZp0v98wvvnIsWkCRH0yRjCKALSABsfolTUQ+Ct5sndbV7W4Sz1xHJzatd5ARzYn84DfIHOj3/vQwIT8RFSuIAMbtZ2wfC5sQJWRRuGGzVcoS4HIbrgkiMlgwgotVNfatMHqZYLw+aiQ58rhqXcvqJoBxNKESK1N9+zqjXuLfH02tnKkZc2/Ly7w8AlzWkHPILFclEqroZqtUHIMb3WgiE5CZiQy+Po0wMxiDiTRK1FRoym8IHss4kqm14V0DKZqtQ71TG4ssNd1M5FiObclyPwrFp6OMR4HxIEz0Dvq4n+tz7N/rghh3Cuh4Dpk+O/XnPJ9cnykY3EZD6rOdItWTBO7pi0Vz+W1lTepZ6yXUIKxeMjmTKfLIsKsVOuHBy7WauWbCvJMJT/EGCfV8EhXuZECRJdwD60+kvdY5lm1EBVlPd1V+ZVNJt53VJ89RFJx9yUBYepteyQHvqI+gVNgMvvVl2mvqgDWYcfuHR75MqDuFu9Em4ac4QlYFsgdLMEUxMubDZ/D2DTC5+n8NSV6hlKRtkd7BmnuWNTBlk6JVoFeTBNBERMF9x4YJHtSDfwhqUKAK1gtdrhdMuMnOgvHgHC4n3RVxNR/8//ihgTL9a7B1awiBdYuoNQ3oUCrU2EWO3bj4C+HU8Jd0N0xA438DdlKIgDRvLM8SwrhllgYIJPmJB1h5V4rZVdqVywHCY269Uf8kY0wBcJKHoChaaYho9qfE4M+wrufqNd/Yup7pg4WixsehGAZGO3BdA5kNs8gUL3MEwR5ynXnu9omrkz25zerY1WWGpHVUL5IoKJjKFLYZgf6PmJY4 V48AM/w/ xrXoKUojzb9K1ZK9F4w0l+1KFwsPfJsdcuRAjWlDUEzAFmli6i9PAXMf0mmsmVczgO7bhL+UReIw2dliMxoWXfSQDWhupKr6WAjQ7hE2VqGxtnC/EgmMjfF1cV83WBijKf8Px6p1uK/UbCknkJ/vv5f24Mr12r9Ci9GaBX8gi5R7qz8os3mwFNbJQR6RwzaZ3hxfUQ9vRkP6CkZ103936NOCa1y8cTTx/aFPFpNWLwnNQ+XDd6XTI2t8lUZSEXDhkAJ+zqxVV/e257w8pp35XaQwwGjtPBk35DWtBeEwj1n+Ij5qzJhXLnR5vpXWvcgv9+EyOjaCgCabfHmDJasLvdWg8keet0ewlkrYQ5oPG88RcC25dFuGIyR7W+8+0i0W6gnrB/GoWqbKLT7isDEe6usK3HAVdYSXR0UXMwCCoxPiyqSphLCGI6ntyEIBBbS8VjXaW X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: v1: https://lore.kernel.org/r/20230613215346.1022773-1-peterx@redhat.com v2: https://lore.kernel.org/r/20230619231044.112894-1-peterx@redhat.com v3: - Added R-bs and A-bs - Patch 2: - s/huge_pte_write/pte_write/ [David] - Add back the WARN_ON_ONCE I used to have in v1; change retval to the real errno from try_grab_page(), rather than NULL - Patch 3: make the page_mask calculation slightly prettier [David] - Other small cosmetic changes Hugetlb has a special path for slow gup that follow_page_mask() is actually skipped completely along with faultin_page(). It's not only confusing, but also duplicating a lot of logics that generic gup already has, making hugetlb slightly special. This patchset tries to dedup the logic, by first touching up the slow gup code to be able to handle hugetlb pages correctly with the current follow page and faultin routines (where we're mostly there.. due to 10 years ago we did try to optimize thp, but half way done; more below), then at the last patch drop the special path, then the hugetlb gup will always go the generic routine too via faultin_page(). Note that hugetlb is still special for gup, mostly due to the pgtable walking (hugetlb_walk()) that we rely on which is currently per-arch. But this is still one small step forward, and the diffstat might be a proof too that this might be worthwhile. Then for the "speed up thp" side: as a side effect, when I'm looking at the chunk of code, I found that thp support is actually partially done. It doesn't mean that thp won't work for gup, but as long as **pages pointer passed over, the optimization will be skipped too. Patch 6 should address that, so for thp we now get full speed gup. For a quick number, "chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10" gives me 13992.50us -> 378.50us. Gup_test is an extreme case, but just to show how it affects thp gups. Patch 1-5: prepares for the switch Patch 6: switchover to the new code and remove the old Patch 7-8: added some gup test matrix into run_vmtests.sh Please review, thanks. Peter Xu (8): mm/hugetlb: Handle FOLL_DUMP well in follow_page_mask() mm/hugetlb: Prepare hugetlb_follow_page_mask() for FOLL_PIN mm/hugetlb: Add page_mask for hugetlb_follow_page_mask() mm/gup: Cleanup next_page handling mm/gup: Accelerate thp gup even for "pages != NULL" mm/gup: Retire follow_hugetlb_page() selftests/mm: Add -a to run_vmtests.sh selftests/mm: Add gup test matrix in run_vmtests.sh fs/userfaultfd.c | 2 +- include/linux/hugetlb.h | 20 +- mm/gup.c | 83 ++++--- mm/hugetlb.c | 263 +++------------------- tools/testing/selftests/mm/run_vmtests.sh | 48 +++- 5 files changed, 124 insertions(+), 292 deletions(-)