From patchwork Fri Dec 16 15:50:52 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13075132 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1B53C001B2 for ; Fri, 16 Dec 2022 15:51:13 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 99AF48E0002; Fri, 16 Dec 2022 10:51:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 90F5C8E000A; Fri, 16 Dec 2022 10:51:12 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 69A9D8E0002; Fri, 16 Dec 2022 10:51:12 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 4EC338E0008 for ; Fri, 16 Dec 2022 10:51:12 -0500 (EST) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 337514115F for ; Fri, 16 Dec 2022 15:51:12 +0000 (UTC) X-FDA: 80248608384.28.70C30A6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf17.hostedemail.com (Postfix) with ESMTP id 3A75A4001B for ; Fri, 16 Dec 2022 15:51:10 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NawBOwIk; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671205870; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=W6TRH+D4KiiNcSkU8trC/D1fuTVk/3U3ig0mjyWFdEw=; b=7prTNVpe6RoPceeuL5TtGtUTtYgnXXi3eEsx+Vl7GouM40BqCK8ekQfpZBhHnB3Moi9iEK D9IqzsxmQxuF4W1NbEW8/1Dp4vuhXv92hdvzg8DxxYR+x7FNeloiig4lYadk2j4eT2Kf91 LuNQhOkd4bXE9jxfjz99imYdNwIRhZk= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=NawBOwIk; spf=pass (imf17.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671205870; a=rsa-sha256; cv=none; b=QzfN7/M8ir+TajLvPvsO7BOKHGNj4YiTRa+ZyOw6jsGgSYb63cXtllHffZ9W9Wr69kuBRK nMvQ34D12y9nhMNNMLo2SV7/CNmZITIUtEjIdhWFmsBI0ZwdzsRlW2zWYqQN7VtLJSqFOq jIKiDRFx7eg0pSQm2li3MuaFztuuKOY= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671205869; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=W6TRH+D4KiiNcSkU8trC/D1fuTVk/3U3ig0mjyWFdEw=; b=NawBOwIkY6pLpbM0P1RyjWF81H0WCF5TVgKkr++GwpiKgU5Q8e6fUMny46VUQkYRyOFnr3 c2BdGNCEXiS5Hu4MDN22b50HCjRjhyzpDDcmC7vIJUrAeJ8NY5nBFxi3FwCRlrGR9ig3N3 n7gvNdTglOnyVTeuxHXJoLC7icky6VY= Received: from mail-qk1-f198.google.com (mail-qk1-f198.google.com [209.85.222.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-373-qUSS89UsOiq4MbxqqH--LA-1; Fri, 16 Dec 2022 10:51:08 -0500 X-MC-Unique: qUSS89UsOiq4MbxqqH--LA-1 Received: by mail-qk1-f198.google.com with SMTP id ay43-20020a05620a17ab00b006fa30ed61fdso2073680qkb.5 for ; Fri, 16 Dec 2022 07:51:08 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=W6TRH+D4KiiNcSkU8trC/D1fuTVk/3U3ig0mjyWFdEw=; b=0CGyXi8uiusqwfJVzCWhw9QF8/D59yxUgO1JxXv1PWQ6ig9Xr5oE5ar0EVLmGs0JcU zsk7LvXtQed2dJ8Oc457edAxl6qBE5CQgfy0B/Ryjeta/dSBECXyZ/wWUBZ1iSGPWU+w 9AczyI9BcSMCcWt5u2luvSY3jut9XKQft6Q1yg9UoaDrjjC/awnt8TwWHbZ0CM1E+rJA 3Gn3YlAXrFVzUGsMDo3o6RJWAcq8wSquzENGQK24Utcdn59zQbh7oRABKCZBh+vuzgU+ LUJdAUhAA0ZkXQHWzvZxY3JYB8tTBTKRPBV2ouYOV3xy0rtoCCKRYih9hIV/PAptVmKH Jptg== X-Gm-Message-State: ANoB5pk8BQh1Ww3auLNFY7FI2dNcXp+m0LNzFkX6Eh2LG6agY5JiHaaI zhAmClI5mjQvIUugzENp8jAUwL8o9SPYsZNN0+0ngfI64NPdMmM2zvo0/xeXBXJ5uHRyIRpsIA0 01CovjYZWo8E= X-Received: by 2002:ad4:5883:0:b0:4c6:f65b:2e16 with SMTP id dz3-20020ad45883000000b004c6f65b2e16mr49280190qvb.21.1671205867778; Fri, 16 Dec 2022 07:51:07 -0800 (PST) X-Google-Smtp-Source: AA0mqf4KCS0PO65p/SFUOeoYSoiml6w+3ib8mlnjzyMFPkaRCZw2NRbc7RAQAd1lnesrPAfcNCB8IQ== X-Received: by 2002:ad4:5883:0:b0:4c6:f65b:2e16 with SMTP id dz3-20020ad45883000000b004c6f65b2e16mr49280167qvb.21.1671205867572; Fri, 16 Dec 2022 07:51:07 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-45-70-31-26-132.dsl.bell.ca. [70.31.26.132]) by smtp.gmail.com with ESMTPSA id s81-20020a37a954000000b006eeb3165554sm1682297qke.19.2022.12.16.07.51.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Dec 2022 07:51:03 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Muchun Song , Miaohe Lin , Andrea Arcangeli , Nadav Amit , James Houghton , peterx@redhat.com, Mike Kravetz , David Hildenbrand , Rik van Riel , John Hubbard , Andrew Morton , Jann Horn Subject: [PATCH v4 1/9] mm/hugetlb: Let vma_offset_start() to return start Date: Fri, 16 Dec 2022 10:50:52 -0500 Message-Id: <20221216155100.2043537-2-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221216155100.2043537-1-peterx@redhat.com> References: <20221216155100.2043537-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 3A75A4001B X-Stat-Signature: 85dnef51ufb4a5pi5363xmp6143bo81w X-Rspam-User: X-HE-Tag: 1671205870-523934 X-HE-Meta: U2FsdGVkX1+BkrKbkNlhAZVGth0G7Nm5WGFt1St5DjO1OLNhX0vBV+yseZXGkyBKcBD/lwqrAwzRIt/gV6lFTMyv+76EKyJpHoqoVK2yGIyc3koNZoKKYeEGVqADUcEa9gIgS+QEowhwd6sljxSFQNw+khOzY7Jq/hSKYRgkimM/UQghez6lE4MxueKTDkK2CW/HXKcCwkYJHGjA72asKo0BX9hjGXfMXJ0kLTMd5rgJyxbovIYu3JX1YHUIsMv0ZK0NuB9hQROfeDUjnEdJvm/7VI7zJ/fvAyc4eRJ9nGRtVZA29BhLR7N09+f8DPiOCbJBaBlLx0+hXJ1kQiRULbIK0Jwag4deRlXFy8x/zlZDvmdyu5p5+oS7G85K/0KyK1LpSEMflFwvMb4HnvuJy/Ir2+5Lij0ORQrqrD3RwSiwRmxDPEHnx0mI+3sfVBBlYwOQguY3JpOeCWAa1rl5jI8LIyxEx+4yZ6cDh8rVHskqPN3VNW2IM85Wm9BtxTPBBou0vBBGqsoHZEg0979qoQYeJX4pObxiHmntkh+Luzk9HkJon9H0UwAL+1X4oqBRywovEYx2diAirCUZu9j1ir0R6P9IfxnbpKklPlWiG/MTtNZLav3wuHogI6pTq25fDDhPqDVtwn14hyJD76zpS56V39vbKUtL0hxfxeUe6w5UA7ln/Bj8FA8A0o46eqrmbF1h0P6umDpBIamGgVejz55GJJyon85fWEz3uA2k47gdlBlu3kvqoaoP4SZPV5qLKB8k8NAedMMdnKupUM0839aI2j7ZYF41UjGoBEPlYYKHwR/an4GIUm31gHc/6s1ojEGjFOIyLf8w8Sahj4mi0IjJqgQqvjYTMbk0HyAUnvEuOLtnUBNmg1uSOX85DsbBaLtg5hBxxhQzFKrTflplfQfMLFHuJqnP1N9gEvQwim75PY9lKEt5KYMRTLkx9gHEuC+NzjATyfUSNccOupi fWGuWb8q H5Gb6gAuSqOQ4nqBP/PAmK/+yK96awbRfMKhltgSaaYEC5Zk99KwkDytl1uTdBORclyESw2AQmiy+Dw0FZ8cJS39ZDHEQLhGFbBVjn7RRtX+fElwPK04BV1cMb2qGJ7lqKnYgqDCcMy5WFRj/8wtGjtoVUmQcf6VzkD5J4fv29exOKHLfu5xdZ2ZmfsMt0w/GW3Mospwl2AuTQTMovD583BNQ3R36pgHPeBBG X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Even though vma_offset_start() is named like that, it's not returning "the start address of the range" but rather the offset we should use to offset the vma->vm_start address. Make it return the real value of the start vaddr, and it also helps for all the callers because whenever the retval is used, it'll be ultimately added into the vma->vm_start anyway, so it's better. Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- fs/hugetlbfs/inode.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 790d2727141a..fdb16246f46e 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -412,10 +412,12 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, */ static unsigned long vma_offset_start(struct vm_area_struct *vma, pgoff_t start) { + unsigned long offset = 0; + if (vma->vm_pgoff < start) - return (start - vma->vm_pgoff) << PAGE_SHIFT; - else - return 0; + offset = (start - vma->vm_pgoff) << PAGE_SHIFT; + + return vma->vm_start + offset; } static unsigned long vma_offset_end(struct vm_area_struct *vma, pgoff_t end) @@ -457,7 +459,7 @@ static void hugetlb_unmap_file_folio(struct hstate *h, v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - if (!hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page)) + if (!hugetlb_vma_maps_page(vma, v_start, page)) continue; if (!hugetlb_vma_trylock_write(vma)) { @@ -473,8 +475,8 @@ static void hugetlb_unmap_file_folio(struct hstate *h, break; } - unmap_hugepage_range(vma, vma->vm_start + v_start, v_end, - NULL, ZAP_FLAG_DROP_MARKER); + unmap_hugepage_range(vma, v_start, v_end, NULL, + ZAP_FLAG_DROP_MARKER); hugetlb_vma_unlock_write(vma); } @@ -507,10 +509,9 @@ static void hugetlb_unmap_file_folio(struct hstate *h, */ v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - if (hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page)) - unmap_hugepage_range(vma, vma->vm_start + v_start, - v_end, NULL, - ZAP_FLAG_DROP_MARKER); + if (hugetlb_vma_maps_page(vma, v_start, page)) + unmap_hugepage_range(vma, v_start, v_end, NULL, + ZAP_FLAG_DROP_MARKER); kref_put(&vma_lock->refs, hugetlb_vma_lock_release); hugetlb_vma_unlock_write(vma); @@ -540,8 +541,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - unmap_hugepage_range(vma, vma->vm_start + v_start, v_end, - NULL, zap_flags); + unmap_hugepage_range(vma, v_start, v_end, NULL, zap_flags); /* * Note that vma lock only exists for shared/non-private From patchwork Fri Dec 16 15:50:53 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13075134 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C275EC4167B for ; Fri, 16 Dec 2022 15:51:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3DE5A8E000A; Fri, 16 Dec 2022 10:51:15 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 33FE58E0008; Fri, 16 Dec 2022 10:51:15 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16B3B8E000A; Fri, 16 Dec 2022 10:51:15 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id F36F38E0008 for ; Fri, 16 Dec 2022 10:51:14 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D002CAB97E for ; Fri, 16 Dec 2022 15:51:14 +0000 (UTC) X-FDA: 80248608468.04.F29FD80 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id D1B42100004 for ; Fri, 16 Dec 2022 15:51:12 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ELVCcT8X; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671205872; a=rsa-sha256; cv=none; b=ukOwQI0/D2RFHP60to7qzpSq1sGD/AMm6jvcqcRqFpBxE03H2vBjkjl46Z9pMdmuTIb+OM oPqxbG8klvGyxmZcmVaYza/DmLTxwUn1qtNPwWGtzjRYhFFNb4+qVrk/rMH3C3DZnCI6ka /PwAk+6SfK8p4djtsNMFuzpLxANl1Os= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ELVCcT8X; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671205872; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=t1N733mGfOZDIeBgBOLPf9hKyMpT23Vbs9QUhNwynbg=; b=AKQQNQWJBbQgbxZ1u2JN+eZmCW327mVxtED/n0jm5i3tQr82QwTUOtZU6cdaUHgKOzBrIQ j23jlRrFRHb9LWp07uvXQ2LALQkaJz6oxhDLKe+swzJVvJr/c7pufdG77CK2/wFQfjxkcZ ffEtFVWJWsqttOagDT/KXH78O2JBHy4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671205872; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=t1N733mGfOZDIeBgBOLPf9hKyMpT23Vbs9QUhNwynbg=; b=ELVCcT8X2+qnYXjMSo4POIva87565guTEAQXoKVa+hIFfm7ohRHqyMc51cDTZ9r20Pjgr1 5bbi2n0F3iTIU4YFWo6PTfy+Lwm9EM3JRTF7clQb+IleaB1wYTjxKE+1iaGk+N+aEoK44s sOl+6QO6dTXZqfgFgHbCUKden7ziVe4= Received: from mail-qk1-f199.google.com (mail-qk1-f199.google.com [209.85.222.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-65-INbXPdXZObm7keFtvjqjYg-1; Fri, 16 Dec 2022 10:51:11 -0500 X-MC-Unique: INbXPdXZObm7keFtvjqjYg-1 Received: by mail-qk1-f199.google.com with SMTP id q20-20020a05620a0d9400b006fcaa1eac9bso2053430qkl.23 for ; Fri, 16 Dec 2022 07:51:11 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=t1N733mGfOZDIeBgBOLPf9hKyMpT23Vbs9QUhNwynbg=; b=r5MVOoSCg+Mqf+JzGxt3gIgQXMOiilAD4XbqqhM61Id/JJoSltowBI/oIjhgDPTdzJ To/3ZGu4lIky8HjrChRlgS5ut9uZX9t8MMV62lO9mYA/LgRxGolqqjj0+d9pu0eA2CrG LPU2+U5amV8FwMvx6BmXdZdaBkBOywZgIRcABmiu1r1TY6m+bqYa2soKXlODsrAhVISw EfyyzeDfuEI3wOovzRd5Q+mXJntydjVVR0YWaXJNPDUFSlBQtRtOiDSA9jh3gUg1ozCZ I+78zCDqON9D8mzC71P/ZvgxVwyTCf0qnKzaGDo1lL7Fr60yK6q6qZ1p/sXVR0kcj0iC k0DA== X-Gm-Message-State: ANoB5pkmO08kraov2WCuPukFw3gVJw079t/p3m6cAlMu9iynoUOAH9IH GkoYjkLGcwJR9u5YiPO3JAIxYHr33dJ2U7zN0RBbNXoL0UI7+isK0FAiVZAbs9HrIZFPFWcS1Qx mOS7XuZFxFd0= X-Received: by 2002:a0c:dd01:0:b0:4c6:fb9f:a327 with SMTP id u1-20020a0cdd01000000b004c6fb9fa327mr41198456qvk.29.1671205870611; Fri, 16 Dec 2022 07:51:10 -0800 (PST) X-Google-Smtp-Source: AA0mqf4FDmXNL5rmIhjVY1GIvpSpY4Qco5LlZXCExcuuFvXBDlMq/YlZ/u02dz6yRyxhRquFj5j+Cw== X-Received: by 2002:a0c:dd01:0:b0:4c6:fb9f:a327 with SMTP id u1-20020a0cdd01000000b004c6fb9fa327mr41198425qvk.29.1671205870379; Fri, 16 Dec 2022 07:51:10 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-45-70-31-26-132.dsl.bell.ca. [70.31.26.132]) by smtp.gmail.com with ESMTPSA id s81-20020a37a954000000b006eeb3165554sm1682297qke.19.2022.12.16.07.51.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Dec 2022 07:51:09 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Muchun Song , Miaohe Lin , Andrea Arcangeli , Nadav Amit , James Houghton , peterx@redhat.com, Mike Kravetz , David Hildenbrand , Rik van Riel , John Hubbard , Andrew Morton , Jann Horn Subject: [PATCH v4 2/9] mm/hugetlb: Don't wait for migration entry during follow page Date: Fri, 16 Dec 2022 10:50:53 -0500 Message-Id: <20221216155100.2043537-3-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221216155100.2043537-1-peterx@redhat.com> References: <20221216155100.2043537-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: D1B42100004 X-Stat-Signature: gg7onrk49d8myfmff3t5uurxde7wpnio X-HE-Tag: 1671205872-926323 X-HE-Meta: U2FsdGVkX18gxplshuRetAG1JvOU6JdpLjhqeXicIEO2zsuGrTEs+rkuqTV1Nff+lga7sFGUjyRN+X5ldn2Xg14vfgCqhD4GIabQtP5Te0vh0/YppvSJPJrP8QBgruUUYX2IN7gxMwP4F7gjbMUkx1Z+MlDumkIS9EeiSBah7OUcy/wIcWi1i2yquwuXIk4u6eGjCv3Kl9U5KQQWztHCbjIlDc+UyxEXc+6InY1rlCltY9AuECdsSwKb/DZCcwJl7hZXswwKilQh8cN6mNcOizbGyqTTa9VxW3aTVmX9v3CYnIkysNV/8sR2LJhN86GcZbzxZIKkHrerdtoDSBNxItr9gxv6n4Cs97Ipk7UWjoGrxgSr+wPvpJPyPr2/N96wGbpO/Wi8h589CC76jTIg9RhZ6e6NhoXCjYU8NlgW41+GSeAybg0ZzzDUvG0dkypQt74Xuqptfkpv4sx47l9Fq0rdG0LFm4yYAr9oGCr8D3bznkK2b8TP09ZggXywX6s7nf16KrHFWxvt+ntMEOabrcnqm4+K8h0CPMUduvFrptymXWr/5zq+0Ia+Gj6++rf+PgtnVKlzxCwFQRXZ4iemUxsmglIWxVUxdJn2zCL8UzOfh3VHF5nlGzXVoahCodc4SzRjPsne1UaAsGdHpJyaYtXgPG/QYRdmHxMLwSrFGgaFN/NBG83Yh6Z1VznXomr9oSfnDzPihRwicZV9SZrhlILoY1BNW8ikTra35gKAf4PZ83HI017NRbwNeK4CSqNMdt8Flk8wZTKR45gASaAL0fmeNE1Ds+480mNxvsS+yXgEXQyfi4P3i5I1020KWE3jTqpQj9s/q1Vu009rBhV9AhErodtebm+l0FClq1eruXyl/vjRothc+j8qqYwH6QlmKysyVgaatYLhD4ndQ8C+S3ZIxo+n/MzpVhmlOLY1fzIM0DgxPddmAaDoTxsclUjKmLT8B1cVVUYzAfLro+q oMN1Dcoa fs6h38fBwa3NSs3YdjCD/DZPYc/rw3t9MJDktGQI4fylm0I8Ij/AJWEGOJLBIZB2VaOPmmgUYGIFmIyBlYMyYN+e0LVMIOS2g8UfMiEP7XhQARLQ19HhvCAwqn1ae+qcxI3chZaTGFLFog3bG/TrSTfq6e5EAaeP7QrpzdqNhRKcbR+TqtWUf/ANR0w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: That's what the code does with !hugetlb pages, so we should logically do the same for hugetlb, so migration entry will also be treated as no page. This is probably also the last piece in follow_page code that may sleep, the last one should be removed in cf994dd8af27 ("mm/gup: remove FOLL_MIGRATION", 2022-11-16). Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- mm/hugetlb.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0dfe441f9f4d..8ccd55f9fbd3 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6380,7 +6380,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, if (WARN_ON_ONCE(flags & FOLL_PIN)) return NULL; -retry: pte = huge_pte_offset(mm, haddr, huge_page_size(h)); if (!pte) return NULL; @@ -6403,16 +6402,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, page = NULL; goto out; } - } else { - if (is_hugetlb_entry_migration(entry)) { - spin_unlock(ptl); - __migration_entry_wait_huge(pte, ptl); - goto retry; - } - /* - * hwpoisoned entry is treated as no_page_table in - * follow_page_mask(). - */ } out: spin_unlock(ptl); From patchwork Fri Dec 16 15:50:54 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13075135 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 61AE9C4332F for ; Fri, 16 Dec 2022 15:51:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E642A8E000B; Fri, 16 Dec 2022 10:51:17 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DED278E0008; Fri, 16 Dec 2022 10:51:17 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C3FD38E000B; Fri, 16 Dec 2022 10:51:17 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id B15A18E0008 for ; Fri, 16 Dec 2022 10:51:17 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 88FDA161163 for ; Fri, 16 Dec 2022 15:51:17 +0000 (UTC) X-FDA: 80248608594.09.ED901E3 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 716C020002 for ; Fri, 16 Dec 2022 15:51:15 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fFpGvVIO; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671205875; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=KF8YBIr+SenpQMPOCGcYKB11m6lZYSBtX9iYqo4LY9E=; b=hEraS+PKenkG8/qInhiCUDw4lCJs8E4h3qn4a34wHCt8mtZXsP4jQsUo1Xb/ZgcnqQLkot IbrUIcWkLoOQCgRB8GGwh2MmsqvH4Vn7HkRpnmZUmX8Cn5YYLPCqk+5PNwuVZAG2tI1xOq HQcaUCJvyZZtdWJj/PL0g19j8Q3hOHM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=fFpGvVIO; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671205875; a=rsa-sha256; cv=none; b=j5ItWDrg/EnCwOzBF88oICV1iqgTGE5pig8N9yLUv95nVS0RBEmjfWmGIhENtFHnv9XPY9 Eo6zhvIj7lLIHPRdCEjNlVREqIoqaP3S/qwSjwXELHB//jgmBOwwU5dRb47stdfmrSyzJx qCEJFceTOTTFcg2xe9ia+LPo1/+4PW0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671205874; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=KF8YBIr+SenpQMPOCGcYKB11m6lZYSBtX9iYqo4LY9E=; b=fFpGvVIO9F6Z+ib+i60gCNJkKYejSxMkfzCK6txsTHZnQGagCwP65sGrrFoHaQe4QS7McM RyNZoutjNdynSV4c48raNHZqopUU7Dz7jOanOexboKrN5giFZYz4YoiedmLBi8FeEDAbo5 5ZUdYQdHiOg+H9monTmyma24ZvWPQqo= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-259-gAvNHcyZPwyKlmZVpPge_w-1; Fri, 16 Dec 2022 10:51:13 -0500 X-MC-Unique: gAvNHcyZPwyKlmZVpPge_w-1 Received: by mail-qk1-f200.google.com with SMTP id bm30-20020a05620a199e00b006ff813575b1so2066751qkb.16 for ; Fri, 16 Dec 2022 07:51:13 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=KF8YBIr+SenpQMPOCGcYKB11m6lZYSBtX9iYqo4LY9E=; b=cFn1VJeXjPYs2OT5ah1cugjsFF3OKiVV+daMpXdV+dsn04cdO8BMpmdH8wGH76ryYq 2/yneyKc7TKeeSPSkRrhilxgjc0rmxpo54kB3TI7A3Tp3P5XqmAqr71UAzB2FQB5ixPt rjnHfbI0eiyrB05TwU7OQnIh1CSAJ3rq414qdqOob7mhy7sQPn//LfJ7qgmXhPo4BNCE eBlOpK6HjH+boYdyjySOES1Of9+3HLW6T9Sy/B8ysqK7l+IasRe9IyWVaJc83hCqNrvH U/HPuJa4MCXNgfWX1st2+2OzLbuy+q8BAgiOe+nlQCSQug/07HekXQ5TeqdSRYpdioHb 43Mw== X-Gm-Message-State: ANoB5pm/imjFIbNrcikIN9tp6nIdZvEmuNXq/WLbFTibQixja5P381dr DfZmiBJe2IhMRFxD5Kqgu32ArD5BuXpFEeLX8cywKIBJJ0BezNwgZ0TREfik42Fmgq6nZlzXlZ+ 5vmS6h5go8ak= X-Received: by 2002:a05:622a:4d8e:b0:3a6:258d:4387 with SMTP id ff14-20020a05622a4d8e00b003a6258d4387mr42497434qtb.13.1671205873201; Fri, 16 Dec 2022 07:51:13 -0800 (PST) X-Google-Smtp-Source: AA0mqf5Jn2Uhau9BzqMwoo3qPaHalj0PlAXaiA0mx+xscLlNHjTHI5SgMhkHZ6FN0+QlzD5tFFX+mQ== X-Received: by 2002:a05:622a:4d8e:b0:3a6:258d:4387 with SMTP id ff14-20020a05622a4d8e00b003a6258d4387mr42497407qtb.13.1671205872924; Fri, 16 Dec 2022 07:51:12 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-45-70-31-26-132.dsl.bell.ca. [70.31.26.132]) by smtp.gmail.com with ESMTPSA id s81-20020a37a954000000b006eeb3165554sm1682297qke.19.2022.12.16.07.51.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Dec 2022 07:51:12 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Muchun Song , Miaohe Lin , Andrea Arcangeli , Nadav Amit , James Houghton , peterx@redhat.com, Mike Kravetz , David Hildenbrand , Rik van Riel , John Hubbard , Andrew Morton , Jann Horn Subject: [PATCH v4 3/9] mm/hugetlb: Document huge_pte_offset usage Date: Fri, 16 Dec 2022 10:50:54 -0500 Message-Id: <20221216155100.2043537-4-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221216155100.2043537-1-peterx@redhat.com> References: <20221216155100.2043537-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 716C020002 X-Stat-Signature: hbbp6rut3fn9jocmsphxc6dqsseea83s X-Rspam-User: X-Rspamd-Server: rspam08 X-HE-Tag: 1671205875-527151 X-HE-Meta: U2FsdGVkX187o7QD2jm3f4ANKIxy/G8TGXbECNrisKYiu86KnfRvNZrjnCjzESmTF7mOkYcjuBB+ZoJyRs9Kbd9sJ6Zj8BaxDt3i45Jv3JTTx5MV2H4AhFkK0pBjh4PZxyBH+sLT/OB3TZQP6f7WtL7VSm5kRbpsfdhDUZOQ8XDDXrjdQUX6biPRLIK9RU2sNEQvshEEcSM8IavmnU/cLPFFHkNlSXaiX8NrvAgUfbFBs0bA9NJKw25I4CvMxyKptPNQnPaJIPRFUWiqRgaUaDj/xaiMZY8ap3XosZQe6j/AC/aP+iFRSdTsP9sgxELoIUCUstm5SlcuntQTUAic6JK7A/ktdvBv8Mr12/nE/7Qh1CQ5Dd3krUggowAy+ECe7aguUPYG1ceu82Vux4IAxk11HLuCK9v48H/EDyyG1oU1LTAhMf3y2QGxkYTK5OD31J7Ppgg8p/q5IelEWxssv/aAi4ZDbNaLApyVz1DvTStdvYCVva+ycg0leWE/adkak57HMWZ+B+rXxHpLY+KdZdLtUKxkP9JzPGvzHS/N5Ubh+RUMFb1obP7orlaIWnRPZYO9imfOoahZaQSCcE31l4lLA1mCgf8f09/+joE8WjIhsWfy90hXbY6BR6s+gaWEnWekd3DBIMOg3l+epcPmdwldhjIuGrvDweXQuwfP1wMhOMkHgU08Wwk05wq5l/huBm+DxxzP6BW/4VFyNe4AWKDuaKq+iGsMKbNkp7DJCjqFEShKHdF3XLyh0HhqZE3duhUURKPe9aanZmvN52vRR56fa8dofzdhPRK/9h91L3RExM87A+7Fs+T/x0KAGCCpmfqanohsKYo2+bzb8mcmy9FLOry5uYrZMrLqsvagmHPWHSGzA51gjSnSLBtMYG4UqMQpt7GEQ232AWMvvc5XuwRoLBe+ysRS0BgvykXx9YbrQrmN49TvajZJrIa5Xdr7lIq5AIbvB5ITQrr/qKR cgBU8l0E wSe58//LnhBiJ8qpDgiUynYPl7EE0PW3i7Gxkmhtja+hdwHWBf1OCyL/1oGSJAw4wVEy52xdcPNnMykknJf9l9v2nF0802CeVw80FbuGysXW/t3E2+izodYobjItyPlaIHL8xNT3aKaEeCkI0msGo9scWAs7cpALENGQmWU8ct/9/uWW3hJaILKCg9/96WAg8ARylHwh3Ysms3oertHsYi0nRA8h+ya6iBYHX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_pte_offset() is potentially a pgtable walker, looking up pte_t* for a hugetlb address. Normally, it's always safe to walk a generic pgtable as long as we're with the mmap lock held for either read or write, because that guarantees the pgtable pages will always be valid during the process. But it's not true for hugetlbfs, especially shared: hugetlbfs can have its pgtable freed by pmd unsharing, it means that even with mmap lock held for current mm, the PMD pgtable page can still go away from under us if pmd unsharing is possible during the walk. So we have two ways to make it safe even for a shared mapping: (1) If we're with the hugetlb vma lock held for either read/write, it's okay because pmd unshare cannot happen at all. (2) If we're with the i_mmap_rwsem lock held for either read/write, it's okay because even if pmd unshare can happen, the pgtable page cannot be freed from under us. Document it. Reviewed-by: John Hubbard Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu --- include/linux/hugetlb.h | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 551834cd5299..d755e2a7c0db 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -192,6 +192,38 @@ extern struct list_head huge_boot_pages; pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz); +/* + * huge_pte_offset(): Walk the hugetlb pgtable until the last level PTE. + * Returns the pte_t* if found, or NULL if the address is not mapped. + * + * Since this function will walk all the pgtable pages (including not only + * high-level pgtable page, but also PUD entry that can be unshared + * concurrently for VM_SHARED), the caller of this function should be + * responsible of its thread safety. One can follow this rule: + * + * (1) For private mappings: pmd unsharing is not possible, so holding the + * mmap_lock for either read or write is sufficient. Most callers + * already hold the mmap_lock, so normally, no special action is + * required. + * + * (2) For shared mappings: pmd unsharing is possible (so the PUD-ranged + * pgtable page can go away from under us! It can be done by a pmd + * unshare with a follow up munmap() on the other process), then we + * need either: + * + * (2.1) hugetlb vma lock read or write held, to make sure pmd unshare + * won't happen upon the range (it also makes sure the pte_t we + * read is the right and stable one), or, + * + * (2.2) hugetlb mapping i_mmap_rwsem lock held read or write, to make + * sure even if unshare happened the racy unmap() will wait until + * i_mmap_rwsem is released. + * + * Option (2.1) is the safest, which guarantees pte stability from pmd + * sharing pov, until the vma lock released. Option (2.2) doesn't protect + * a concurrent pmd unshare, but it makes sure the pgtable page is safe to + * access. + */ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz); unsigned long hugetlb_mask_last_page(struct hstate *h); From patchwork Fri Dec 16 15:50:55 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13075136 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 93477C4167B for ; Fri, 16 Dec 2022 15:51:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2A43B8E0008; Fri, 16 Dec 2022 10:51:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 22DA3900002; Fri, 16 Dec 2022 10:51:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0803A8E000C; Fri, 16 Dec 2022 10:51:20 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id E9C838E0008 for ; Fri, 16 Dec 2022 10:51:19 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id C0B2D1207B8 for ; Fri, 16 Dec 2022 15:51:19 +0000 (UTC) X-FDA: 80248608678.18.CC5E5FE Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id DA51840018 for ; Fri, 16 Dec 2022 15:51:17 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iVTBbnFc; spf=pass (imf12.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671205877; a=rsa-sha256; cv=none; b=MT5fyFUkKomhhbJeGKmwXiKhqiWMZZ6Oioql5wbigMcYYVWcfB2EepwJ7DZGjdZ8Tz7k2O xF3Igv6Vynt+ry6+IzBWoJvyzyVMl73tooqfIdkeCQVc3XpipJEXEB+zSJEcCa8cugtFZx GrVz+V7+j0VvF71tfGy+QddZkYf7tOk= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iVTBbnFc; spf=pass (imf12.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671205877; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=vsQ4xrOeNHA8Em2t3SYPnGMugJW9C8Z5spFzSFBIRTk=; b=zQPkUqHISlDE4ZfJUs2YTwfeBMvRKGj7ls1AXRcDR60kFF1nHEqmbXFVEP6M52moBj7tjT rbTizxmimH4FOUbFpNpAqVjrzqGe7EJIV7Vk6QdFtoKjCjC4wMCI5MHyA0YofqYtWwNX19 tE2OkxSodFHwPp7G3atlPvB4qBxOIAk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671205877; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=vsQ4xrOeNHA8Em2t3SYPnGMugJW9C8Z5spFzSFBIRTk=; b=iVTBbnFcz5yro4LmhsLBZ1+sZW4/M9qjY6OGwtlVz/SbyOJhHzyAtdXnDm4HN/6thcQBlX NoNTNtBUpyvZQ4j7wKoE3V64Nx6BAN0H+KJdJjuzb4fqSPpoZtsRhsE8tAz8VintREGRvw umxt1F+w3Xxzz4/X6SjC1l36JrfsLVY= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-277-vTqT24PBMgircqiXkkaL8A-1; Fri, 16 Dec 2022 10:51:16 -0500 X-MC-Unique: vTqT24PBMgircqiXkkaL8A-1 Received: by mail-qv1-f69.google.com with SMTP id r10-20020ad4522a000000b004d28fcbfe17so1672583qvq.4 for ; Fri, 16 Dec 2022 07:51:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vsQ4xrOeNHA8Em2t3SYPnGMugJW9C8Z5spFzSFBIRTk=; b=xs8HIkuoi0OYIXk0oQpxeKKK2kJQjN3FNJDy8tuEEqow4Ikri6HbXIlBs4V1+DPL0Q 5pAsGiJ23dF/tUTb8dWseoup63RqyAk/xZmQd1gsDhgABPCiejxQKkArZsj1SC/Ibh9d X2JvDhgYdDyu94cfgK1T1RBA8VB8sYasi/Oy82AhpYyc280q2obIcQVTZHttHfY1brLO 8hcYpTaBIt0sTT0vaaQRO/2dsQzc3/v7CPet/JtRaqfSCv+X1o6C/e4U5bT9Qm1PYEiY 0KNBpplgEVaQ+CSBhgQ15ghQ4pDjZ9Q8UwXx1ApK17MbFs0fnerT5GUBcmOKVdCkpTnY +kPg== X-Gm-Message-State: AFqh2kp6AGz8w7umB4r6cRyIdf31PXM6VzoM/2+627pWRHppS7PWt7XP XQC7WIU6ejn7d7HaHpHxo1+EhODr1UchNz3fVcSRmMyNBVOJAQc4ecsc6QV2iUqimj2pRHX3wfE jumUaV4y6oDI= X-Received: by 2002:a05:6214:3787:b0:4f3:7d92:13a9 with SMTP id ni7-20020a056214378700b004f37d9213a9mr8375879qvb.15.1671205875660; Fri, 16 Dec 2022 07:51:15 -0800 (PST) X-Google-Smtp-Source: AMrXdXuI37Cnbiq7WTJZi91OO/hWpXTz/+s2VaXzLPKoSNSCTyQ3nllT7XSFGoanS+lfNocZLgfsog== X-Received: by 2002:a05:6214:3787:b0:4f3:7d92:13a9 with SMTP id ni7-20020a056214378700b004f37d9213a9mr8375847qvb.15.1671205875427; Fri, 16 Dec 2022 07:51:15 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-45-70-31-26-132.dsl.bell.ca. [70.31.26.132]) by smtp.gmail.com with ESMTPSA id s81-20020a37a954000000b006eeb3165554sm1682297qke.19.2022.12.16.07.51.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Dec 2022 07:51:14 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: Muchun Song , Miaohe Lin , Andrea Arcangeli , Nadav Amit , James Houghton , peterx@redhat.com, Mike Kravetz , David Hildenbrand , Rik van Riel , John Hubbard , Andrew Morton , Jann Horn Subject: [PATCH v4 4/9] mm/hugetlb: Move swap entry handling into vma lock when faulted Date: Fri, 16 Dec 2022 10:50:55 -0500 Message-Id: <20221216155100.2043537-5-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221216155100.2043537-1-peterx@redhat.com> References: <20221216155100.2043537-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: DA51840018 X-Stat-Signature: k85frze77w7t44m55b865r1w53ck3yiz X-HE-Tag: 1671205877-59162 X-HE-Meta: U2FsdGVkX1/geprLu0+ET0xqiRl4oav8VxvJZ+p39WUZGzKs6ff6j7UNEHv7J67PIkQydkp7u3CQVeMTcJatllyoTwl00aFNWgE2iSnkibIJLWuEjSPj5KB+TKEmTmszJZZ5s7iSN7cDis5BbD8x5r1J7t7R0okdgIlLhycm58I2WcDkD3KVQqsLcHsrwLth/JHhg9o8nijGQYFcmVKx6DHo18P+i2E6C7RZs6cfLd5xCtA+ofxPPvZiPd4+5f26yweYhCAwKAyY17dsRquU0QmRE/7wqOpFFpOuTZI581PJvXOXLQmIaiy9m3u42pCb0jfD+anU8slB6u/DTjS2vmq+ygRnYm9/TPrQQgiof/OPMhxDcV9ohGNAHYh4seTvmV5kkBjVefbItXCNwNvf28+z+wyYKGmHnz+H0uiGQjNvM0fD2d9Zvl3NEhFTFc2N+MTpgEnuR4ndhmSG2p7IzpnVvUM50PxX0VXoLFLZaKKRuhVN7zLfQyUVvgW2ap+41kbYA4bb1Rvx1poDsd34nv3JDQm01gtuWYLNGdUz37YVE61f7d38r29OmatA/1m6R46zDZ1x1ibQtufy2rYacJLGr/PAFiJZw5KegpXRCwZxjiRiGSucukpJ3LvgF7x2r69dj7wyIohIpBJALGIGCAPC2xTz9dz1wm/VVYkmX5lFKh0dGv/brsD+GS7ZYuOsOoGNzOTF6ShRbi7B4Kb9TRKfRKKk3vKpUZJ8ZjU1NUrujfFJYNZP57a2OHsXyAFgfT4HTLWWZRRd5HlipKGjOX4bfuu7jUCMl9OQbjfxDvhlnHZtx3WX3nHCDN9Qrbszb3TIAVMv0Gc7fb4SbPqisjYKSrG3jRQjom31buma0GjyBTET1Lj8QKjMsvNBzqCpojPzygfU8mQCttitIDeBzEv74oatX9WZhUEj9/Tu+ixk4j3V61bz8yGXsUh2xZ2jc6CajG2RODRDLse7h8P 8hBsOHTt 2qsPEGhr+PxRmBSsle6Va4f5p2tg4fRZS7OtT+OsLIZb6ZCmoMv33/dQiQGOf3YQTN+hZySo1ATHeVwKggBkFyXKj/Nfm57nTYYjfCtDQXAjjMbp+lIbfdx+cewZ+KqO39FeKqR0WmhDQIPtrC0YvZrM13Rg+7/4XjWkG32lfyAcmQ2j8BxAFung1fw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In hugetlb_fault(), there used to have a special path to handle swap entry at the entrance using huge_pte_offset(). That's unsafe because huge_pte_offset() for a pmd sharable range can access freed pgtables if without any lock to protect the pgtable from being freed after pmd unshare. Here the simplest solution to make it safe is to move the swap handling to be after the vma lock being held. We may need to take the fault mutex on either migration or hwpoison entries now (also the vma lock, but that's really needed), however neither of them is hot path. Note that the vma lock cannot be released in hugetlb_fault() when the migration entry is detected, because in migration_entry_wait_huge() the pgtable page will be used again (by taking the pgtable lock), so that also need to be protected by the vma lock. Modify migration_entry_wait_huge() so that it must be called with vma read lock held, and properly release the lock in __migration_entry_wait_huge(). Reviewed-by: Mike Kravetz Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- include/linux/swapops.h | 6 ++++-- mm/hugetlb.c | 37 ++++++++++++++++--------------------- mm/migrate.c | 25 +++++++++++++++++++++---- 3 files changed, 41 insertions(+), 27 deletions(-) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index b982dd614572..3a451b7afcb3 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -337,7 +337,8 @@ extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address); #ifdef CONFIG_HUGETLB_PAGE -extern void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl); +extern void __migration_entry_wait_huge(struct vm_area_struct *vma, + pte_t *ptep, spinlock_t *ptl); extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte); #endif /* CONFIG_HUGETLB_PAGE */ #else /* CONFIG_MIGRATION */ @@ -366,7 +367,8 @@ static inline void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address) { } #ifdef CONFIG_HUGETLB_PAGE -static inline void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl) { } +static inline void __migration_entry_wait_huge(struct vm_area_struct *vma, + pte_t *ptep, spinlock_t *ptl) { } static inline void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { } #endif /* CONFIG_HUGETLB_PAGE */ static inline int is_writable_migration_entry(swp_entry_t entry) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 8ccd55f9fbd3..64512a151567 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5972,22 +5972,6 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, int need_wait_lock = 0; unsigned long haddr = address & huge_page_mask(h); - ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); - if (ptep) { - /* - * Since we hold no locks, ptep could be stale. That is - * OK as we are only making decisions based on content and - * not actually modifying content here. - */ - entry = huge_ptep_get(ptep); - if (unlikely(is_hugetlb_entry_migration(entry))) { - migration_entry_wait_huge(vma, ptep); - return 0; - } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) - return VM_FAULT_HWPOISON_LARGE | - VM_FAULT_SET_HINDEX(hstate_index(h)); - } - /* * Serialize hugepage allocation and instantiation, so that we don't * get spurious allocation failures if two CPUs race to instantiate @@ -6002,10 +5986,6 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * Acquire vma lock before calling huge_pte_alloc and hold * until finished with ptep. This prevents huge_pmd_unshare from * being called elsewhere and making the ptep no longer valid. - * - * ptep could have already be assigned via huge_pte_offset. That - * is OK, as huge_pte_alloc will return the same value unless - * something has changed. */ hugetlb_vma_lock_read(vma); ptep = huge_pte_alloc(mm, vma, haddr, huge_page_size(h)); @@ -6034,8 +6014,23 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * fault, and is_hugetlb_entry_(migration|hwpoisoned) check will * properly handle it. */ - if (!pte_present(entry)) + if (!pte_present(entry)) { + if (unlikely(is_hugetlb_entry_migration(entry))) { + /* + * Release the hugetlb fault lock now, but retain + * the vma lock, because it is needed to guard the + * huge_pte_lockptr() later in + * migration_entry_wait_huge(). The vma lock will + * be released there. + */ + mutex_unlock(&hugetlb_fault_mutex_table[hash]); + migration_entry_wait_huge(vma, ptep); + return 0; + } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) + ret = VM_FAULT_HWPOISON_LARGE | + VM_FAULT_SET_HINDEX(hstate_index(h)); goto out_mutex; + } /* * If we are going to COW/unshare the mapping later, we examine the diff --git a/mm/migrate.c b/mm/migrate.c index a4d3fc65085f..98de7ce2b576 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -329,24 +329,41 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, } #ifdef CONFIG_HUGETLB_PAGE -void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl) +/* + * The vma read lock must be held upon entry. Holding that lock prevents either + * the pte or the ptl from being freed. + * + * This function will release the vma lock before returning. + */ +void __migration_entry_wait_huge(struct vm_area_struct *vma, + pte_t *ptep, spinlock_t *ptl) { pte_t pte; + hugetlb_vma_assert_locked(vma); spin_lock(ptl); pte = huge_ptep_get(ptep); - if (unlikely(!is_hugetlb_entry_migration(pte))) + if (unlikely(!is_hugetlb_entry_migration(pte))) { spin_unlock(ptl); - else + hugetlb_vma_unlock_read(vma); + } else { + /* + * If migration entry existed, safe to release vma lock + * here because the pgtable page won't be freed without the + * pgtable lock released. See comment right above pgtable + * lock release in migration_entry_wait_on_locked(). + */ + hugetlb_vma_unlock_read(vma); migration_entry_wait_on_locked(pte_to_swp_entry(pte), NULL, ptl); + } } void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), vma->vm_mm, pte); - __migration_entry_wait_huge(pte, ptl); + __migration_entry_wait_huge(vma, pte, ptl); } #endif From patchwork Fri Dec 16 15:52:17 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13075138 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3266C001B2 for ; Fri, 16 Dec 2022 15:52:29 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B2AB8E0006; Fri, 16 Dec 2022 10:52:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 450428E0002; Fri, 16 Dec 2022 10:52:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 22DC18E0006; Fri, 16 Dec 2022 10:52:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 06D158E0002 for ; Fri, 16 Dec 2022 10:52:29 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 961B740669 for ; Fri, 16 Dec 2022 15:52:28 +0000 (UTC) X-FDA: 80248611576.20.92AF16B Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf26.hostedemail.com (Postfix) with ESMTP id A025914000E for ; Fri, 16 Dec 2022 15:52:26 +0000 (UTC) Authentication-Results: imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Fq9Y01sw; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671205946; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oQxWaXmf/BdMLd8AroQPz0fxFz+2cpe/6LbLYuumgUQ=; b=X80zj/nuYu6XJoPRxoPRXvDLDjsPcXBnnXdL+Tr8bB2oZWMGHdp2kimItlyhrNask+UEdO 8jYeVJAdOoQfUsUiezyxqqkI7uro/3MAE5JzY348RmZYWFx+7WCXGuGAuSexpT1Z2pxUKa uaENcHfvEn8YnmROE/wzS2MQ0Dv61dI= ARC-Authentication-Results: i=1; imf26.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Fq9Y01sw; spf=pass (imf26.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671205946; a=rsa-sha256; cv=none; b=5mugqQptBE0W8XM7AeOuZIVPQNRDvSncJjJdh2CmPbnpyhMZ1tIdlO2HHeVHWvpH2xeXBe A8llRpwF9bhovOJjVnjNDsSswzSn+F8b8Qz4ZbUgNUyRJbvKT2t3kGrwLtGAXjqL6OPxFE I6i/JzZ+XvAneLBfG8MF8HLNPNHHeto= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671205946; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oQxWaXmf/BdMLd8AroQPz0fxFz+2cpe/6LbLYuumgUQ=; b=Fq9Y01swPG7tPVzWweQ9JfC9o7xRGfxd3pXVTlPhd9+ulDNKdPNsq3eUWFVnd3I3wI51Bo giFMNgR3iYUGmxTQCzkikm3uOx334czRiJRW39weuwuDvXekF0ggPf6O715fVDk8dSG9lN TB2SgDZV/SINBRvc9ywd0mariP54MbM= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-78-nyX4EDnUNouD5ljbR3wW4w-1; Fri, 16 Dec 2022 10:52:20 -0500 X-MC-Unique: nyX4EDnUNouD5ljbR3wW4w-1 Received: by mail-qk1-f200.google.com with SMTP id q20-20020a05620a0d9400b006fcaa1eac9bso2055771qkl.23 for ; Fri, 16 Dec 2022 07:52:19 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oQxWaXmf/BdMLd8AroQPz0fxFz+2cpe/6LbLYuumgUQ=; b=lhk8MrY44HWAXgM1TTZ1S7DOxyt//hXzRtQJSSnA2kACF9gGMnYe48CDX/CZ6leZCr kxjv4WR/7Odnx4VfZXwJl8ewsEqs1UoGfqqElhv0BWBiwqx6X3v80syFtwx/rDUe3JJC JgIQzJesOmaPy1I422V1zeJBs60rOYk5szb6pqvFPf/EJrmxZn/BTq+1kMmqSlk0kjWF HqkK93/P2dPJS56nDTlaBhC+6fXxhHADPbo5FguJezz0biCoJwiVQQXnJ4vJtEgd+AMq CGrdlE4pVFcX3lXO2tNmeate+m+ZFZKDk+xU3acTWH0Zf//IVNfv/V/zkqFcYSLhKFe2 vliA== X-Gm-Message-State: ANoB5plwXzNjb9pvc+gwYrZH02uPtrhbFDWL6jlUuYlcK8xtDQJoohvV jwtVzE7s4+lEyd+AD+Motdccm1SIYy8QuZV5t9abJJo1XFUBi5JaUA4q/b6ES7RzCCdP4DPBHDV Ao3i3KByjpbU= X-Received: by 2002:a05:622a:4c11:b0:3a5:2f80:5b08 with SMTP id ey17-20020a05622a4c1100b003a52f805b08mr46852438qtb.1.1671205939596; Fri, 16 Dec 2022 07:52:19 -0800 (PST) X-Google-Smtp-Source: AA0mqf7dhghRe2b9Hr/nL0Vtk5vE2Zf1v91HSpvXyjbmaYYF/40Yyuz87fscH2VZ1GmjeQmVyYE2LA== X-Received: by 2002:a05:622a:4c11:b0:3a5:2f80:5b08 with SMTP id ey17-20020a05622a4c1100b003a52f805b08mr46852414qtb.1.1671205939371; Fri, 16 Dec 2022 07:52:19 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-45-70-31-26-132.dsl.bell.ca. [70.31.26.132]) by smtp.gmail.com with ESMTPSA id x13-20020ac8120d000000b003a6847d6386sm1483787qti.68.2022.12.16.07.52.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Dec 2022 07:52:18 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: David Hildenbrand , John Hubbard , Muchun Song , Mike Kravetz , Nadav Amit , Andrea Arcangeli , Rik van Riel , peterx@redhat.com, Miaohe Lin , Jann Horn , James Houghton , Andrew Morton Subject: [PATCH v4 5/9] mm/hugetlb: Make userfaultfd_huge_must_wait() safe to pmd unshare Date: Fri, 16 Dec 2022 10:52:17 -0500 Message-Id: <20221216155217.2043700-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221216155100.2043537-1-peterx@redhat.com> References: <20221216155100.2043537-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A025914000E X-Stat-Signature: zc9dyd5g3znfzkt7iww3sh3dwdorfwg3 X-Rspam-User: X-HE-Tag: 1671205946-229045 X-HE-Meta: U2FsdGVkX193RbXp5kyqgdNjfr1mTMeXs4LKqALNx5LjmtdGssK/1Zf2Teo4KmfEQ5jS8YeOHiaaC02zQXBF/PkQc0Lf6/YW5p2UEW0wRncxffqxzn4LyntxXTv9tAQtVyzvn/rOwFbEMmPaytbObw+rMAULMCcHS+XEnwF1dRvjqbs8+EynuBVM0Q9FTrFjO5MoFy7QvV4bLgZ3eRwOEDhJumGLzefJZXjPPlCpnNGirIGVw9vdmIkciiAJuYMMMKBvvYd1BwzmV4ApYdrGRu7LT/R+Autdbp9451rFlrdD0x4VO+2QBZISwSB06yYiGnd+NfcLM3xPyOcA/4MqrlRZcGOp7ZrTk/VmTFkesDhlh+YG6m4fctzpOrzPgZnqQyp/7AO4Dda2se1KS9+prrUxzdF/Re4wkJvp18vC1TkAeiJETrZGTwNfYrmJJsrgwXEoeyb0z3vrZ6sSu1QOA1wLJZCNemyyD4ju23H0Grlv/+S9zqhYFzpvew/UhdB34fWCZziGNI+zsx8+OAW701x7LaQBs9dtKetNG//IAyhvDyqshdUi9XdYSld0F6cBgObCX84pfRRbMUm97mdkSGkR7InZHI5rlr0+orLHONAbQAIwV/IC4zYzqC9trUz//2Uw0PXiGKFYta2jzL9Ar0T2AzY1iXLMKienlVYdEJoyBHSvEug6eFe8nNc529FWL5sUdgVqcRidqmkfATPRo2P8lRTdOzYDJ4hWnnucc+pXDItUQzs9Wu4mhVQM8nnU8txiQxPPZbDDdfs75ADbndms6qDGV/WiHp4Mj8zpfh5RALzfOU71glPSLsc/7ERwBTHNvZoTSJcBxxD5fKui/5Jo1srjLWK5dtBm5WzFeQtz4/6HWgJ3onycaP3PISXFFe254p+AtcAUgEgOZOWUVRz/oxAdoihoH8XrHXh8L6cHaVw5Nzo16eSw7/8mysVYkb5L+8IdpEPtJEZRxqO qJ4AY4MW F6MDPR5xfX0jFn+VeRX/8Il+ObiPD154QJImpBqAG1obz0NibfxXZWS9U22+i+nrBiqcYzaX+LJfBU9tuzOiBe1TDeNQRVdmPW6GzGUAVB+CnIt46GHQ2AkJd6NZyZz85PzToaHpBgcGSmHLb5XX1xEenqY4fLmxCIMdBuSa29sB0VfosXQrhhCNYOOWMVyHBZii3Qh4uhf5ozV1P7zFrwWN9WXaSiqrR5lup X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can take the hugetlb walker lock, here taking vma lock directly. Reviewed-by: David Hildenbrand Reviewed-by: Mike Kravetz Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- fs/userfaultfd.c | 20 ++++++++++++++++---- 1 file changed, 16 insertions(+), 4 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 98ac37e34e3d..887e20472051 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -376,7 +376,8 @@ static inline unsigned int userfaultfd_get_blocking_state(unsigned int flags) */ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) { - struct mm_struct *mm = vmf->vma->vm_mm; + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; struct userfaultfd_ctx *ctx; struct userfaultfd_wait_queue uwq; vm_fault_t ret = VM_FAULT_SIGBUS; @@ -403,7 +404,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) */ mmap_assert_locked(mm); - ctx = vmf->vma->vm_userfaultfd_ctx.ctx; + ctx = vma->vm_userfaultfd_ctx.ctx; if (!ctx) goto out; @@ -493,6 +494,15 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) blocking_state = userfaultfd_get_blocking_state(vmf->flags); + /* + * Take the vma lock now, in order to safely call + * userfaultfd_huge_must_wait() later. Since acquiring the + * (sleepable) vma lock can modify the current task state, that + * must be before explicitly calling set_current_state(). + */ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_lock_read(vma); + spin_lock_irq(&ctx->fault_pending_wqh.lock); /* * After the __add_wait_queue the uwq is visible to userland @@ -507,13 +517,15 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) set_current_state(blocking_state); spin_unlock_irq(&ctx->fault_pending_wqh.lock); - if (!is_vm_hugetlb_page(vmf->vma)) + if (!is_vm_hugetlb_page(vma)) must_wait = userfaultfd_must_wait(ctx, vmf->address, vmf->flags, reason); else - must_wait = userfaultfd_huge_must_wait(ctx, vmf->vma, + must_wait = userfaultfd_huge_must_wait(ctx, vma, vmf->address, vmf->flags, reason); + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_unlock_read(vma); mmap_read_unlock(mm); if (likely(must_wait && !READ_ONCE(ctx->released))) { From patchwork Fri Dec 16 15:52:19 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13075137 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1A7CC4332F for ; Fri, 16 Dec 2022 15:52:28 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6256A8E0005; Fri, 16 Dec 2022 10:52:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D5938E0002; Fri, 16 Dec 2022 10:52:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 476CC8E0005; Fri, 16 Dec 2022 10:52:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 35F188E0002 for ; Fri, 16 Dec 2022 10:52:28 -0500 (EST) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 117098111F for ; Fri, 16 Dec 2022 15:52:28 +0000 (UTC) X-FDA: 80248611576.27.9CA0C78 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 2AF71A001E for ; Fri, 16 Dec 2022 15:52:26 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="HDrueCX/"; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671205946; a=rsa-sha256; cv=none; b=A8D+1O4ArcmVhAfqbuuU6XVc9yI0Dr4X76/jlTt43YPogc/QUm1FjCRX8cq3F6rOtxBNjo q2iAL+zaS6Xw8s057llLd1NZomdWLg2pvW56vfPWasfqGzJyaMxkzo/pVmqRZ+ZrwDT7Ww t44AsBXIO/fP0G28JzqnsZtH9bzACA8= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="HDrueCX/"; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671205946; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7a/XOpLMoccJVDSJCL0inQbFYq9WEos4O0UHGEG2GQ8=; b=wa+Io4pdJ38ZCg5UCJrz2op/NLED3ugoNMRwrJ0YoAuclEctsCNyTZee5I62reToozjoIq jQeRmWjHmlI/bxUR2e0Id1TcovRipodIbG8e7VVWh+EqtwedoE/CE6UJRowBUyyJyFAj8o X/4SmQD+XoL3RvSn5OH6p7FQ8AZOhE4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671205945; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=7a/XOpLMoccJVDSJCL0inQbFYq9WEos4O0UHGEG2GQ8=; b=HDrueCX/cVDYX/Kg6bCPPhhJl1NWa4x2q498z1bjq6V/ab4HoWmNkHhH5sRz4ctYpOEbAH iWt1hCeJi9iGYd4W1hXrTsAr9TjYT5Lxp9od8m7+LrrqIRySRWW0mRV6NOciiw1t2w62bn OrXqT4DbmcFEJmkWd2pNf6Gkd5kgByM= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-104-htPPHwGgOGaFWjED0wNC_g-1; Fri, 16 Dec 2022 10:52:24 -0500 X-MC-Unique: htPPHwGgOGaFWjED0wNC_g-1 Received: by mail-qt1-f200.google.com with SMTP id fg11-20020a05622a580b00b003a7eaa5cb47so1772332qtb.15 for ; Fri, 16 Dec 2022 07:52:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7a/XOpLMoccJVDSJCL0inQbFYq9WEos4O0UHGEG2GQ8=; b=gNDysy7qpnr4dp2Sf+ofYXALPiPfPkCdk2D6QVZeqzW7w8J3Q6Nq3mbDcJglmbeU47 rp1XGu9XJS/FMao53tSnpdbwX7cc08Us4ode4ydrcflLGZ7p9l2xsNTkpERXUHLDUeyA Ts/xiEEA04IrvKtJa1VW4DdQJkhAz9cJQwtnlWyo4yy+uAxlo80n2Y5vuNujVDJ9WxLi CaPoT4daGfan5vYc9mBHJncOGkOkWdtDyaKBC+ZJ6zHqHHLAyoWr1+OjNNIQ8DU0+uCw FQZNy2oG+q1syNF4wCN/NlxNIGu4+7xuQE8Oo+5ig18/o1A3pNJ20y4K5ugwPc+wYZWF /nkQ== X-Gm-Message-State: ANoB5pk5hVgiNi+dxtv2ZikG5XhsIjwVc6nzqVjNWhcEQSh7VFQ62vtL 0ctpy6LgDZ0oLh2v7TQ3kFIz6D0gr8Ix4IiWqE1utpK7BT3pDzMhAPzVFcd/MTA6df9zgLZzsSl u6uzQCwtVY00= X-Received: by 2002:ac8:5755:0:b0:3a6:9162:245e with SMTP id 21-20020ac85755000000b003a69162245emr61455512qtx.36.1671205943494; Fri, 16 Dec 2022 07:52:23 -0800 (PST) X-Google-Smtp-Source: AA0mqf4eRO6j6nITJR7U5O1saoqGPaZuUJ+DbwInDy4iHwhtjCTw5HCWKMaEHXGZqHgIE74QZNrEYA== X-Received: by 2002:ac8:5755:0:b0:3a6:9162:245e with SMTP id 21-20020ac85755000000b003a69162245emr61455484qtx.36.1671205943290; Fri, 16 Dec 2022 07:52:23 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-45-70-31-26-132.dsl.bell.ca. [70.31.26.132]) by smtp.gmail.com with ESMTPSA id d136-20020ae9ef8e000000b006fef157c8aesm1673001qkg.36.2022.12.16.07.52.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Dec 2022 07:52:22 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: David Hildenbrand , John Hubbard , Muchun Song , Mike Kravetz , Nadav Amit , Andrea Arcangeli , Rik van Riel , peterx@redhat.com, Miaohe Lin , Jann Horn , James Houghton , Andrew Morton Subject: [PATCH v4 6/9] mm/hugetlb: Make hugetlb_follow_page_mask() safe to pmd unshare Date: Fri, 16 Dec 2022 10:52:19 -0500 Message-Id: <20221216155219.2043714-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221216155100.2043537-1-peterx@redhat.com> References: <20221216155100.2043537-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 2AF71A001E X-Stat-Signature: wjc3icrk6fi73wf18jwybg64ggjxfawp X-HE-Tag: 1671205946-649475 X-HE-Meta: U2FsdGVkX18OzGf0XoIroG3JnS47xsJu3MIPg5CTspYqkMztkw8FraQpB8pAK2QqGirvoE9G4ky70/NC3rimgL/8DMn5xTnmJ1RfrR/PQNDH2W+/WtqUX+n3/N+KCxXMBzXW9/zKDcxN87pZ6TPZYjP2YvlGz7ZM+7cCuM2dWiNj5sVq6rMUfh24Fy3lz6hCegEyHpSnXU24ro3r4XXr7Xme4oEip3vfS/RDjIMY+iyl2+nu5z2UFyvQ3lwx+bnOwzGL9bQjR6X2giYgtqEUCgLLEdxxgEUzFbEzBTf2lqDyheRoboSPgzR1NduekjHcJoBxwraY2uZJRC89xb1c60e0p/+MuSMOqigsHfZp7N3ksmjNGnCIYWRySW7XztDO/PWdzetEt0kqJEQLUFA87RlWzHckBQf3iaSOiqhaPuJRjACT6qVuSF8GVzCiJ7t26A867LX4LcOJmdTW8XBv4sIEaXADmo/kA0bdq76NXIggtR2pRK5wZnaUvGHgS0xPVaiO5r4UCb2djbs/dauThriXF25UiLxxWYYaEdhpruln6H8bBpvzrtTUoCgVMffV7K4ztNwgEwQDOz04z01QVKLplsUdXhhqmYZ2yHbdm3688x2NnIkfezkRj2Kd+MRKDOJV9Hp2ssn5wmMwH8hW5xaVXcE8Kw4xl3j/IO4KAJdwbGJkNKpDNnWsfrLjE7/c3iBmC/v7TsCdOB5MwrPRa5Xmq8wtE1Umyxz3nSEFQ7DvHX0j8HPnxBXZ/Ls7ySSpG387faikw1bzalZ3CcdlImOcrJdAg1dt1LeQBENGdweyo2WtMfEly3UIOpVJ/xQg7YdG2uo/Qk60/ke1jliRAtq+6CwkKxy/KbPL+JuaRxvaVsZhTxdVlu86a/8tkhdHfJcHpPeqTyoHuYCIDMAmGj1Dj8PcP+h0LMev3bMLHsI3nSjTzMxnUs5FCiP2OoX+4mICf3moigFsAr7p2PY 1oL+M4kH yETYYSTtM/UMWBbeK3x9xXj8rIYVH3Vgu5meeEaAgmvQlrSPcAmIXtCqstn/LK1BE1Q8uO7T05oRXmMiJ3UsO/yMNZxUi7wBNW6skfg2e7BLeUXfxFVqtHA8tLMPzuHLdzComD5ZavW8bBgjp5o6KLB7xnZANEHCUZK2FMpmlfenhI9mBn3xnnUPnbKBib6P7t+mRBFDFeC03KSm89jZoxWWyCeDTNz/sc7yR X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since hugetlb_follow_page_mask() walks the pgtable, it needs the vma lock to make sure the pgtable page will not be freed concurrently. Acked-by: David Hildenbrand Reviewed-by: Mike Kravetz Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- mm/hugetlb.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 64512a151567..0bf0abea388d 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6375,9 +6375,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, if (WARN_ON_ONCE(flags & FOLL_PIN)) return NULL; + hugetlb_vma_lock_read(vma); pte = huge_pte_offset(mm, haddr, huge_page_size(h)); if (!pte) - return NULL; + goto out_unlock; ptl = huge_pte_lock(h, mm, pte); entry = huge_ptep_get(pte); @@ -6400,6 +6401,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, } out: spin_unlock(ptl); +out_unlock: + hugetlb_vma_unlock_read(vma); return page; } From patchwork Fri Dec 16 15:52:23 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13075139 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B3321C4332F for ; Fri, 16 Dec 2022 15:52:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 515B08E0007; Fri, 16 Dec 2022 10:52:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4C59C8E0002; Fri, 16 Dec 2022 10:52:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 27DC38E0007; Fri, 16 Dec 2022 10:52:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 06B4A8E0002 for ; Fri, 16 Dec 2022 10:52:31 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BFE6880234 for ; Fri, 16 Dec 2022 15:52:30 +0000 (UTC) X-FDA: 80248611660.24.5A3421A Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf08.hostedemail.com (Postfix) with ESMTP id D2D9716001B for ; Fri, 16 Dec 2022 15:52:28 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IHHOUX2z; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671205948; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rIRGvziR/92Z54UoPp5pwcNYu+IBI1cm3v25qmGTr7U=; b=VHV1mG5KynerjSppZ9tZwr/fGyr12AXJgmnEMN8sdKFKW0Zx5uvIYQ3OsSnljSX6fk1XWx mVTZkpkAQlCv7hz2XWXL3pobCZBBNjU/JrMA4Khlzd4PX9ebEcnsJoQuqunoQOZ0TRvSNN B3oSi6O8AVbYKPN4vNjb0g9uQwPRaUw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=IHHOUX2z; spf=pass (imf08.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671205948; a=rsa-sha256; cv=none; b=yQIX7cs7CAXFUr3FBOa/aB4imgbbsH62BoW+OXmBHbaUk/bSCAVFyBlLPUymsT+viSC7dO oHlZP9vdcJCK5RW76q3/DXAsdQcPxngv0JdEX/bs5nW7rMy86H18Y00pKYECV812Nv8ZAK gkEydoXjUeDFUX+9GO/yFGpqx/06AmA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671205948; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rIRGvziR/92Z54UoPp5pwcNYu+IBI1cm3v25qmGTr7U=; b=IHHOUX2zHuY7iyny7BCjYIhp5LfU+qUWgZi1FdSW1h1BZRgEfEAinmOa4XLVb+B7Rl1vEZ FL5q+N5VG/F5JnmJkIF4WpYL5MVHOKy71HHGa7n+J7JMVEhHvHTZ0R0dLJPgVK8R5lCIO7 +bE8Asm+Yoj2S05s6GNItt7FOp2r5B8= Received: from mail-qk1-f200.google.com (mail-qk1-f200.google.com [209.85.222.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-613-UJkRhB8LOcKi3989o7dI6A-1; Fri, 16 Dec 2022 10:52:27 -0500 X-MC-Unique: UJkRhB8LOcKi3989o7dI6A-1 Received: by mail-qk1-f200.google.com with SMTP id bq39-20020a05620a46a700b006ffd5db9fe9so2095651qkb.2 for ; Fri, 16 Dec 2022 07:52:27 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rIRGvziR/92Z54UoPp5pwcNYu+IBI1cm3v25qmGTr7U=; b=Cmwxi1ojjQSeRz0gw6PPt+0UM39ZbzI2VIluVSFGEAN+PVVaRFOPtbUZ9Kwn0kA66f vm6KOjW9XjZSDxeiwIKmnvt1HVvP87z6c6nSAd69uPH18d0JLPX3moINOlNJrLVTmUf6 JzBx53EkDpnO1P24sh37HK4b+p1++Lj5jwS88vlBcsZMtaxsfMlNaJnrcGZTLFdrIMg9 Vb1GxRA6N1VVj/wz35q8FwH/GTppxh6WiS+EAEYpzEgMFo1TszqYYognMey7R25l39Xn q/Zsooa2gEV/nbcNVZHO47CDiITS6wKb7C8lHjS4bdGe0lJ4/i62pXOnBdudssGvetvd vAcA== X-Gm-Message-State: ANoB5pkgMiijhw3XTVoTPbGdiDxbljf07Ow90cbbnAJEpapUDpliKNjI 9atPHDzBHm9++brAmpgwGb0rDHiU+0ppTX2z8KT063Ss8Z2cuPuxfwErmgBiC01Mx0h0SnKcSD/ ZYdKG4coBWZg= X-Received: by 2002:ad4:4353:0:b0:4e8:c749:7f3a with SMTP id q19-20020ad44353000000b004e8c7497f3amr20525132qvs.50.1671205946729; Fri, 16 Dec 2022 07:52:26 -0800 (PST) X-Google-Smtp-Source: AA0mqf4PGgI1eYNhoIKqqtlFBB21vIxd73/3ENYwzCz1J5KG++K1lTB3/aO/vYUZa2W/DRv/E7u/0g== X-Received: by 2002:ad4:4353:0:b0:4e8:c749:7f3a with SMTP id q19-20020ad44353000000b004e8c7497f3amr20525095qvs.50.1671205946508; Fri, 16 Dec 2022 07:52:26 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-45-70-31-26-132.dsl.bell.ca. [70.31.26.132]) by smtp.gmail.com with ESMTPSA id d136-20020ae9ef8e000000b006fef157c8aesm1673097qkg.36.2022.12.16.07.52.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Dec 2022 07:52:25 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: David Hildenbrand , John Hubbard , Muchun Song , Mike Kravetz , Nadav Amit , Andrea Arcangeli , Rik van Riel , peterx@redhat.com, Miaohe Lin , Jann Horn , James Houghton , Andrew Morton Subject: [PATCH v4 7/9] mm/hugetlb: Make follow_hugetlb_page() safe to pmd unshare Date: Fri, 16 Dec 2022 10:52:23 -0500 Message-Id: <20221216155223.2043727-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221216155100.2043537-1-peterx@redhat.com> References: <20221216155100.2043537-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: D2D9716001B X-Rspam-User: X-Stat-Signature: ihcrw6xxntrcsufw3n4q8wuh9cp7eje9 X-HE-Tag: 1671205948-347998 X-HE-Meta: U2FsdGVkX1+Kq4jcLYXIadFEjDHrFpbUmuuC9sTfnonF/pTLzqdky2IQau+T0QGDxSpcUDKh2af4QYBnWncDOdcpMq0RFFINW3kKlrO8A2ltD1c3Rd0mcKSILyf1rkedHLgpFlhvpUsCRRdzOwnDCSE+e4ujq/+HcZcFJBXfronO34NhVpd1bvpb+cw6kY/rSFy7eTnDEBdi4lPg30r+JXWTeuOW2nsTM3IUmQqjRcnSdhTy5yhvDw9+swG0Kz55ZXTPtG9G+lLsSHTr4z4je/yaPMeB0NL+CYHKau+mka/USjKfm+V9QK3Utf+yDoCcj13qdKVGPyLWEkfb5f5+kSCzTS63StaSMGH/HtgGdBLA96SCjP1BjP3j6y40H++ujkHaxE2fWksONtsXVTGn/IEM5qYAJ+k05jKoobPP443/FQA9JFLaNQL/2SSkkJ3hi3KrSBPosi1+CrxpA+cvjg7ybrXVvLPKJCCZnUjedgkb09kGd+LmvidwPLcFrNW0Dl4lb0G/niMAH4MuIBh/TQWOZ9IYEuR/y6dV5fP0nG7Hv5bcSv8lCPRGuURIn0nebCnMmRYqPw70WYLno0APnY4m4W542o8jkyeeDnxfn+BGtbUqTyhWi8R5SPfMMJ6X7kyOujbnhgn2z2W7mx94I7aywzhI7QJWTzF0DL/iUcsGesCn1LR4IXOvYf2H8YfV0jvdV+xS8lYI91d9l4kR+vWP6ivXh0/pnVnqe+eTtG1VpUgOY6K1rHQB0FvEvOXQE6OA26u5LDu3fxz820dhT2V4mTZCtWMOloIvhJRYsg2Ad+E4inHYeoacyrwO0w3HexVqwnp6z7+W9JZLp7wa/WrkbGabkQTs8BRbnZ5byAXr4j7Ysrn0ptzXtnELcReyvdrYZuvSPgYLifxUZj/d+gMEDzA3SYNXoVszSXQ65baxY4S1xzv5P0lhedgPTieRkQa9OYvYbhCQPN1RfCL cU5Oa+n5 cvcwiasxXoLwR9NrPizHnVtsI+lN+kv56ctWJ4zTozZxqV+OHAk+w+fHUwwbJ0K5BJrWnw8zvjI/wfXIJou3RaopFx+50czDL5RbnmLdVpalpzas8GpTRyzUKEMUcuLPLyl93hNjs0/jgNclJQAG6c/u3zP90X16tMzxey+VdVmGxDVmFIB55o9xWZZ207sP4KTUpzvolYgF65XgHd16xUsu67RNnkfEXyNqp X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since follow_hugetlb_page() walks the pgtable, it needs the vma lock to make sure the pgtable page will not be freed concurrently. Acked-by: David Hildenbrand Reviewed-by: Mike Kravetz Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- mm/hugetlb.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 0bf0abea388d..33fe73e1e589 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6433,6 +6433,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, break; } + hugetlb_vma_lock_read(vma); /* * Some archs (sparc64, sh*) have multiple pte_ts to * each hugepage. We have to make sure we get the @@ -6457,6 +6458,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, !hugetlbfs_pagecache_present(h, vma, vaddr)) { if (pte) spin_unlock(ptl); + hugetlb_vma_unlock_read(vma); remainder = 0; break; } @@ -6478,6 +6480,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, if (pte) spin_unlock(ptl); + hugetlb_vma_unlock_read(vma); + if (flags & FOLL_WRITE) fault_flags |= FAULT_FLAG_WRITE; else if (unshare) @@ -6540,6 +6544,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, remainder -= pages_per_huge_page(h); i += pages_per_huge_page(h); spin_unlock(ptl); + hugetlb_vma_unlock_read(vma); continue; } @@ -6569,6 +6574,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, if (WARN_ON_ONCE(!try_grab_folio(pages[i], refs, flags))) { spin_unlock(ptl); + hugetlb_vma_unlock_read(vma); remainder = 0; err = -ENOMEM; break; @@ -6580,6 +6586,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, i += refs; spin_unlock(ptl); + hugetlb_vma_unlock_read(vma); } *nr_pages = remainder; /* From patchwork Fri Dec 16 15:52:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13075140 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B7D7C4332F for ; Fri, 16 Dec 2022 15:52:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 27CC88E0008; Fri, 16 Dec 2022 10:52:35 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 22C688E0002; Fri, 16 Dec 2022 10:52:35 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0CE488E0008; Fri, 16 Dec 2022 10:52:35 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id EE2DF8E0002 for ; Fri, 16 Dec 2022 10:52:34 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 9DF6EA775F for ; Fri, 16 Dec 2022 15:52:34 +0000 (UTC) X-FDA: 80248611828.19.5017F14 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf13.hostedemail.com (Postfix) with ESMTP id 9CFDC2000B for ; Fri, 16 Dec 2022 15:52:32 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Jnm0OOLM; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671205952; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=SUHaufgIZGXg7uWbITubevKDDOKyb4JchyF1bpwW4KQ=; b=1xPctR5V39UOS86HvR3uiURSgooFuuZ0HMGI8MqMX+ztubxs5yzPr7Wcn2R0mGA3IX2GsW 1c+pyg6OdtdQ/U3abj+ptQglvRq9FS4HikqOeV4i0LbkSv2ox2wKl80oYvnAdr8/2jfKbj SlImGrSuaoKqRDTwvmzyiYle+QWVCjM= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Jnm0OOLM; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf13.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671205952; a=rsa-sha256; cv=none; b=41SdDFDK1TiNbLkbxGcTxl5A88lU4fX5S/t+LmOxQ5fYdYiBFETGU/CtEfLrwQzV9X/UYO M1wb4gjCN6AGkLmE8eZLhwOo118GOrxXTR8E8w0rNPS+PQamkHkPCsnlz+gTe4PkTVpwfn HF62RFF4Wzzh/Q07Hyg1vOrXsC+79cs= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671205952; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=SUHaufgIZGXg7uWbITubevKDDOKyb4JchyF1bpwW4KQ=; b=Jnm0OOLMGa8ZduX8a8Wt4+UK/2SuR3A/7tuy4+lRA7+xDAuw8/aNHB2diLLy8cMcpQAF4Y CPPnd5O1jt4bMEuSlCcxqUOgt2sGXrY3DpAr5MAVtjsCYHkxQt20ofk8neCc8Niux1mIyd 5I59y2ah3ZHjMaMQH4Q1L8M3huskxQE= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-468-JG4wsg4cOhmtkoA4zEBfkQ-1; Fri, 16 Dec 2022 10:52:30 -0500 X-MC-Unique: JG4wsg4cOhmtkoA4zEBfkQ-1 Received: by mail-qv1-f71.google.com with SMTP id kl19-20020a056214519300b004e174020eebso1655618qvb.23 for ; Fri, 16 Dec 2022 07:52:30 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=SUHaufgIZGXg7uWbITubevKDDOKyb4JchyF1bpwW4KQ=; b=C8HIlJGeaSrgd6mZTT6gmtUKNBdLOtIjUHnHz3O7h4AM/afAR9YlG95X9q3IJUp3X/ zzxW6yZyxwkrnvnINK9eAJAscHyogtyyeLxnbKNDLhtURRL6ptlqKex+ir63enGShs/n 0vJJrx2orEzGV+0ZMKoQVVZ2DAz8v5WSe0xtvrS3XPMN+y/DSQ5h53T+5fGfXAFu24/Q 9WKDC/XmXKPad0KD1Dp0U49NSvfrWM6/2yq3B5dBUjOSZNWN1szeIVGS9wmXtuhvAc/N bUYkNiFUxb5nmQmMeSwGTC1o63xovyg6jwoNWxxl6cK8PLGPpywCzlbpGPO/tfGjH9H1 fcfQ== X-Gm-Message-State: ANoB5pmLByl36gbc8ldif+loptOgh1019uBQPUWYpPjtxaRB2IO3e4ro 1Qc5ixTQLZWlRmZWgWco2apiDwas06n3xrkehVTRvTgRDSpIlevFneWudLsUwkzhusG+zZJijhq Kc+gr1p3y7qA= X-Received: by 2002:a05:622a:1652:b0:3a8:1600:e60f with SMTP id y18-20020a05622a165200b003a81600e60fmr42089078qtj.14.1671205949523; Fri, 16 Dec 2022 07:52:29 -0800 (PST) X-Google-Smtp-Source: AA0mqf6EvF97ux4AdUPswvdTGfH5GWS4ADEWRFVyxzLx9ScIIRjScwUEcGsBcdGYmcjjmGBVg0epbg== X-Received: by 2002:a05:622a:1652:b0:3a8:1600:e60f with SMTP id y18-20020a05622a165200b003a81600e60fmr42089044qtj.14.1671205949310; Fri, 16 Dec 2022 07:52:29 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-45-70-31-26-132.dsl.bell.ca. [70.31.26.132]) by smtp.gmail.com with ESMTPSA id h9-20020ac81389000000b003a530a32f67sm1472717qtj.65.2022.12.16.07.52.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Dec 2022 07:52:28 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: David Hildenbrand , John Hubbard , Muchun Song , Mike Kravetz , Nadav Amit , Andrea Arcangeli , Rik van Riel , peterx@redhat.com, Miaohe Lin , Jann Horn , James Houghton , Andrew Morton Subject: [PATCH v4 8/9] mm/hugetlb: Make walk_hugetlb_range() safe to pmd unshare Date: Fri, 16 Dec 2022 10:52:26 -0500 Message-Id: <20221216155226.2043738-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221216155100.2043537-1-peterx@redhat.com> References: <20221216155100.2043537-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Rspamd-Queue-Id: 9CFDC2000B X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: x1pmin6ncxo1ez8u5e4cm19tfyusd9ug X-HE-Tag: 1671205952-221471 X-HE-Meta: U2FsdGVkX182fSDXjzqHHSEnGoKvKTI0kTajXquKVdFYJSCtmrsmDrhAa9g6RLH3sosk9GaoklzXkWZ0hyIYEvG4d4AeEr5igUfzx2fNTr8iu1Xt+Lp/6IxOzuSb8YLEhGq3xBgjKdOJI16Iq/GmZW9RaO45RD5+/Uu8aK6oxszZkcM+hT5iBfzy793r1i6I+eKVMO99rq5ZVe7YxUyyG0bMoHPfMH9lAlovtYbcGSTwi2L30n6Vosve3+heanpqXKC6Ak8YroY3MpUKXoFBcg4pXPXtISdIl7p8RSa+5/gjia3+Xw4SJFpHqVMejK3VbY9fvSgOjGfOmQFdsEszWKORrjp3sspJz9jjK1KB72WBBl80IqpXse10W/p9c672ftwZk7c/IpRbFyisbQiX39i7tshBSQxEkZQJIV7qz+OhELKhd1YYJMZR6/sy06TODhPpPV6a3ir1jnI7e2QvKPC6rdPURygm7UpmIWF1XxnlbFKHpFfbCqQnJ1+pV+i3ULj6x2HMDNH9/HsU9rC+2ySasrNnTdja1Lg8iPVKaBYAQjxgC6DQqIvp9dpeTM3/vj2omYLb9gtKhAvAdEjsntv7zGBVWsEiwK46mmz8UxVPpBu+C8/VZMqyuaPkZL0VnEGd3jhX9lnioXVU+FQZnZhyroL6GsiLPJRf/BcETdlox70jony1YZTBHR5yJTmeHowv5j4jBdIBHJ62Nc9J2uuUkt6J8WibhKO3YAOAdpexI4OfY7vBaOW/8LG3VMFQjJoL2ABJvq+ZNgekTd5lqQZsshSpNMHFm53pwI/imtjsLJq8E1AkHNtIRQjhSguX4Uu9kci57p81GI0EAK15Isg4zrueMVASr0iVEti2QTs2NgVX93YWeiuNrARq8jJgdkiTILadbI3x8PKCZQgMYhCPSSCWF7vvIrvjX6beGrGueI+bGHEi5N+9Gd/S4iB6fsr9N6LmoYJINPFumOk 8ZkXJM42 HxdpnafeQ8rbeej0DSu/+AlwdVdTGP3OpbYjC5Kxw6+5mRXNVozMtk0KvKkLx7/aDlZB86kVbB3u8iwAf4rtrObYtVczDBOGHGPXlyZ4L+nHkcAu7y942Da2om1h0Xd2Rkz5nOGoeiLbOnEOYiOGhI7rKP7vOV0na2su0Rlh742/Fv/gv+oAE2yQbTpvNVbe6O3tl0p1TVGyWICuxWI8bHTC0laDgSe4VsIMg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since walk_hugetlb_range() walks the pgtable, it needs the vma lock to make sure the pgtable page will not be freed concurrently. Reviewed-by: Mike Kravetz Reviewed-by: John Hubbard Signed-off-by: Peter Xu --- include/linux/pagewalk.h | 11 ++++++++++- mm/hmm.c | 15 ++++++++++++++- mm/pagewalk.c | 2 ++ 3 files changed, 26 insertions(+), 2 deletions(-) diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index 959f52e5867d..27a6df448ee5 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -21,7 +21,16 @@ struct mm_walk; * depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD. * Any folded depths (where PTRS_PER_P?D is equal to 1) * are skipped. - * @hugetlb_entry: if set, called for each hugetlb entry + * @hugetlb_entry: if set, called for each hugetlb entry. This hook + * function is called with the vma lock held, in order to + * protect against a concurrent freeing of the pte_t* or + * the ptl. In some cases, the hook function needs to drop + * and retake the vma lock in order to avoid deadlocks + * while calling other functions. In such cases the hook + * function must either refrain from accessing the pte or + * ptl after dropping the vma lock, or else revalidate + * those items after re-acquiring the vma lock and before + * accessing them. * @test_walk: caller specific callback function to determine whether * we walk over the current vma or not. Returning 0 means * "do page table walk over the current vma", returning diff --git a/mm/hmm.c b/mm/hmm.c index 3850fb625dda..796de6866089 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -493,8 +493,21 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, cpu_flags); if (required_fault) { + int ret; + spin_unlock(ptl); - return hmm_vma_fault(addr, end, required_fault, walk); + hugetlb_vma_unlock_read(vma); + /* + * Avoid deadlock: drop the vma lock before calling + * hmm_vma_fault(), which will itself potentially take and + * drop the vma lock. This is also correct from a + * protection point of view, because there is no further + * use here of either pte or ptl after dropping the vma + * lock. + */ + ret = hmm_vma_fault(addr, end, required_fault, walk); + hugetlb_vma_lock_read(vma); + return ret; } pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT); diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 7f1c9b274906..d98564a7be57 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -302,6 +302,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, const struct mm_walk_ops *ops = walk->ops; int err = 0; + hugetlb_vma_lock_read(vma); do { next = hugetlb_entry_end(h, addr, end); pte = huge_pte_offset(walk->mm, addr & hmask, sz); @@ -314,6 +315,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, if (err) break; } while (addr = next, addr != end); + hugetlb_vma_unlock_read(vma); return err; } From patchwork Fri Dec 16 15:52:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13075141 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 59CCDC4332F for ; Fri, 16 Dec 2022 15:52:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E8A1B8E0009; Fri, 16 Dec 2022 10:52:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E3A508E0002; Fri, 16 Dec 2022 10:52:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C66078E0009; Fri, 16 Dec 2022 10:52:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id B5FBC8E0002 for ; Fri, 16 Dec 2022 10:52:37 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 778C2121146 for ; Fri, 16 Dec 2022 15:52:37 +0000 (UTC) X-FDA: 80248611954.15.7F2D77F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf07.hostedemail.com (Postfix) with ESMTP id 802EE40005 for ; Fri, 16 Dec 2022 15:52:35 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GJDPfVU4; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671205955; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=75WLafdrG2VCO1SJG5HZ0fON/RGaffQASLSh3g0vcGk=; b=xtpnfSYdJKbnE6KW1q+M0Ao64WCJFgc836/G0oF70i/NbGiQ1g4vzOIXZjqXsnxpUtLWEC XyWco7AuVtJPiQCU/sQ8E+ii94ys4NfxUpVoDnjCD8Gv5ACOuLd91q2mnuawgX9t82i9Jl kG3tPNXhkoY9PQTDiLiiQ8CGa2mpiY8= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=GJDPfVU4; spf=pass (imf07.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671205955; a=rsa-sha256; cv=none; b=2S3PFx+dlRELOgTkADZBkwfMIh2t5ZZJZ79bRiwIJJ5INqKn2RbC3BAFGwCmZoxUnshcAG xyCmw6IOq34crRy1cfsLXztI2zRwmWztONkRzyFhE90EUFS5q9Te/Wqw9MRL1gTYzwM21C gTbQzqlqDKuJCihKMJw3X3mHiFXW6qk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1671205954; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=75WLafdrG2VCO1SJG5HZ0fON/RGaffQASLSh3g0vcGk=; b=GJDPfVU420pVr8uf0PzGkzDOwjIHIatnl8BqyQbsZ1MrFPkCLkd/qGmpFyMaUE/Q8d0MaR HUs80WplDn6dmIH5cHnPN5CTg65rzD2KJ7CUO1fmPAJhXTdhOxPDb4s8n6gfXRF0KfesSw /u4hhqR+hmeOtJSWTgXqQPfvXZueCVo= Received: from mail-qt1-f197.google.com (mail-qt1-f197.google.com [209.85.160.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-328-goQvR5RPPaGtm0Q77du7cA-1; Fri, 16 Dec 2022 10:52:33 -0500 X-MC-Unique: goQvR5RPPaGtm0Q77du7cA-1 Received: by mail-qt1-f197.google.com with SMTP id z16-20020ac84550000000b003a807d816b2so1791989qtn.4 for ; Fri, 16 Dec 2022 07:52:33 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=75WLafdrG2VCO1SJG5HZ0fON/RGaffQASLSh3g0vcGk=; b=rZoIeUf8sp/4EMugmz0mupWqgdIY5jcXhDdBkXL5Mb+PaGpLVmmXOLnhnPzVnuM/bz TdV2OtEw9U92zgaNkwDHTyc0L4vu98NAMlX7xHMltJGArFggtqfbyc/IkQu9FaPoXVWR zAHQXLJnB++yC0KgxjNveVlwQEdv3UynOy4uf/l4b0AGFT2Fzl8L3LDdHlefOY9mCtXX /Y0G2A8SJ0HeALrm40jVVdeLvDf/hz9AOzVTCCnMZhmGYYOUP+sDrjcW542cZzzIvEwD HtmK2RgZb+F9yyygESNV+yIt+JSCXwbeLgqSnzO51SX9XRMKMII/AkMdG0nrVQ7/3KxC 7H4w== X-Gm-Message-State: AFqh2kq8NXWRQwkD/O6z0blvDp2wmMtVdlxC7zU2qgDWg7ocNwP0sGNv YREUSX+ntmk2Rzx4PkR9JsN6eEotljSAKwWUoRypEdQIOGTx/VJ/yQjn/txITgyUWsxUpMh2I+2 bw/0pG1I0OM4= X-Received: by 2002:ac8:545a:0:b0:3a9:763b:4a6d with SMTP id d26-20020ac8545a000000b003a9763b4a6dmr2397190qtq.10.1671205952994; Fri, 16 Dec 2022 07:52:32 -0800 (PST) X-Google-Smtp-Source: AMrXdXuv8NHiTHmsEqW/Az++9jPo/C9YVLUnT2LppWFbkrYlLQllWbZoKp1kPpJu4n0rC4ntRJAVZA== X-Received: by 2002:ac8:545a:0:b0:3a9:763b:4a6d with SMTP id d26-20020ac8545a000000b003a9763b4a6dmr2397157qtq.10.1671205952717; Fri, 16 Dec 2022 07:52:32 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-45-70-31-26-132.dsl.bell.ca. [70.31.26.132]) by smtp.gmail.com with ESMTPSA id s21-20020a05620a0bd500b006fa4ac86bfbsm202359qki.55.2022.12.16.07.52.30 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 16 Dec 2022 07:52:32 -0800 (PST) From: Peter Xu To: linux-kernel@vger.kernel.org, linux-mm@kvack.org Cc: David Hildenbrand , John Hubbard , Muchun Song , Mike Kravetz , Nadav Amit , Andrea Arcangeli , Rik van Riel , peterx@redhat.com, Miaohe Lin , Jann Horn , James Houghton , Andrew Morton Subject: [PATCH v4 9/9] mm/hugetlb: Introduce hugetlb_walk() Date: Fri, 16 Dec 2022 10:52:29 -0500 Message-Id: <20221216155229.2043750-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221216155100.2043537-1-peterx@redhat.com> References: <20221216155100.2043537-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-Stat-Signature: ns831qffkf56akm5qbj6sxa1i6765b3i X-Rspam-User: X-Rspamd-Queue-Id: 802EE40005 X-Rspamd-Server: rspam06 X-HE-Tag: 1671205955-371112 X-HE-Meta: U2FsdGVkX1/V4d9VxlHCdYLq2PfZOUvPZRT1xaFuep9hjO4tOgT3vXISqwVTzh9wxxI2tlzlElsL6Lu7uDVwZLruUzLx9lAYT3znpstmGHPio4uDYHonZGPhPywpgvlzg4BdbFu/TVseJPD0jbaxXN7SkdTIgyGC4Vv3zXqIAXtCg4yD/LPtN9H8PPAe6alNllZghTBuJqNjJiYLstRlzfoOeM8fK72x5MB1++IN9Y2GxyvsC3y0wJD4Uz0GmDJaFgE6JutFvxXY7ofYi7FkFhnqjLx3oOX63+4v1yV7soQMomnPydYn307u4wJ2t8739c1QbujRrUaflyFPYOgOvIPGnpEdMXarHLo7KfKgVSTKtOG34qX4toItVnSK7063LSFmV68prGT7uuE51m4R3VMhqX+vKTdsfqN03URFz8beEfexRmX3kvqnAsKIXo5sxdvK6K0J7BGB2+rVvp2iAdLUr59CZDw14gyeWw/pGmCuD3O4aBgMsTZWwnztncmsrNBgzuEDPpV1i+E7fopZzOAGDrJVFnpio+sdnxn/4Y25a93ee82lueDjeuBc5s1KujkIVqSfbqzxiWWMUXXN5P6u+pvpojraV8P1z0b6Z8IKxAcp4nyQ99i+RigieF4HfqxPGYhBt6vZkcGFyTMFQRxiJwEX7XzrL7QuFhlNStjUjj5Xzzl1cG0//Z+TfQtWqXQHTbA9hDtCeNp6lG1tH8ri4afPTr3E1hmIBu8vansAK0sNQ8OpKccPzLVdrmjhhbnk0LZfEb7ytrvVchd9tsqoHqXZ6DPGpx6r5QIpoA2L9mX+fJmGXutA/ceCnoPuWHE91bsJn6dhBbETJ0+KD/QF/JY3JcU/IZdiISglvIEK0RpGD61aRNU80CCyUpe3CEb6aPxKdnJnlJolW8DYmXaZFlKARFySg7Sasz3tq1ps9PcyFpp6jWu+lDZO2r2mnmklhdjEAqcABvIo6lj wdb32USx bmejI1zLQK9u7rBMKkkf36ulWvf/a8mld/+bZRLfsOWqqecDDoc5/IPwaOG+Yuiaa+sZUgWtwXRgFeLtvIWGkGqgDPAh154kTXinqn+Gu0UmQedn4qFRkP02ZkxAoC0QuaMM+B92uz7OqPUFlEB/0NR0h2aQ1JnQjJv8IpMS0e5bsgPKt1IH+2IQhSV5vsH/DOLSu8sN6gYxKxpSoIqDBHANVE2AxKxcP2asg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_pte_offset() is the main walker function for hugetlb pgtables. The name is not really representing what it does, though. Instead of renaming it, introduce a wrapper function called hugetlb_walk() which will use huge_pte_offset() inside. Assert on the locks when walking the pgtable. Note, the vma lock assertion will be a no-op for private mappings. Document the last special case in the page_vma_mapped_walk() path where we don't need any more lock to call hugetlb_walk(). Taking vma lock there is not needed because either: (1) potential callers of hugetlb pvmw holds i_mmap_rwsem already (from one rmap_walk()), or (2) the caller will not walk a hugetlb vma at all so the hugetlb code path not reachable (e.g. in ksm or uprobe paths). It's slightly implicit for future page_vma_mapped_walk() callers on that lock requirement. But anyway, when one day this rule breaks, one will get a straightforward warning in hugetlb_walk() with lockdep, then there'll be a way out. Reviewed-by: Mike Kravetz Reviewed-by: John Hubbard Signed-off-by: Peter Xu Reviewed-by: David Hildenbrand --- fs/hugetlbfs/inode.c | 4 +--- fs/userfaultfd.c | 6 ++---- include/linux/hugetlb.h | 37 +++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 31 +++++++++++++------------------ mm/page_vma_mapped.c | 9 ++++++--- mm/pagewalk.c | 4 +--- 6 files changed, 60 insertions(+), 31 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index fdb16246f46e..48f1a8ad2243 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -388,9 +388,7 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, { pte_t *ptep, pte; - ptep = huge_pte_offset(vma->vm_mm, addr, - huge_page_size(hstate_vma(vma))); - + ptep = hugetlb_walk(vma, addr, huge_page_size(hstate_vma(vma))); if (!ptep) return false; diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 887e20472051..4e27ff526873 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -237,14 +237,12 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, unsigned long flags, unsigned long reason) { - struct mm_struct *mm = ctx->mm; pte_t *ptep, pte; bool ret = true; - mmap_assert_locked(mm); - - ptep = huge_pte_offset(mm, address, vma_mmu_pagesize(vma)); + mmap_assert_locked(ctx->mm); + ptep = hugetlb_walk(vma, address, vma_mmu_pagesize(vma)); if (!ptep) goto out; diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index d755e2a7c0db..b6b10101bea7 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -2,6 +2,7 @@ #ifndef _LINUX_HUGETLB_H #define _LINUX_HUGETLB_H +#include #include #include #include @@ -196,6 +197,11 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, * huge_pte_offset(): Walk the hugetlb pgtable until the last level PTE. * Returns the pte_t* if found, or NULL if the address is not mapped. * + * IMPORTANT: we should normally not directly call this function, instead + * this is only a common interface to implement arch-specific + * walker. Please use hugetlb_walk() instead, because that will attempt to + * verify the locking for you. + * * Since this function will walk all the pgtable pages (including not only * high-level pgtable page, but also PUD entry that can be unshared * concurrently for VM_SHARED), the caller of this function should be @@ -1229,4 +1235,35 @@ bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr); #define flush_hugetlb_tlb_range(vma, addr, end) flush_tlb_range(vma, addr, end) #endif +static inline bool __vma_shareable_lock(struct vm_area_struct *vma) +{ + return (vma->vm_flags & VM_MAYSHARE) && vma->vm_private_data; +} + +/* + * Safe version of huge_pte_offset() to check the locks. See comments + * above huge_pte_offset(). + */ +static inline pte_t * +hugetlb_walk(struct vm_area_struct *vma, unsigned long addr, unsigned long sz) +{ +#if defined(CONFIG_HUGETLB_PAGE) && \ + defined(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) && defined(CONFIG_LOCKDEP) + struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; + + /* + * If pmd sharing possible, locking needed to safely walk the + * hugetlb pgtables. More information can be found at the comment + * above huge_pte_offset() in the same file. + * + * NOTE: lockdep_is_held() is only defined with CONFIG_LOCKDEP. + */ + if (__vma_shareable_lock(vma)) + WARN_ON_ONCE(!lockdep_is_held(&vma_lock->rw_sema) && + !lockdep_is_held( + &vma->vm_file->f_mapping->i_mmap_rwsem)); +#endif + return huge_pte_offset(vma->vm_mm, addr, sz); +} + #endif /* _LINUX_HUGETLB_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 33fe73e1e589..21dc37ff0896 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -258,11 +258,6 @@ static inline struct hugepage_subpool *subpool_vma(struct vm_area_struct *vma) /* * hugetlb vma_lock helper routines */ -static bool __vma_shareable_lock(struct vm_area_struct *vma) -{ - return vma->vm_flags & VM_MAYSHARE && vma->vm_private_data; -} - void hugetlb_vma_lock_read(struct vm_area_struct *vma) { if (__vma_shareable_lock(vma)) { @@ -4959,7 +4954,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } else { /* * For shared mappings the vma lock must be held before - * calling huge_pte_offset in the src vma. Otherwise, the + * calling hugetlb_walk() in the src vma. Otherwise, the * returned ptep could go away if part of a shared pmd and * another thread calls huge_pmd_unshare. */ @@ -4969,7 +4964,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, last_addr_mask = hugetlb_mask_last_page(h); for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; - src_pte = huge_pte_offset(src, addr, sz); + src_pte = hugetlb_walk(src_vma, addr, sz); if (!src_pte) { addr |= last_addr_mask; continue; @@ -5176,7 +5171,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, hugetlb_vma_lock_write(vma); i_mmap_lock_write(mapping); for (; old_addr < old_end; old_addr += sz, new_addr += sz) { - src_pte = huge_pte_offset(mm, old_addr, sz); + src_pte = hugetlb_walk(vma, old_addr, sz); if (!src_pte) { old_addr |= last_addr_mask; new_addr |= last_addr_mask; @@ -5239,7 +5234,7 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct last_addr_mask = hugetlb_mask_last_page(h); address = start; for (; address < end; address += sz) { - ptep = huge_pte_offset(mm, address, sz); + ptep = hugetlb_walk(vma, address, sz); if (!ptep) { address |= last_addr_mask; continue; @@ -5552,7 +5547,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vma_lock_read(vma); spin_lock(ptl); - ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); + ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) goto retry_avoidcopy; @@ -5590,7 +5585,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * before the page tables are altered */ spin_lock(ptl); - ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); + ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); @@ -6376,7 +6371,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, return NULL; hugetlb_vma_lock_read(vma); - pte = huge_pte_offset(mm, haddr, huge_page_size(h)); + pte = hugetlb_walk(vma, haddr, huge_page_size(h)); if (!pte) goto out_unlock; @@ -6441,8 +6436,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, * * Note that page table lock is not held when pte is null. */ - pte = huge_pte_offset(mm, vaddr & huge_page_mask(h), - huge_page_size(h)); + pte = hugetlb_walk(vma, vaddr & huge_page_mask(h), + huge_page_size(h)); if (pte) ptl = huge_pte_lock(h, mm, pte); absent = !pte || huge_pte_none(huge_ptep_get(pte)); @@ -6633,7 +6628,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, last_addr_mask = hugetlb_mask_last_page(h); for (; address < end; address += psize) { spinlock_t *ptl; - ptep = huge_pte_offset(mm, address, psize); + ptep = hugetlb_walk(vma, address, psize); if (!ptep) { address |= last_addr_mask; continue; @@ -7040,8 +7035,8 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, saddr = page_table_shareable(svma, vma, addr, idx); if (saddr) { - spte = huge_pte_offset(svma->vm_mm, saddr, - vma_mmu_pagesize(svma)); + spte = hugetlb_walk(svma, saddr, + vma_mmu_pagesize(svma)); if (spte) { get_page(virt_to_page(spte)); break; @@ -7358,7 +7353,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) hugetlb_vma_lock_write(vma); i_mmap_lock_write(vma->vm_file->f_mapping); for (address = start; address < end; address += PUD_SIZE) { - ptep = huge_pte_offset(mm, address, sz); + ptep = hugetlb_walk(vma, address, sz); if (!ptep) continue; ptl = huge_pte_lock(h, mm, ptep); diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 93e13fc17d3c..f3729b23dd0e 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -168,9 +168,12 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) /* The only possible mapping was handled on last iteration */ if (pvmw->pte) return not_found(pvmw); - - /* when pud is not present, pte will be NULL */ - pvmw->pte = huge_pte_offset(mm, pvmw->address, size); + /* + * All callers that get here will already hold the + * i_mmap_rwsem. Therefore, no additional locks need to be + * taken before calling hugetlb_walk(). + */ + pvmw->pte = hugetlb_walk(vma, pvmw->address, size); if (!pvmw->pte) return false; diff --git a/mm/pagewalk.c b/mm/pagewalk.c index d98564a7be57..cb23f8a15c13 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -305,13 +305,11 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, hugetlb_vma_lock_read(vma); do { next = hugetlb_entry_end(h, addr, end); - pte = huge_pte_offset(walk->mm, addr & hmask, sz); - + pte = hugetlb_walk(vma, addr & hmask, sz); if (pte) err = ops->hugetlb_entry(pte, hmask, addr, next, walk); else if (ops->pte_hole) err = ops->pte_hole(addr, next, -1, walk); - if (err) break; } while (addr = next, addr != end);