From patchwork Wed Dec 7 20:30:25 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13067577 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 959BAC63705 for ; Wed, 7 Dec 2022 20:30:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E2178E0001; Wed, 7 Dec 2022 15:30:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 96B0A8E0005; Wed, 7 Dec 2022 15:30:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7BD728E0001; Wed, 7 Dec 2022 15:30:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 542BD8E0005 for ; Wed, 7 Dec 2022 15:30:43 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 21C97141025 for ; Wed, 7 Dec 2022 20:30:43 +0000 (UTC) X-FDA: 80216653566.15.0381D93 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf09.hostedemail.com (Postfix) with ESMTP id A64EA140011 for ; Wed, 7 Dec 2022 20:30:42 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Jplkhv66; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670445042; a=rsa-sha256; cv=none; b=TOvbAjapAvrh+1TCEDiZeaySfWtUFXY81oXL00C6LilXetGVpRrUDi4du6DZ04GXDizrrl nX6nHM/81c630fquV7nndo2g01UdHAEo4mBMIyjhu7nPmiDqUFPvE3I70ln0srqXoIj8y1 NQCh55HpcdcILNj0IFzc6fPdgWfnuwk= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Jplkhv66; spf=pass (imf09.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670445042; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=kXTqB5FubF2+0tcQcQ/EoMxi6C2oCNf5KVcHgua8JL0=; b=fJeKdzBBoOkFeCMd4XNMfGDfhte8KcxVEdcvrg2kVtcU5O71Ug8pqnvqOVUJKPoU6e/psX Jx4tt61/tqSajn0DvWm6lVjJaIUBdfuvb0cEnUYNvqNh1xpzFRt/5jZPBU1iUa87jAoa8A 1Leh1W77/u26abPtehIrjSSdCLT/nGw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670445042; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=kXTqB5FubF2+0tcQcQ/EoMxi6C2oCNf5KVcHgua8JL0=; b=Jplkhv66+/xlI1uUbT+rjsNq/Wj8B3aGSyuMo6Raarc3s4gFkslI1whjCth46a0K57KsbJ oSk7mSfrUsEjLYRW74M8pqiimXQ2XfnFuxJzhxU7oTmXOH/3A1L2bgmv9AKNh0CGNscuR6 yWQLxzZ1pHfc/LWBwCJJg6H5YY2iaEI= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-671-iz7lCSIvPKyIjOwypYBADA-1; Wed, 07 Dec 2022 15:30:41 -0500 X-MC-Unique: iz7lCSIvPKyIjOwypYBADA-1 Received: by mail-qk1-f197.google.com with SMTP id bq13-20020a05620a468d00b006fa5a75759aso26384710qkb.13 for ; Wed, 07 Dec 2022 12:30:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kXTqB5FubF2+0tcQcQ/EoMxi6C2oCNf5KVcHgua8JL0=; b=ErCl1/6OoQAiR5wcO7raDp6ZM94gDgclopO6ibpgp43oTvjfqJjAAiUJagx5VDoF3P C5RZXZ2R032hTAlFjDUuT6gcGDQx2guv1OW6+M3wstmTUqgcNRa0ajIT7z0atidB+ekz epGBAD+zqvqyYlSGPnF+h+C8qp3w+2e+GfF+cMqeNv9wK2cnaWuTyqKBSnK6OHMmpWGq 8ybxa44p0W5tPbgxE01nBzoNXlolMlByO3TZ/abwIpztIuDMBG1Schbe7f/IIyH5LTTM l+7DpMa13RHtIrwJCgovQ+RAf0BdVJRdge7A5CWeWsaw3ZzHjNHLkRY/t4pCNncot8XH 3R1w== X-Gm-Message-State: ANoB5pmuP+mpdV7c4qn3u8IHZTqL5ylua37VbaAQ6KdpqewTkta0gVaO HvThJqrpvthFss51lt9zeY1/wELEEm5BImkymIAGrCm2TEmivAlLCuOAVTiqQWsK8CY92O0Ts8H PFhqeu1jXXItdCLjmDEnXD8opO6uxFPDwXrqK/nS62YE5eCswrwC5yOitM1kF X-Received: by 2002:ac8:4e47:0:b0:3a5:63ef:cf4e with SMTP id e7-20020ac84e47000000b003a563efcf4emr1902955qtw.16.1670445040174; Wed, 07 Dec 2022 12:30:40 -0800 (PST) X-Google-Smtp-Source: AA0mqf6IQwsRhSSOAs+i008iOESp/4MgizW47uNvKVfGDqpV4y6exkH0YuDqLS4fPXFFuTpd369j9Q== X-Received: by 2002:ac8:4e47:0:b0:3a5:63ef:cf4e with SMTP id e7-20020ac84e47000000b003a563efcf4emr1902931qtw.16.1670445039881; Wed, 07 Dec 2022 12:30:39 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id dc53-20020a05620a523500b006fefa5f7fcesm855594qkb.10.2022.12.07.12.30.36 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 12:30:37 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , John Hubbard , Andrea Arcangeli , James Houghton , Jann Horn , Rik van Riel , Miaohe Lin , Andrew Morton , Mike Kravetz , peterx@redhat.com, David Hildenbrand , Nadav Amit Subject: [PATCH v2 01/10] mm/hugetlb: Let vma_offset_start() to return start Date: Wed, 7 Dec 2022 15:30:25 -0500 Message-Id: <20221207203034.650899-2-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221207203034.650899-1-peterx@redhat.com> References: <20221207203034.650899-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain X-Spamd-Result: default: False [3.60 / 9.00]; BAYES_HAM(-5.50)[98.90%]; SORBS_IRL_BL(3.00)[209.85.222.197:received]; R_MISSING_CHARSET(2.50)[]; SUSPICIOUS_RECIPS(1.50)[]; SUBJECT_HAS_UNDERSCORES(1.00)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DKIM_TRACE(0.00)[redhat.com:+]; R_SPF_ALLOW(0.00)[+ip4:170.10.129.0/24]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: A64EA140011 X-Stat-Signature: g6bqi3qpq5u48ht6imk5ksiq87qd7p5o X-HE-Tag: 1670445042-807483 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Even though vma_offset_start() is named like that, it's not returning "the start address of the range" but rather the offset we should use to offset the vma->vm_start address. Make it return the real value of the start vaddr, and it also helps for all the callers because whenever the retval is used, it'll be ultimately added into the vma->vm_start anyway, so it's better. Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- fs/hugetlbfs/inode.c | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index 790d2727141a..fdb16246f46e 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -412,10 +412,12 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, */ static unsigned long vma_offset_start(struct vm_area_struct *vma, pgoff_t start) { + unsigned long offset = 0; + if (vma->vm_pgoff < start) - return (start - vma->vm_pgoff) << PAGE_SHIFT; - else - return 0; + offset = (start - vma->vm_pgoff) << PAGE_SHIFT; + + return vma->vm_start + offset; } static unsigned long vma_offset_end(struct vm_area_struct *vma, pgoff_t end) @@ -457,7 +459,7 @@ static void hugetlb_unmap_file_folio(struct hstate *h, v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - if (!hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page)) + if (!hugetlb_vma_maps_page(vma, v_start, page)) continue; if (!hugetlb_vma_trylock_write(vma)) { @@ -473,8 +475,8 @@ static void hugetlb_unmap_file_folio(struct hstate *h, break; } - unmap_hugepage_range(vma, vma->vm_start + v_start, v_end, - NULL, ZAP_FLAG_DROP_MARKER); + unmap_hugepage_range(vma, v_start, v_end, NULL, + ZAP_FLAG_DROP_MARKER); hugetlb_vma_unlock_write(vma); } @@ -507,10 +509,9 @@ static void hugetlb_unmap_file_folio(struct hstate *h, */ v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - if (hugetlb_vma_maps_page(vma, vma->vm_start + v_start, page)) - unmap_hugepage_range(vma, vma->vm_start + v_start, - v_end, NULL, - ZAP_FLAG_DROP_MARKER); + if (hugetlb_vma_maps_page(vma, v_start, page)) + unmap_hugepage_range(vma, v_start, v_end, NULL, + ZAP_FLAG_DROP_MARKER); kref_put(&vma_lock->refs, hugetlb_vma_lock_release); hugetlb_vma_unlock_write(vma); @@ -540,8 +541,7 @@ hugetlb_vmdelete_list(struct rb_root_cached *root, pgoff_t start, pgoff_t end, v_start = vma_offset_start(vma, start); v_end = vma_offset_end(vma, end); - unmap_hugepage_range(vma, vma->vm_start + v_start, v_end, - NULL, zap_flags); + unmap_hugepage_range(vma, v_start, v_end, NULL, zap_flags); /* * Note that vma lock only exists for shared/non-private From patchwork Wed Dec 7 20:30:26 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13067579 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D7A2C63709 for ; Wed, 7 Dec 2022 20:30:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 355598E0006; Wed, 7 Dec 2022 15:30:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 2DDE98E0005; Wed, 7 Dec 2022 15:30:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0935E8E0006; Wed, 7 Dec 2022 15:30:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D67908E0005 for ; Wed, 7 Dec 2022 15:30:44 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A92DBA0F24 for ; Wed, 7 Dec 2022 20:30:44 +0000 (UTC) X-FDA: 80216653608.30.FD392DC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf04.hostedemail.com (Postfix) with ESMTP id 0666040019 for ; Wed, 7 Dec 2022 20:30:43 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="evZwOO/r"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670445044; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RTEeCRjKpb4wZGC2IHTK/YLghk1hCkfDBIBgYSHgVC0=; b=a9P5eZd+zUwyV48YqrtZQts/kuUsMU8Hthn22qTkUkGLbRxBoyNQvvGPVRJLOQN5mFDviD VCgqvV7v9ynFBb/JNTflnBeGCdvq7do2cCyKi2dwigySUb2VKem1LPixA/RwgguWYk0bbt XVt+53hMWpB2M+Wr+AVITEnnYGey8Zg= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="evZwOO/r"; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf04.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670445044; a=rsa-sha256; cv=none; b=HRxNKDFAxJoO9SY69Yp3yGTS0k6cqUMgTxBPteFUwcuep5bxNBTRIwP3Syfxwn4TCu+Kji iY8e0/8pXMc4mwhD8jaJVAe1OdOKHqad7nDb5sbiXcGvghnjyqW4vXl6YM3CzNN+4s34HH a2nTEGfVDoId07jzqiW/83Tnjo5ec1U= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670445043; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=RTEeCRjKpb4wZGC2IHTK/YLghk1hCkfDBIBgYSHgVC0=; b=evZwOO/reQhgKX78m90wFfpYs0rFtY1fq+Cxif56xRNB+CUFF17eR/kzpN5JQzu/NfKv+S 2/pU0BAM51YhINEEIzjPB+pZE7tRI3889YnCqAvVm3qXP1BDS+7qClDsau2YHodaZYp0ch iuLX4hhdq6x2WwIprAJh6v9iv5UToU0= Received: from mail-qv1-f69.google.com (mail-qv1-f69.google.com [209.85.219.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-35-bIwh04kmOpShYQwxzNd3FQ-1; Wed, 07 Dec 2022 15:30:42 -0500 X-MC-Unique: bIwh04kmOpShYQwxzNd3FQ-1 Received: by mail-qv1-f69.google.com with SMTP id on28-20020a056214449c00b004bbf12d7976so37739758qvb.18 for ; Wed, 07 Dec 2022 12:30:42 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RTEeCRjKpb4wZGC2IHTK/YLghk1hCkfDBIBgYSHgVC0=; b=UN6oOaXsiMarE7bh0ZpN+lXdM57GdOJixxi6kRE8aBUvjsWjWbrv8ChwnzR5fht/pj lMGE4tsZu+hcjCgeiATGCgHJMqCChEp15hWQTlcSAPM+yAsPnHSNvr7815asCE4jX1oj +qL3RIZnZIoKufaJHfYe4Cn5gf6S7NoZcPEwicZ73/lZDUVz16ORq37z+rBrQSisonjy u9JZyIfnUmk3YpwDmkd+8dNb4kyKNl6AiYD0TNl4pFA+Wvs+F9qADy7hka/6J3fjXgLi sJLTVxrfxMeFjn1D+OeNpolfiRQjfhGfcrSqTlrPMfT2OaypFoGsF2fe6BTaUusMK1YA MWsA== X-Gm-Message-State: ANoB5pkcYvNAIQ77QGhrtMkpKmxyGleZC14nMq5PvTisUM4tIWonTBVW k1/DPYywFbNPHpTB1qOauMeNA+pSjA8FCX/2rPXVeh7WW35qhMD+2rButSnsmOgYAQUxoJ9jLYD 79IruSgfyOdBDm8rIGyCq2s+CZsWbkV72OCsl92DGQ6zFXwCBZTaNr8lPEvr8 X-Received: by 2002:ac8:6b92:0:b0:3a6:8a53:b8ab with SMTP id z18-20020ac86b92000000b003a68a53b8abmr1339926qts.36.1670445041545; Wed, 07 Dec 2022 12:30:41 -0800 (PST) X-Google-Smtp-Source: AA0mqf4sNFnBMyehihPsNzLXJQMBLFYcojqEtB+STHm+CXTTITN32QZQYRMPzOE5zmyAwmPrrPQM7g== X-Received: by 2002:ac8:6b92:0:b0:3a6:8a53:b8ab with SMTP id z18-20020ac86b92000000b003a68a53b8abmr1339904qts.36.1670445041289; Wed, 07 Dec 2022 12:30:41 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id dc53-20020a05620a523500b006fefa5f7fcesm855594qkb.10.2022.12.07.12.30.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 12:30:40 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , John Hubbard , Andrea Arcangeli , James Houghton , Jann Horn , Rik van Riel , Miaohe Lin , Andrew Morton , Mike Kravetz , peterx@redhat.com, David Hildenbrand , Nadav Amit Subject: [PATCH v2 02/10] mm/hugetlb: Don't wait for migration entry during follow page Date: Wed, 7 Dec 2022 15:30:26 -0500 Message-Id: <20221207203034.650899-3-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221207203034.650899-1-peterx@redhat.com> References: <20221207203034.650899-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 0666040019 X-Stat-Signature: ec344p6jdd8kd88bbaofxyjjrs9oah73 X-Spamd-Result: default: False [6.47 / 9.00]; SORBS_IRL_BL(3.00)[209.85.219.69:received]; R_MISSING_CHARSET(2.50)[]; BAYES_HAM(-1.63)[84.94%]; SUSPICIOUS_RECIPS(1.50)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DKIM_TRACE(0.00)[redhat.com:+]; R_SPF_ALLOW(0.00)[+ip4:170.10.129.0/24]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-HE-Tag: 1670445043-338273 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: That's what the code does with !hugetlb pages, so we should logically do the same for hugetlb, so migration entry will also be treated as no page. This is probably also the last piece in follow_page code that may sleep, the last one should be removed in cf994dd8af27 ("mm/gup: remove FOLL_MIGRATION", 2022-11-16). Reviewed-by: Mike Kravetz Reviewed-by: David Hildenbrand Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/hugetlb.c | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 1088f2f41c88..c8a6673fe5b4 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6232,7 +6232,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, if (WARN_ON_ONCE(flags & FOLL_PIN)) return NULL; -retry: pte = huge_pte_offset(mm, haddr, huge_page_size(h)); if (!pte) return NULL; @@ -6255,16 +6254,6 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, page = NULL; goto out; } - } else { - if (is_hugetlb_entry_migration(entry)) { - spin_unlock(ptl); - __migration_entry_wait_huge(pte, ptl); - goto retry; - } - /* - * hwpoisoned entry is treated as no_page_table in - * follow_page_mask(). - */ } out: spin_unlock(ptl); From patchwork Wed Dec 7 20:30:27 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13067580 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0BFACC63705 for ; Wed, 7 Dec 2022 20:30:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EC4FA8E0007; Wed, 7 Dec 2022 15:30:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E0C3C8E0005; Wed, 7 Dec 2022 15:30:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BDDEC8E0007; Wed, 7 Dec 2022 15:30:46 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 9D4B18E0005 for ; Wed, 7 Dec 2022 15:30:46 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 50D551A0F31 for ; Wed, 7 Dec 2022 20:30:46 +0000 (UTC) X-FDA: 80216653692.30.DC26D46 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id A8E178001A for ; Wed, 7 Dec 2022 20:30:45 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZlOcxqMN; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670445045; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=1A7zSSAdH3Q+Nn9HaLXwUXJ3cHWzn45QfpKMkITSm0w=; b=Q7wi+6H7MsnY03AbUDWi+rU+gCb4azmtR5II9rRX/JaHs+sBJPC/p/ydgSyqbUvsTr5r0z MK6l/Qqtpl+nSsPYllIYkrIpoZ/8kj9d8UvvhHihOnH9sr/otDJ9vBl083stAsMP0JUpQF +MFupoE2iIL/enPZbHzsQeq0wu+uycs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZlOcxqMN; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670445045; a=rsa-sha256; cv=none; b=elFsk41xilKGHVE9oNvoZtKXLXpAHCHAHMPW1YP4R8yKqsXe0ktn1xnskjfiIz3jqxxqsi A8fPgbqVXDdQLktzI6fn7cemPpmqyzbSHQLcnj/h3O1MuNMmBG8Mgh7XkJNlMLwNuZ003T 98TKoRC7xP4uGYByMZiAVXe9ginFpJE= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670445045; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1A7zSSAdH3Q+Nn9HaLXwUXJ3cHWzn45QfpKMkITSm0w=; b=ZlOcxqMNAb8KYh2XPxVOfnwl0Vtd3WqIGafTrqiPpLYh9R0VBE0UC31b673AUFJeWdq3DO bNT6q4GJlaf2b1PbmYMIWry0ffR0DB3A8b40PSHeh5mkWHQKD4BCOe3HXCSK51NMQQVQNK R99XkjrgyOF6cFjro9C+w0UWD0Qtc/w= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-659-nK-YC4PLO0qxTlpLUmx_Ag-1; Wed, 07 Dec 2022 15:30:44 -0500 X-MC-Unique: nK-YC4PLO0qxTlpLUmx_Ag-1 Received: by mail-qt1-f199.google.com with SMTP id s14-20020a05622a1a8e00b00397eacd9c1aso40512947qtc.21 for ; Wed, 07 Dec 2022 12:30:43 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=1A7zSSAdH3Q+Nn9HaLXwUXJ3cHWzn45QfpKMkITSm0w=; b=BdFL2CqOssSXkxy2aJMyiodaYMMugCE/J5HSXWDYTXfMSytYHLL8qCbskVqJFG6FFr Un50wHjbdh/e2h0X9Bdqg7iWcKCvnQsGXjIolhfumfc8QMfhokPzIB5+OL589108tE29 sN3uMvXuW7IUwvNTAjoCIQHyCxW9WxGww+E2UDJvjhl+5GL1JCOPHuUH5YS5WzB6TOOI wPoNmUlSFOjlYGx3pgKFgz8pVfcNUPUwh3CFVCP1tM21c6FygaKzvKA5dgzv9CK8p8mr 1Pady4w3pPUwG3dgDDbWPcACsLoC75H+4CANF0kmu1SRVJ4YxW9v9AiGMXzwcQZhBUiF c0JA== X-Gm-Message-State: ANoB5pmakTw9fUqXNcN37MlGQjdGO3JQuUDwWRrJ7aZAHUnhlEQUhch6 fj3F0RdTumglOcAikK6sriXfEhxf/1kWcuBVzCObMJe5PkYd3bhFMAm7DyPUUmJ5cV1NfRzarej xTDF+oWowgpHYnt3sCPnebQx7HuSwSAcsDsgfq5CzCNkGkAoNS4bpcFDEBzly X-Received: by 2002:a0c:e109:0:b0:4c6:ecbf:e47e with SMTP id w9-20020a0ce109000000b004c6ecbfe47emr1918700qvk.44.1670445043021; Wed, 07 Dec 2022 12:30:43 -0800 (PST) X-Google-Smtp-Source: AA0mqf7c6SH17+R0AvUqKo+tJdpohz+daUf26kZ0d9KHAqt+n6XDiZgwzv4YTz/R34Mrel5se5ut4w== X-Received: by 2002:a0c:e109:0:b0:4c6:ecbf:e47e with SMTP id w9-20020a0ce109000000b004c6ecbfe47emr1918677qvk.44.1670445042671; Wed, 07 Dec 2022 12:30:42 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id dc53-20020a05620a523500b006fefa5f7fcesm855594qkb.10.2022.12.07.12.30.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 12:30:42 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , John Hubbard , Andrea Arcangeli , James Houghton , Jann Horn , Rik van Riel , Miaohe Lin , Andrew Morton , Mike Kravetz , peterx@redhat.com, David Hildenbrand , Nadav Amit Subject: [PATCH v2 03/10] mm/hugetlb: Document huge_pte_offset usage Date: Wed, 7 Dec 2022 15:30:27 -0500 Message-Id: <20221207203034.650899-4-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221207203034.650899-1-peterx@redhat.com> References: <20221207203034.650899-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain X-Spamd-Result: default: False [5.07 / 9.00]; BAYES_HAM(-4.03)[95.15%]; SORBS_IRL_BL(3.00)[209.85.160.199:received]; R_MISSING_CHARSET(2.50)[]; SUSPICIOUS_RECIPS(1.50)[]; SUBJECT_HAS_UNDERSCORES(1.00)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DKIM_TRACE(0.00)[redhat.com:+]; R_SPF_ALLOW(0.00)[+ip4:170.10.129.0/24]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A8E178001A X-Stat-Signature: p7cgfmm7g1b9f8heya4gfaoff6x4uryi X-HE-Tag: 1670445045-219726 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_pte_offset() is potentially a pgtable walker, looking up pte_t* for a hugetlb address. Normally, it's always safe to walk a generic pgtable as long as we're with the mmap lock held for either read or write, because that guarantees the pgtable pages will always be valid during the process. But it's not true for hugetlbfs, especially shared: hugetlbfs can have its pgtable freed by pmd unsharing, it means that even with mmap lock held for current mm, the PMD pgtable page can still go away from under us if pmd unsharing is possible during the walk. So we have two ways to make it safe even for a shared mapping: (1) If we're with the hugetlb vma lock held for either read/write, it's okay because pmd unshare cannot happen at all. (2) If we're with the i_mmap_rwsem lock held for either read/write, it's okay because even if pmd unshare can happen, the pgtable page cannot be freed from under us. Document it. Signed-off-by: Peter Xu Reviewed-by: John Hubbard Reviewed-by: David Hildenbrand --- include/linux/hugetlb.h | 32 ++++++++++++++++++++++++++++++++ 1 file changed, 32 insertions(+) diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 551834cd5299..81efd9b9baa2 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -192,6 +192,38 @@ extern struct list_head huge_boot_pages; pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz); +/* + * huge_pte_offset(): Walk the hugetlb pgtable until the last level PTE. + * Returns the pte_t* if found, or NULL if the address is not mapped. + * + * Since this function will walk all the pgtable pages (including not only + * high-level pgtable page, but also PUD entry that can be unshared + * concurrently for VM_SHARED), the caller of this function should be + * responsible of its thread safety. One can follow this rule: + * + * (1) For private mappings: pmd unsharing is not possible, so it'll + * always be safe if we're with the mmap sem for either read or write. + * This is normally always the case, IOW we don't need to do anything + * special. + * + * (2) For shared mappings: pmd unsharing is possible (so the PUD-ranged + * pgtable page can go away from under us! It can be done by a pmd + * unshare with a follow up munmap() on the other process), then we + * need either: + * + * (2.1) hugetlb vma lock read or write held, to make sure pmd unshare + * won't happen upon the range (it also makes sure the pte_t we + * read is the right and stable one), or, + * + * (2.2) hugetlb mapping i_mmap_rwsem lock held read or write, to make + * sure even if unshare happened the racy unmap() will wait until + * i_mmap_rwsem is released. + * + * Option (2.1) is the safest, which guarantees pte stability from pmd + * sharing pov, until the vma lock released. Option (2.2) doesn't protect + * a concurrent pmd unshare, but it makes sure the pgtable page is safe to + * access. + */ pte_t *huge_pte_offset(struct mm_struct *mm, unsigned long addr, unsigned long sz); unsigned long hugetlb_mask_last_page(struct hstate *h); From patchwork Wed Dec 7 20:30:28 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13067581 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 20EFAC4708D for ; Wed, 7 Dec 2022 20:30:49 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96D498E0008; Wed, 7 Dec 2022 15:30:48 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8F68D8E0005; Wed, 7 Dec 2022 15:30:48 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5C5018E0008; Wed, 7 Dec 2022 15:30:48 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 2E7628E0005 for ; Wed, 7 Dec 2022 15:30:48 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id F3D68AB67F for ; Wed, 7 Dec 2022 20:30:47 +0000 (UTC) X-FDA: 80216653776.02.4703D46 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 8F65BC0005 for ; Wed, 7 Dec 2022 20:30:47 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YBL683Qo; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670445047; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XK/5vqUd1QlikRxebc+rXesIul2oExyAtkR9smyGVX0=; b=bPPSCGqziATJZlez2YTFqZ8RO1iWkhov0t+lbu2KXBLS4NzCBNxdChw2eSNW9bP+u/CaS1 YLZUwCJEngNK+SgJSBJ2f1z0lYoArNZhbTxhKlbILURWWG5rO1GZtNDdcnRjT0yWHt1g/V LcLJkX3clj6NPeSvm6jx7TH/ob7GfTU= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=YBL683Qo; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670445047; a=rsa-sha256; cv=none; b=VTJD69zeJfrXqdLubwfoAn+rQ6OsAWutGn4dpRVRePmyfJ/ASp4Ufqfm2t0K3M3Yt9YVUU wduZcrxgqFfuzjXQCBlCO00eTlhM0k9z5VsdeiDXvVW2FeL/q/ERFQ+ZBH8Oxl4K4mgmw3 f9QxZOHf5N2M4VAeZJbB0h8IA2Tzay0= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670445047; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XK/5vqUd1QlikRxebc+rXesIul2oExyAtkR9smyGVX0=; b=YBL683QoLr3TGrxzVQ4yz2/Iqwhw2O1mFP0n+9bmqVjfrZcrsWKxCL0OT5e4tUpSBPaTTx OHQKh+7qFAQ1yhBuAOD4f48fJiQezMoOuVZ9tgr/NFxu3Mi8k8/82g04FLhjQuR3haAZUU A2fsklmEXNoySCuOpwva2FP1IZTkxm8= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-367-xhHXBTEIMd-D8DO2Mc0D_A-1; Wed, 07 Dec 2022 15:30:45 -0500 X-MC-Unique: xhHXBTEIMd-D8DO2Mc0D_A-1 Received: by mail-qv1-f71.google.com with SMTP id on28-20020a056214449c00b004bbf12d7976so37739937qvb.18 for ; Wed, 07 Dec 2022 12:30:45 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=XK/5vqUd1QlikRxebc+rXesIul2oExyAtkR9smyGVX0=; b=EfzfqwFrBpghvJGXeV6CiSjpfD19A8DTUWBTdBSex5r4OH+MAUqLuLqJYEOUnhI2Wf SfByu9B8P0LPN+GiJGKQuwLMLLZgYpqES5IXKifWaJPiWuLw8i2nUtWIZbtAK2DoX5hO My0qS0s6cOVAtsomqKHc+wgQYHeHa4acKWEwdxeKPBTRVNIxXQK/D26CXmhC1c6PuemN Tkf6ADABb59tkBqYiAY0vMzm/bRr1VZ3NYasoQiz3E91uzov5O7efUj6l2Wo38nrJ5Ed 5qM6Ur8Pey6kFBZqvi8qdTQPFR2NcZhdLJk4h91cIphsoAv6IR5sCY67xsfhOpw9slYi YI1A== X-Gm-Message-State: ANoB5pmRe3jEXtD7ogITTCt2EPnJU21y1ajrt2WMTrxdeQUuh2XRIFru IWmSZzWumAaZhGzYHlC/Uis4ELMPkEDWeyDPT+N4cK+w8XvwjWcPua6e/zDKoFf9iDDdmiVW0Em WFyXPXbxWpiNKfQNQNcbittIFYDn3vidT0gNI4ezly09QhPX8QrbSUeOijvfF X-Received: by 2002:ac8:57cd:0:b0:3a6:7b53:9b20 with SMTP id w13-20020ac857cd000000b003a67b539b20mr2152374qta.12.1670445044556; Wed, 07 Dec 2022 12:30:44 -0800 (PST) X-Google-Smtp-Source: AA0mqf5LbXfVhI/+vq/w+LG4yNKtnSGIgip174q5+ZLxE8DPe02Msdaz4r+WtR6QivWpqpG5VnnIiA== X-Received: by 2002:ac8:57cd:0:b0:3a6:7b53:9b20 with SMTP id w13-20020ac857cd000000b003a67b539b20mr2152353qta.12.1670445044228; Wed, 07 Dec 2022 12:30:44 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id dc53-20020a05620a523500b006fefa5f7fcesm855594qkb.10.2022.12.07.12.30.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 12:30:43 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , John Hubbard , Andrea Arcangeli , James Houghton , Jann Horn , Rik van Riel , Miaohe Lin , Andrew Morton , Mike Kravetz , peterx@redhat.com, David Hildenbrand , Nadav Amit Subject: [PATCH v2 04/10] mm/hugetlb: Move swap entry handling into vma lock when faulted Date: Wed, 7 Dec 2022 15:30:28 -0500 Message-Id: <20221207203034.650899-5-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221207203034.650899-1-peterx@redhat.com> References: <20221207203034.650899-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain X-Rspam-User: X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 8F65BC0005 X-Stat-Signature: h13ryikgsfwtddm1xfr4bxbjg3i1hxxy X-Spamd-Result: default: False [2.10 / 9.00]; BAYES_HAM(-6.00)[100.00%]; SORBS_IRL_BL(3.00)[209.85.219.71:received]; R_MISSING_CHARSET(2.50)[]; SUSPICIOUS_RECIPS(1.50)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DKIM_TRACE(0.00)[redhat.com:+]; R_SPF_ALLOW(0.00)[+ip4:170.10.129.0/24]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-HE-Tag: 1670445047-684177 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In hugetlb_fault(), there used to have a special path to handle swap entry at the entrance using huge_pte_offset(). That's unsafe because huge_pte_offset() for a pmd sharable range can access freed pgtables if without any lock to protect the pgtable from being freed after pmd unshare. Here the simplest solution to make it safe is to move the swap handling to be after the vma lock being held. We may need to take the fault mutex on either migration or hwpoison entries now (also the vma lock, but that's really needed), however neither of them is hot path. Note that the vma lock cannot be released in hugetlb_fault() when the migration entry is detected, because in migration_entry_wait_huge() the pgtable page will be used again (by taking the pgtable lock), so that also need to be protected by the vma lock. Modify migration_entry_wait_huge() so that it must be called with vma read lock held, and properly release the lock in __migration_entry_wait_huge(). Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- include/linux/swapops.h | 6 ++++-- mm/hugetlb.c | 36 +++++++++++++++--------------------- mm/migrate.c | 25 +++++++++++++++++++++---- 3 files changed, 40 insertions(+), 27 deletions(-) diff --git a/include/linux/swapops.h b/include/linux/swapops.h index a70b5c3a68d7..b134c5eb75cb 100644 --- a/include/linux/swapops.h +++ b/include/linux/swapops.h @@ -337,7 +337,8 @@ extern void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, extern void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address); #ifdef CONFIG_HUGETLB_PAGE -extern void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl); +extern void __migration_entry_wait_huge(struct vm_area_struct *vma, + pte_t *ptep, spinlock_t *ptl); extern void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte); #endif /* CONFIG_HUGETLB_PAGE */ #else /* CONFIG_MIGRATION */ @@ -366,7 +367,8 @@ static inline void __migration_entry_wait(struct mm_struct *mm, pte_t *ptep, static inline void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, unsigned long address) { } #ifdef CONFIG_HUGETLB_PAGE -static inline void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl) { } +static inline void __migration_entry_wait_huge(struct vm_area_struct *vma, + pte_t *ptep, spinlock_t *ptl) { } static inline void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { } #endif /* CONFIG_HUGETLB_PAGE */ static inline int is_writable_migration_entry(swp_entry_t entry) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index c8a6673fe5b4..49f73677a418 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5824,22 +5824,6 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, int need_wait_lock = 0; unsigned long haddr = address & huge_page_mask(h); - ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); - if (ptep) { - /* - * Since we hold no locks, ptep could be stale. That is - * OK as we are only making decisions based on content and - * not actually modifying content here. - */ - entry = huge_ptep_get(ptep); - if (unlikely(is_hugetlb_entry_migration(entry))) { - migration_entry_wait_huge(vma, ptep); - return 0; - } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) - return VM_FAULT_HWPOISON_LARGE | - VM_FAULT_SET_HINDEX(hstate_index(h)); - } - /* * Serialize hugepage allocation and instantiation, so that we don't * get spurious allocation failures if two CPUs race to instantiate @@ -5854,10 +5838,6 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * Acquire vma lock before calling huge_pte_alloc and hold * until finished with ptep. This prevents huge_pmd_unshare from * being called elsewhere and making the ptep no longer valid. - * - * ptep could have already be assigned via huge_pte_offset. That - * is OK, as huge_pte_alloc will return the same value unless - * something has changed. */ hugetlb_vma_lock_read(vma); ptep = huge_pte_alloc(mm, vma, haddr, huge_page_size(h)); @@ -5886,8 +5866,22 @@ vm_fault_t hugetlb_fault(struct mm_struct *mm, struct vm_area_struct *vma, * fault, and is_hugetlb_entry_(migration|hwpoisoned) check will * properly handle it. */ - if (!pte_present(entry)) + if (!pte_present(entry)) { + if (unlikely(is_hugetlb_entry_migration(entry))) { + /* + * Release fault lock first because the vma lock is + * needed to guard the huge_pte_lockptr() later in + * migration_entry_wait_huge(). The vma lock will + * be released there. + */ + mutex_unlock(&hugetlb_fault_mutex_table[hash]); + migration_entry_wait_huge(vma, ptep); + return 0; + } else if (unlikely(is_hugetlb_entry_hwpoisoned(entry))) + ret = VM_FAULT_HWPOISON_LARGE | + VM_FAULT_SET_HINDEX(hstate_index(h)); goto out_mutex; + } /* * If we are going to COW/unshare the mapping later, we examine the diff --git a/mm/migrate.c b/mm/migrate.c index 48584b032ea9..d14f1f3ab073 100644 --- a/mm/migrate.c +++ b/mm/migrate.c @@ -333,24 +333,41 @@ void migration_entry_wait(struct mm_struct *mm, pmd_t *pmd, } #ifdef CONFIG_HUGETLB_PAGE -void __migration_entry_wait_huge(pte_t *ptep, spinlock_t *ptl) +void __migration_entry_wait_huge(struct vm_area_struct *vma, + pte_t *ptep, spinlock_t *ptl) { pte_t pte; + /* + * The vma read lock must be taken, which will be released before + * the function returns. It makes sure the pgtable page (along + * with its spin lock) not be freed in parallel. + */ + hugetlb_vma_assert_locked(vma); + spin_lock(ptl); pte = huge_ptep_get(ptep); - if (unlikely(!is_hugetlb_entry_migration(pte))) + if (unlikely(!is_hugetlb_entry_migration(pte))) { spin_unlock(ptl); - else + hugetlb_vma_unlock_read(vma); + } else { + /* + * If migration entry existed, safe to release vma lock + * here because the pgtable page won't be freed without the + * pgtable lock released. See comment right above pgtable + * lock release in migration_entry_wait_on_locked(). + */ + hugetlb_vma_unlock_read(vma); migration_entry_wait_on_locked(pte_to_swp_entry(pte), NULL, ptl); + } } void migration_entry_wait_huge(struct vm_area_struct *vma, pte_t *pte) { spinlock_t *ptl = huge_pte_lockptr(hstate_vma(vma), vma->vm_mm, pte); - __migration_entry_wait_huge(pte, ptl); + __migration_entry_wait_huge(vma, pte, ptl); } #endif From patchwork Wed Dec 7 20:30:29 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13067582 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D183C63708 for ; Wed, 7 Dec 2022 20:30:50 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 96DA48E0009; Wed, 7 Dec 2022 15:30:49 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 91F2A8E0005; Wed, 7 Dec 2022 15:30:49 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 721568E0009; Wed, 7 Dec 2022 15:30:49 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 3A43E8E0005 for ; Wed, 7 Dec 2022 15:30:49 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 12F84A0F0F for ; Wed, 7 Dec 2022 20:30:49 +0000 (UTC) X-FDA: 80216653818.22.F09B4E6 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf30.hostedemail.com (Postfix) with ESMTP id 927B680016 for ; Wed, 7 Dec 2022 20:30:48 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EuFYZeWa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670445048; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=v+Krxeq6Jc+PcaZJFIdSBIisnuJapI9S7rvolPGgBf8=; b=fyCRDaISRKGxX685IiXtCfGmBlO+wvW4sjDtI3oR0U3oqw4cbmtiUiStRaOXEd68WxDLlV S+b5iw19jEIvtfg02ABYK1qW5CQN8bdP7oMMLENewvl3iOSolPsbEHX+N0AJTPkmCLpjoA xWeDbYChIpg5Jni2SxuOk3i/zJ/AkUs= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=EuFYZeWa; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670445048; a=rsa-sha256; cv=none; b=Pnc8Z0nWd2e73HeVzt9bNyC5TjoC5Ee/gh8taCa3xx8XOs0ib5c4S/5J1YnIeTSj2SGfbu JkyVxqx3LFTvEkl7lb7QsTexCiqn1NYQqg9lP2VA5n2aOfJMQEThKIDF3/moFqppDE80nt dY/qXvujcgm8aDxiJo5sMD9fwU/EKcQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670445048; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=v+Krxeq6Jc+PcaZJFIdSBIisnuJapI9S7rvolPGgBf8=; b=EuFYZeWa63kT2UHXBI0WtvoxqGG46FM31/eM35yXM7vEVGDjg9j5vfYZLXn5E/M5v0T92N gbUhSxEB0IMWtAAtdwIcyYTmgvlylRIq746wf8x34Am5gq+26TDlARFoIWmyYPWr7chjf2 8J5OSW+QonL++ghxpvH68u0mAPthb04= Received: from mail-qt1-f199.google.com (mail-qt1-f199.google.com [209.85.160.199]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-224-rb2SeheMN5-0kxxqmHSLLQ-1; Wed, 07 Dec 2022 15:30:47 -0500 X-MC-Unique: rb2SeheMN5-0kxxqmHSLLQ-1 Received: by mail-qt1-f199.google.com with SMTP id ay12-20020a05622a228c00b003a52bd33749so38181988qtb.8 for ; Wed, 07 Dec 2022 12:30:47 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=v+Krxeq6Jc+PcaZJFIdSBIisnuJapI9S7rvolPGgBf8=; b=uxxza7YbZVf3dCnws+96I6gnxkhfsIa9A4j5pqjbDLD6Zs0m6G/0ICkMwF2vs2ALkt eg7AzYzN7GecAeKIxFOw89Ri0WJGpNgmvhI7beAHtYjl5YJRRQGckXmE8ImNgX/7vAQr iRhAHx6Tisq2cF4xUrMp5rxb4544L+3HZZInbqMJCpOyeU69dx7CY0qacYLscGV3eYbh oMpPGnx/O7CvEUO1hS+IcmAduWeBYx68e+8Uvm1RJZW86EHwlal3GQzFZfmm8z4FSIAU 2K007g0xuLLVjPH7+6ICACs46iUgcCVkP0SGAv43laB35Ue7jaMkOpSgq9sAJVQ198iH Gdew== X-Gm-Message-State: ANoB5pmppLEGQ1Ug1+rFeUcwLSR/vnQZmkKufheuX7aUIwTVe5CbQi6E 3eTt87zZi7ucnkDSpUQ0fsvSNalsOoP9JXkoPqxJf20XsTmY8SqNZJyQxp52jG9DO7ITcYaYgZl 7Z4qL1aH0IsNklYtPVueEYqyfCESl+ldDIJizoOpk+FDQtfzVaK7Pb48SyT6L X-Received: by 2002:ad4:4049:0:b0:4c6:e720:39ff with SMTP id r9-20020ad44049000000b004c6e72039ffmr1560036qvp.28.1670445046289; Wed, 07 Dec 2022 12:30:46 -0800 (PST) X-Google-Smtp-Source: AA0mqf4HMr/hq5EZlF5E6QU5x135bvh3KiJsjh1/wzJyvKRsivrnjQIaSzqnKC1C9heD+L0fqJzg4A== X-Received: by 2002:ad4:4049:0:b0:4c6:e720:39ff with SMTP id r9-20020ad44049000000b004c6e72039ffmr1560015qvp.28.1670445045854; Wed, 07 Dec 2022 12:30:45 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id dc53-20020a05620a523500b006fefa5f7fcesm855594qkb.10.2022.12.07.12.30.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 12:30:45 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , John Hubbard , Andrea Arcangeli , James Houghton , Jann Horn , Rik van Riel , Miaohe Lin , Andrew Morton , Mike Kravetz , peterx@redhat.com, David Hildenbrand , Nadav Amit Subject: [PATCH v2 05/10] mm/hugetlb: Make userfaultfd_huge_must_wait() safe to pmd unshare Date: Wed, 7 Dec 2022 15:30:29 -0500 Message-Id: <20221207203034.650899-6-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221207203034.650899-1-peterx@redhat.com> References: <20221207203034.650899-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain X-Spamd-Result: default: False [3.85 / 9.00]; BAYES_HAM(-5.25)[98.33%]; SORBS_IRL_BL(3.00)[209.85.160.199:received]; R_MISSING_CHARSET(2.50)[]; SUSPICIOUS_RECIPS(1.50)[]; SUBJECT_HAS_UNDERSCORES(1.00)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DKIM_TRACE(0.00)[redhat.com:+]; R_SPF_ALLOW(0.00)[+ip4:170.10.133.0/24]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 927B680016 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: 51ag5om8n5yadzjhayidcgzaf9q8ibod X-HE-Tag: 1670445048-426255 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We can take the hugetlb walker lock, here taking vma lock directly. Reviewed-by: David Hildenbrand Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- fs/userfaultfd.c | 18 ++++++++++++++---- 1 file changed, 14 insertions(+), 4 deletions(-) diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 07c81ab3fd4d..a602f008dde5 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -376,7 +376,8 @@ static inline unsigned int userfaultfd_get_blocking_state(unsigned int flags) */ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) { - struct mm_struct *mm = vmf->vma->vm_mm; + struct vm_area_struct *vma = vmf->vma; + struct mm_struct *mm = vma->vm_mm; struct userfaultfd_ctx *ctx; struct userfaultfd_wait_queue uwq; vm_fault_t ret = VM_FAULT_SIGBUS; @@ -403,7 +404,7 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) */ mmap_assert_locked(mm); - ctx = vmf->vma->vm_userfaultfd_ctx.ctx; + ctx = vma->vm_userfaultfd_ctx.ctx; if (!ctx) goto out; @@ -493,6 +494,13 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) blocking_state = userfaultfd_get_blocking_state(vmf->flags); + /* + * This stablizes pgtable for hugetlb on e.g. pmd unsharing. Need + * to be before setting current state. + */ + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_lock_read(vma); + spin_lock_irq(&ctx->fault_pending_wqh.lock); /* * After the __add_wait_queue the uwq is visible to userland @@ -507,13 +515,15 @@ vm_fault_t handle_userfault(struct vm_fault *vmf, unsigned long reason) set_current_state(blocking_state); spin_unlock_irq(&ctx->fault_pending_wqh.lock); - if (!is_vm_hugetlb_page(vmf->vma)) + if (!is_vm_hugetlb_page(vma)) must_wait = userfaultfd_must_wait(ctx, vmf->address, vmf->flags, reason); else - must_wait = userfaultfd_huge_must_wait(ctx, vmf->vma, + must_wait = userfaultfd_huge_must_wait(ctx, vma, vmf->address, vmf->flags, reason); + if (is_vm_hugetlb_page(vma)) + hugetlb_vma_unlock_read(vma); mmap_read_unlock(mm); if (likely(must_wait && !READ_ONCE(ctx->released))) { From patchwork Wed Dec 7 20:30:30 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13067584 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id ACC1AC4708D for ; Wed, 7 Dec 2022 20:30:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CEF248E0005; Wed, 7 Dec 2022 15:30:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C9FCB8E000A; Wed, 7 Dec 2022 15:30:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A7C488E0005; Wed, 7 Dec 2022 15:30:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6DD238E000A for ; Wed, 7 Dec 2022 15:30:53 -0500 (EST) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 408B4A0EC8 for ; Wed, 7 Dec 2022 20:30:53 +0000 (UTC) X-FDA: 80216653986.13.F86E907 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf30.hostedemail.com (Postfix) with ESMTP id CA7C680016 for ; Wed, 7 Dec 2022 20:30:52 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="GS+AOg/+"; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670445052; a=rsa-sha256; cv=none; b=GXrbi9F1g/f+VvGCrL0VNEcYCxaHGPcIVyBLk1KOHb2+HuatOflVuhRAjcKXR8XUFfv0uf /rKjNlaFGiSDsd8d65CSTK9ieD9QizfAYkizsEiKhoGW/C9LVtvymysOQkB5AquMT9vNnp Yt/v+8mK9ZTOlioHVEzwsJw0FGBQkE4= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b="GS+AOg/+"; spf=pass (imf30.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670445052; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=PxW1+7FnblUXilUivNjEMBQoO6EpHq3+X52yjUcQpsM=; b=imNmEFvRyAG9u+MovO8j7Bg48LIWLwGjaX30fG+sRGWGmKqEtCkbDP08Yd2pL6tJiOTwjD PECexcsdm0mP2DvqsiqCfUNcJphsrAAVioutAnbJn4wl16AUd75C1AePcB9VbUBCuHErGZ lXcFBJdSI5qd4Mo8+z9vw6dewIF7AAk= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670445052; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=PxW1+7FnblUXilUivNjEMBQoO6EpHq3+X52yjUcQpsM=; b=GS+AOg/+Xwe6W7r8CL5nK3G0RitZE/m6YPWFAZo9hwlJ5xbjbdm0jeRiW5YX3sxWwhVhy+ Z2ckqlG7Z+GexpDB39hIi25pfssGTBVA8dLVygbEX5u3roVyPOsuLEdngY6N/i49l/gzNk 2i6E6J5IMXUsGNUWBqBfzkv9N4DVyH0= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-411-bz1CFdliNE2Vqf20W98KFw-1; Wed, 07 Dec 2022 15:30:48 -0500 X-MC-Unique: bz1CFdliNE2Vqf20W98KFw-1 Received: by mail-qt1-f200.google.com with SMTP id s14-20020a05622a1a8e00b00397eacd9c1aso40513200qtc.21 for ; Wed, 07 Dec 2022 12:30:48 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=PxW1+7FnblUXilUivNjEMBQoO6EpHq3+X52yjUcQpsM=; b=E/bu3qtS2YN4/4iE/LKFuqnOsW/xOjNuOqB5YHnrycFRjW/JNKoJ6nzYUZRs7cVWJR GCEjx6MNGPMPCfrlPl3R6Nt7UU3gQKD9n0s5fNmeBqJMh68W+tEozYUfwIzibmMtwLKb tpfANSKGj08/4ifes1JLMqpl4jTMnTmWcRNbY2B41nktqoYOHzefmzCVLush+U8TCnvv TxZr/8KdBfARHF4BEl1Hbif9CnQFa7gFACBj5wkATAkAzvcBSB8PA4lU+O6UrguPmrRv LM3QPdm5dgWuBaTz2FMcqkgMuKKOtQ/St77E0BWQOd/vhs7UXDJBNlWMATTo3HD/fG/i Yc0Q== X-Gm-Message-State: ANoB5pkJi7p4rveDOfiDyZEEYGQYCuP8xAzETIWL/BGzdVwqBUznUpP8 qrBlufjN+IBPx5/nmcdgks/ELNpOu8m6LoWGtu18ZNXOG+0f/JFsT8cETQXUKf6bvv4SIwJ5e2W o/LS4MhmIWiiYaQkZscuJtiYCbSD1TZOoPFdrLJU7onafTWGH+5NgVJc2d6UF X-Received: by 2002:a05:622a:1b1c:b0:3a6:a205:741f with SMTP id bb28-20020a05622a1b1c00b003a6a205741fmr1433049qtb.32.1670445047827; Wed, 07 Dec 2022 12:30:47 -0800 (PST) X-Google-Smtp-Source: AA0mqf4LgYx0Tk+k6P2vLtvZSUdKvr8Brzp1Zc7q0zTbDSasDnwn5BMIAU2eNX9tOidMqIznOg+0Lg== X-Received: by 2002:a05:622a:1b1c:b0:3a6:a205:741f with SMTP id bb28-20020a05622a1b1c00b003a6a205741fmr1433023qtb.32.1670445047366; Wed, 07 Dec 2022 12:30:47 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id dc53-20020a05620a523500b006fefa5f7fcesm855594qkb.10.2022.12.07.12.30.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 12:30:47 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , John Hubbard , Andrea Arcangeli , James Houghton , Jann Horn , Rik van Riel , Miaohe Lin , Andrew Morton , Mike Kravetz , peterx@redhat.com, David Hildenbrand , Nadav Amit Subject: [PATCH v2 06/10] mm/hugetlb: Make hugetlb_follow_page_mask() safe to pmd unshare Date: Wed, 7 Dec 2022 15:30:30 -0500 Message-Id: <20221207203034.650899-7-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221207203034.650899-1-peterx@redhat.com> References: <20221207203034.650899-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain X-Spamd-Result: default: False [4.30 / 9.00]; BAYES_HAM(-4.80)[97.25%]; SORBS_IRL_BL(3.00)[209.85.160.200:received]; R_MISSING_CHARSET(2.50)[]; SUSPICIOUS_RECIPS(1.50)[]; SUBJECT_HAS_UNDERSCORES(1.00)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DKIM_TRACE(0.00)[redhat.com:+]; R_SPF_ALLOW(0.00)[+ip4:170.10.129.0/24]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: CA7C680016 X-Stat-Signature: i5h164pcbgjiissqqgedunawzog3towr X-HE-Tag: 1670445052-520402 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since hugetlb_follow_page_mask() walks the pgtable, it needs the vma lock to make sure the pgtable page will not be freed concurrently. Acked-by: David Hildenbrand Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/hugetlb.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 49f73677a418..3fbbd599d015 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6226,9 +6226,10 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, if (WARN_ON_ONCE(flags & FOLL_PIN)) return NULL; + hugetlb_vma_lock_read(vma); pte = huge_pte_offset(mm, haddr, huge_page_size(h)); if (!pte) - return NULL; + goto out_unlock; ptl = huge_pte_lock(h, mm, pte); entry = huge_ptep_get(pte); @@ -6251,6 +6252,8 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, } out: spin_unlock(ptl); +out_unlock: + hugetlb_vma_unlock_read(vma); return page; } From patchwork Wed Dec 7 20:30:31 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13067583 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D9FE9C63705 for ; Wed, 7 Dec 2022 20:30:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 722DC8E000B; Wed, 7 Dec 2022 15:30:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6AD858E0005; Wed, 7 Dec 2022 15:30:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4891A8E000A; Wed, 7 Dec 2022 15:30:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 2FA6B8E0005 for ; Wed, 7 Dec 2022 15:30:53 -0500 (EST) Received: from smtpin25.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D9AC7AB683 for ; Wed, 7 Dec 2022 20:30:52 +0000 (UTC) X-FDA: 80216653944.25.2663362 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf12.hostedemail.com (Postfix) with ESMTP id 1E4674001E for ; Wed, 7 Dec 2022 20:30:51 +0000 (UTC) Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WnJmySBC; spf=pass (imf12.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670445052; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=NwIY6qGYW16H7RKgjgoL5wZJoRvSXS6wmhu5l+J439g=; b=2zACSvXljcylKaVy3Xc2HoKvRT9t/qbj8ZX4NZRk7Ptqf6iU1LtUGrq3LyDCN02AL4iEi/ IOIfFRwKelLR1UELF0vx/wqs5+d169+EWBodpTYtNvW80aLF/Y5B+OuDfs9CRoD4dyoKv8 gWCqnxsz4zfnhc4u17bA+usTZ/dWYLE= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=WnJmySBC; spf=pass (imf12.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670445052; a=rsa-sha256; cv=none; b=Y7qfyvaPxbTCBit3ltLEVnnMwFd3gXTsh+rGtVT56VAABazSWcm3K1b7Ym1iRSUa10a5Qq HiUXxrwqM3mKH7XcnI8exhR0hSN1zez47ZlwCAAMEmpTlHzK4vO6Ex2WWRgnKB+V0n7RSw kMCERv+10h/kHheF29O+N9tkdpCZ7wQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670445051; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NwIY6qGYW16H7RKgjgoL5wZJoRvSXS6wmhu5l+J439g=; b=WnJmySBCHDfTfkxjIFsTO61nvAbyOelrftN2sOY3a+/kjYHomQd9Vy7qVd4d+Y6k9wwraf M+je51BXgp0IegbvO4KgQeI+eP9CPwGuu+Aug0AAcGYV3lj1tqVsKarVQlPHDpDXjzGQ3r 03BZA82z5mFQAFmdq1lDgx83xyo13JY= Received: from mail-qk1-f197.google.com (mail-qk1-f197.google.com [209.85.222.197]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-551-Ca5eooBpNsSyDeRqo-b_jQ-1; Wed, 07 Dec 2022 15:30:50 -0500 X-MC-Unique: Ca5eooBpNsSyDeRqo-b_jQ-1 Received: by mail-qk1-f197.google.com with SMTP id h13-20020a05620a244d00b006fb713618b8so25949820qkn.0 for ; Wed, 07 Dec 2022 12:30:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NwIY6qGYW16H7RKgjgoL5wZJoRvSXS6wmhu5l+J439g=; b=IokHajl005tWVCcz/MRL59HWPbiEvgM4HW20Ks/+WZfMcTYqsjeYoYX/8PTpnZDmQz X8x7hOezsefF0JeMvQA9F1wkQsMWg01HXZKagilDfEcOxENDLiJyTXlJnPOi3CER3EZh QbzqHFEEEMHtRxfkgPqFBeGHj6Hq5uX/tUkrTYSQ878WNKxBR3FsVVysvJvuV5AK+5es 2T2KEcrUNRE9Q6xw2F4kHyWap4ulnsDR9GGXzIfl+E7MeaTTGJGl7tCPcqN+aeZC1eOD vIo/xXjG6FMDhkWR+wxQEsea6cfujc9hu2fJXpO7SdFNOdTOp3zG92BZbj0L0doei0yP ea/g== X-Gm-Message-State: ANoB5plgneHFHTSYCdcNI9iHMwaseUzVbE2RpgdSmDKn42H6OnfN6IKO lL4Sizyb9EkvLDA+Y9PQVHWRUsKH1gYqMxKTBOC2tlVrbBOD47kTpc3G2g38U1XRjuqNZ7zapEF e6TGWM9CE5Zmb7EwIoARe5+aUBL4GrRZ6oC8LacTJfvQ+x9glYurjbqNt1kBX X-Received: by 2002:a05:6214:284:b0:4c7:4f6d:d9bd with SMTP id l4-20020a056214028400b004c74f6dd9bdmr1356006qvv.38.1670445049332; Wed, 07 Dec 2022 12:30:49 -0800 (PST) X-Google-Smtp-Source: AA0mqf6gwEUm1ExsgjT3GOzwaF3cPo9sE8A57vdvavUjxQytQjXm37D0mKl9btfYXk/zG1ZmitVzEg== X-Received: by 2002:a05:6214:284:b0:4c7:4f6d:d9bd with SMTP id l4-20020a056214028400b004c74f6dd9bdmr1355984qvv.38.1670445048921; Wed, 07 Dec 2022 12:30:48 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id dc53-20020a05620a523500b006fefa5f7fcesm855594qkb.10.2022.12.07.12.30.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 12:30:48 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , John Hubbard , Andrea Arcangeli , James Houghton , Jann Horn , Rik van Riel , Miaohe Lin , Andrew Morton , Mike Kravetz , peterx@redhat.com, David Hildenbrand , Nadav Amit Subject: [PATCH v2 07/10] mm/hugetlb: Make follow_hugetlb_page() safe to pmd unshare Date: Wed, 7 Dec 2022 15:30:31 -0500 Message-Id: <20221207203034.650899-8-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221207203034.650899-1-peterx@redhat.com> References: <20221207203034.650899-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 1E4674001E X-Rspam-User: X-Stat-Signature: g5abdk83rkhbdhn14r1dudj41k1n8cas X-Spamd-Result: default: False [7.67 / 9.00]; SORBS_IRL_BL(3.00)[209.85.222.197:received]; R_MISSING_CHARSET(2.50)[]; SUSPICIOUS_RECIPS(1.50)[]; BAYES_HAM(-1.43)[83.56%]; MID_CONTAINS_FROM(1.00)[]; SUBJECT_HAS_UNDERSCORES(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DKIM_TRACE(0.00)[redhat.com:+]; R_SPF_ALLOW(0.00)[+ip4:170.10.129.0/24]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-HE-Tag: 1670445051-60104 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since follow_hugetlb_page() walks the pgtable, it needs the vma lock to make sure the pgtable page will not be freed concurrently. Acked-by: David Hildenbrand Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- mm/hugetlb.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/hugetlb.c b/mm/hugetlb.c index 3fbbd599d015..f42399522805 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -6284,6 +6284,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, break; } + hugetlb_vma_lock_read(vma); /* * Some archs (sparc64, sh*) have multiple pte_ts to * each hugepage. We have to make sure we get the @@ -6308,6 +6309,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, !hugetlbfs_pagecache_present(h, vma, vaddr)) { if (pte) spin_unlock(ptl); + hugetlb_vma_unlock_read(vma); remainder = 0; break; } @@ -6329,6 +6331,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, if (pte) spin_unlock(ptl); + hugetlb_vma_unlock_read(vma); + if (flags & FOLL_WRITE) fault_flags |= FAULT_FLAG_WRITE; else if (unshare) @@ -6388,6 +6392,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, remainder -= pages_per_huge_page(h); i += pages_per_huge_page(h); spin_unlock(ptl); + hugetlb_vma_unlock_read(vma); continue; } @@ -6415,6 +6420,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, if (WARN_ON_ONCE(!try_grab_folio(pages[i], refs, flags))) { spin_unlock(ptl); + hugetlb_vma_unlock_read(vma); remainder = 0; err = -ENOMEM; break; @@ -6426,6 +6432,7 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, i += refs; spin_unlock(ptl); + hugetlb_vma_unlock_read(vma); } *nr_pages = remainder; /* From patchwork Wed Dec 7 20:30:32 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13067585 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C208FC63708 for ; Wed, 7 Dec 2022 20:30:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5784C8E000C; Wed, 7 Dec 2022 15:30:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 463048E000A; Wed, 7 Dec 2022 15:30:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2665A8E000C; Wed, 7 Dec 2022 15:30:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 0F67B8E000A for ; Wed, 7 Dec 2022 15:30:56 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CB36040407 for ; Wed, 7 Dec 2022 20:30:55 +0000 (UTC) X-FDA: 80216654070.09.034219F Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf14.hostedemail.com (Postfix) with ESMTP id 4C068100022 for ; Wed, 7 Dec 2022 20:30:55 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Lsk2f1d5; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670445055; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dnUJ2i61erGKHgVVBS7fwgSAnx0w9hoDMCwKHPZDtzw=; b=snOmzE9INBAAuS58dNpVsf8jdnMyZ7G5+h74MJ+og/0iKDjs5jFt7k1oRV2ozRcv8m+PEv Pw070ewKVbTh/t7gDPqUkeXlPp3myfMBDtE130YgAKMt76R25EtxtHCqnR57HMGq6xVxWq SH3mhJ2Lb9+s5JfXs6iY2Hn7ItWnXSI= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Lsk2f1d5; spf=pass (imf14.hostedemail.com: domain of peterx@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670445055; a=rsa-sha256; cv=none; b=nHnuHBzbfn14+Iatch6hZg2KubFrjzvLul4Z9El7dKu30/UoG/GsPeBABS8SoXVnLYcRgx /sU5PEO+Rg+oPyT6CohekFLyWHtbhGZJbJUtNU4/CJvT/++hTI6IOYt8udVJDhB2UWwG4H L4EZ7UURYEtKuB7iQPNmyala/m1ry3A= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670445054; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=dnUJ2i61erGKHgVVBS7fwgSAnx0w9hoDMCwKHPZDtzw=; b=Lsk2f1d5ikGELoe05WsV9Yw14VXgfTHGA+BJ3jFEMRYLswI1xUsgDqyiTn5A8NXFY6BxWZ HMm9eG4jTdiHKV3ig03CBO55Xl0VSyV+3lrBNBg8XQRKJBQTClETJBlcqzI46TZMtcQDR2 lnlXrikhqNHoTkqQzsoCUmbo+WU1oFA= Received: from mail-qt1-f198.google.com (mail-qt1-f198.google.com [209.85.160.198]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-629-TMo4UtMLPqyKQiZpJxsfXA-1; Wed, 07 Dec 2022 15:30:53 -0500 X-MC-Unique: TMo4UtMLPqyKQiZpJxsfXA-1 Received: by mail-qt1-f198.google.com with SMTP id cm12-20020a05622a250c00b003a521f66e8eso39653445qtb.17 for ; Wed, 07 Dec 2022 12:30:53 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dnUJ2i61erGKHgVVBS7fwgSAnx0w9hoDMCwKHPZDtzw=; b=RBoPrsljh126K98Uub1gE25NuE7tUW/mzo1etiH3TlhbscN9RPErOZ0NLaduCerxLw g2a+zIse3LB767Vt1rAioQaaUmE51JvUcj+uAPpxyak4Goa3+LgnouOOU8P1rJ6/lchM 9peI1TWDzh3AXqnsaVwMT9FT6/doEwb6RLzavFCTkXG6/f9l3xwfTtKMGjlHSKjKMNR8 0mQxDG+lsg4UtAa5xVcToZ+vdXbDcrbcdTMciYfRW10j7HPWS8ghQ+zcbXgKmTQs8Q+H G6XjDyuIRkZlG1GdLXQ1hozdZl6tbrCwypxAz+tXpAQUjFsTRZOpHEXod1csz+E02mg1 YzEw== X-Gm-Message-State: ANoB5plsSp7IQ1aWM2cy6Gxb8glDo2wJrVF2J5F+BiEUCCp+WqfgnoiB exnolIfMX6QjRSObYvaRx8q6YJ+04tZVV8dfHRgCcP0rMfb5l6ggYOGUu5hGYZ56rKI///rD0g6 /P06t5Wmy0I6TjggodzuOckjyqEwxG3noMaLkKVW5tkIZfb+Uk6Zf/wDJKnMr X-Received: by 2002:a05:622a:4116:b0:39c:e5bf:8162 with SMTP id cc22-20020a05622a411600b0039ce5bf8162mr1523518qtb.55.1670445050907; Wed, 07 Dec 2022 12:30:50 -0800 (PST) X-Google-Smtp-Source: AA0mqf4Dm6o+mvrRgKNPfSdavyDzIrF0yVxDNEuk8J2b1g1jeHdYzQXkU2H9nY0+acDohHOQ2RPx0Q== X-Received: by 2002:a05:622a:4116:b0:39c:e5bf:8162 with SMTP id cc22-20020a05622a411600b0039ce5bf8162mr1523492qtb.55.1670445050551; Wed, 07 Dec 2022 12:30:50 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id dc53-20020a05620a523500b006fefa5f7fcesm855594qkb.10.2022.12.07.12.30.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 12:30:50 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Muchun Song , John Hubbard , Andrea Arcangeli , James Houghton , Jann Horn , Rik van Riel , Miaohe Lin , Andrew Morton , Mike Kravetz , peterx@redhat.com, David Hildenbrand , Nadav Amit Subject: [PATCH v2 08/10] mm/hugetlb: Make walk_hugetlb_range() safe to pmd unshare Date: Wed, 7 Dec 2022 15:30:32 -0500 Message-Id: <20221207203034.650899-9-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221207203034.650899-1-peterx@redhat.com> References: <20221207203034.650899-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 4C068100022 X-Stat-Signature: i1tdzm67ci7dqjk3wxsjedaoo1eei1n5 X-Spamd-Result: default: False [3.10 / 9.00]; BAYES_HAM(-6.00)[100.00%]; SORBS_IRL_BL(3.00)[209.85.160.198:received]; R_MISSING_CHARSET(2.50)[]; SUSPICIOUS_RECIPS(1.50)[]; SUBJECT_HAS_UNDERSCORES(1.00)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DKIM_TRACE(0.00)[redhat.com:+]; R_SPF_ALLOW(0.00)[+ip4:170.10.129.0/24]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspam-User: X-HE-Tag: 1670445055-870373 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Since walk_hugetlb_range() walks the pgtable, it needs the vma lock to make sure the pgtable page will not be freed concurrently. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu Reviewed-by: John Hubbard --- arch/s390/mm/gmap.c | 2 ++ fs/proc/task_mmu.c | 2 ++ include/linux/pagewalk.h | 11 ++++++++++- mm/hmm.c | 15 ++++++++++++++- mm/pagewalk.c | 2 ++ 5 files changed, 30 insertions(+), 2 deletions(-) diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 8947451ae021..292a54c490d4 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -2643,7 +2643,9 @@ static int __s390_enable_skey_hugetlb(pte_t *pte, unsigned long addr, end = start + HPAGE_SIZE - 1; __storage_key_init_range(start, end); set_bit(PG_arch_1, &page->flags); + hugetlb_vma_unlock_read(walk->vma); cond_resched(); + hugetlb_vma_lock_read(walk->vma); return 0; } diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index e35a0398db63..cf3887fb2905 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -1613,7 +1613,9 @@ static int pagemap_hugetlb_range(pte_t *ptep, unsigned long hmask, frame++; } + hugetlb_vma_unlock_read(walk->vma); cond_resched(); + hugetlb_vma_lock_read(walk->vma); return err; } diff --git a/include/linux/pagewalk.h b/include/linux/pagewalk.h index 959f52e5867d..27a6df448ee5 100644 --- a/include/linux/pagewalk.h +++ b/include/linux/pagewalk.h @@ -21,7 +21,16 @@ struct mm_walk; * depth is -1 if not known, 0:PGD, 1:P4D, 2:PUD, 3:PMD. * Any folded depths (where PTRS_PER_P?D is equal to 1) * are skipped. - * @hugetlb_entry: if set, called for each hugetlb entry + * @hugetlb_entry: if set, called for each hugetlb entry. This hook + * function is called with the vma lock held, in order to + * protect against a concurrent freeing of the pte_t* or + * the ptl. In some cases, the hook function needs to drop + * and retake the vma lock in order to avoid deadlocks + * while calling other functions. In such cases the hook + * function must either refrain from accessing the pte or + * ptl after dropping the vma lock, or else revalidate + * those items after re-acquiring the vma lock and before + * accessing them. * @test_walk: caller specific callback function to determine whether * we walk over the current vma or not. Returning 0 means * "do page table walk over the current vma", returning diff --git a/mm/hmm.c b/mm/hmm.c index 3850fb625dda..796de6866089 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -493,8 +493,21 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, cpu_flags); if (required_fault) { + int ret; + spin_unlock(ptl); - return hmm_vma_fault(addr, end, required_fault, walk); + hugetlb_vma_unlock_read(vma); + /* + * Avoid deadlock: drop the vma lock before calling + * hmm_vma_fault(), which will itself potentially take and + * drop the vma lock. This is also correct from a + * protection point of view, because there is no further + * use here of either pte or ptl after dropping the vma + * lock. + */ + ret = hmm_vma_fault(addr, end, required_fault, walk); + hugetlb_vma_lock_read(vma); + return ret; } pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT); diff --git a/mm/pagewalk.c b/mm/pagewalk.c index 7f1c9b274906..d98564a7be57 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -302,6 +302,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, const struct mm_walk_ops *ops = walk->ops; int err = 0; + hugetlb_vma_lock_read(vma); do { next = hugetlb_entry_end(h, addr, end); pte = huge_pte_offset(walk->mm, addr & hmask, sz); @@ -314,6 +315,7 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, if (err) break; } while (addr = next, addr != end); + hugetlb_vma_unlock_read(vma); return err; } From patchwork Wed Dec 7 20:31:56 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13067587 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 14706C4708D for ; Wed, 7 Dec 2022 20:32:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AC9E28E0006; Wed, 7 Dec 2022 15:32:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A7A068E0001; Wed, 7 Dec 2022 15:32:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8F4298E0006; Wed, 7 Dec 2022 15:32:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 731938E0001 for ; Wed, 7 Dec 2022 15:32:43 -0500 (EST) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 4A698C0AB9 for ; Wed, 7 Dec 2022 20:32:43 +0000 (UTC) X-FDA: 80216658606.11.4223A4D Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf01.hostedemail.com (Postfix) with ESMTP id 235E94001A for ; Wed, 7 Dec 2022 20:32:40 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=STEESIXx; spf=pass (imf01.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670445161; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DOnWN4xjgNQJFSPMI7LR+EFJPU3TVyZrxc0oybMi1t4=; b=k+faclMsxF4LkCOY5ASdPK+YQL3LGeuSJCKp7gEHfkXjpqKaRtujxurTP55maBGl9vvy2E rMscKBwvHWTh0j7LM3+wgjyKu4O2Pe+o8r1rSI06gL36UNiPUV3arDgsuS5EussyZ8CCn6 a3cVh8TxJXv0R7LK/TyEANL/h3DcM3U= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=STEESIXx; spf=pass (imf01.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670445161; a=rsa-sha256; cv=none; b=5UMGJWf1Mgcy53kZRIWlygEdCFCl98cWU7qQxIYxVA9HpKgUfq4SGyDgHSLnzhjy48T2Lm w8BX29I4/Mt0KR3dBvuQNOYHDZhIm2i12giBLcwjxjwr4LolglUKkB4Io41Xg1OU8hzWqg C9SvI64XxDb5sOKnU5Bu8ud4GQv77XQ= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670445130; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=DOnWN4xjgNQJFSPMI7LR+EFJPU3TVyZrxc0oybMi1t4=; b=STEESIXxQ4Su06FjOvLQ/U+qSk6a3t+4gMZZvMaCdICid2j64kTGgPrfw2A3e0UUU1Y9zP b2IZ7UgGV29PA7MxhGU8g3rmub//BU4eHHEha0JS63vcD7JYt/XUAafkpt07w/VCrL0Htl eTweb8m5+hHKFCS2I5J8xtRKpf9h+sw= Received: from mail-qt1-f200.google.com (mail-qt1-f200.google.com [209.85.160.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-626-q97mcHK-NCCbScsbB59rnQ-1; Wed, 07 Dec 2022 15:31:59 -0500 X-MC-Unique: q97mcHK-NCCbScsbB59rnQ-1 Received: by mail-qt1-f200.google.com with SMTP id e18-20020ac845d2000000b003a6a5cbbebcso17640898qto.20 for ; Wed, 07 Dec 2022 12:31:59 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DOnWN4xjgNQJFSPMI7LR+EFJPU3TVyZrxc0oybMi1t4=; b=XJS+xDnOBWlnLCSOGWUnqAYUYhVgzl6tq1zk311SlbH8fvUtXdyEZ5l/YaJLObKOLO LmYFWtRNv1sxM35AnPXvrujYCurZkQF9ca3InRQA3ffe7cJhuP/6XpuOTRdhK9OvEf7M HoRTz2OyI4DXdx/R8SioDXgQnschKXpj00nnzesDHoRRAeF0sxYCzSN4e/nFyPZurV3f 096dgoGteM2UV0AKHFo6AtL+GdVqYdhDXpDlpCFPjZ8S5L6APLK/YpF/ZDJ+jVhvK8Qh W5L4dq52Ak2ftlOfPdm95ho+R7JXyTPtoGRY30M8lzA+hr9GuzynTMj9xU57xE/G+SxG tFcQ== X-Gm-Message-State: ANoB5pmzbJWP6vZgDtbUJ3eVmj37iohR5rhl+w43Lq4lF3sSdWRGFZRU 63tKfI6/K4FmcLsCD5LBYJh8CFlcuNp8BvUZjZ7c/GG1j2SPI0EU3a9v1ei3TWqitVYsT28rb25 iuP9HonYtj90ZkCMEjOOVlriMauqsoHDQNNFwoivd8iGySkTK8WrlcBLJZXlL X-Received: by 2002:ac8:7112:0:b0:3a6:88c8:2c36 with SMTP id z18-20020ac87112000000b003a688c82c36mr1643155qto.49.1670445118872; Wed, 07 Dec 2022 12:31:58 -0800 (PST) X-Google-Smtp-Source: AA0mqf7SKKVjMc7L3Uy3ewffwC9JgUfvzhyXrbwHzt/3mnovhEfXvkyWEB/JDyZxPGC4rLCS8j2F8A== X-Received: by 2002:ac8:7112:0:b0:3a6:88c8:2c36 with SMTP id z18-20020ac87112000000b003a688c82c36mr1643131qto.49.1670445118505; Wed, 07 Dec 2022 12:31:58 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id u17-20020a05620a455100b006fa22f0494bsm17916710qkp.117.2022.12.07.12.31.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 12:31:58 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Jann Horn , Andrea Arcangeli , peterx@redhat.com, James Houghton , Rik van Riel , Miaohe Lin , Nadav Amit , John Hubbard , Mike Kravetz , David Hildenbrand , Andrew Morton , Muchun Song Subject: [PATCH v2 09/10] mm/hugetlb: Introduce hugetlb_walk() Date: Wed, 7 Dec 2022 15:31:56 -0500 Message-Id: <20221207203156.651077-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221207203034.650899-1-peterx@redhat.com> References: <20221207203034.650899-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 235E94001A X-Rspam-User: X-Stat-Signature: cjph7e15x5qocaejcndkkzuco9fj5mt1 X-Spamd-Result: default: False [3.10 / 9.00]; BAYES_HAM(-6.00)[100.00%]; SORBS_IRL_BL(3.00)[209.85.160.200:received]; R_MISSING_CHARSET(2.50)[]; SUSPICIOUS_RECIPS(1.50)[]; SUBJECT_HAS_UNDERSCORES(1.00)[]; MID_CONTAINS_FROM(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DKIM_TRACE(0.00)[redhat.com:+]; R_SPF_ALLOW(0.00)[+ip4:170.10.133.0/24]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-HE-Tag: 1670445160-798657 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: huge_pte_offset() is the main walker function for hugetlb pgtables. The name is not really representing what it does, though. Instead of renaming it, introduce a wrapper function called hugetlb_walk() which will use huge_pte_offset() inside. Assert on the locks when walking the pgtable. Note, the vma lock assertion will be a no-op for private mappings. Signed-off-by: Peter Xu Reviewed-by: Mike Kravetz --- fs/hugetlbfs/inode.c | 4 +--- fs/userfaultfd.c | 6 ++---- include/linux/hugetlb.h | 39 +++++++++++++++++++++++++++++++++++++++ mm/hugetlb.c | 32 +++++++++++++------------------- mm/page_vma_mapped.c | 2 +- mm/pagewalk.c | 4 +--- 6 files changed, 57 insertions(+), 30 deletions(-) diff --git a/fs/hugetlbfs/inode.c b/fs/hugetlbfs/inode.c index fdb16246f46e..48f1a8ad2243 100644 --- a/fs/hugetlbfs/inode.c +++ b/fs/hugetlbfs/inode.c @@ -388,9 +388,7 @@ static bool hugetlb_vma_maps_page(struct vm_area_struct *vma, { pte_t *ptep, pte; - ptep = huge_pte_offset(vma->vm_mm, addr, - huge_page_size(hstate_vma(vma))); - + ptep = hugetlb_walk(vma, addr, huge_page_size(hstate_vma(vma))); if (!ptep) return false; diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index a602f008dde5..f31fe1a9f4c5 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -237,14 +237,12 @@ static inline bool userfaultfd_huge_must_wait(struct userfaultfd_ctx *ctx, unsigned long flags, unsigned long reason) { - struct mm_struct *mm = ctx->mm; pte_t *ptep, pte; bool ret = true; - mmap_assert_locked(mm); - - ptep = huge_pte_offset(mm, address, vma_mmu_pagesize(vma)); + mmap_assert_locked(ctx->mm); + ptep = hugetlb_walk(vma, address, vma_mmu_pagesize(vma)); if (!ptep) goto out; diff --git a/include/linux/hugetlb.h b/include/linux/hugetlb.h index 81efd9b9baa2..1c20cbbf3d22 100644 --- a/include/linux/hugetlb.h +++ b/include/linux/hugetlb.h @@ -2,6 +2,7 @@ #ifndef _LINUX_HUGETLB_H #define _LINUX_HUGETLB_H +#include #include #include #include @@ -196,6 +197,11 @@ pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, * huge_pte_offset(): Walk the hugetlb pgtable until the last level PTE. * Returns the pte_t* if found, or NULL if the address is not mapped. * + * IMPORTANT: we should normally not directly call this function, instead + * this is only a common interface to implement arch-specific walker. + * Please consider using the hugetlb_walk() helper to make sure of the + * correct locking is satisfied. + * * Since this function will walk all the pgtable pages (including not only * high-level pgtable page, but also PUD entry that can be unshared * concurrently for VM_SHARED), the caller of this function should be @@ -1229,4 +1235,37 @@ bool want_pmd_share(struct vm_area_struct *vma, unsigned long addr); #define flush_hugetlb_tlb_range(vma, addr, end) flush_tlb_range(vma, addr, end) #endif +static inline bool +__vma_shareable_flags_pmd(struct vm_area_struct *vma) +{ + return vma->vm_flags & (VM_MAYSHARE | VM_SHARED) && + vma->vm_private_data; +} + +/* + * Safe version of huge_pte_offset() to check the locks. See comments + * above huge_pte_offset(). + */ +static inline pte_t * +hugetlb_walk(struct vm_area_struct *vma, unsigned long addr, unsigned long sz) +{ +#if defined(CONFIG_HUGETLB_PAGE) && \ + defined(CONFIG_ARCH_WANT_HUGE_PMD_SHARE) && defined(CONFIG_LOCKDEP) + struct hugetlb_vma_lock *vma_lock = vma->vm_private_data; + + /* + * If pmd sharing possible, locking needed to safely walk the + * hugetlb pgtables. More information can be found at the comment + * above huge_pte_offset() in the same file. + * + * NOTE: lockdep_is_held() is only defined with CONFIG_LOCKDEP. + */ + if (__vma_shareable_flags_pmd(vma)) + WARN_ON_ONCE(!lockdep_is_held(&vma_lock->rw_sema) && + !lockdep_is_held( + &vma->vm_file->f_mapping->i_mmap_rwsem)); +#endif + return huge_pte_offset(vma->vm_mm, addr, sz); +} + #endif /* _LINUX_HUGETLB_H */ diff --git a/mm/hugetlb.c b/mm/hugetlb.c index f42399522805..e3500c087893 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -4814,7 +4814,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, } else { /* * For shared mappings the vma lock must be held before - * calling huge_pte_offset in the src vma. Otherwise, the + * calling hugetlb_walk() in the src vma. Otherwise, the * returned ptep could go away if part of a shared pmd and * another thread calls huge_pmd_unshare. */ @@ -4824,7 +4824,7 @@ int copy_hugetlb_page_range(struct mm_struct *dst, struct mm_struct *src, last_addr_mask = hugetlb_mask_last_page(h); for (addr = src_vma->vm_start; addr < src_vma->vm_end; addr += sz) { spinlock_t *src_ptl, *dst_ptl; - src_pte = huge_pte_offset(src, addr, sz); + src_pte = hugetlb_walk(src_vma, addr, sz); if (!src_pte) { addr |= last_addr_mask; continue; @@ -5028,7 +5028,7 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, hugetlb_vma_lock_write(vma); i_mmap_lock_write(mapping); for (; old_addr < old_end; old_addr += sz, new_addr += sz) { - src_pte = huge_pte_offset(mm, old_addr, sz); + src_pte = hugetlb_walk(vma, old_addr, sz); if (!src_pte) { old_addr |= last_addr_mask; new_addr |= last_addr_mask; @@ -5091,7 +5091,7 @@ static void __unmap_hugepage_range(struct mmu_gather *tlb, struct vm_area_struct last_addr_mask = hugetlb_mask_last_page(h); address = start; for (; address < end; address += sz) { - ptep = huge_pte_offset(mm, address, sz); + ptep = hugetlb_walk(vma, address, sz); if (!ptep) { address |= last_addr_mask; continue; @@ -5404,7 +5404,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, mutex_lock(&hugetlb_fault_mutex_table[hash]); hugetlb_vma_lock_read(vma); spin_lock(ptl); - ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); + ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) goto retry_avoidcopy; @@ -5442,7 +5442,7 @@ static vm_fault_t hugetlb_wp(struct mm_struct *mm, struct vm_area_struct *vma, * before the page tables are altered */ spin_lock(ptl); - ptep = huge_pte_offset(mm, haddr, huge_page_size(h)); + ptep = hugetlb_walk(vma, haddr, huge_page_size(h)); if (likely(ptep && pte_same(huge_ptep_get(ptep), pte))) { /* Break COW or unshare */ huge_ptep_clear_flush(vma, haddr, ptep); @@ -6227,7 +6227,7 @@ struct page *hugetlb_follow_page_mask(struct vm_area_struct *vma, return NULL; hugetlb_vma_lock_read(vma); - pte = huge_pte_offset(mm, haddr, huge_page_size(h)); + pte = hugetlb_walk(vma, haddr, huge_page_size(h)); if (!pte) goto out_unlock; @@ -6292,8 +6292,8 @@ long follow_hugetlb_page(struct mm_struct *mm, struct vm_area_struct *vma, * * Note that page table lock is not held when pte is null. */ - pte = huge_pte_offset(mm, vaddr & huge_page_mask(h), - huge_page_size(h)); + pte = hugetlb_walk(vma, vaddr & huge_page_mask(h), + huge_page_size(h)); if (pte) ptl = huge_pte_lock(h, mm, pte); absent = !pte || huge_pte_none(huge_ptep_get(pte)); @@ -6479,7 +6479,7 @@ unsigned long hugetlb_change_protection(struct vm_area_struct *vma, last_addr_mask = hugetlb_mask_last_page(h); for (; address < end; address += psize) { spinlock_t *ptl; - ptep = huge_pte_offset(mm, address, psize); + ptep = hugetlb_walk(vma, address, psize); if (!ptep) { address |= last_addr_mask; continue; @@ -6857,12 +6857,6 @@ void adjust_range_if_pmd_sharing_possible(struct vm_area_struct *vma, *end = ALIGN(*end, PUD_SIZE); } -static bool __vma_shareable_flags_pmd(struct vm_area_struct *vma) -{ - return vma->vm_flags & (VM_MAYSHARE | VM_SHARED) && - vma->vm_private_data; -} - void hugetlb_vma_lock_read(struct vm_area_struct *vma) { if (__vma_shareable_flags_pmd(vma)) { @@ -7028,8 +7022,8 @@ pte_t *huge_pmd_share(struct mm_struct *mm, struct vm_area_struct *vma, saddr = page_table_shareable(svma, vma, addr, idx); if (saddr) { - spte = huge_pte_offset(svma->vm_mm, saddr, - vma_mmu_pagesize(svma)); + spte = hugetlb_walk(svma, saddr, + vma_mmu_pagesize(svma)); if (spte) { get_page(virt_to_page(spte)); break; @@ -7387,7 +7381,7 @@ void hugetlb_unshare_all_pmds(struct vm_area_struct *vma) hugetlb_vma_lock_write(vma); i_mmap_lock_write(vma->vm_file->f_mapping); for (address = start; address < end; address += PUD_SIZE) { - ptep = huge_pte_offset(mm, address, sz); + ptep = hugetlb_walk(vma, address, sz); if (!ptep) continue; ptl = huge_pte_lock(h, mm, ptep); diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index 93e13fc17d3c..e97b2e23bd28 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -170,7 +170,7 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) return not_found(pvmw); /* when pud is not present, pte will be NULL */ - pvmw->pte = huge_pte_offset(mm, pvmw->address, size); + pvmw->pte = hugetlb_walk(vma, pvmw->address, size); if (!pvmw->pte) return false; diff --git a/mm/pagewalk.c b/mm/pagewalk.c index d98564a7be57..cb23f8a15c13 100644 --- a/mm/pagewalk.c +++ b/mm/pagewalk.c @@ -305,13 +305,11 @@ static int walk_hugetlb_range(unsigned long addr, unsigned long end, hugetlb_vma_lock_read(vma); do { next = hugetlb_entry_end(h, addr, end); - pte = huge_pte_offset(walk->mm, addr & hmask, sz); - + pte = hugetlb_walk(vma, addr & hmask, sz); if (pte) err = ops->hugetlb_entry(pte, hmask, addr, next, walk); else if (ops->pte_hole) err = ops->pte_hole(addr, next, -1, walk); - if (err) break; } while (addr = next, addr != end); From patchwork Wed Dec 7 20:31:58 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Peter Xu X-Patchwork-Id: 13067586 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6F816C63708 for ; Wed, 7 Dec 2022 20:32:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1555C8E0003; Wed, 7 Dec 2022 15:32:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DECB8E0001; Wed, 7 Dec 2022 15:32:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9ACA8E0003; Wed, 7 Dec 2022 15:32:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D3BCC8E0001 for ; Wed, 7 Dec 2022 15:32:03 -0500 (EST) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id A747BA0F23 for ; Wed, 7 Dec 2022 20:32:03 +0000 (UTC) X-FDA: 80216656926.02.ECD106C Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf25.hostedemail.com (Postfix) with ESMTP id 3E0CFA0015 for ; Wed, 7 Dec 2022 20:32:03 +0000 (UTC) Authentication-Results: imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iC7vgpMD; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1670445123; a=rsa-sha256; cv=none; b=Hu/UrtYIKzzIFy4mPe+tORb9ulyj4/h1yDFe2gzDdMv+hyecbFCM82oNWL8pNJU6nDbtc2 70BU7+rwsHUf1iN9Wnl93g8czfJfNWgq6+6nBjmPnCj1CoSFoYOIm7XDDxqj0Nuz9tTsfN UgmhCMO4ccreyaPYAeXWuul65MMfavk= ARC-Authentication-Results: i=1; imf25.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=iC7vgpMD; spf=pass (imf25.hostedemail.com: domain of peterx@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=peterx@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1670445123; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IJ6TW2E1hLLncShjpLbEpjeIsJ0SxMbIuMnmoV2Wa4Y=; b=wOi4/PueETuuBF0Rt/IlhnIJ98RmK1uRY9OzzV8B/NV92MAabcYIE2apzYXbae6pTA3xgh CcBl1DJqETtSahs6RJ5DsYhcwZUV9Pv3s4fRM07K1FrrpGO9pENN/DCnacUJmpcKYGImFN Bh5Jn8SZDhGYqe5QwzCVfjrDg5rqr64= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1670445122; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=IJ6TW2E1hLLncShjpLbEpjeIsJ0SxMbIuMnmoV2Wa4Y=; b=iC7vgpMD1kprpPJTbKrRyUt3svh17KKgXEa/L3QmSGYlHFLn5DPIs5YYg3IDRDvsC9643J Bp5b5a3GiA4/E/fI7Ir1alKClmjdETTQVhkdcpKsMc3JYNPb8fjbG+JO3RFVoeFcjT2TO6 Drg9L25wi3K1gZNMP1QY2DsK57xU/4w= Received: from mail-qv1-f71.google.com (mail-qv1-f71.google.com [209.85.219.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-526-G-J9OgZuNtmb4r7hUdKi1Q-1; Wed, 07 Dec 2022 15:32:01 -0500 X-MC-Unique: G-J9OgZuNtmb4r7hUdKi1Q-1 Received: by mail-qv1-f71.google.com with SMTP id o13-20020a056214108d00b004c6fb4f16dcso36479452qvr.6 for ; Wed, 07 Dec 2022 12:32:01 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=IJ6TW2E1hLLncShjpLbEpjeIsJ0SxMbIuMnmoV2Wa4Y=; b=BQ4Li0HbMzvBREgMAyhUu2LICKLWxIX5XvwIjlPtLvN+E5XBhnLOwe37pmbnhuF4GC kLEJHjbUjaJ3vey4/uQm+lRGk4ovg5Mun8HJZOap00d+7FeC3W36vny/fPgRVJvdWV7b Lxq8iVnRr6eD/rG/+FTz3ZAS/M6AKMaYMGB8pLbHZBsnD9g9WsyqYNtIH/9LSOtNwfYG W9Z/lAzi5XXz8R+xydibh2SKcfgvzey+WIao7GvXrb8blNKlKvUoiPCAzDf+kLAwJXtJ 90/lgw+JMr5Iz2GE541hXaaY6kPlYxnxUDDIx0VhJRdHyxrIR4B4NNpNOua/PHgt+T9i Ir0w== X-Gm-Message-State: ANoB5pnQdXXLl8YIIcauzSBSw5lx9IZPhBJr7jSSaC9d3OIBSGdDJbTu AxJdmTxsT0gyuw162T5RntSDG4Nz2uTW7826TmuVm4aGqDDtRBO/eewQE6m+uxVNTOreyF0Li0x K708HRzkHQL/tvPWmGQP8dJUlxTEoeGs9KLnlJlTeUUmapeY5jh53SU+MJzw1 X-Received: by 2002:ad4:4709:0:b0:4c7:629e:7a70 with SMTP id qb9-20020ad44709000000b004c7629e7a70mr1636782qvb.44.1670445120796; Wed, 07 Dec 2022 12:32:00 -0800 (PST) X-Google-Smtp-Source: AA0mqf55/5Slqm2ol0UT7nKwE62yK11661Qb4rAHiirqSdnnW+OXObDyf5lhkoXKQEtl720joxSvoQ== X-Received: by 2002:ad4:4709:0:b0:4c7:629e:7a70 with SMTP id qb9-20020ad44709000000b004c7629e7a70mr1636758qvb.44.1670445120489; Wed, 07 Dec 2022 12:32:00 -0800 (PST) Received: from x1n.redhat.com (bras-base-aurron9127w-grc-46-70-31-27-79.dsl.bell.ca. [70.31.27.79]) by smtp.gmail.com with ESMTPSA id bm8-20020a05620a198800b006fa8299b4d5sm18007118qkb.100.2022.12.07.12.31.59 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 07 Dec 2022 12:32:00 -0800 (PST) From: Peter Xu To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Jann Horn , Andrea Arcangeli , peterx@redhat.com, James Houghton , Rik van Riel , Miaohe Lin , Nadav Amit , John Hubbard , Mike Kravetz , David Hildenbrand , Andrew Morton , Muchun Song Subject: [PATCH v2 10/10] mm/hugetlb: Document why page_vma_mapped_walk() is safe to walk Date: Wed, 7 Dec 2022 15:31:58 -0500 Message-Id: <20221207203158.651092-1-peterx@redhat.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20221207203034.650899-1-peterx@redhat.com> References: <20221207203034.650899-1-peterx@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-type: text/plain X-Rspam-User: X-Spamd-Result: default: False [7.55 / 9.00]; SORBS_IRL_BL(3.00)[209.85.219.71:received]; R_MISSING_CHARSET(2.50)[]; BAYES_HAM(-1.55)[84.40%]; SUSPICIOUS_RECIPS(1.50)[]; MID_CONTAINS_FROM(1.00)[]; SUBJECT_HAS_UNDERSCORES(1.00)[]; RCVD_NO_TLS_LAST(0.10)[]; MIME_GOOD(-0.10)[text/plain]; BAD_REP_POLICIES(0.10)[]; PREVIOUSLY_DELIVERED(0.00)[linux-mm@kvack.org]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_TWELVE(0.00)[14]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_ALLOW(0.00)[redhat.com:s=mimecast20190719]; MIME_TRACE(0.00)[0:+]; TAGGED_RCPT(0.00)[]; DMARC_POLICY_ALLOW(0.00)[redhat.com,none]; TO_MATCH_ENVRCPT_SOME(0.00)[]; ARC_SIGNED(0.00)[hostedemail.com:s=arc-20220608:i=1]; DKIM_TRACE(0.00)[redhat.com:+]; R_SPF_ALLOW(0.00)[+ip4:170.10.133.0/24]; RCVD_COUNT_THREE(0.00)[4]; TO_DN_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[] X-Rspamd-Queue-Id: 3E0CFA0015 X-Rspamd-Server: rspam01 X-Stat-Signature: 45niftg3o3as3ygjzg4cikweegm95nom X-HE-Tag: 1670445123-448413 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Taking vma lock here is not needed for now because all potential hugetlb walkers here should have i_mmap_rwsem held. Document the fact. Reviewed-by: Mike Kravetz Signed-off-by: Peter Xu --- mm/page_vma_mapped.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/mm/page_vma_mapped.c b/mm/page_vma_mapped.c index e97b2e23bd28..2e59a0419d22 100644 --- a/mm/page_vma_mapped.c +++ b/mm/page_vma_mapped.c @@ -168,8 +168,14 @@ bool page_vma_mapped_walk(struct page_vma_mapped_walk *pvmw) /* The only possible mapping was handled on last iteration */ if (pvmw->pte) return not_found(pvmw); - - /* when pud is not present, pte will be NULL */ + /* + * NOTE: we don't need explicit lock here to walk the + * hugetlb pgtable because either (1) potential callers of + * hugetlb pvmw currently holds i_mmap_rwsem, or (2) the + * caller will not walk a hugetlb vma (e.g. ksm or uprobe). + * When one day this rule breaks, one will get a warning + * in hugetlb_walk(), and then we'll figure out what to do. + */ pvmw->pte = hugetlb_walk(vma, pvmw->address, size); if (!pvmw->pte) return false;