From patchwork Mon Jan 27 09:35:22 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13951123 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C2686C0218C for ; Mon, 27 Jan 2025 09:38:22 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=woVTLCWPTbNjUwt1e6fz+gqM2HazmtNFdMz/pHjJKmU=; b=rKgyjSGPHkpZq8 R6HhcHjZQoskH9H3tUNJ8OFxPXFUXSCd6qTkAqqogWCYDjt5CYM5hP5rsp7txcny2+d4hmm5+DRe0 ZlEl3MnunQz32SG3B0zApLOJ5e4KAnrh9oMqSBGG2A4e2OnzzxFN5MkVLKFbO0jNRJcEf7vPQerYt 25+cA5g2NKKIgWiwPJ4psbVIa+GrxMJw8V/3GREXZ8ek7gwwWRjiNWqk1dLLAt24xwEHDFh/a1Uql ds/sQfgCaIVDAD62omfAiNeJDkpELdYGSyQl6O6BR0c5IwGkK/rWajTkok+B/y7fpL7Bi7sUD0jyY JGBmMzIzz7jVYDCQ5lzg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tcLZS-000000021eW-07rg; Mon, 27 Jan 2025 09:38:18 +0000 Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tcLXu-000000021Is-0okx for linux-riscv@lists.infradead.org; Mon, 27 Jan 2025 09:36:43 +0000 Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-43622267b2eso43420725e9.0 for ; Mon, 27 Jan 2025 01:36:41 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1737970601; x=1738575401; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=YlgUGAmuTWGVKaOGr6DQU8bhY7gyR7CYcPHPJxmYPTE=; b=miwJcCjaGI49ihA7LbNxXDyRqi2KrkBbF968D7K4yD4KX3HMy7EGT0xzxh4ifANlS6 tAuQyC9Gafoym96jMd81oxfxikjvksNOdvLJkDzPH3q98e2tmUYDNLRRAS+ncAyXuuI7 B7Ew+vvYBJP7aH+Bomuqt4amu96PQmbjLhV1BhASNKjKtvOTXf4aXpPM/3cGKixA12Lx CLNBudYry9u3rY0oXPn1Ap84NQBz4DBQ/X+MT+Y6TH07ceYOttS57yfTzLbCW1JL3hAA NV5M8mfuQsOxuvTjcpyz1HH96ZqotHZ3Mt8pZs8SvCEUatA8Ov8+QFuEezIqnObUr3sF y9WQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737970601; x=1738575401; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=YlgUGAmuTWGVKaOGr6DQU8bhY7gyR7CYcPHPJxmYPTE=; b=PstwHt6GKW6UeG2fw0Bfa9pai/64oUZ5RyDUBqTvMqd7wWug7wPY5Y4m5pubG2jBLp 4VKW1qwFu00ku/7OjijlzhVzg/yJtYEiHr4MBOxnTMihYnIdnUiFmRkDiQim/MGNEjOh QmXR2qJtxhxloxrk131Hxkt6H6A/9xdzr4QnSQ7YdhLlvirnZVlWkMZU+/ktCJeGYEq4 8HfO7TpbS7R6Dm6Ap7cBD81lwU2Rg+r8zSU1LB3aNwUz74p2lFBCzCuWwhgUAVoX2C4G cPRZd87grkjrGERtrw3Ane3iosqvs2ChuZEfpvahZHN6YKDfl8lpK1K3CVjyP3R4aL/3 s3sQ== X-Forwarded-Encrypted: i=1; AJvYcCWYuVa0VgLok71VqNZZ7KgEUg8P5S0lFZaLy6ftBWiZvaHk6fP5YUAC1zIpHKVSv0HzoG7oO0jNqziQbg==@lists.infradead.org X-Gm-Message-State: AOJu0YxXV2hPx2VFuS6deg4/i4EKkx5njlAYOoGgvUBQnJsXaNFF25xc i6tNPypnsxxi+JYKQ+R5hNYDC79R9SJipBSxCdfeZpM/692tIH07swBVeXEIQF0= X-Gm-Gg: ASbGnctM6cYjHeS/rSvVvMl+2UKZxytwLpxLsOoODPcB185fwdnkeRDUR5QKmuWdMxh Ti/a3c2r/dPxh7H2CTiBhQGhvggwLx0lKuZdJzFEVsf/BbsfMWFKMZYiwWb0vbNOoR4+HLSyi6I 7LzRcMU/q9nHDxcuz/BbprhzavZ3R1K4sowMyaaxIiOxOZKAvYMM+u+Yo/FrCciTAwCy9mcSNYa typSOVeP2K3Oj2gNkKQ/tg7YNgXNsQuj5j5JW0YblYFuYxX+QIgHS1u7CwEp8L9Y0LDy5jXRYzT cnR+MpdDrmIqWac4CRDJyUzFJNQDk+E= X-Google-Smtp-Source: AGHT+IHUUXHJ5coMzJl68ATLI/2WB7Q+b+nPXfERL8/PDwCduzlj4e0CTpfsp9bAQ08qd/jskgeAvg== X-Received: by 2002:a05:6000:1445:b0:386:37f5:99f6 with SMTP id ffacd0b85a97d-38bf59ed62fmr33321912f8f.53.1737970600263; Mon, 27 Jan 2025 01:36:40 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3e22:3e78:ce5a:32c3]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c2a176490sm10314205f8f.1.2025.01.27.01.36.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jan 2025 01:36:39 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v4 1/9] riscv: Safely remove huge_pte_offset() when manipulating NAPOT ptes Date: Mon, 27 Jan 2025 10:35:22 +0100 Message-Id: <20250127093530.19548-2-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250127093530.19548-1-alexghiti@rivosinc.com> References: <20250127093530.19548-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250127_013642_230444_505BB185 X-CRM114-Status: UNSURE ( 9.80 ) X-CRM114-Notice: Please train this message. X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The pte_t pointer is expected to point to the first entry of the NAPOT mapping so no need to use huge_pte_offset(), similarly to what is done in arm64. Signed-off-by: Alexandre Ghiti --- arch/riscv/mm/hugetlbpage.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 42314f093922..6b09cd1ef41c 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -276,7 +276,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, order = napot_cont_order(pte); pte_num = napot_pte_num(order); - ptep = huge_pte_offset(mm, addr, napot_cont_size(order)); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); if (pte_dirty(orig_pte)) @@ -322,7 +321,6 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, order = napot_cont_order(pte); pte_num = napot_pte_num(order); - ptep = huge_pte_offset(mm, addr, napot_cont_size(order)); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); From patchwork Mon Jan 27 09:35:23 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13951143 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F2FC7C0218C for ; Mon, 27 Jan 2025 09:39:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CCu4mRWzudTXH+JWJtuYCBJWHHe728RHJS98PVCI4GE=; b=0P6o6dTjtAKV1b jjW9rof2KkpNzlJjLxDShQ2dCCMxakSKA+Nq0lovcv0JOE5sFCQ7YVR6tFQ9X++TJdyeSOurcUmUy apAgcqkNcFcqRJ6H80KwKsz5pvd14HqA94XG5CZQuikxJvtKgWF8a85FW01xYPF3+47NjYxjrE8NL tW9sQkyFvvdVwLq3lBuPtFnpxjM7KYky0VBGr90O25ZRJrZcs0ptcs9eGIDTLwLpw/qj7GsphS8l8 hI/G7BKwqLCHMscfbUjehr6HJ+fhnwMORBYs19B9OLPzc/nxQkp4D8/KGOi9Py712PRSHzI29KDQR g5DQc3gReqbLALsztOXQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tcLah-000000021yS-2bpf; Mon, 27 Jan 2025 09:39:35 +0000 Received: from mail-wm1-x336.google.com ([2a00:1450:4864:20::336]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tcLYu-000000021a0-0uGp for linux-riscv@lists.infradead.org; Mon, 27 Jan 2025 09:37:45 +0000 Received: by mail-wm1-x336.google.com with SMTP id 5b1f17b1804b1-4361b0ec57aso44708245e9.0 for ; Mon, 27 Jan 2025 01:37:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1737970662; x=1738575462; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Pa3kN5Pj/U9dm1V/qwQhRimY+zw1O6zUoccW70uHpSw=; b=VmJpENrpN+vkDYTmpDYsIe4wB6gvavnoG9JYIy73Q0cmbhOXQYGKQ8eeFI9x9Jq8cR rxPYO4rLswF3o3OZVR5W2/cTB70etlDHjrMe/yF4yVTwH3xCMJkqVhjhZ+J1ttLktF5l s2suiG7hrWPFNCip3y0ZfhiFxEndnaO3eke7nsZML+ih08gL4LYFjUdzc12P1KLeJ+jD r708VaV05rDGoJ3MSDNwrSA9w6nTzTe3EJRDA5iyJnZGC6irEL6QiJFCvbtHM86NMkMx fNMOkVNDW8Q1p7G4k4ntajurS0wz2csCIJ8EXL49vRv+upuu9FsJjiLIo9/TGsfa5266 QtUA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737970662; x=1738575462; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Pa3kN5Pj/U9dm1V/qwQhRimY+zw1O6zUoccW70uHpSw=; b=KM36ZHet3aMiOeMdLAOaf2mXWsAhC/DgtFnQw7JACWdCQxh5Yx10OeVHs9oPdN7sN+ O4nS4VyWH424A2vBZeS7U2BKXgFTcICqnSM/t+HJ53jUVfFI28GvJUYcKX2R5cIk1iHV S7+tyeUai2gsy4ZV4cnRy/Y18LhGzpVCadqiq0+vFk1BFC6KpSJxLAFEMU1GNIz49woJ +qFbcoTT0QmG7sP3l3NIkc7fAHpxSbv1W70T+UrXAv2YfiWFGXQgddjAG3BdoSyHU77J 9hXIlNXa8yuq2w8Z+zchHP4QHC51XMhe+zTf+k48IqHKPOkQRAwOAZd1M0yQ3JnNCevO WrgQ== X-Forwarded-Encrypted: i=1; AJvYcCUW0u4MzNoQilUiBrDh1pcxgwwosVbdCnAH1+cKbqfd2jIvBEE9iQ5I8n9lAKkL4YTMlDC6G2SHzuBwrw==@lists.infradead.org X-Gm-Message-State: AOJu0YziyyLauYr0Rt7HNgxPTZKlhCZFOL4l/nH7F+s6mHIs5uJoD4Ke yt5IbmOlV65uPEPcBuhJE/4PlFXq0eYBxY1Yj7XdE3CqOGnZN8AqLDd3uk6bO/Q= X-Gm-Gg: ASbGnctxw7WLAumDltu8hr/+SKGEHkn0/6xpnZ7iqKetadmg/cReRxvE/41FzpefBlk ZJoRk/jbv3nbGbgwEwxI/c7M7qQwozHlkcYUrLeMTXHsLmd2JJ0ZcbN1sN0uUqWUnoQSnQd9Des MVPtVM/3bEnSI+v86z6kFiKnlq4Y8AnXBRJvhDXMDPWNtGI9Me8BhWOv3q7WjjyvU1ECqSLTcza ntMa632r/+RzQVarn2yPlIL8yI4TctpdqeQiLNwPAxb2t7Q7DyLqkevWrZ7DxtLyxAL904u/OmH /6a992BqU03VSs81IRLuGuPYkL3TEW8= X-Google-Smtp-Source: AGHT+IHlygR1Rem42sW1brqnRYg+b7cppikaFDk94wrGvPPrCI76J4t5Wx9vdAYW3HFrhkgLyStnBg== X-Received: by 2002:a05:600c:4f05:b0:434:fe62:28c1 with SMTP id 5b1f17b1804b1-438913ef83bmr346516855e9.18.1737970662539; Mon, 27 Jan 2025 01:37:42 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3e22:3e78:ce5a:32c3]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-438b1718741sm100844135e9.0.2025.01.27.01.37.41 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jan 2025 01:37:42 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v4 2/9] riscv: Restore the pfn in a NAPOT pte when manipulated by core mm code Date: Mon, 27 Jan 2025 10:35:23 +0100 Message-Id: <20250127093530.19548-3-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250127093530.19548-1-alexghiti@rivosinc.com> References: <20250127093530.19548-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250127_013744_248554_A8379761 X-CRM114-Status: GOOD ( 24.76 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org The core mm code expects to be able to extract the pfn from a pte. NAPOT mappings work differently since its ptes actually point to the first pfn of the mapping, the other bits being used to encode the size of the mapping. So modify ptep_get() so that it returns a pte value that contains the *real* pfn (which is then different from what the HW expects) and right before storing the ptes to the page table, reset the pfn LSBs to the size of the mapping. And make sure that all NAPOT mappings are set using set_ptes(). Signed-off-by: Alexandre Ghiti --- arch/riscv/include/asm/pgtable-64.h | 11 ++++ arch/riscv/include/asm/pgtable.h | 91 ++++++++++++++++++++++++++--- arch/riscv/mm/hugetlbpage.c | 9 +-- 3 files changed, 96 insertions(+), 15 deletions(-) diff --git a/arch/riscv/include/asm/pgtable-64.h b/arch/riscv/include/asm/pgtable-64.h index 0897dd99ab8d..cddbe426f618 100644 --- a/arch/riscv/include/asm/pgtable-64.h +++ b/arch/riscv/include/asm/pgtable-64.h @@ -104,6 +104,17 @@ enum napot_cont_order { #define napot_cont_mask(order) (~(napot_cont_size(order) - 1UL)) #define napot_pte_num(order) BIT(order) +static inline bool is_napot_order(unsigned int order) +{ + unsigned int napot_order; + + for_each_napot_order(napot_order) + if (order == napot_order) + return true; + + return false; +} + #ifdef CONFIG_RISCV_ISA_SVNAPOT #define HUGE_MAX_HSTATE (2 + (NAPOT_ORDER_MAX - NAPOT_CONT_ORDER_BASE)) #else diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 050fdc49b5ad..82b264423b25 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -296,6 +296,8 @@ static inline unsigned long pte_napot(pte_t pte) return pte_val(pte) & _PAGE_NAPOT; } +#define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) + static inline pte_t pte_mknapot(pte_t pte, unsigned int order) { int pos = order - 1 + _PAGE_PFN_SHIFT; @@ -305,6 +307,12 @@ static inline pte_t pte_mknapot(pte_t pte, unsigned int order) return __pte((pte_val(pte) & napot_mask) | napot_bit | _PAGE_NAPOT); } +/* pte at entry must *not* encode the mapping size in the pfn LSBs. */ +static inline pte_t pte_clear_napot(pte_t pte) +{ + return __pte(pte_val(pte) & ~_PAGE_NAPOT); +} + #else static __always_inline bool has_svnapot(void) { return false; } @@ -314,17 +322,14 @@ static inline unsigned long pte_napot(pte_t pte) return 0; } +#define pte_valid_napot(pte) false + #endif /* CONFIG_RISCV_ISA_SVNAPOT */ /* Yields the page frame number (PFN) of a page table entry */ static inline unsigned long pte_pfn(pte_t pte) { - unsigned long res = __page_val_to_pfn(pte_val(pte)); - - if (has_svnapot() && pte_napot(pte)) - res = res & (res - 1UL); - - return res; + return __page_val_to_pfn(pte_val(pte)); } #define pte_page(x) pfn_to_page(pte_pfn(x)) @@ -559,8 +564,13 @@ static inline void __set_pte_at(struct mm_struct *mm, pte_t *ptep, pte_t pteval) #define PFN_PTE_SHIFT _PAGE_PFN_SHIFT -static inline void set_ptes(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pteval, unsigned int nr) +static inline pte_t __ptep_get(pte_t *ptep) +{ + return READ_ONCE(*ptep); +} + +static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pteval, unsigned int nr) { page_table_check_ptes_set(mm, ptep, pteval, nr); @@ -569,10 +579,13 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, if (--nr == 0) break; ptep++; + + if (unlikely(pte_valid_napot(pteval))) + continue; + pte_val(pteval) += 1 << _PAGE_PFN_SHIFT; } } -#define set_ptes set_ptes static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -627,6 +640,66 @@ static inline int ptep_clear_flush_young(struct vm_area_struct *vma, return ptep_test_and_clear_young(vma, address, ptep); } +#ifdef CONFIG_RISCV_ISA_SVNAPOT +static inline void set_ptes(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pteval, unsigned int nr) +{ + if (unlikely(pte_valid_napot(pteval))) { + unsigned int order = ilog2(nr); + + if (!is_napot_order(order)) { + /* + * Something's weird, we are given a NAPOT pte but the + * size of the mapping is not a known NAPOT mapping + * size, so clear the NAPOT bit and map this without + * NAPOT support: core mm only manipulates pte with the + * real pfn so we know the pte is valid without the N + * bit. + */ + pr_err("Incorrect NAPOT mapping, resetting.\n"); + pteval = pte_clear_napot(pteval); + } else { + /* + * NAPOT ptes that arrive here only have the N bit set + * and their pfn does not contain the mapping size, so + * set that here. + */ + pteval = pte_mknapot(pteval, order); + } + } + + __set_ptes(mm, addr, ptep, pteval, nr); +} +#define set_ptes set_ptes + +static inline pte_t ptep_get(pte_t *ptep) +{ + pte_t pte = __ptep_get(ptep); + + /* + * The pte we load has the N bit set and the size of the mapping in + * the pfn LSBs: keep the N bit and replace the mapping size with + * the *real* pfn since the core mm code expects to find it there. + * The mapping size will be reset just before being written to the + * page table in set_ptes(). + */ + if (unlikely(pte_valid_napot(pte))) { + unsigned int order = napot_cont_order(pte); + int pos = order - 1 + _PAGE_PFN_SHIFT; + unsigned long napot_mask = ~GENMASK(pos, _PAGE_PFN_SHIFT); + pte_t *orig_ptep = PTR_ALIGN_DOWN(ptep, sizeof(*ptep) * napot_pte_num(order)); + + pte = __pte((pte_val(pte) & napot_mask) + ((ptep - orig_ptep) << _PAGE_PFN_SHIFT)); + } + + return pte; +} +#define ptep_get ptep_get +#else +#define set_ptes __set_ptes +#define ptep_get __ptep_get +#endif /* CONFIG_RISCV_ISA_SVNAPOT */ + #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 6b09cd1ef41c..59ed26ce6857 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -256,8 +256,7 @@ void set_huge_pte_at(struct mm_struct *mm, clear_flush(mm, addr, ptep, pgsize, pte_num); - for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) - set_pte_at(mm, addr, ptep, pte); + set_ptes(mm, addr, ptep, pte, pte_num); } int huge_ptep_set_access_flags(struct vm_area_struct *vma, @@ -284,8 +283,7 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (pte_young(orig_pte)) pte = pte_mkyoung(pte); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) - set_pte_at(mm, addr, ptep, pte); + set_ptes(mm, addr, ptep, pte, pte_num); return true; } @@ -325,8 +323,7 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, orig_pte = pte_wrprotect(orig_pte); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) - set_pte_at(mm, addr, ptep, orig_pte); + set_ptes(mm, addr, ptep, orig_pte, pte_num); } pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, From patchwork Mon Jan 27 09:35:24 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13951145 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 94431C0218F for ; Mon, 27 Jan 2025 09:41:00 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=SsHQhor+kOWXP5pX2mlHyLvYM84uChcTqhBTtepwXJw=; b=BZOZoWAmLue7pX XVpGjYORRFwCv5FgsY+NmtOq40wqykDzxkdwEpRQCOT+F/EY6JBw9tHUHDX5dh4p4JEClAfhDZVL4 11UemdBvUmGmnE8OD+fDKCiw4y47mUmPjIz5BKC4unEZc58FeaqlricZj980L3uLhI7ezSvFb3pkG L+k1UzXqSqWRsaIk522qIg60lkxOkSg8JfvXxReGxy9i6rV9FZmE6GwWDLcoBUge5hBzIr6MMLOX4 drHfHONFxL+p/ED+dIEo756FFUAVf7ogFR1PdqjdI8uBjIvqe6pbF5tquHyToSSrqn3kBTBIvxv6N yPhT7Jc79YJyGfr5jGpA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tcLbx-000000022Ox-3exQ; Mon, 27 Jan 2025 09:40:53 +0000 Received: from mail-wr1-x429.google.com ([2a00:1450:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tcLZt-000000021kq-48Bo for linux-riscv@lists.infradead.org; Mon, 27 Jan 2025 09:38:48 +0000 Received: by mail-wr1-x429.google.com with SMTP id ffacd0b85a97d-38634c35129so3842787f8f.3 for ; Mon, 27 Jan 2025 01:38:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1737970724; x=1738575524; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=kgdyYq7XD2Yilje0JoNAbIWlWrTDbtDHltbsUNmCh+Y=; b=vyREzZY/83tfkPxoEZaN+E9zAcnIut7/dp6ET7Dk9u4upgWRJ7y7BeR6yZjEdj5stD VRX0vp9fGVPHEWwS7QeXBtuvEK0QbMfQjMi1E2tXGQjSq7MD5l1oVP3xu4AhX0M8OZtD LH8bfJxYwt/+/lxbvPNvql+OlwtYdH1cu1qRAvcfMET/9JaC/AwmcOqvZ8N4vRg1JIIa XUR4hKPl6zsQOb/P8qVw+uU6VyM6VtrFL7IDOx7gGrv4Qg5E6X0PzRF49ahyGYL+nmQ/ qo1jlEtocAFOc1pdtsuHo0CYQGY669WwTyLCcIG56YDj3CuQckadmeT9stitZWBqAl0B RebQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737970724; x=1738575524; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=kgdyYq7XD2Yilje0JoNAbIWlWrTDbtDHltbsUNmCh+Y=; b=MbcoW19xkYVKGCPg63htr651Xjv81lEMUVK47S7TzgoAVR5Lw+h5p/NI0hg85euCtc Eyo3isAlrZAyXYtaSXjty92dObYIR1g9nQ3hYbIlvdGNYt9Co99AYY9B0AaLr6L8PDj6 H5aYtmMIDkX1+D8+JLKTdcbzaB63pmHw9gmdzA8k/K1xPUmhZUPLbNICq0XLraaIpaRA trCr01M9y4OFLNCiJ+52KyxE154STOBcGZz0Q9+LvP/gLGrjn62326VuONOhOcj2nPwz k6v9KZ31WEW79dlOh7YWShxA5Nlmm3FuJlX6r5xAa6JTHaMm9pz1wcEpqGBzrc2M+8pG Nkmg== X-Forwarded-Encrypted: i=1; AJvYcCWbqr544HmSo5k1x34zkchmijY1b3hkP6o3pWR778XF4gZ0P7CS5O49sK/q7kEfuDsPOxQVQBEaht4JhQ==@lists.infradead.org X-Gm-Message-State: AOJu0Yxi4qOW395RLtXyfcv9AaX7rkD0IR2oyIviTeu7tzQ2WYKpu9ct yuxmS7YKO/1AAn/xgR6/xXNr6S8Fg5DugZmX851ZbV24ErfOtUFphgQEoRxAPcg= X-Gm-Gg: ASbGnctoiC97Z8p3C4ZkWix7fWD19r/T4w/+f+Ijx6DOwmft7JSRYalB2evOLW62OQ0 MCZnxaDr+u5UplgSKz+qUVo4+nsiq8uRXjGuWW2poLNT8xHyHJdOjC4y6bSKwq13YcP2QgIqH75 5WmJK0SLORmpKfDOcbg+HhzpJX4X8ocOyn0Ebsu+ITVpKP3V2vwkCku0wdPUv+9lBSxonB+HwZg VvwbJQG3fjKi1pKXo8rrxu6/BiYiMZ3IZWHUbCPuUBq89HfRZpZjS3/Admqj/ufruD5u1WHa4Bs HV40JYVmV+jDomRvElFrCozZ+2DNYp0= X-Google-Smtp-Source: AGHT+IHF4Wz5gcDgdKwuRpBVbD6zHBbXFDhSzsPT6V4k0QqTqrpmiB0ymgyz8XwZE3Dv5j9kSPHaVA== X-Received: by 2002:a5d:4312:0:b0:388:e377:8a1b with SMTP id ffacd0b85a97d-38bf57950eemr28290614f8f.28.1737970724282; Mon, 27 Jan 2025 01:38:44 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3e22:3e78:ce5a:32c3]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c2a1baf65sm10687893f8f.64.2025.01.27.01.38.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jan 2025 01:38:43 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v4 3/9] mm: Use common huge_ptep_get() function for riscv/arm64 Date: Mon, 27 Jan 2025 10:35:24 +0100 Message-Id: <20250127093530.19548-4-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250127093530.19548-1-alexghiti@rivosinc.com> References: <20250127093530.19548-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250127_013846_017863_E5AAF811 X-CRM114-Status: GOOD ( 23.38 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/Kconfig | 1 + arch/arm64/include/asm/hugetlb.h | 3 +- arch/arm64/include/asm/pgtable.h | 48 +++++++++++++++++++++++++--- arch/arm64/mm/hugetlbpage.c | 55 ++------------------------------ arch/riscv/Kconfig | 1 + arch/riscv/include/asm/hugetlb.h | 6 ++-- arch/riscv/include/asm/pgtable.h | 36 +++++++++++++++++++++ arch/riscv/mm/hugetlbpage.c | 45 ++++++-------------------- include/linux/hugetlb_contpte.h | 12 +++++++ mm/Kconfig | 3 ++ mm/Makefile | 1 + mm/hugetlb_contpte.c | 44 +++++++++++++++++++++++++ 12 files changed, 157 insertions(+), 98 deletions(-) create mode 100644 include/linux/hugetlb_contpte.h create mode 100644 mm/hugetlb_contpte.c diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 100570a048c5..fb85d33bfe98 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -111,6 +111,7 @@ config ARM64 select ARCH_WANT_DEFAULT_BPF_JIT select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT select ARCH_WANT_FRAME_POINTERS + select ARCH_WANT_GENERAL_HUGETLB_CONTPTE select ARCH_WANT_HUGE_PMD_SHARE if ARM64_4K_PAGES || (ARM64_16K_PAGES && !ARM64_VA_BITS_36) select ARCH_WANT_LD_ORPHAN_WARN select ARCH_WANTS_EXECMEM_LATE if EXECMEM diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index c6dff3e69539..27d7f4bdd724 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -13,6 +13,7 @@ #include #include #include +#include #ifdef CONFIG_ARCH_ENABLE_HUGEPAGE_MIGRATION #define arch_hugetlb_migration_supported arch_hugetlb_migration_supported @@ -53,8 +54,6 @@ extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, #define __HAVE_ARCH_HUGE_PTE_CLEAR extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz); -#define __HAVE_ARCH_HUGE_PTEP_GET -extern pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); void __init arm64_hugetlb_cma_reserve(void); diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index 6986345b537a..cebbfcfb0e53 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -420,9 +420,10 @@ static inline pte_t pte_advance_pfn(pte_t pte, unsigned long nr) return pfn_pte(pte_pfn(pte) + nr, pte_pgprot(pte)); } -static inline void __set_ptes(struct mm_struct *mm, - unsigned long __always_unused addr, - pte_t *ptep, pte_t pte, unsigned int nr) +static inline void ___set_ptes(struct mm_struct *mm, + unsigned long __always_unused addr, + pte_t *ptep, pte_t pte, unsigned int nr, + size_t pgsize) { page_table_check_ptes_set(mm, ptep, pte, nr); __sync_cache_and_tags(pte, nr); @@ -433,10 +434,15 @@ static inline void __set_ptes(struct mm_struct *mm, if (--nr == 0) break; ptep++; - pte = pte_advance_pfn(pte, 1); + pte = pte_advance_pfn(pte, pgsize >> PAGE_SHIFT); } } +#define __set_ptes(mm, addr, ptep, pte, nr) \ + ___set_ptes(mm, addr, ptep, pte, nr, PAGE_SIZE) + +#define set_contptes ___set_ptes + /* * Hugetlb definitions. */ @@ -1825,6 +1831,40 @@ static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, #endif /* CONFIG_ARM64_CONTPTE */ +static inline int arch_contpte_get_num_contig(pte_t *ptep, + unsigned long size, + size_t *pgsize) +{ + int contig_ptes = 0; + + if (pgsize) + *pgsize = size; + + switch (size) { +#ifndef __PAGETABLE_PMD_FOLDED + case PUD_SIZE: + if (pud_sect_supported()) + contig_ptes = 1; + break; +#endif + case PMD_SIZE: + contig_ptes = 1; + break; + case CONT_PMD_SIZE: + if (pgsize) + *pgsize = PMD_SIZE; + contig_ptes = CONT_PMDS; + break; + case CONT_PTE_SIZE: + if (pgsize) + *pgsize = PAGE_SIZE; + contig_ptes = CONT_PTES; + break; + } + + return contig_ptes; +} + #endif /* !__ASSEMBLY__ */ #endif /* __ASM_PGTABLE_H */ diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 3215adf48a1b..3458461adb90 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -98,57 +98,6 @@ static int find_num_contig(struct mm_struct *mm, unsigned long addr, return CONT_PTES; } -static inline int num_contig_ptes(unsigned long size, size_t *pgsize) -{ - int contig_ptes = 0; - - *pgsize = size; - - switch (size) { -#ifndef __PAGETABLE_PMD_FOLDED - case PUD_SIZE: - if (pud_sect_supported()) - contig_ptes = 1; - break; -#endif - case PMD_SIZE: - contig_ptes = 1; - break; - case CONT_PMD_SIZE: - *pgsize = PMD_SIZE; - contig_ptes = CONT_PMDS; - break; - case CONT_PTE_SIZE: - *pgsize = PAGE_SIZE; - contig_ptes = CONT_PTES; - break; - } - - return contig_ptes; -} - -pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) -{ - int ncontig, i; - size_t pgsize; - pte_t orig_pte = __ptep_get(ptep); - - if (!pte_present(orig_pte) || !pte_cont(orig_pte)) - return orig_pte; - - ncontig = num_contig_ptes(page_size(pte_page(orig_pte)), &pgsize); - for (i = 0; i < ncontig; i++, ptep++) { - pte_t pte = __ptep_get(ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - return orig_pte; -} - /* * Changing some bits of contiguous entries requires us to follow a * Break-Before-Make approach, breaking the whole contiguous set @@ -229,7 +178,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, unsigned long pfn, dpfn; pgprot_t hugeprot; - ncontig = num_contig_ptes(sz, &pgsize); + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) @@ -390,7 +339,7 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, int i, ncontig; size_t pgsize; - ncontig = num_contig_ptes(sz, &pgsize); + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) __pte_clear(mm, addr, ptep); diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig index d4a7ca0388c0..2fe8c68fba85 100644 --- a/arch/riscv/Kconfig +++ b/arch/riscv/Kconfig @@ -78,6 +78,7 @@ config RISCV select ARCH_WANT_DEFAULT_TOPDOWN_MMAP_LAYOUT if MMU select ARCH_WANT_FRAME_POINTERS select ARCH_WANT_GENERAL_HUGETLB if !RISCV_ISA_SVNAPOT + select ARCH_WANT_GENERAL_HUGETLB_CONTPTE if RISCV_ISA_SVNAPOT select ARCH_WANT_HUGE_PMD_SHARE if 64BIT select ARCH_WANT_LD_ORPHAN_WARN if !XIP_KERNEL select ARCH_WANT_OPTIMIZE_DAX_VMEMMAP diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index faf3624d8057..d9f9bfb84908 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -4,6 +4,9 @@ #include #include +#ifdef CONFIG_ARCH_WANT_GENERAL_HUGETLB_CONTPTE +#include +#endif static inline void arch_clear_hugetlb_flags(struct folio *folio) { @@ -43,9 +46,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty); -#define __HAVE_ARCH_HUGE_PTEP_GET -pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 82b264423b25..d4e6427b8ca9 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -296,6 +296,8 @@ static inline unsigned long pte_napot(pte_t pte) return pte_val(pte) & _PAGE_NAPOT; } +#define pte_cont pte_napot + #define pte_valid_napot(pte) (pte_present(pte) && pte_napot(pte)) static inline pte_t pte_mknapot(pte_t pte, unsigned int order) @@ -587,6 +589,38 @@ static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, } } +#ifdef CONFIG_RISCV_ISA_SVNAPOT +static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, + size_t *pgsize) +{ + unsigned long hugepage_shift; + pte_t __pte; + + if (size >= PGDIR_SIZE) + hugepage_shift = PGDIR_SHIFT; + else if (size >= P4D_SIZE) + hugepage_shift = P4D_SHIFT; + else if (size >= PUD_SIZE) + hugepage_shift = PUD_SHIFT; + else if (size >= PMD_SIZE) + hugepage_shift = PMD_SHIFT; + else + hugepage_shift = PAGE_SHIFT; + + if (pgsize) + *pgsize = BIT(hugepage_shift); + + /* We must read the raw value of the pte to get the size of the mapping */ + __pte = __ptep_get(ptep); + + /* Make sure __pte is not a swap entry */ + if (pte_valid_napot(__pte)) + return napot_pte_num(napot_cont_order(__pte)); + + return size >> hugepage_shift; +} +#endif + static inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { @@ -671,6 +705,8 @@ static inline void set_ptes(struct mm_struct *mm, unsigned long addr, __set_ptes(mm, addr, ptep, pteval, nr); } #define set_ptes set_ptes +#define set_contptes(mm, addr, ptep, pte, nr, pgsize) \ + set_ptes(mm, addr, ptep, pte, nr) static inline pte_t ptep_get(pte_t *ptep) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 59ed26ce6857..d51863824540 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -3,30 +3,6 @@ #include #ifdef CONFIG_RISCV_ISA_SVNAPOT -pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) -{ - unsigned long pte_num; - int i; - pte_t orig_pte = ptep_get(ptep); - - if (!pte_present(orig_pte) || !pte_napot(orig_pte)) - return orig_pte; - - pte_num = napot_pte_num(napot_cont_order(orig_pte)); - - for (i = 0; i < pte_num; i++, ptep++) { - pte_t pte = ptep_get(ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - - return orig_pte; -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, @@ -266,15 +242,13 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, int dirty) { struct mm_struct *mm = vma->vm_mm; - unsigned long order; pte_t orig_pte; - int i, pte_num; + int pte_num; if (!pte_napot(pte)) return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - order = napot_cont_order(pte); - pte_num = napot_pte_num(order); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); if (pte_dirty(orig_pte)) @@ -298,7 +272,7 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, if (!pte_napot(orig_pte)) return ptep_get_and_clear(mm, addr, ptep); - pte_num = napot_pte_num(napot_cont_order(orig_pte)); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); return get_clear_contig(mm, addr, ptep, pte_num); } @@ -308,17 +282,15 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, pte_t *ptep) { pte_t pte = ptep_get(ptep); - unsigned long order; pte_t orig_pte; - int i, pte_num; + int pte_num; if (!pte_napot(pte)) { ptep_set_wrprotect(mm, addr, ptep); return; } - order = napot_cont_order(pte); - pte_num = napot_pte_num(order); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); @@ -336,7 +308,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_clear_flush(vma, addr, ptep); - pte_num = napot_pte_num(napot_cont_order(pte)); + pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } @@ -346,6 +318,7 @@ void huge_pte_clear(struct mm_struct *mm, pte_t *ptep, unsigned long sz) { + size_t pgsize; pte_t pte = ptep_get(ptep); int i, pte_num; @@ -354,8 +327,8 @@ void huge_pte_clear(struct mm_struct *mm, return; } - pte_num = napot_pte_num(napot_cont_order(pte)); - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) + pte_num = arch_contpte_get_num_contig(ptep, sz, &pgsize); + for (i = 0; i < pte_num; i++, addr += pgsize, ptep++) pte_clear(mm, addr, ptep); } diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h new file mode 100644 index 000000000000..ec4189cd65b8 --- /dev/null +++ b/include/linux/hugetlb_contpte.h @@ -0,0 +1,12 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2024 Rivos Inc. + */ + +#ifndef _LINUX_HUGETLB_CONTPTE_H +#define _LINUX_HUGETLB_CONTPTE_H + +#define __HAVE_ARCH_HUGE_PTEP_GET +extern pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); + +#endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/Kconfig b/mm/Kconfig index 84000b016808..8cd38de612ce 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -810,6 +810,9 @@ config NOMMU_INITIAL_TRIM_EXCESS config ARCH_WANT_GENERAL_HUGETLB bool +config ARCH_WANT_GENERAL_HUGETLB_CONTPTE + bool + config ARCH_WANTS_THP_SWAP def_bool n diff --git a/mm/Makefile b/mm/Makefile index dba52bb0da8a..1c1250fbb020 100644 --- a/mm/Makefile +++ b/mm/Makefile @@ -96,6 +96,7 @@ obj-$(CONFIG_MIGRATION) += migrate.o obj-$(CONFIG_NUMA) += memory-tiers.o obj-$(CONFIG_DEVICE_MIGRATION) += migrate_device.o obj-$(CONFIG_TRANSPARENT_HUGEPAGE) += huge_memory.o khugepaged.o +obj-$(CONFIG_ARCH_WANT_GENERAL_HUGETLB_CONTPTE) += hugetlb_contpte.o obj-$(CONFIG_PAGE_COUNTER) += page_counter.o obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c new file mode 100644 index 000000000000..a03e91d3efb1 --- /dev/null +++ b/mm/hugetlb_contpte.c @@ -0,0 +1,44 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright 2025 Rivos Inc. + */ + +#include +#include +#include + +/* + * Any arch that wants to use that needs to define: + * - __ptep_get() + * - pte_cont() + * - arch_contpte_get_num_contig() + */ + +/* + * This file implements the following contpte aware API: + * - huge_ptep_get() + */ + +pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) +{ + int ncontig, i; + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_present(orig_pte) || !pte_cont(orig_pte)) + return orig_pte; + + ncontig = arch_contpte_get_num_contig(ptep, + page_size(pte_page(orig_pte)), + NULL); + + for (i = 0; i < ncontig; i++, ptep++) { + pte_t pte = __ptep_get(ptep); + + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + return orig_pte; +} From patchwork Mon Jan 27 09:35:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13951146 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5770CC0218C for ; Mon, 27 Jan 2025 09:42:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=vJRiKBdBfEQllvMQc1NjWrSVEvSZ8QHbE/Fus3C3FvQ=; b=IbS30Cnk0Hy+1a kUyBk8mMvK+SRJXwxSyH7ckr1tR9O5zFN+wp0Q9Ae+tbQ2r0y8O1tMBo5Pka+I3fXyvU6EJO0ncFo YAfqXQaPXWHg8OMaiUdDLuEvCbkbQv2b9HR0slmdJD+r+/XZ0/dX+4I7/0UHPG7/dvg9o4zC+0MCC i1yqspw/H1UV7ki5ukUfwRTCsOrzRjlYN7hlri/ypQZm+FknWP8Xn/PLW3lAEqnXx6J6ByJerRQUu OOKOy83RCThYJ6PO9m1xrGoqEkU3h03w9YUF9mjnZoBax/DhosEXnMzHrIkLnEpsrEBQGBJOu7Pzg XfvY8i1PR6t0CeG4NDZA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tcLdF-000000022lF-3TXv; Mon, 27 Jan 2025 09:42:13 +0000 Received: from mail-wm1-x32d.google.com ([2a00:1450:4864:20::32d]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tcLau-0000000222r-04vZ for linux-riscv@lists.infradead.org; Mon, 27 Jan 2025 09:39:51 +0000 Received: by mail-wm1-x32d.google.com with SMTP id 5b1f17b1804b1-4361dc6322fso27270915e9.3 for ; Mon, 27 Jan 2025 01:39:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1737970786; x=1738575586; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=8mhN3d8PsNley3p53cGFp5ftMvicJk5x5kFgKHD2MsE=; b=IMNnKrnnV7BVNJsFCWovMrK584DQJS+CvUyj9NF6EUDy+eUb1hA88Mva3eEThOiQA0 U5WbjO55HH6ebGd0UH5Mvx6oqBz64JVGZ/WwQqfsX4HsaEDL0nHdpI1OKfBw/YWqrqjg RwSFPTmyi5e3eX5x3uUbmbymO90sbbiZpcAvgS9TnnqFq821kCLbXtve+sd0ZC1ZBiYb +2OU+dgjFnvKZAjOLTf4yN+77KOG5QGCPbLJlgjNIVVPkGL2z23IqiBPgbBfQEWiQ9dS GjL5QDNVRkInxEPBdKkc4l2zttSIq/aA3TWyPu+UoRMg5PQICUe8g9tBT//PN38lV650 zfbA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737970786; x=1738575586; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=8mhN3d8PsNley3p53cGFp5ftMvicJk5x5kFgKHD2MsE=; b=Xp9OvlhxuSPXY5HdVsiFT9C6HEB+gQ4HsHzMJZI7lMermig448NlR/rn/G5ZjuoYC1 z9yYVdTOJaiPtrsNxxDNZzbsqKIdbCWupPKds8kxVZS0Qm2OkskeNvkI2f1/BXtp16pq 3jmwZCN2r53XIALO44g2ratssKtmisj8vWz1/Lfa+fxd7C/Ga54VjaW5B5a2Un0Inirr k87lFca3++Pd5CtvB0VQXJ5m16yFVxz6l2ArZCBL12x98IMiYgRXCDUsKRLXWp04VF+t IaBd3JGh10NFarBrL3J+7ekDDPnM5jO3PXcPN4Y1YU31AdD4Q8JiYn0DX/FSvQMCelK6 +FLA== X-Forwarded-Encrypted: i=1; AJvYcCVw8ZtaFr9lXvWarJiI6QjNGT8dROqFZ1pjw966FnD6lABuiniK/ti8FfYkLbbVFz6CKT3XeZ6q7gdwyA==@lists.infradead.org X-Gm-Message-State: AOJu0YzynluwH/QXmOOu3gyB96aVavGEvgVH7oGVYC8CdgdciROgWlbU PaOla32vfp9Po77AFBV5xjbGtzTThvsUT9GedAh3UAmEXVlYPuUn3d8HAZxnz3k= X-Gm-Gg: ASbGncuSb0SOu2bNnLlXS6vvwbqI4jPYdjCnoUXshJ+QH63w+lMMPFArAJ30lJEwbfR gq5UionOMIs9ijCzS4sDxMmRnXgOV3HtQYCOwiG3V2aaWBeEla1sNuQCylK6PE7vbwZEvV/rR7w EZvv7q9zRtrZCSPm+jZlvDE+oIs7EzwiA3zT4rS1NsfvoPi/uec2uHAbrNORMN1DZtHx7P69zYy 7cikenaHHI9X9iGmA66og877KPFubpRWGV1iiaLc77u8s/JX1B33jtpQ+QcUu2hW63w3RWSiDTK bMzHCbvChffZT8HA1UryYNTtuPIK3PM= X-Google-Smtp-Source: AGHT+IFb60eslNQdePOaLfXv36Z0OpFdyDEXUXYChc5sJwAPhp8i6cvPTkFCm9BP5tmum6xqjAkWnA== X-Received: by 2002:a05:6000:4022:b0:385:e013:73f6 with SMTP id ffacd0b85a97d-38bf59eff21mr37315469f8f.50.1737970786287; Mon, 27 Jan 2025 01:39:46 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3e22:3e78:ce5a:32c3]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c2a1bb040sm10590537f8f.67.2025.01.27.01.39.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jan 2025 01:39:46 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v4 4/9] mm: Use common set_huge_pte_at() function for riscv/arm64 Date: Mon, 27 Jan 2025 10:35:25 +0100 Message-Id: <20250127093530.19548-5-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250127093530.19548-1-alexghiti@rivosinc.com> References: <20250127093530.19548-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250127_013948_065342_FA2D976B X-CRM114-Status: GOOD ( 20.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 -- arch/arm64/mm/hugetlbpage.c | 56 ----------------------------- arch/riscv/include/asm/hugetlb.h | 5 --- arch/riscv/include/asm/pgtable.h | 8 +++-- arch/riscv/mm/hugetlbpage.c | 62 -------------------------------- include/linux/hugetlb_contpte.h | 5 +++ mm/hugetlb_contpte.c | 59 ++++++++++++++++++++++++++++++ 7 files changed, 69 insertions(+), 129 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 27d7f4bdd724..40d87a563093 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -35,9 +35,6 @@ static inline void arch_clear_hugetlb_flags(struct folio *folio) pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT -extern void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned long sz); #define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 3458461adb90..02de680a6a0d 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -145,62 +145,6 @@ static pte_t get_clear_contig_flush(struct mm_struct *mm, return orig_pte; } -/* - * Changing some bits of contiguous entries requires us to follow a - * Break-Before-Make approach, breaking the whole contiguous set - * before we can change any entries. See ARM DDI 0487A.k_iss10775, - * "Misprogramming of the Contiguous bit", page D4-1762. - * - * This helper performs the break step for use cases where the - * original pte is not needed. - */ -static void clear_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long i, saddr = addr; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - __ptep_get_and_clear(mm, addr, ptep); - - flush_tlb_range(&vma, saddr, addr); -} - -void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, pte_t pte, unsigned long sz) -{ - size_t pgsize; - int i; - int ncontig; - unsigned long pfn, dpfn; - pgprot_t hugeprot; - - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); - - if (!pte_present(pte)) { - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - if (!pte_cont(pte)) { - __set_ptes(mm, addr, ptep, pte, 1); - return; - } - - pfn = pte_pfn(pte); - dpfn = pgsize >> PAGE_SHIFT; - hugeprot = pte_pgprot(pte); - - clear_flush(mm, addr, ptep, pgsize, ncontig); - - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz) { diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index d9f9bfb84908..28cbf5d761e1 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -24,11 +24,6 @@ bool arch_hugetlb_migration_supported(struct hstate *h); void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz); -#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT -void set_huge_pte_at(struct mm_struct *mm, - unsigned long addr, pte_t *ptep, pte_t pte, - unsigned long sz); - #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index d4e6427b8ca9..74d29d0af172 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -634,9 +634,8 @@ extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addre extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); -#define __HAVE_ARCH_PTEP_GET_AND_CLEAR -static inline pte_t ptep_get_and_clear(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { pte_t pte = __pte(atomic_long_xchg((atomic_long_t *)ptep, 0)); @@ -736,6 +735,9 @@ static inline pte_t ptep_get(pte_t *ptep) #define ptep_get __ptep_get #endif /* CONFIG_RISCV_ISA_SVNAPOT */ +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR +#define ptep_get_and_clear __ptep_get_and_clear + #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) { diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index d51863824540..0ecb2846c3f0 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -173,68 +173,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -static void clear_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - unsigned long i, saddr = addr; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - ptep_get_and_clear(mm, addr, ptep); - - flush_tlb_range(&vma, saddr, addr); -} - -/* - * When dealing with NAPOT mappings, the privileged specification indicates that - * "if an update needs to be made, the OS generally should first mark all of the - * PTEs invalid, then issue SFENCE.VMA instruction(s) covering all 4 KiB regions - * within the range, [...] then update the PTE(s), as described in Section - * 4.2.1.". That's the equivalent of the Break-Before-Make approach used by - * arm64. - */ -void set_huge_pte_at(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - pte_t pte, - unsigned long sz) -{ - unsigned long hugepage_shift, pgsize; - int i, pte_num; - - if (sz >= PGDIR_SIZE) - hugepage_shift = PGDIR_SHIFT; - else if (sz >= P4D_SIZE) - hugepage_shift = P4D_SHIFT; - else if (sz >= PUD_SIZE) - hugepage_shift = PUD_SHIFT; - else if (sz >= PMD_SIZE) - hugepage_shift = PMD_SHIFT; - else - hugepage_shift = PAGE_SHIFT; - - pte_num = sz >> hugepage_shift; - pgsize = 1 << hugepage_shift; - - if (!pte_present(pte)) { - for (i = 0; i < pte_num; i++, ptep++, addr += pgsize) - set_ptes(mm, addr, ptep, pte, 1); - return; - } - - if (!pte_napot(pte)) { - set_ptes(mm, addr, ptep, pte, 1); - return; - } - - clear_flush(mm, addr, ptep, pgsize, pte_num); - - set_ptes(mm, addr, ptep, pte, pte_num); -} - int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index ec4189cd65b8..7acd734a75e8 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -9,4 +9,9 @@ #define __HAVE_ARCH_HUGE_PTEP_GET extern pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_HUGE_SET_HUGE_PTE_AT +extern void set_huge_pte_at(struct mm_struct *mm, + unsigned long addr, pte_t *ptep, pte_t pte, + unsigned long sz); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index a03e91d3efb1..677d714fd10d 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -10,6 +10,8 @@ /* * Any arch that wants to use that needs to define: * - __ptep_get() + * - __set_ptes() + * - __ptep_get_and_clear() * - pte_cont() * - arch_contpte_get_num_contig() */ @@ -17,6 +19,7 @@ /* * This file implements the following contpte aware API: * - huge_ptep_get() + * - set_huge_pte_at() */ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -42,3 +45,59 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) } return orig_pte; } + +/* + * ARM64: Changing some bits of contiguous entries requires us to follow a + * Break-Before-Make approach, breaking the whole contiguous set + * before we can change any entries. See ARM DDI 0487A.k_iss10775, + * "Misprogramming of the Contiguous bit", page D4-1762. + * + * RISCV: When dealing with NAPOT mappings, the privileged specification + * indicates that "if an update needs to be made, the OS generally should first + * mark all of the PTEs invalid, then issue SFENCE.VMA instruction(s) covering + * all 4 KiB regions within the range, [...] then update the PTE(s), as + * described in Section 4.2.1.". That's the equivalent of the Break-Before-Make + * approach used by arm64. + * + * This helper performs the break step for use cases where the + * original pte is not needed. + */ +static void clear_flush(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + unsigned long i, saddr = addr; + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) + __ptep_get_and_clear(mm, addr, ptep); + + flush_tlb_range(&vma, saddr, addr); +} + +void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, pte_t pte, unsigned long sz) +{ + size_t pgsize; + int i; + int ncontig; + + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + + if (!pte_present(pte)) { + for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) + __set_ptes(mm, addr, ptep, pte, 1); + return; + } + + if (!pte_cont(pte)) { + __set_ptes(mm, addr, ptep, pte, 1); + return; + } + + clear_flush(mm, addr, ptep, pgsize, ncontig); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); +} From patchwork Mon Jan 27 09:35:26 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13951147 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DA805C02190 for ; Mon, 27 Jan 2025 09:42:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=CyHzABVKmhkG6npZP/nQiXh4FekTgO3JDym/OcXUUOI=; b=TXW3B3Ck+9/s3L okQM8gL3Ju9YyOUVL9ngiHaFZnxLEGEE8hkncGffvf+0J6Pv0rr/l0CgRlU4XWwtENT3XnC1be1gD omekdNLl59Y/RWWGNOUETgdtiZFYZCwAor6EYxsh9JsYLu6zmX8F86BLd4SpUswPQ/1A4BGSB9KpY axWztDHAp7IIcelqwy6qqeexoptXN4ldqmZic+czEWQSfnS7O+44MkavL7j43EU3hlqn0KOCA0haL saDJ8eeuwFYbv4fGbAtOIBJJrQz/mG1xcyfY4wiJm7tz1t2JaVkECC6Xm8nCq1wd+HCHny1MhuEYH SvJDCvX/O2rga8vlCccw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tcLdG-000000022lg-21I0; Mon, 27 Jan 2025 09:42:14 +0000 Received: from mail-wm1-x32c.google.com ([2a00:1450:4864:20::32c]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tcLbt-000000022My-2r43 for linux-riscv@lists.infradead.org; Mon, 27 Jan 2025 09:40:50 +0000 Received: by mail-wm1-x32c.google.com with SMTP id 5b1f17b1804b1-4362bae4d7dso29056195e9.1 for ; Mon, 27 Jan 2025 01:40:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1737970848; x=1738575648; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WRFnXJIF4/2q4SeWU0XB4ffUEopf6Q+wLHMkPdh3cGw=; b=ImzJuwDIx2ePO++CDbSc9Py86TnFDTZsZqDKSNVbGpIgdqh9NJZM08BoK0A5viI6U2 /m4oYSc2U7VODGi+4So9SlbRB+9npY5Xl71pid2ajjBdzF6+Pao2vmkPty5KP1TkGSng I5c82+R8RahLNg1qzCybT6J4UOBZKgf9Bhh4DPWdSJazSXMMaEMdWuiTczfENLuaTrkk rNnOHR1hD8OnzbExKZBSQVq3l/SOrU+izSZb9Qtzk3t4RlV2h25S5ztWbvsJzsKHrKLu 6wi7/W6E8fsjGOlB78f43V696aKft5QLVyhWmcwFyM8W5QhMpjV69v0cJqmY2lgrvpH+ 1tKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737970848; x=1738575648; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WRFnXJIF4/2q4SeWU0XB4ffUEopf6Q+wLHMkPdh3cGw=; b=PV5ILamIoiy9FxpS8LZJgODweNwfAHtgD6XvEkmDVE+miLwOCUdH1wN/nF/6jdbDeu ONlZKEEh45b/QtDVS9M7JV29uvZQZ3rDmns/u6PL+ObqgETnrurnYv75R6fyqZjEvAk9 g1urGm+Lfba5cGTSb2utorpAp9g5/PZ9jdGxHRUDeL3qzGKegtu3v0+pX0tgLJO7xhoP GKtfR9mtB3pW8o34s89AXl0LOElmCC0vhy5kaLnf14d3pdxTc8G7YMFqHc4JmoRVs2bQ 6LISZZ6BHsqoYlnaJ3Zd7v+lv2VnEvr1L+gCLfist+sZN+TBHaYUWH+3Foe6BWxRpTux 1mHw== X-Forwarded-Encrypted: i=1; AJvYcCVb8VReICrP36jmNz1Ic+wmJfwx8gd1xjtBuohq3ScL4K/yBoXrdKqTrekF70+q+VAmTVAMhcgr1Jq8bA==@lists.infradead.org X-Gm-Message-State: AOJu0YxTTDRQ3gLMl05hsGbu6h0tWlmXwLFHUdbJdB7v69Gpp/GY2NTY OUOEhRphswnz/yF4q9k4lsE4MW/y8OvBIp0Wl1ViwuN4SeEPyFFVtuZxGfZJYs0= X-Gm-Gg: ASbGncsShkOl9DfYRlWEBUNTnkf4gvyZNTZgLR287iLGRNT1KfUwPGvJ1GKHPp1rclw Gb+Q8NMOatPX/tCUcUkDcdku0XirZMp74rNotf8etQw2tUALf4XkolCakCNhuejs3zuIk1uUYez Hjf5KV2z2WGUbsdNBYuZFW+nHKd/rwlaHVbIqIh/Td3+cMWOXIvbtIYX0Aft9gjcMx2ODvRJJI1 yBa0DN9r89oZKUPrE+BJNI4BR/JD7Q+k9E0Wr4MEGhx4XpL2wjjTDYd2u6tb65xYz4vDbDVsr4t nIC4voB51BNv3woaBpo+CnSwP8MYcE8= X-Google-Smtp-Source: AGHT+IHaxudRMH2a6X9Kau2FejbNNa+aNZH7OQ4Z4KpCCzwiphcNEJX34OXuuFlJfN/dPYRcVT3pPw== X-Received: by 2002:a05:600c:6c9a:b0:438:a240:c54 with SMTP id 5b1f17b1804b1-438a2400d95mr279659705e9.9.1737970847952; Mon, 27 Jan 2025 01:40:47 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3e22:3e78:ce5a:32c3]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-438bd57517fsm122917795e9.38.2025.01.27.01.40.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jan 2025 01:40:47 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v4 5/9] mm: Use common huge_pte_clear() function for riscv/arm64 Date: Mon, 27 Jan 2025 10:35:26 +0100 Message-Id: <20250127093530.19548-6-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250127093530.19548-1-alexghiti@rivosinc.com> References: <20250127093530.19548-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250127_014049_721373_EB494E5A X-CRM114-Status: GOOD ( 13.62 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Both architectures have the same implementation so move it to generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 --- arch/arm64/mm/hugetlbpage.c | 12 ------------ arch/riscv/include/asm/hugetlb.h | 4 ---- arch/riscv/include/asm/pgtable.h | 5 +++-- arch/riscv/mm/hugetlbpage.c | 19 ------------------- include/linux/hugetlb_contpte.h | 4 ++++ mm/hugetlb_contpte.c | 14 ++++++++++++++ 7 files changed, 21 insertions(+), 40 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 40d87a563093..e4acaedea149 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -48,9 +48,6 @@ extern void huge_ptep_set_wrprotect(struct mm_struct *mm, #define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); -#define __HAVE_ARCH_HUGE_PTE_CLEAR -extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long sz); void __init arm64_hugetlb_cma_reserve(void); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 02de680a6a0d..541358f50b64 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -277,18 +277,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -void huge_pte_clear(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long sz) -{ - int i, ncontig; - size_t pgsize; - - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) - __pte_clear(mm, addr, ptep); -} - pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 28cbf5d761e1..ca9930cdf2e6 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -20,10 +20,6 @@ bool arch_hugetlb_migration_supported(struct hstate *h); #endif #ifdef CONFIG_RISCV_ISA_SVNAPOT -#define __HAVE_ARCH_HUGE_PTE_CLEAR -void huge_pte_clear(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, unsigned long sz); - #define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 74d29d0af172..08b24c0a579b 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -621,8 +621,8 @@ static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, } #endif -static inline void pte_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) +static inline void __pte_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) { __set_pte_at(mm, ptep, __pte(0)); } @@ -737,6 +737,7 @@ static inline pte_t ptep_get(pte_t *ptep) #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear +#define pte_clear __pte_clear #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 0ecb2846c3f0..e2093e7266a5 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -251,25 +251,6 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } -void huge_pte_clear(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long sz) -{ - size_t pgsize; - pte_t pte = ptep_get(ptep); - int i, pte_num; - - if (!pte_napot(pte)) { - pte_clear(mm, addr, ptep); - return; - } - - pte_num = arch_contpte_get_num_contig(ptep, sz, &pgsize); - for (i = 0; i < pte_num; i++, addr += pgsize, ptep++) - pte_clear(mm, addr, ptep); -} - static bool is_napot_size(unsigned long size) { unsigned long order; diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index 7acd734a75e8..d9892a047b2b 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -14,4 +14,8 @@ extern void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, pte_t *ptep, pte_t pte, unsigned long sz); +#define __HAVE_ARCH_HUGE_PTE_CLEAR +extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned long sz); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index 677d714fd10d..c76d6b3d0121 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -12,6 +12,7 @@ * - __ptep_get() * - __set_ptes() * - __ptep_get_and_clear() + * - __pte_clear() * - pte_cont() * - arch_contpte_get_num_contig() */ @@ -20,6 +21,7 @@ * This file implements the following contpte aware API: * - huge_ptep_get() * - set_huge_pte_at() + * - huge_pte_clear() */ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -101,3 +103,15 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, set_contptes(mm, addr, ptep, pte, ncontig, pgsize); } + +void huge_pte_clear(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, unsigned long sz) +{ + int i, ncontig; + size_t pgsize; + + ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) + __pte_clear(mm, addr, ptep); +} From patchwork Mon Jan 27 09:35:27 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13951148 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1E690C0218C for ; Mon, 27 Jan 2025 09:43:39 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1noKm8ONUjBrNxL1tUq1r9bVv1xoMbfBJrw3Z1AwNc0=; b=g6ahznjTe4cYhs K7rp/46ByX48jNjNCS3b2TIbN2VOx3WraO01kJGvsHsB4jNS82kjZpcKqquUUWT8aaWTVld7F91cs S3cdpo/SHNG0Oz3t356FP0zkxYhFKSxVpKHCESLvGGlAu/exoXbLTULcHjTxqfCw+/VT6XGH0iS2s ZvGv8peEo+3WKzJACxusYDGKSJR0d+v0dPh36M2yXjIfOn9SrfQ/3oDPdh1CqrmYqvHMp07GFMeYW AbS/zhjOkAJIQ7Z//dDezIRFgRzPF5DSZCmICyeoWi9oRlqyn22l0sgFmUd49igSUy5iH6lDooKES UCFZc73JCISg2OE2MwtA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tcLeY-0000000239p-3G7U; Mon, 27 Jan 2025 09:43:34 +0000 Received: from mail-wr1-x431.google.com ([2a00:1450:4864:20::431]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tcLct-000000022eo-0CUj for linux-riscv@lists.infradead.org; Mon, 27 Jan 2025 09:41:52 +0000 Received: by mail-wr1-x431.google.com with SMTP id ffacd0b85a97d-385d7b4da2bso3927873f8f.1 for ; Mon, 27 Jan 2025 01:41:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1737970909; x=1738575709; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=/Ib7BO4DbRcQ5HoSRv1ee1R7qySu3xzgQBO2hzqpTQY=; b=0C9TUI3yxLa/ppsGi0hDrXgYW1ecu38qmn/GaXSgthjXL8ao+UBfCNFSmpwXNpplsK 84Bm2EqjHn/ga/vIDdPLWpIeQUAG4I6Lo4ddX/3iCeYcLAls5/OGzygBqW75p9OlPV+J u+SsqZwOFKGFfiH+TEYTPzlqOg9zQGF/jBdDdBbPsM5jU8NpLX/8dhJE3AtoxhjBD9+t mUnd5EkN4luc1b6Bv9hQD3r1AsPtn6S6mKTJFD+DCqgSbNSwqF63Tp2V82XbtVfAtcEg FMvodncgwP2ZVbo47kbvgEKaRNDwR1XamhCZ+WZjCj4Vsnkt07/QvXCYLvgGEvx7L8rr k61w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737970909; x=1738575709; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/Ib7BO4DbRcQ5HoSRv1ee1R7qySu3xzgQBO2hzqpTQY=; b=s4WULkNg1SaDQTm1iF6TPb79VXy4H5d8Muvvoimox9tFHXyDp9kCOsEVpkhzfYPpYg azwgCdPchwLtX1NeS2Ph8KhFCDDEU0vooBd+XFMCe9wRd0n0e7chdJ9NtoaeDwqs2MUu l9lGJaoayIedL90ewMyMTGrSue4IdJASL4ObCSsFaCQlQBrLVEqlClAHIplp9x+5TfY3 CLqtyWnTMuxCj4EqpoOoUQRvLZLY5If/vP40mm5v2ANROUKdOeBStRkZGJD5XtIoikYc YByYMztq21ne720L+pOnBArLJneufV+zfZy9iFmezAriNNeVo6E8AQ8j6+L87I1WEA4G Oa9Q== X-Forwarded-Encrypted: i=1; AJvYcCVEKT8j2xSOXGwphzK6G6UyJJ23SQNxgEfMUm4N6qlxXr2j9Df49DW7k5zHO1s79VPQvfP1uEXF+JpgQQ==@lists.infradead.org X-Gm-Message-State: AOJu0Yyt12HyWX/stThvRcSxcGVFVff8k+L1VAtQtAZAsPl8DG0kPbHT VN2IdS3UeehugCNFuwfIf1Pe9wEpTvDSmUUQZjVZBDCLIRHCA5AJC5Iu3+SuWos= X-Gm-Gg: ASbGncux8xOzIykANqTbFaCM/k05H4D4cCx5xWj+9GYBZ/Ct9esXLxbGjsJy+blOQDT b2nEWFRGMcT7vHR8Jk+7fgEJc7qg9E+YvQjMCf044rq86oc1I/Qhoz/ASqKx8QrLLuFtOQysbvI nr9htmPHemMwOQa9u+T5fALyyb1GI/VqCfj2UnIwpNFSDdpb2Y/Em0C7pWqP13/cAuMkmW2QgJK UPUBf1P6bMeH31Xsb1+449wayxfhvohhR2jSnAA+gjVaiDoUjOcSSRGjpDdSOGfTwy4THQij5M8 TTZGohJig28b0xb31dDAQZZ+MH5uunE= X-Google-Smtp-Source: AGHT+IE0wvAkHX9Lz2nK7eyxGx0auM1eq+y4AS6IYBZCccECBQROh0UBY1MLjeLntJwSdARX6LIQ6w== X-Received: by 2002:a5d:64a3:0:b0:385:f909:eb2c with SMTP id ffacd0b85a97d-38bf57a77a2mr48324115f8f.38.1737970909255; Mon, 27 Jan 2025 01:41:49 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3e22:3e78:ce5a:32c3]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c2a1c42fcsm10707051f8f.96.2025.01.27.01.41.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jan 2025 01:41:49 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v4 6/9] mm: Use common huge_ptep_get_and_clear() function for riscv/arm64 Date: Mon, 27 Jan 2025 10:35:27 +0100 Message-Id: <20250127093530.19548-7-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250127093530.19548-1-alexghiti@rivosinc.com> References: <20250127093530.19548-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250127_014151_090446_51092EC3 X-CRM114-Status: GOOD ( 20.08 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Note that get_clear_contig() function is duplicated in the generic and the arm64 code because it is still used by some arm64 functions that will, in the next commits, be moved to the generic code. Once all have been moved, the arm64 version will be removed. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 -- arch/arm64/include/asm/pgtable.h | 15 ++++++++-- arch/arm64/mm/hugetlbpage.c | 19 ++----------- arch/riscv/include/asm/hugetlb.h | 4 --- arch/riscv/include/asm/pgtable.h | 4 ++- arch/riscv/mm/hugetlbpage.c | 23 ++++----------- include/linux/hugetlb_contpte.h | 4 +++ mm/hugetlb_contpte.c | 48 ++++++++++++++++++++++++++++++-- 8 files changed, 72 insertions(+), 48 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index e4acaedea149..5c605a0a2017 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -39,9 +39,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty); -#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR -extern pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT extern void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep); diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h index cebbfcfb0e53..c339b568ac51 100644 --- a/arch/arm64/include/asm/pgtable.h +++ b/arch/arm64/include/asm/pgtable.h @@ -1831,12 +1831,23 @@ static inline void clear_young_dirty_ptes(struct vm_area_struct *vma, #endif /* CONFIG_ARM64_CONTPTE */ -static inline int arch_contpte_get_num_contig(pte_t *ptep, - unsigned long size, +extern int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize); + +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { int contig_ptes = 0; + /* + * If the size is not passed, we need to go through the page table to + * find out the number of contiguous ptes. + */ + if (size == 0) + return find_num_contig(mm, addr, ptep, pgsize); + if (pgsize) *pgsize = size; diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 541358f50b64..0b7a53fee55d 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -79,8 +79,8 @@ bool arch_hugetlb_migration_supported(struct hstate *h) } #endif -static int find_num_contig(struct mm_struct *mm, unsigned long addr, - pte_t *ptep, size_t *pgsize) +int find_num_contig(struct mm_struct *mm, unsigned long addr, + pte_t *ptep, size_t *pgsize) { pgd_t *pgdp = pgd_offset(mm, addr); p4d_t *p4dp; @@ -277,21 +277,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - int ncontig; - size_t pgsize; - pte_t orig_pte = __ptep_get(ptep); - - if (!pte_cont(orig_pte)) - return __ptep_get_and_clear(mm, addr, ptep); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - - return get_clear_contig(mm, addr, ptep, pgsize, ncontig); -} - /* * huge_ptep_set_access_flags will update access flags (dirty, accesssed) * and write permission. diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index ca9930cdf2e6..0fbb6b19df79 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -20,10 +20,6 @@ bool arch_hugetlb_migration_supported(struct hstate *h); #endif #ifdef CONFIG_RISCV_ISA_SVNAPOT -#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, pte_t *ptep); - #define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 08b24c0a579b..705d666e014d 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -590,7 +590,9 @@ static inline void __set_ptes(struct mm_struct *mm, unsigned long addr, } #ifdef CONFIG_RISCV_ISA_SVNAPOT -static inline int arch_contpte_get_num_contig(pte_t *ptep, unsigned long size, +static inline int arch_contpte_get_num_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, unsigned long size, size_t *pgsize) { unsigned long hugepage_shift; diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index e2093e7266a5..b44023336fd9 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -186,7 +186,8 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); + orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); if (pte_dirty(orig_pte)) @@ -200,21 +201,6 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return true; } -pte_t huge_ptep_get_and_clear(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep) -{ - pte_t orig_pte = ptep_get(ptep); - int pte_num; - - if (!pte_napot(orig_pte)) - return ptep_get_and_clear(mm, addr, ptep); - - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); - - return get_clear_contig(mm, addr, ptep, pte_num); -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -228,7 +214,8 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, return; } - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); + pte_num = arch_contpte_get_num_contig(mm, addr, ptep, 0, NULL); + orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); orig_pte = pte_wrprotect(orig_pte); @@ -246,7 +233,7 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, if (!pte_napot(pte)) return ptep_clear_flush(vma, addr, ptep); - pte_num = arch_contpte_get_num_contig(ptep, 0, NULL); + pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); } diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index d9892a047b2b..20d3a3e14e14 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -18,4 +18,8 @@ extern void set_huge_pte_at(struct mm_struct *mm, extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep, unsigned long sz); +#define __HAVE_ARCH_HUGE_PTEP_GET_AND_CLEAR +extern pte_t huge_ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index c76d6b3d0121..0c86c6f77c29 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -22,6 +22,7 @@ * - huge_ptep_get() * - set_huge_pte_at() * - huge_pte_clear() + * - huge_ptep_get_and_clear() */ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -32,7 +33,7 @@ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) if (!pte_present(orig_pte) || !pte_cont(orig_pte)) return orig_pte; - ncontig = arch_contpte_get_num_contig(ptep, + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, page_size(pte_page(orig_pte)), NULL); @@ -86,7 +87,7 @@ void set_huge_pte_at(struct mm_struct *mm, unsigned long addr, int i; int ncontig; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); if (!pte_present(pte)) { for (i = 0; i < ncontig; i++, ptep++, addr += pgsize) @@ -110,8 +111,49 @@ void huge_pte_clear(struct mm_struct *mm, unsigned long addr, int i, ncontig; size_t pgsize; - ncontig = arch_contpte_get_num_contig(ptep, sz, &pgsize); + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, sz, &pgsize); for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) __pte_clear(mm, addr, ptep); } + +static pte_t get_clear_contig(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + pte_t orig_pte = __ptep_get(ptep); + unsigned long i; + + for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { + pte_t pte = __ptep_get_and_clear(mm, addr, ptep); + + /* + * If HW_AFDBM (arm64) or Svadu (riscv) is enabled, then the HW + * could turn on the dirty or accessed bit for any page in the + * set, so check them all. + */ + if (pte_dirty(pte)) + orig_pte = pte_mkdirty(orig_pte); + + if (pte_young(pte)) + orig_pte = pte_mkyoung(orig_pte); + } + return orig_pte; +} + +pte_t huge_ptep_get_and_clear(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + int ncontig; + size_t pgsize; + pte_t orig_pte = __ptep_get(ptep); + + if (!pte_cont(orig_pte)) + return __ptep_get_and_clear(mm, addr, ptep); + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + return get_clear_contig(mm, addr, ptep, pgsize, ncontig); +} From patchwork Mon Jan 27 09:35:28 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13951164 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 569CAC02190 for ; Mon, 27 Jan 2025 09:44:59 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=nKytl+WBMvweVjrRNrBuIz4WptmEU6RwCsjHprIHUF8=; b=oVuFEPKdiB6xGG Rnzn/8c0JvpnKgzX4s6EfRdJgGEIcsk6riP9nv78zkqqkH7WeAyRvEOSlPXg6HyaZv2vXzzYdUuAz NVCNE7wHqmqY2ULgFRzTOPRb2nrYIL306HG9tkwKek+ZAoFGjiZBzc7QZ+EANxmDrhbAEZqR86Z4g o3ajgVDZ+M18gtobR88q14ppUGmdu8RfVmLUwOR59YMrtzwQFdJi6A/NMeIzsrZovcf+bLt7q/WLc 9t0ObTSBKh0oclduUT6Dq3nVsFiPJVzeZS+b5j3ilGqBwFxKS2Yi7dQBHJVsSf9ib4eKFUyPX9hLB P/NdyDsYDcFa4W21hjCg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tcLfq-000000023Rr-46vv; Mon, 27 Jan 2025 09:44:54 +0000 Received: from mail-wm1-x335.google.com ([2a00:1450:4864:20::335]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tcLds-00000002302-1uAr for linux-riscv@lists.infradead.org; Mon, 27 Jan 2025 09:42:53 +0000 Received: by mail-wm1-x335.google.com with SMTP id 5b1f17b1804b1-436202dd7f6so48504705e9.0 for ; Mon, 27 Jan 2025 01:42:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1737970971; x=1738575771; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=GkmyUQAgkdeDfCGOVLuC8IWvmfEC8o5jHyW1gtJjCzs=; b=Mp8/ysZkGLTnOSR8t40ymYnbHsuleB/vMyKPbI7FCVenFIgm7S1Z1cjbnNtxkH1Z2H zI+lEMBMOdvBiZbEP+0wbRJSqYpgW2FccTXQ54Fp0+5yOTFYBf1eSqmqRHvp3ubMH6Pt twQjiOBfiOw9mc97x8FB84ttA/FJITdDfR1aS4Bjyv7pYcytyh//NwjMvByh2OSaqiFE CLsCwdxYRP3nfMcPX2il9YFWnXcS42ww9m2P7wjtRd2v1Dj+PoHFbYoY1oNjsQu1FbVX 6wKMDvi8A2i2+llN/+VCDDrMuh5QMDJ9ytsRrXcUe4ddqwIwmw6txL/JZwfyNmqnLBNI PbEw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737970971; x=1738575771; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GkmyUQAgkdeDfCGOVLuC8IWvmfEC8o5jHyW1gtJjCzs=; b=mzVHeaVj/jBPEiPLuOHQAGZogMYAduiUaO94A268PcuD9puJtNkCy/RAn+ZjrWCx4t psxIbWlreW9+tvMu4qIMeCb+XtFuMvWLDOf420iBvTAK8Ep0tQImv7PooWJzYv1zictk 1T1kCWgdyI7ceUxpR0JG5Nm2ZdaG443OMBZSZtYIDw1BKpwYtH0Xg54yYtOGHlFYVHsq MrQU7bU5H1Up/PlfNayI0wZ4ZPg5dvZoLOKtuVtNfQ6+OJZoxincjqgBmUMPdU//375w ebKNwrZJheG8rQymCWXSbFUrw3UGZElVy5EoNTHekbobr6JIhWbJnOyzGHD+wzz112Ox zamw== X-Forwarded-Encrypted: i=1; AJvYcCUS0cAdpKwp4/OK9H77xlyw6tUZGhyMXcAGctsYuZgknpecQw7M87uOHyw/7ts/QKMXzuO67XPVlGuDUA==@lists.infradead.org X-Gm-Message-State: AOJu0Yx9uHw9UUz1VNx9iMYMxSqRxyqys2QZvZVXO0PVuVJiztGWLujM 0iKBlJJwS7qlH0PW+DMSkP6IOhuFoHGLGz9gSnV1PqVR39CoBxiwF+O+m9/SlKc= X-Gm-Gg: ASbGncuCIh2YwGd/0sDqVrdnRdJRTUxozqCCVz9CQTw7mNy6VtuS8B4h956e8DvSlFv cWZ/yYZHwPlOCfsmzlb6fMlj89VlJ1iXHYSvoJsQkDH3N0a99dY3jwCk/b3rcrG8cMW/4ktIpcp 4heyvPlVCZTmiMcpDt+LQJW5+YBPXQaoi/M2oKIRNJVB0q0dYY0gcmusi61oRBAL7hRa7NBDA0/ bxkCEkdJesTXVVxEZR7gaKuFv94DCKAijmNAsYTMsMi7jzECDB8ox1hoJW87b1TgfChCvCyAv3I cOV1n3sgxYRa9PJpz8xXmcyDKGaPrpM= X-Google-Smtp-Source: AGHT+IFkGD+876PDgJO8tAmQAHU0ol+3F5plCgX8+RfXXi3SGv5kqaXmgh3Sp+SaS77G8Nc/FNbA7g== X-Received: by 2002:a05:6000:1ac6:b0:385:f07b:93da with SMTP id ffacd0b85a97d-38bf57bf69bmr33979049f8f.47.1737970970947; Mon, 27 Jan 2025 01:42:50 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3e22:3e78:ce5a:32c3]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c2a17d6edsm10380540f8f.40.2025.01.27.01.42.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jan 2025 01:42:50 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v4 7/9] mm: Use common huge_ptep_set_access_flags() function for riscv/arm64 Date: Mon, 27 Jan 2025 10:35:28 +0100 Message-Id: <20250127093530.19548-8-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250127093530.19548-1-alexghiti@rivosinc.com> References: <20250127093530.19548-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250127_014252_523668_0D6D3FDF X-CRM114-Status: GOOD ( 22.37 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org Both architectures have almost the same implementation: __cont_access_flags_changed() is also correct on riscv and brings the same benefits (ie don't do anything if the flags are unchanged). As in the previous commit, get_clear_contig_flush() is duplicated in both the arch and the generic codes, it will be removed from the arch code when the last reference there gets moved to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 4 -- arch/arm64/mm/hugetlbpage.c | 65 --------------------------- arch/riscv/include/asm/hugetlb.h | 5 --- arch/riscv/include/asm/pgtable.h | 7 +-- arch/riscv/mm/hugetlbpage.c | 28 ------------ arch/riscv/mm/pgtable.c | 6 +-- include/linux/hugetlb_contpte.h | 5 +++ mm/hugetlb_contpte.c | 75 ++++++++++++++++++++++++++++++++ 8 files changed, 87 insertions(+), 108 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 5c605a0a2017..654f5f2f03a3 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -35,10 +35,6 @@ static inline void arch_clear_hugetlb_flags(struct folio *folio) pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS -extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t pte, int dirty); #define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT extern void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 0b7a53fee55d..643ba2043f0f 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -277,71 +277,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -/* - * huge_ptep_set_access_flags will update access flags (dirty, accesssed) - * and write permission. - * - * For a contiguous huge pte range we need to check whether or not write - * permission has to change only on the first pte in the set. Then for - * all the contiguous ptes we need to check whether or not there is a - * discrepancy between dirty or young. - */ -static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig) -{ - int i; - - if (pte_write(pte) != pte_write(__ptep_get(ptep))) - return 1; - - for (i = 0; i < ncontig; i++) { - pte_t orig_pte = __ptep_get(ptep + i); - - if (pte_dirty(pte) != pte_dirty(orig_pte)) - return 1; - - if (pte_young(pte) != pte_young(orig_pte)) - return 1; - } - - return 0; -} - -int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t pte, int dirty) -{ - int ncontig, i; - size_t pgsize = 0; - unsigned long pfn = pte_pfn(pte), dpfn; - struct mm_struct *mm = vma->vm_mm; - pgprot_t hugeprot; - pte_t orig_pte; - - if (!pte_cont(pte)) - return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - dpfn = pgsize >> PAGE_SHIFT; - - if (!__cont_access_flags_changed(ptep, pte, ncontig)) - return 0; - - orig_pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); - - /* Make sure we don't lose the dirty or young state */ - if (pte_dirty(orig_pte)) - pte = pte_mkdirty(pte); - - if (pte_young(orig_pte)) - pte = pte_mkyoung(pte); - - hugeprot = pte_pgprot(pte); - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); - - return 1; -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 0fbb6b19df79..bf533c2cef84 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -28,11 +28,6 @@ pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep); -#define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS -int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep, - pte_t pte, int dirty); - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 705d666e014d..290d5fbfe031 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -629,9 +629,8 @@ static inline void __pte_clear(struct mm_struct *mm, __set_pte_at(mm, ptep, __pte(0)); } -#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS /* defined in mm/pgtable.c */ -extern int ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, - pte_t *ptep, pte_t entry, int dirty); +extern int __ptep_set_access_flags(struct vm_area_struct *vma, unsigned long address, + pte_t *ptep, pte_t entry, int dirty); #define __HAVE_ARCH_PTEP_TEST_AND_CLEAR_YOUNG /* defined in mm/pgtable.c */ extern int ptep_test_and_clear_young(struct vm_area_struct *vma, unsigned long address, pte_t *ptep); @@ -740,6 +739,8 @@ static inline pte_t ptep_get(pte_t *ptep) #define __HAVE_ARCH_PTEP_GET_AND_CLEAR #define ptep_get_and_clear __ptep_get_and_clear #define pte_clear __pte_clear +#define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS +#define ptep_set_access_flags __ptep_set_access_flags #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index b44023336fd9..0e2ca7327479 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -173,34 +173,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -int huge_ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long addr, - pte_t *ptep, - pte_t pte, - int dirty) -{ - struct mm_struct *mm = vma->vm_mm; - pte_t orig_pte; - int pte_num; - - if (!pte_napot(pte)) - return ptep_set_access_flags(vma, addr, ptep, pte, dirty); - - pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); - - orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); - - if (pte_dirty(orig_pte)) - pte = pte_mkdirty(pte); - - if (pte_young(orig_pte)) - pte = pte_mkyoung(pte); - - set_ptes(mm, addr, ptep, pte, pte_num); - - return true; -} - void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep) diff --git a/arch/riscv/mm/pgtable.c b/arch/riscv/mm/pgtable.c index 4ae67324f992..af8b3769a349 100644 --- a/arch/riscv/mm/pgtable.c +++ b/arch/riscv/mm/pgtable.c @@ -5,9 +5,9 @@ #include #include -int ptep_set_access_flags(struct vm_area_struct *vma, - unsigned long address, pte_t *ptep, - pte_t entry, int dirty) +int __ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long address, pte_t *ptep, + pte_t entry, int dirty) { asm goto(ALTERNATIVE("nop", "j %l[svvptc]", 0, RISCV_ISA_EXT_SVVPTC, 1) : : : : svvptc); diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index 20d3a3e14e14..fea47035ac38 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -22,4 +22,9 @@ extern void huge_pte_clear(struct mm_struct *mm, unsigned long addr, extern pte_t huge_ptep_get_and_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_HUGE_PTEP_SET_ACCESS_FLAGS +extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, int dirty); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index 0c86c6f77c29..49950c1ce615 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -13,6 +13,7 @@ * - __set_ptes() * - __ptep_get_and_clear() * - __pte_clear() + * - __ptep_set_access_flags() * - pte_cont() * - arch_contpte_get_num_contig() */ @@ -23,6 +24,7 @@ * - set_huge_pte_at() * - huge_pte_clear() * - huge_ptep_get_and_clear() + * - huge_ptep_set_access_flags() */ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -157,3 +159,76 @@ pte_t huge_ptep_get_and_clear(struct mm_struct *mm, return get_clear_contig(mm, addr, ptep, pgsize, ncontig); } + +/* + * huge_ptep_set_access_flags will update access flags (dirty, accesssed) + * and write permission. + * + * For a contiguous huge pte range we need to check whether or not write + * permission has to change only on the first pte in the set. Then for + * all the contiguous ptes we need to check whether or not there is a + * discrepancy between dirty or young. + */ +static int __cont_access_flags_changed(pte_t *ptep, pte_t pte, int ncontig) +{ + int i; + + if (pte_write(pte) != pte_write(__ptep_get(ptep))) + return 1; + + for (i = 0; i < ncontig; i++) { + pte_t orig_pte = __ptep_get(ptep + i); + + if (pte_dirty(pte) != pte_dirty(orig_pte)) + return 1; + + if (pte_young(pte) != pte_young(orig_pte)) + return 1; + } + + return 0; +} + +static pte_t get_clear_contig_flush(struct mm_struct *mm, + unsigned long addr, + pte_t *ptep, + unsigned long pgsize, + unsigned long ncontig) +{ + pte_t orig_pte = get_clear_contig(mm, addr, ptep, pgsize, ncontig); + struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); + + flush_tlb_range(&vma, addr, addr + (pgsize * ncontig)); + return orig_pte; +} + +int huge_ptep_set_access_flags(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep, + pte_t pte, int dirty) +{ + int ncontig; + size_t pgsize = 0; + struct mm_struct *mm = vma->vm_mm; + pte_t orig_pte; + + if (!pte_cont(pte)) + return __ptep_set_access_flags(vma, addr, ptep, pte, dirty); + + ncontig = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, &pgsize); + + if (!__cont_access_flags_changed(ptep, pte, ncontig)) + return 0; + + orig_pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); + + /* Make sure we don't lose the dirty or young state */ + if (pte_dirty(orig_pte)) + pte = pte_mkdirty(pte); + + if (pte_young(orig_pte)) + pte = pte_mkyoung(pte); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); + + return 1; +} From patchwork Mon Jan 27 09:35:29 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13951165 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 61D6BC0218C for ; Mon, 27 Jan 2025 09:47:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Gdn3xfyvWiyfQY+IjxBA23hd2Qy3KMaDgMuvdiQ5ouY=; b=Wu8SnqYzNHGfnE wFu1CbGfewfdZ5rNA5JjWg7E7H5/a9qylF2aC5bMAW2XFHZwjZEYZ9xj7d8N1xj5R3iEN69dlhTHX TZS/rqvvFlVVytoUKN6FIPPvwc2cC6PgBx5156zv1iaVXavbZcLvOlsPwn9RJdWg9/B4G1nKiy93V nxeLvfPAN7NUxhwpt5+baeoZfOdsTHXxlkOfqODWb/yH01vSzccZkEUszdfutLV0ONzP/bfS+Gl/U qmAxJmjEEBpYsbk2yBUL79a2RHFYLRi2IeNZmDPOf4Eqm/h7WeC4BFl2+BMw44/Rgd0fNyY+8Yaqi zYjw1q+3WNXAkLYfB/sg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tcLiW-000000024HQ-0ByU; Mon, 27 Jan 2025 09:47:40 +0000 Received: from mail-wr1-x42a.google.com ([2a00:1450:4864:20::42a]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tcLes-000000023Do-0mwg for linux-riscv@lists.infradead.org; Mon, 27 Jan 2025 09:43:56 +0000 Received: by mail-wr1-x42a.google.com with SMTP id ffacd0b85a97d-386329da1d9so2116873f8f.1 for ; Mon, 27 Jan 2025 01:43:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1737971032; x=1738575832; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=C5hVtVN5ts8qRfTru3U6IxaXR2nRBQvNb9/UQuUojb8=; b=fX9bd/0O9A8gsb5s5gsmSyBZf8wUDqUzXo6aDtGwG3CZesAU1UdGHGthq1PWMFkn7r JReEXjAzq4TD4RsctKoHwBZY34QxfMmehdJxKY+JPxVkBNcURWwtsyrMIKOqMkyU3BCh 6O7f9B0WJL8CaPedFRDFZatakwp5cJltcplbKDDv6ojOnrOAddu6XcgXqJNgYU2gEEQm 6wjuDwmP/vji0du3Rl4cOvIlIZvnBIwNBLfviZsJnILKWyKmdZ2hH3CRxDDzhgqrdCBi 7kuJo/x8cHS7TUdP9k2F1sjr0XOamB1f9PcaNtKhKm+smMljXDS1Uv2shxzytFcPUQpH FA5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737971032; x=1738575832; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=C5hVtVN5ts8qRfTru3U6IxaXR2nRBQvNb9/UQuUojb8=; b=Snag1aTelIKV+uRLMpnj8XaRefmYRgcXMUwa8/wA8u+dO+PkrESkW0t94/zsPy8q5O uJeEvAuaA+mU323zFS59rYT1Fr7jf/zp7klQ7G9S3nrK4dEIs4d/Fy5WgyCp5ewhafm0 xU7pz1SBrd0Afp/KIn2WhdCtfpGXWmi/QxxWNtMvQYueBlK85722EIt0NAJ3e6TBK5/r B3T/YWzLsHnlALgNfa1aN+KIQBDZewWhWRji6mpdwYYh03u17TSF2CS4d0AG+SWsxOkB PxcP4sIlPVGTU92KBXfJ3XvoAFSH4e1AcTKTlgaos8hIt81vnl9+GtNGgG8mHsNPe5uO 6rdA== X-Forwarded-Encrypted: i=1; AJvYcCXm8pfP41djFmpJoiIjkRrJ9OBcdbb116Zb1ofPThvPByXW/9z4mnzU1jD2PA2syEJa6gmkudNfxQoqhw==@lists.infradead.org X-Gm-Message-State: AOJu0Yz2zs0GXS9uC8dRl0zqW9hxZKKs1/f4BvP6+LsiMi+T7sjdWBvD OjpRA+i8PJmmNbNLW6P37QdrfaddKQMIL6T9rFGr5WVwhSO8dayDiC7scBlXdp4= X-Gm-Gg: ASbGnctEeNwHsrW+YL5mXIFeMw7WcIdoLDw+7kjvcAqnRb0H6eUiFvl2peDPuDdsoGE LRNyfj6DViOoXzugFHMPURa2CUbs1Gy1yUCBgdy/M2dgtF1PDTLc+l5BDk56MfNStAPaGMOlHsq R+ypPHaRjWlFJQaH/jxHxI24m5A7ueDjNLd9d+gy/fkooH+ZZ5sp22LXxh7ER8V9QTmQje3WsfQ /PF2BOvxycSHWuApcCeBKoYIN2u9vc1jBhCh125bXNnT14IZqJLSxvSe0yRW7RXyZDcY5BD+ZGt 14vIn+TfqhQs9d229/Cn12Cfu5iwUNs= X-Google-Smtp-Source: AGHT+IFXkE/IMZA1k4v5FDtRJn2j5Onh6MT51lwkeaLHVsBqz96ACklS2Q489STBeMVgKrxepukPbg== X-Received: by 2002:a05:6000:18a8:b0:385:f69a:7e5f with SMTP id ffacd0b85a97d-38bf57a9a86mr46329622f8f.38.1737971032476; Mon, 27 Jan 2025 01:43:52 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3e22:3e78:ce5a:32c3]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-38c2a17d6a6sm10405111f8f.26.2025.01.27.01.43.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jan 2025 01:43:52 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v4 8/9] mm: Use common huge_ptep_set_wrprotect() function for riscv/arm64 Date: Mon, 27 Jan 2025 10:35:29 +0100 Message-Id: <20250127093530.19548-9-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250127093530.19548-1-alexghiti@rivosinc.com> References: <20250127093530.19548-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250127_014354_224581_B8FDF0E0 X-CRM114-Status: GOOD ( 13.33 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 --- arch/arm64/mm/hugetlbpage.c | 27 --------------------------- arch/riscv/include/asm/hugetlb.h | 4 ---- arch/riscv/include/asm/pgtable.h | 7 ++++--- arch/riscv/mm/hugetlbpage.c | 22 ---------------------- include/linux/hugetlb_contpte.h | 4 ++++ mm/hugetlb_contpte.c | 22 ++++++++++++++++++++++ 7 files changed, 30 insertions(+), 59 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index 654f5f2f03a3..fd1de0caad3f 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -35,9 +35,6 @@ static inline void arch_clear_hugetlb_flags(struct folio *folio) pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT -extern void huge_ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep); #define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 643ba2043f0f..0430cb41f381 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -277,33 +277,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -void huge_ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep) -{ - unsigned long pfn, dpfn; - pgprot_t hugeprot; - int ncontig, i; - size_t pgsize; - pte_t pte; - - if (!pte_cont(__ptep_get(ptep))) { - __ptep_set_wrprotect(mm, addr, ptep); - return; - } - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - dpfn = pgsize >> PAGE_SHIFT; - - pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); - pte = pte_wrprotect(pte); - - hugeprot = pte_pgprot(pte); - pfn = pte_pfn(pte); - - for (i = 0; i < ncontig; i++, ptep++, addr += pgsize, pfn += dpfn) - __set_ptes(mm, addr, ptep, pfn_pte(pfn, hugeprot), 1); -} - pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) { diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index bf533c2cef84..4c692dd82779 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -24,10 +24,6 @@ bool arch_hugetlb_migration_supported(struct hstate *h); pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep); -#define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT -void huge_ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, pte_t *ptep); - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte diff --git a/arch/riscv/include/asm/pgtable.h b/arch/riscv/include/asm/pgtable.h index 290d5fbfe031..5a29153a4013 100644 --- a/arch/riscv/include/asm/pgtable.h +++ b/arch/riscv/include/asm/pgtable.h @@ -645,9 +645,8 @@ static inline pte_t __ptep_get_and_clear(struct mm_struct *mm, return pte; } -#define __HAVE_ARCH_PTEP_SET_WRPROTECT -static inline void ptep_set_wrprotect(struct mm_struct *mm, - unsigned long address, pte_t *ptep) +static inline void __ptep_set_wrprotect(struct mm_struct *mm, + unsigned long address, pte_t *ptep) { atomic_long_and(~(unsigned long)_PAGE_WRITE, (atomic_long_t *)ptep); } @@ -741,6 +740,8 @@ static inline pte_t ptep_get(pte_t *ptep) #define pte_clear __pte_clear #define __HAVE_ARCH_PTEP_SET_ACCESS_FLAGS #define ptep_set_access_flags __ptep_set_access_flags +#define __HAVE_ARCH_PTEP_SET_WRPROTECT +#define ptep_set_wrprotect __ptep_set_wrprotect #define pgprot_nx pgprot_nx static inline pgprot_t pgprot_nx(pgprot_t _prot) diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 0e2ca7327479..8963a4e77742 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -173,28 +173,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -void huge_ptep_set_wrprotect(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep) -{ - pte_t pte = ptep_get(ptep); - pte_t orig_pte; - int pte_num; - - if (!pte_napot(pte)) { - ptep_set_wrprotect(mm, addr, ptep); - return; - } - - pte_num = arch_contpte_get_num_contig(mm, addr, ptep, 0, NULL); - - orig_pte = get_clear_contig_flush(mm, addr, ptep, pte_num); - - orig_pte = pte_wrprotect(orig_pte); - - set_ptes(mm, addr, ptep, orig_pte, pte_num); -} - pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep) diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index fea47035ac38..02bce0ed93d8 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -27,4 +27,8 @@ extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, unsigned long addr, pte_t *ptep, pte_t pte, int dirty); +#define __HAVE_ARCH_HUGE_PTEP_SET_WRPROTECT +extern void huge_ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index 49950c1ce615..de505350ef48 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -14,6 +14,7 @@ * - __ptep_get_and_clear() * - __pte_clear() * - __ptep_set_access_flags() + * - __ptep_set_wrprotect() * - pte_cont() * - arch_contpte_get_num_contig() */ @@ -25,6 +26,7 @@ * - huge_pte_clear() * - huge_ptep_get_and_clear() * - huge_ptep_set_access_flags() + * - huge_ptep_set_wrprotect() */ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -232,3 +234,23 @@ int huge_ptep_set_access_flags(struct vm_area_struct *vma, return 1; } + +void huge_ptep_set_wrprotect(struct mm_struct *mm, + unsigned long addr, pte_t *ptep) +{ + int ncontig; + size_t pgsize; + pte_t pte; + + if (!pte_cont(__ptep_get(ptep))) { + __ptep_set_wrprotect(mm, addr, ptep); + return; + } + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + + pte = get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); + pte = pte_wrprotect(pte); + + set_contptes(mm, addr, ptep, pte, ncontig, pgsize); +} From patchwork Mon Jan 27 09:35:30 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexandre Ghiti X-Patchwork-Id: 13951168 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C9B3FC0218C for ; Mon, 27 Jan 2025 09:50:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=B8ndKthdpGf1PTPaWMZBmLT4+xWQknmAOOXXUrj030M=; b=hXr+IyprzrGNcC 7HRFpAj4Lmcei0pLq8mxy3778RWnYdWpIWGHcmtHKTL3W03rTeUydbwtqXtkpRkFRY200cv1ff7H7 ZHqN1QXum4TW7t3CnDmKutVNLV4sghyG5xF2oRxsC3F+bM1O/TSd1DT+OX0qLvyBTFLg9kneuonJ1 fYt0IaiXOgdpL1nIMrT391z171UglY+PCU9nMr8N55Yhh1euOY5KFEtxO/Qfn3LynHKx52NJqskH3 j89G301xjdzek6SRb8YNK3P8PzHuL26C3LnVxekpVlC2C1JhCvQP0kUxfDitFBT3K6OykGN6iF1KO zAMWTFiGswhIclX9e0qA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.98 #2 (Red Hat Linux)) id 1tcLl9-000000024fj-42AA; Mon, 27 Jan 2025 09:50:23 +0000 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by bombadil.infradead.org with esmtps (Exim 4.98 #2 (Red Hat Linux)) id 1tcLfr-000000023Ro-3CwL for linux-riscv@lists.infradead.org; Mon, 27 Jan 2025 09:44:57 +0000 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-437a92d7b96so43306005e9.2 for ; Mon, 27 Jan 2025 01:44:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1737971094; x=1738575894; darn=lists.infradead.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jsGp3DbjRPLkoj2kwFjv1+EJYiVm/DSEknC/OoKntbQ=; b=gi2jtG0AjyFnAET+fhZEGaeuAymsRSLJ4WZ7Zkd8pUMIBvbOu3uZjXpQeDXTUm2qZK Sw1Qq0aD+OC1AhC4mFyPGygs04fU8AQR/pumTxZ5VeOEGA9K4TdFU2a2zPY303E/Nq+D 24SnfB0iPCHum2UIqtYlhIYhG5CtoxTKybaVq4oieYhqqBmkliITB1mDQK5czOKSTlKU eizn9LxAbjrJcog11cfjI1gzZyORDM0jomtLKUTuKhRsN0MxWrt3g6e8+boddYaSu8Gv hQr5nFI4SzQnu3ozNOHOijXX/IR8LRvHOPtuSJeYUNg8OWLdhrIyV4zQspzPA5Dg6eIq Vqyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737971094; x=1738575894; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jsGp3DbjRPLkoj2kwFjv1+EJYiVm/DSEknC/OoKntbQ=; b=TR1ycLJWpLSQUTugZUYC7iOsHZNcgmL2oLD2Di3X2kXI95FeouEwm12ltbgylAGyft eDec5lTriAYBWKTw5qRXp2XYzvRNdUDRxMngSae2UDRDkEassd5SmqJ5gMukD8V0WD6V Z4llVGrCu5nn2dv9v4WrX53m06CwSTOrhAUhF0rf0XXTmmVOoPKvdtb/3mkilaZ88DK4 K738qGRUfnFjxOVX7iSa4AfaS/x7UytD5nIZEJvVJVQSdQ+8eDZd/N1Y2sRgmu9NeqGi 3U4l6R3rr1ujJMOIrAU++2RlUEjbXc96K8RApVQIyTcf2kcaHEYwa/mjXw8CAc+YRJLM ZONQ== X-Forwarded-Encrypted: i=1; AJvYcCXLmlGuAY/LmDXzLZ5pqEesPtiHqcqLc39kXg2Yl7AAF9DPMH+qNqaW1SdRr/2DuZ4s21M1V18BhPEe0g==@lists.infradead.org X-Gm-Message-State: AOJu0YyvffD6op5RVG+xEi8mhLT/xe9dsEd+KH4tyARD4B4QZDVWAIts Q4BDaLGF6q1sV1JYPV9zay/E0zafYSForTi2HOsa0jGTVpVjzcaUl0BuPhJdgr4= X-Gm-Gg: ASbGncsBkYsYLC2qXRWxJU2tb30cDv8qyCe8Pyxb8pmJwQ5YS1EC61iyBKRp+jh+8b6 G8s+heIxgheqGyRAH8KVUbZsCZ+ylLzMNtIjqIEOUCPkmcEu4sJCI9HjRsKEpQFg6h7tTfURiD9 tqqKoVpk0UuFP8emJsycRy+5KNV3Xx6OgDYaIUwhB6DFU/QarI03iPN4cVx0HnH0i/x90JMcggT TouhJGlH1T8BGsO3f4+sJ4IsrE0QQnVajb0olxBgMaDXByysZgI4ooaNexcSPkgF0oqafK5AGYG PpFdcTLK0D9lEDbeXDoI3dydlLVcDvk= X-Google-Smtp-Source: AGHT+IFDaK9cNMo23MROO6bkzC2nQlBGexsaacNWVhEd87bHI/6OEZGcK91epmkFjxQ01HSL43+DVA== X-Received: by 2002:a05:600c:1ca9:b0:434:f297:8e78 with SMTP id 5b1f17b1804b1-438913cb65fmr390606995e9.7.1737971094091; Mon, 27 Jan 2025 01:44:54 -0800 (PST) Received: from alex-rivos.ba.rivosinc.com ([2001:861:3382:ef90:3e22:3e78:ce5a:32c3]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-438bd47eecasm124006275e9.6.2025.01.27.01.44.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jan 2025 01:44:53 -0800 (PST) From: Alexandre Ghiti To: Catalin Marinas , Will Deacon , Ryan Roberts , Mark Rutland , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Cc: Alexandre Ghiti Subject: [PATCH v4 9/9] mm: Use common huge_ptep_clear_flush() function for riscv/arm64 Date: Mon, 27 Jan 2025 10:35:30 +0100 Message-Id: <20250127093530.19548-10-alexghiti@rivosinc.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20250127093530.19548-1-alexghiti@rivosinc.com> References: <20250127093530.19548-1-alexghiti@rivosinc.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20250127_014455_814869_AD0654B8 X-CRM114-Status: GOOD ( 17.61 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org After some adjustments, both architectures have the same implementation so move it to the generic code. Signed-off-by: Alexandre Ghiti --- arch/arm64/include/asm/hugetlb.h | 3 -- arch/arm64/mm/hugetlbpage.c | 61 -------------------------------- arch/riscv/include/asm/hugetlb.h | 7 +--- arch/riscv/mm/hugetlbpage.c | 51 -------------------------- include/linux/hugetlb_contpte.h | 4 +++ mm/hugetlb_contpte.c | 15 ++++++++ 6 files changed, 20 insertions(+), 121 deletions(-) diff --git a/arch/arm64/include/asm/hugetlb.h b/arch/arm64/include/asm/hugetlb.h index fd1de0caad3f..3f79e4b76711 100644 --- a/arch/arm64/include/asm/hugetlb.h +++ b/arch/arm64/include/asm/hugetlb.h @@ -35,9 +35,6 @@ static inline void arch_clear_hugetlb_flags(struct folio *folio) pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte -#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH -extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep); void __init arm64_hugetlb_cma_reserve(void); diff --git a/arch/arm64/mm/hugetlbpage.c b/arch/arm64/mm/hugetlbpage.c index 0430cb41f381..270e4580e12a 100644 --- a/arch/arm64/mm/hugetlbpage.c +++ b/arch/arm64/mm/hugetlbpage.c @@ -98,53 +98,6 @@ int find_num_contig(struct mm_struct *mm, unsigned long addr, return CONT_PTES; } -/* - * Changing some bits of contiguous entries requires us to follow a - * Break-Before-Make approach, breaking the whole contiguous set - * before we can change any entries. See ARM DDI 0487A.k_iss10775, - * "Misprogramming of the Contiguous bit", page D4-1762. - * - * This helper performs the break step. - */ -static pte_t get_clear_contig(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - pte_t orig_pte = __ptep_get(ptep); - unsigned long i; - - for (i = 0; i < ncontig; i++, addr += pgsize, ptep++) { - pte_t pte = __ptep_get_and_clear(mm, addr, ptep); - - /* - * If HW_AFDBM is enabled, then the HW could turn on - * the dirty or accessed bit for any page in the set, - * so check them all. - */ - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - return orig_pte; -} - -static pte_t get_clear_contig_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pgsize, - unsigned long ncontig) -{ - pte_t orig_pte = get_clear_contig(mm, addr, ptep, pgsize, ncontig); - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - - flush_tlb_range(&vma, addr, addr + (pgsize * ncontig)); - return orig_pte; -} - pte_t *huge_pte_alloc(struct mm_struct *mm, struct vm_area_struct *vma, unsigned long addr, unsigned long sz) { @@ -277,20 +230,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep) -{ - struct mm_struct *mm = vma->vm_mm; - size_t pgsize; - int ncontig; - - if (!pte_cont(__ptep_get(ptep))) - return ptep_clear_flush(vma, addr, ptep); - - ncontig = find_num_contig(mm, addr, ptep, &pgsize); - return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); -} - static int __init hugetlbpage_init(void) { if (pud_sect_supported()) diff --git a/arch/riscv/include/asm/hugetlb.h b/arch/riscv/include/asm/hugetlb.h index 4c692dd82779..63c7e4fa342a 100644 --- a/arch/riscv/include/asm/hugetlb.h +++ b/arch/riscv/include/asm/hugetlb.h @@ -20,14 +20,9 @@ bool arch_hugetlb_migration_supported(struct hstate *h); #endif #ifdef CONFIG_RISCV_ISA_SVNAPOT -#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH -pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, - unsigned long addr, pte_t *ptep); - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags); #define arch_make_huge_pte arch_make_huge_pte - -#endif /*CONFIG_RISCV_ISA_SVNAPOT*/ +#endif /* CONFIG_RISCV_ISA_SVNAPOT */ #include diff --git a/arch/riscv/mm/hugetlbpage.c b/arch/riscv/mm/hugetlbpage.c index 8963a4e77742..ea1ae3a43d45 100644 --- a/arch/riscv/mm/hugetlbpage.c +++ b/arch/riscv/mm/hugetlbpage.c @@ -121,42 +121,6 @@ unsigned long hugetlb_mask_last_page(struct hstate *h) return 0UL; } -static pte_t get_clear_contig(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pte_num) -{ - pte_t orig_pte = ptep_get(ptep); - unsigned long i; - - for (i = 0; i < pte_num; i++, addr += PAGE_SIZE, ptep++) { - pte_t pte = ptep_get_and_clear(mm, addr, ptep); - - if (pte_dirty(pte)) - orig_pte = pte_mkdirty(orig_pte); - - if (pte_young(pte)) - orig_pte = pte_mkyoung(orig_pte); - } - - return orig_pte; -} - -static pte_t get_clear_contig_flush(struct mm_struct *mm, - unsigned long addr, - pte_t *ptep, - unsigned long pte_num) -{ - pte_t orig_pte = get_clear_contig(mm, addr, ptep, pte_num); - struct vm_area_struct vma = TLB_FLUSH_VMA(mm, 0); - bool valid = !pte_none(orig_pte); - - if (valid) - flush_tlb_range(&vma, addr, addr + (PAGE_SIZE * pte_num)); - - return orig_pte; -} - pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) { unsigned long order; @@ -173,21 +137,6 @@ pte_t arch_make_huge_pte(pte_t entry, unsigned int shift, vm_flags_t flags) return entry; } -pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, - unsigned long addr, - pte_t *ptep) -{ - pte_t pte = ptep_get(ptep); - int pte_num; - - if (!pte_napot(pte)) - return ptep_clear_flush(vma, addr, ptep); - - pte_num = arch_contpte_get_num_contig(vma->vm_mm, addr, ptep, 0, NULL); - - return get_clear_contig_flush(vma->vm_mm, addr, ptep, pte_num); -} - static bool is_napot_size(unsigned long size) { unsigned long order; diff --git a/include/linux/hugetlb_contpte.h b/include/linux/hugetlb_contpte.h index 02bce0ed93d8..911b9cd4aa4d 100644 --- a/include/linux/hugetlb_contpte.h +++ b/include/linux/hugetlb_contpte.h @@ -31,4 +31,8 @@ extern int huge_ptep_set_access_flags(struct vm_area_struct *vma, extern void huge_ptep_set_wrprotect(struct mm_struct *mm, unsigned long addr, pte_t *ptep); +#define __HAVE_ARCH_HUGE_PTEP_CLEAR_FLUSH +extern pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep); + #endif /* _LINUX_HUGETLB_CONTPTE_H */ diff --git a/mm/hugetlb_contpte.c b/mm/hugetlb_contpte.c index de505350ef48..d27c7599ce74 100644 --- a/mm/hugetlb_contpte.c +++ b/mm/hugetlb_contpte.c @@ -27,6 +27,7 @@ * - huge_ptep_get_and_clear() * - huge_ptep_set_access_flags() * - huge_ptep_set_wrprotect() + * - huge_ptep_clear_flush() */ pte_t huge_ptep_get(struct mm_struct *mm, unsigned long addr, pte_t *ptep) @@ -254,3 +255,17 @@ void huge_ptep_set_wrprotect(struct mm_struct *mm, set_contptes(mm, addr, ptep, pte, ncontig, pgsize); } + +pte_t huge_ptep_clear_flush(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + struct mm_struct *mm = vma->vm_mm; + size_t pgsize; + int ncontig; + + if (!pte_cont(__ptep_get(ptep))) + return ptep_clear_flush(vma, addr, ptep); + + ncontig = arch_contpte_get_num_contig(mm, addr, ptep, 0, &pgsize); + return get_clear_contig_flush(mm, addr, ptep, pgsize, ncontig); +}