From patchwork Tue Mar 5 11:18:32 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13582150 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BD88C54E41 for ; Tue, 5 Mar 2024 11:19:02 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CD23A6B00BC; Tue, 5 Mar 2024 06:19:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C5B796B00BD; Tue, 5 Mar 2024 06:19:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AACC46B00BE; Tue, 5 Mar 2024 06:19:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 973796B00BC for ; Tue, 5 Mar 2024 06:19:01 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 2E8D080DC8 for ; Tue, 5 Mar 2024 11:19:01 +0000 (UTC) X-FDA: 81862738482.15.8DE0C8D Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf13.hostedemail.com (Postfix) with ESMTP id 8A7E520007 for ; Tue, 5 Mar 2024 11:18:59 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=oDVDa6DQ; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709637539; a=rsa-sha256; cv=none; b=Ixf818dSGzZxeu5EhOXO5PrZIIcwNfQHA3BQLGYr7pzEPY+nrQ1+PAoc13OR134yr53tCy co7uJ3KVr5QsC31MKAZAKgbzicDQENoHjMZXVEj56+d71EdXYROvbrefrDowkmgDLqaz+Y qiG/KQ1H1DvSEzga7YPTF38Jd1u4ENI= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=oDVDa6DQ; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf13.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709637539; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VGKYLFPfUGJHtT00zVJnn7onAYX1D5CI2UtbLSnte14=; b=uu8ndoPj7yTzCTN8Zxv29CE6oupLIchIWXrpCkIzO5RC6vYIvZK6jSbs9zjkDb0ikJgl4z YuMnnhw+PckuFdzQ7n6TB1BCKXB54sMNPjSKogMpkqPBS6wTcvkjD6lDjwTp94wy69sziU GK1+bqvG7P8FICS5jBp3pRTrr0HYgEg= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id B32A861320; Tue, 5 Mar 2024 11:18:58 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9A2EEC43390; Tue, 5 Mar 2024 11:18:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1709637538; bh=EE9BRQu4WZxxi1OZrE2CKyiBDx8STpwb4qcA00ZbW2s=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=oDVDa6DQwNTcv3NwnWsgiiMKOIV7/K6NDlVTdgGBiYwnB7EisVZPVobiznADeFy7f ZNkpViGakTvQNDN5lcqWymg90055eoDxJGkjcEo0SLPgVGObALAW8gzuwJbHHbIrnQ GzaG57EN9Qa1s6MgxTTglPO3tlZGinvm+yuj5Uw6p0CXEuATlMU768QtIgpbaacA4s QEjkVrnPb4cNOVhk/MGHgtTp878F1cJdcZAkgg9j458WXiok8jtBXFcnRaKQl71Tny m6tk3S6ClgeMTJJuB+n/Eg16neudzyOU+cRnBAYeXVb8c2Fit16B8JJyUe8jG5cqdP Ikg+UfLv0Xv6A== From: Leon Romanovsky To: Christoph Hellwig , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Jason Gunthorpe , Chaitanya Kulkarni Cc: Leon Romanovsky , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?utf-8?b?SsOpcsO0bWUgR2xp?= =?utf-8?b?c3Nl?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, linux-mm@kvack.org, Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , Dan Williams , "jack@suse.com" , Zhu Yanjun Subject: [RFC RESEND 01/16] mm/hmm: let users to tag specific PFNs Date: Tue, 5 Mar 2024 13:18:32 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 In-Reply-To: References: MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: 8A7E520007 X-Stat-Signature: pd43yeiq9kkoctdywsto3hchtawxr6up X-HE-Tag: 1709637539-502802 X-HE-Meta: U2FsdGVkX1+QVmoW2tA4Lp6S+BGDZl9HdMakmpxxpFCKT+iE7ZGeY6TMv8rVQ01wDK7PRTkZpgD0W5JsyqGmGLnLoG0OP5dnjlvB/7GKzMsco38ib4x6eviawqY25r9UYcV+5cDFbtuLQRvI9asCA8+ZgjQL4wxAOdT0yDlGi27q7I1fAy8yp8OSSwt2vNKjAVrXGnQqqsz7E1lYp/ziaSMoHMixN0xCT9WD78TajB3QvEHuJsBnvrH7HUBJm/owOP6SWYsdU+sybxrcjjA7lSRBIEZsAhI4K9qpwVon+nVFB49szthreFvLO6S+jCV3buFRzy9FEaYtJs3gwgNmgMvhKOkRllyk1Mv1lg4RYFNSw1DBF/+dlLfd4MG/HDuEnwjfikd5x9coWAEW+p22QhLk3cWKvFzvgrkR2EE0GVg/rzkasFCftBMYkiAxy9AibKJGRIWN+evzVJ8f/kPlaQClQNDANZz1s6suVWck63tr4VmQQRRhntqA2n62D8WKTmipZO3y3CpY389QZwSVxP0a9ObKa/4KXWv61JZ7tSSuqBcGi49PeSnRwwAlBymOeSdk4KE5Sv/FG6hJVzbSEfFrkO/LMvjM+2mi8wKE+C/EoTIq0iOIyAzX648PXoTnkljWDb1YjmY/k6eoiWJLrKT8c6VFdGppKs5tA6V5jvGEzUwlHNFcvdrv+PJr4tUCOzEdX+hV2wETtU9f2y3pkPQF0uh68Yn/CuzfHULxw3R+Gu8iNgcdpNDKi3/s2lMOTVd3gkTXKaYpuaFBVO+4Ff1cfavUxak8r1RKGgnDkUVO7IRDvEUVOImk6Akt2a+/vRV4jsoFbA0NijikDwKegBg1tJggWH8oDK6Zl1gqi0KBE+x3cnjd26MlgJRImZd1lz191ZeG7nFceWqQWKb7Y0toEXjsT/6Nx/v5iSM2aCjz+FC/8NGjki41oihgNYsrFL1SHmzbE9flBkY22vb Cm6LjpF4 19+NfPwd74RZTHq19iNzF98HBFqNWADIDn7TqSGLqyKJADZsF/Kd8Mci9xcd4UIvDrWhToPZurbhAKrLYywOcza1otAF31DlYNaCITMwO5cBjlXjIO9NV27q+yfPXcw8OKl3oBQxWbhQDFctynm8whd+TtL7J1ijMcvGNxoozchMrf/77cKuU1VlfTG7Ec0h24oB28V/7A1uf5SwuYSXS/69KXpf4UBHuleWh4rXD3jcoeScMpOZsgKrgqg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky Introduce new sticky flag, which isn't overwritten by HMM range fault. Such flag allows users to tag specific PFNs with extra data in addition to already filled by HMM. Signed-off-by: Leon Romanovsky --- include/linux/hmm.h | 3 +++ mm/hmm.c | 34 +++++++++++++++++++++------------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 126a36571667..b90902baa593 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -23,6 +23,7 @@ struct mmu_interval_notifier; * HMM_PFN_WRITE - if the page memory can be written to (requires HMM_PFN_VALID) * HMM_PFN_ERROR - accessing the pfn is impossible and the device should * fail. ie poisoned memory, special pages, no vma, etc + * HMM_PFN_STICKY - Flag preserved on input-to-output transformation * * On input: * 0 - Return the current state of the page, do not fault it. @@ -36,6 +37,8 @@ enum hmm_pfn_flags { HMM_PFN_VALID = 1UL << (BITS_PER_LONG - 1), HMM_PFN_WRITE = 1UL << (BITS_PER_LONG - 2), HMM_PFN_ERROR = 1UL << (BITS_PER_LONG - 3), + /* Sticky lag, carried from Input to Output */ + HMM_PFN_STICKY = 1UL << (BITS_PER_LONG - 7), HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 8), /* Input flags */ diff --git a/mm/hmm.c b/mm/hmm.c index 277ddcab4947..9645a72beec0 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -44,8 +44,10 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end, { unsigned long i = (addr - range->start) >> PAGE_SHIFT; - for (; addr < end; addr += PAGE_SIZE, i++) - range->hmm_pfns[i] = cpu_flags; + for (; addr < end; addr += PAGE_SIZE, i++) { + range->hmm_pfns[i] &= HMM_PFN_STICKY; + range->hmm_pfns[i] |= cpu_flags; + } return 0; } @@ -202,8 +204,10 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, return hmm_vma_fault(addr, end, required_fault, walk); pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) - hmm_pfns[i] = pfn | cpu_flags; + for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) { + hmm_pfns[i] &= HMM_PFN_STICKY; + hmm_pfns[i] |= pfn | cpu_flags; + } return 0; } #else /* CONFIG_TRANSPARENT_HUGEPAGE */ @@ -236,7 +240,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (required_fault) goto fault; - *hmm_pfn = 0; + *hmm_pfn = *hmm_pfn & HMM_PFN_STICKY; return 0; } @@ -253,14 +257,14 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, cpu_flags = HMM_PFN_VALID; if (is_writable_device_private_entry(entry)) cpu_flags |= HMM_PFN_WRITE; - *hmm_pfn = swp_offset_pfn(entry) | cpu_flags; + *hmm_pfn = (*hmm_pfn & HMM_PFN_STICKY) | swp_offset_pfn(entry) | cpu_flags; return 0; } required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (!required_fault) { - *hmm_pfn = 0; + *hmm_pfn = *hmm_pfn & HMM_PFN_STICKY; return 0; } @@ -304,11 +308,11 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, pte_unmap(ptep); return -EFAULT; } - *hmm_pfn = HMM_PFN_ERROR; + *hmm_pfn = (*hmm_pfn & HMM_PFN_STICKY) | HMM_PFN_ERROR; return 0; } - *hmm_pfn = pte_pfn(pte) | cpu_flags; + *hmm_pfn = (*hmm_pfn & HMM_PFN_STICKY) | pte_pfn(pte) | cpu_flags; return 0; fault: @@ -453,8 +457,10 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, } pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - for (i = 0; i < npages; ++i, ++pfn) - hmm_pfns[i] = pfn | cpu_flags; + for (i = 0; i < npages; ++i, ++pfn) { + hmm_pfns[i] &= HMM_PFN_STICKY; + hmm_pfns[i] |= pfn | cpu_flags; + } goto out_unlock; } @@ -512,8 +518,10 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, } pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT); - for (; addr < end; addr += PAGE_SIZE, i++, pfn++) - range->hmm_pfns[i] = pfn | cpu_flags; + for (; addr < end; addr += PAGE_SIZE, i++, pfn++) { + range->hmm_pfns[i] &= HMM_PFN_STICKY; + range->hmm_pfns[i] |= pfn | cpu_flags; + } spin_unlock(ptl); return 0;