From patchwork Tue Mar 5 10:15:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Leon Romanovsky X-Patchwork-Id: 13581928 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A349CC54E41 for ; Tue, 5 Mar 2024 10:15:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3A5FC6B0096; Tue, 5 Mar 2024 05:15:44 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3576E6B00A8; Tue, 5 Mar 2024 05:15:44 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 184D56B00AA; Tue, 5 Mar 2024 05:15:44 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 03AAD6B0096 for ; Tue, 5 Mar 2024 05:15:44 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B8AC280D7C for ; Tue, 5 Mar 2024 10:15:43 +0000 (UTC) X-FDA: 81862578966.06.A6EDC35 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf09.hostedemail.com (Postfix) with ESMTP id 15B04140017 for ; Tue, 5 Mar 2024 10:15:40 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=qOqLaLn3; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709633741; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=VGKYLFPfUGJHtT00zVJnn7onAYX1D5CI2UtbLSnte14=; b=B0eWJh8nord4QahAnGHyzRu07WtYMsnTA6/qBUdSLD8N0HGvcQDjqw0wW2kcOpc+O7VXrm ik+nZp5iL0brlwpLxq/fPXcJwhnXg4SRn85f6WyAPFHpieVCoK8ULKXAapgVVe4LkQcryn RFte0myU2LHJ5pRQTjEgjYGy7GgzxVo= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=qOqLaLn3; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf09.hostedemail.com: domain of leon@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=leon@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709633741; a=rsa-sha256; cv=none; b=EK0sc6Sp8EmxpgH95VCumqKfikAXNnL6KcRhlq+0OPD2arW/V6iwEeubZgDv0f5uNYVyvs eYLUL6kbIoJNtlDY9vLcd6eT+BYW4RWy0u7kFUC16NKXTPbbF6xIwEtI85Laclcb2CSi0c yETKwE3G5uhflaLb23EZZNZOeigO08Y= Received: from smtp.kernel.org (transwarp.subspace.kernel.org [100.75.92.58]) by dfw.source.kernel.org (Postfix) with ESMTP id 3BC006148C; Tue, 5 Mar 2024 10:15:40 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id F3B5AC43390; Tue, 5 Mar 2024 10:15:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1709633739; bh=EE9BRQu4WZxxi1OZrE2CKyiBDx8STpwb4qcA00ZbW2s=; h=From:To:Cc:Subject:Date:From; b=qOqLaLn3iBOQV8Vpk9C7fIIoeo4gt3eJqv3Qk0ac57+rTecyJfbAwk5F9oV2CYlYi iCxE76g9a4VsFGyienbFOKBwslCX8QnHQSkps7BVaKOrdU3YgbTgSp3wgD5vuXtzjH Auf9D1w6HpIVDYu45nDqaeQDo5Qxg7SqI71cb4CF5bX33POEIF22qvG6dZC3feFMSL L8/YYU6bMkn9HMPttYWzUIqXiOvpy+fN5ejAW5p5z2cSEsV2UMHJjs77k01LVnvlWp tAQhYEiHcEmzmfYrM9sBM2FOv7/h5MX0oMiIYnb/sIlyxLNKRLomqakayy9734ligO 6YKcLLle3sv0Q== From: Leon Romanovsky To: Christoph Hellwig , Robin Murphy , Marek Szyprowski , Joerg Roedel , Will Deacon , Jason Gunthorpe , Chaitanya Kulkarni Cc: Leon Romanovsky , Jonathan Corbet , Jens Axboe , Keith Busch , Sagi Grimberg , Yishai Hadas , Shameer Kolothum , Kevin Tian , Alex Williamson , =?utf-8?b?SsOpcsO0bWUgR2xp?= =?utf-8?b?c3Nl?= , Andrew Morton , linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-block@vger.kernel.org, linux-rdma@vger.kernel.org, iommu@lists.linux.dev, linux-nvme@lists.infradead.org, kvm@vger.kernel.org, linux-mm@kvack.org, Bart Van Assche , Damien Le Moal , Amir Goldstein , "josef@toxicpanda.com" , "Martin K. Petersen" , "daniel@iogearbox.net" , Dan Williams , "jack@suse.com" , Zhu Yanjun Subject: [RFC 01/16] mm/hmm: let users to tag specific PFNs Date: Tue, 5 Mar 2024 12:15:11 +0200 Message-ID: X-Mailer: git-send-email 2.44.0 MIME-Version: 1.0 X-Rspamd-Queue-Id: 15B04140017 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: abcaoxqrxenqcnudzp9w6qj7pi9pjwmx X-HE-Tag: 1709633740-29801 X-HE-Meta: U2FsdGVkX18gwy+Mjh1AFJCmS8tLzOWV/hl1Z7gvv51dEN/2O/ejsEcV5VWS1QNoV1TBhT5evIAh+ZE6nol9irKGyprv5QLt+WjiTahYzivz6Yo9Q9iariGS6K3ZbJj+JhAOJYJc78IBlk27sSzq/NrG63UTcIKSEcq7QsiGnPoDhlmcCE52bYKnS3+IZXmsyWsu+RDsYliaYeNd2wJ/jqhZ57VQLiURpcp9yrLooVHFGb9H+b/mVet4Zx5MD1p+OkO8OcBx+xI9g9/kN7+8xqroKdMpx9HaJcJu71ch7GhRMoRv9E0fru0zVLVxIXFvuBo0T7dXoocmDAjEKXSdJxS99agYDmvRlu+WhToD37phrpVvLcp0C+u5sCH9cYZSInYCaTG/IyqohxJYU4wKX7fpn+13nHaCT0QdNc3WeZhSClaDldDz2toUJMhJQtRRUTPZNgEDYo4LLC9G+6sN3XoIDR7Xfx6cOtulZ3uR4PWLa+QsHAsmcF0tuTopGI9KPPHxaqHiTQ1HRVrAj+yI5hP4/2c/m3oThpTCh+CiTaNZisBhDIvUY7QmzYkwc5IhoWXctj9zvgUbTOfrdJSz1G9De3b4zXusNbqs7e1eeYfajccfiG4ouiFIpas6yaxb0OLqXrLTlgsWiqFd5e6dx04toHR00WF26nNletGvlIqRnANuUT/RJmjZDHeHbwr1anbl7FrEF8UiLIVlV20V3NHRs2chq9bNJ4DPoHnDpzRfOBCFiLucz1hVBarKak98+vqGY8CjxS4ZamTK/Dcn8MzIQcbymx39eFpcndwcoxkZsGt0WGik034EKZqKliB7hYQ+I7AmlP/kzcfbtP6uUb8xe9u+qIjoCXgU6V8Qzc+FgSL0bAeKY1IESEdm0Qat1F1OsOoQEQiFquwxNaGOssr4IIDnmw2a3FlxJWEHrfI5fp5RiCfN5FOSeRiNVPKooqV6Hy9tdmCujT9M4+t wX/rBTt/ HN+O+pGsp2bI0m9sxGj0dZxly8KtqFn/cR18o+ClvVzK1NHTa3Cw++oanx4A4ggJfa4cxsP7jV+cJFj5SKseNDmfro9oWAZddbDk9NiKfegiMSKWXfrAFw3ajJPS43WPoxkc1j/bt3cGGAaNb2MhADENuewGyVua4jHiQsoxU4+VuaRNZ/eKw4At59JtK3sCj/AvzmmwY5j2GnIIaVfaVxRhapab+3vt+9qBZIs/0Fj3uS3E8VBAFWPiGfw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Leon Romanovsky Introduce new sticky flag, which isn't overwritten by HMM range fault. Such flag allows users to tag specific PFNs with extra data in addition to already filled by HMM. Signed-off-by: Leon Romanovsky --- include/linux/hmm.h | 3 +++ mm/hmm.c | 34 +++++++++++++++++++++------------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index 126a36571667..b90902baa593 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -23,6 +23,7 @@ struct mmu_interval_notifier; * HMM_PFN_WRITE - if the page memory can be written to (requires HMM_PFN_VALID) * HMM_PFN_ERROR - accessing the pfn is impossible and the device should * fail. ie poisoned memory, special pages, no vma, etc + * HMM_PFN_STICKY - Flag preserved on input-to-output transformation * * On input: * 0 - Return the current state of the page, do not fault it. @@ -36,6 +37,8 @@ enum hmm_pfn_flags { HMM_PFN_VALID = 1UL << (BITS_PER_LONG - 1), HMM_PFN_WRITE = 1UL << (BITS_PER_LONG - 2), HMM_PFN_ERROR = 1UL << (BITS_PER_LONG - 3), + /* Sticky lag, carried from Input to Output */ + HMM_PFN_STICKY = 1UL << (BITS_PER_LONG - 7), HMM_PFN_ORDER_SHIFT = (BITS_PER_LONG - 8), /* Input flags */ diff --git a/mm/hmm.c b/mm/hmm.c index 277ddcab4947..9645a72beec0 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -44,8 +44,10 @@ static int hmm_pfns_fill(unsigned long addr, unsigned long end, { unsigned long i = (addr - range->start) >> PAGE_SHIFT; - for (; addr < end; addr += PAGE_SIZE, i++) - range->hmm_pfns[i] = cpu_flags; + for (; addr < end; addr += PAGE_SIZE, i++) { + range->hmm_pfns[i] &= HMM_PFN_STICKY; + range->hmm_pfns[i] |= cpu_flags; + } return 0; } @@ -202,8 +204,10 @@ static int hmm_vma_handle_pmd(struct mm_walk *walk, unsigned long addr, return hmm_vma_fault(addr, end, required_fault, walk); pfn = pmd_pfn(pmd) + ((addr & ~PMD_MASK) >> PAGE_SHIFT); - for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) - hmm_pfns[i] = pfn | cpu_flags; + for (i = 0; addr < end; addr += PAGE_SIZE, i++, pfn++) { + hmm_pfns[i] &= HMM_PFN_STICKY; + hmm_pfns[i] |= pfn | cpu_flags; + } return 0; } #else /* CONFIG_TRANSPARENT_HUGEPAGE */ @@ -236,7 +240,7 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (required_fault) goto fault; - *hmm_pfn = 0; + *hmm_pfn = *hmm_pfn & HMM_PFN_STICKY; return 0; } @@ -253,14 +257,14 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, cpu_flags = HMM_PFN_VALID; if (is_writable_device_private_entry(entry)) cpu_flags |= HMM_PFN_WRITE; - *hmm_pfn = swp_offset_pfn(entry) | cpu_flags; + *hmm_pfn = (*hmm_pfn & HMM_PFN_STICKY) | swp_offset_pfn(entry) | cpu_flags; return 0; } required_fault = hmm_pte_need_fault(hmm_vma_walk, pfn_req_flags, 0); if (!required_fault) { - *hmm_pfn = 0; + *hmm_pfn = *hmm_pfn & HMM_PFN_STICKY; return 0; } @@ -304,11 +308,11 @@ static int hmm_vma_handle_pte(struct mm_walk *walk, unsigned long addr, pte_unmap(ptep); return -EFAULT; } - *hmm_pfn = HMM_PFN_ERROR; + *hmm_pfn = (*hmm_pfn & HMM_PFN_STICKY) | HMM_PFN_ERROR; return 0; } - *hmm_pfn = pte_pfn(pte) | cpu_flags; + *hmm_pfn = (*hmm_pfn & HMM_PFN_STICKY) | pte_pfn(pte) | cpu_flags; return 0; fault: @@ -453,8 +457,10 @@ static int hmm_vma_walk_pud(pud_t *pudp, unsigned long start, unsigned long end, } pfn = pud_pfn(pud) + ((addr & ~PUD_MASK) >> PAGE_SHIFT); - for (i = 0; i < npages; ++i, ++pfn) - hmm_pfns[i] = pfn | cpu_flags; + for (i = 0; i < npages; ++i, ++pfn) { + hmm_pfns[i] &= HMM_PFN_STICKY; + hmm_pfns[i] |= pfn | cpu_flags; + } goto out_unlock; } @@ -512,8 +518,10 @@ static int hmm_vma_walk_hugetlb_entry(pte_t *pte, unsigned long hmask, } pfn = pte_pfn(entry) + ((start & ~hmask) >> PAGE_SHIFT); - for (; addr < end; addr += PAGE_SIZE, i++, pfn++) - range->hmm_pfns[i] = pfn | cpu_flags; + for (; addr < end; addr += PAGE_SIZE, i++, pfn++) { + range->hmm_pfns[i] &= HMM_PFN_STICKY; + range->hmm_pfns[i] |= pfn | cpu_flags; + } spin_unlock(ptl); return 0;