From patchwork Tue Oct 12 17:12:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: "Sierra Guiza, Alejandro (Alex)" X-Patchwork-Id: 12553281 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70766C433EF for ; Tue, 12 Oct 2021 17:13:24 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0FEB2610C9 for ; Tue, 12 Oct 2021 17:13:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 0FEB2610C9 Authentication-Results: mail.kernel.org; dmarc=fail (p=quarantine dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id A127490000C; Tue, 12 Oct 2021 13:13:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C26E900005; Tue, 12 Oct 2021 13:13:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83C0290000C; Tue, 12 Oct 2021 13:13:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0175.hostedemail.com [216.40.44.175]) by kanga.kvack.org (Postfix) with ESMTP id 6A80A900005 for ; Tue, 12 Oct 2021 13:13:11 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 3061E1802EC3D for ; Tue, 12 Oct 2021 17:13:11 +0000 (UTC) X-FDA: 78688430982.19.1D28A17 Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2078.outbound.protection.outlook.com [40.107.237.78]) by imf10.hostedemail.com (Postfix) with ESMTP id A4C9F6001987 for ; Tue, 12 Oct 2021 17:13:10 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bYuc6nPCRdHqQljtSwaei3Aur5OopUXtHwO1SiYiqylgKGvIj+4S5lApenyq4jR2KYwTwhOpbMYjFv5IUj/xAwyxCpc86SNHmlefWHRJsNECw3MQBjAg/hBUxwg0o43kgpuCPHU3qTaFR7VSN/AVuHSJyOOeKOO7b2YSIfzFQG0eKmfrecNshVR1O43S9H8/MdsChFJW6yS8n2IQ6XTlF1sqkmdUFHNdSIwFD+seVO3SMqAbY9GIe2GKV57mBvHzsYTYlq71W8iep4K3EY6MVK9qxE6W6/TE5CEF4OwswsaKA9YnJI+IC11NSPV3IXoVHH6YBtLhKvir/2jdp3ubcA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=1ZOiCyZwhLuHxcR5uQwpzd4k0pBr38tyPpHPo6FypKE=; b=D9uM5OOCr+HasHrtkySYt7rDhufw/+yGHSyc1JZUXwWLQLa0dnXxh45pB9v6hbMgkk/CVzSt+ATqNkmo1SxMRgZvbdTCXXggfQhIKws3USJG7SV0nFO7lGZGVqgJgTb5smbTAmjlIZMvwDH8LzcbMsPO+PtcqcHpDACeGirFytQq+lvp+R+Rx7A9ZA7A/pYIZw8LprsFkCuqqyFDL9oPQ6RrZnpPzkOkRJ56sY4BdeJjkNrnJApnXEj5rUzMvjNNJSHheH/9nsd5rwD2EX1eYsUIpMsSERleuTMXRLxHaW4g/+reP/QLC4Ci+vd3doDRYGZ/+OpIO+3MPTCY1PugPQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 165.204.84.17) smtp.rcpttodomain=linux-foundation.org smtp.mailfrom=amd.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=1ZOiCyZwhLuHxcR5uQwpzd4k0pBr38tyPpHPo6FypKE=; b=BLryTInlrrBdab1Exje+WMrBO4GXAOitMAdTL78t0Ba3OQUK1CBLgA14fGSMZBB/uUI2jJcA4YTh1VLfxm6ZfFJBLGN9OcMOdN4/Tim4ZhHM4kDtghs/hZB2XATY2VadmuMM8xjiDLW1zA8P1tzmb13bOUW7HjeY3VWp8H5HrWc= Received: from MWHPR14CA0012.namprd14.prod.outlook.com (2603:10b6:300:ae::22) by DM6PR12MB2716.namprd12.prod.outlook.com (2603:10b6:5:49::25) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.25; Tue, 12 Oct 2021 17:13:08 +0000 Received: from CO1NAM11FT018.eop-nam11.prod.protection.outlook.com (2603:10b6:300:ae:cafe::a6) by MWHPR14CA0012.outlook.office365.com (2603:10b6:300:ae::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4587.24 via Frontend Transport; Tue, 12 Oct 2021 17:13:08 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; linux-foundation.org; dkim=none (message not signed) header.d=none;linux-foundation.org; dmarc=pass action=none header.from=amd.com; Received-SPF: Pass (protection.outlook.com: domain of amd.com designates 165.204.84.17 as permitted sender) receiver=protection.outlook.com; client-ip=165.204.84.17; helo=SATLEXMB04.amd.com; Received: from SATLEXMB04.amd.com (165.204.84.17) by CO1NAM11FT018.mail.protection.outlook.com (10.13.175.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.4587.18 via Frontend Transport; Tue, 12 Oct 2021 17:13:07 +0000 Received: from alex-MS-7B09.amd.com (10.180.168.240) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2308.8; Tue, 12 Oct 2021 12:13:04 -0500 From: Alex Sierra To: , , , , , CC: , , , , , Subject: [PATCH v1 10/12] lib: add support for device coherent type in test_hmm Date: Tue, 12 Oct 2021 12:12:45 -0500 Message-ID: <20211012171247.2861-11-alex.sierra@amd.com> X-Mailer: git-send-email 2.32.0 In-Reply-To: <20211012171247.2861-1-alex.sierra@amd.com> References: <20211012171247.2861-1-alex.sierra@amd.com> MIME-Version: 1.0 X-Originating-IP: [10.180.168.240] X-ClientProxiedBy: SATLEXMB03.amd.com (10.181.40.144) To SATLEXMB04.amd.com (10.181.40.145) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 9ea873c8-9c64-49bf-a066-08d98da38f8a X-MS-TrafficTypeDiagnostic: DM6PR12MB2716: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4sIcHxhKHeHvWkwJY8NT8nwh2WQaY1WHLxJZmm4z8tlyyqHKU8rcRy1ZO+4yRrRdf/Yukh3c5fvTL+hJOyCW1x2RItUg1LRXuZQ4P8JsdZkdUP/NLpO05JjBcib7ivLqiQ3h9MVTpqDl5NVxgF+vnv9pNZMWonsCREPT6CIH9lO1Qvo4lPyZpVH4ponEAFWYjq4v/UXY0XYPoLzKZo5e8HKZvjoPA+WC3wGICOrQr1V3XD4mv4iHZsnGFbtByaeq6C7oey7bBFO5RnxTu8R7OVRThg3LyoGMXG+r6dZ9trZzUu9ODktqetDJAxGEDq+TVIfl977CvTDDQveEV6zQ+4XVXptFdPY1uk3/PiJM5AWx/MurlL2A0chczibHYPH6swdVvjgqZ+xkZ5qrRxDVxfU9HRHAd2GU2iI0mZv2eg0XsSmA+NNgybSpfI9tcR6wIrnmrN/oW6cb9Xlgi293B4AzrEmrLrHNUell0Pupxq1ZCGaqc992rXnIkNK+zB3F6qicgNm9Euu4elDLrF9nPBnDqlEiTqFeDtcqsUvm0WFpxJmHgDcT17Zcw+y5f7WjEmqiOEKm2EvGIXpG8/qc+L5D9zRyUAgaUYNfFwnff9Xxtb85zJCAw5yBRExRNoXDO9NpTWs+ePN86FW2dLufprwy5azLezTB+m4bbPe/HRsTly8jmKcLvDguAU+7XMh9ON6FNYRG41IJE7mT89hE0083YJ3fRQb1hYGzvYuCjD8= X-Forefront-Antispam-Report: CIP:165.204.84.17;CTRY:US;LANG:en;SCL:1;SRV:;IPV:CAL;SFV:NSPM;H:SATLEXMB04.amd.com;PTR:InfoDomainNonexistent;CAT:NONE;SFS:(4636009)(36840700001)(46966006)(54906003)(86362001)(30864003)(508600001)(81166007)(6666004)(7416002)(82310400003)(1076003)(316002)(70206006)(83380400001)(70586007)(356005)(8676002)(110136005)(426003)(5660300002)(2616005)(336012)(36860700001)(26005)(44832011)(4326008)(16526019)(186003)(36756003)(8936002)(7696005)(47076005)(2906002)(36900700001);DIR:OUT;SFP:1101; X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 12 Oct 2021 17:13:07.5301 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 9ea873c8-9c64-49bf-a066-08d98da38f8a X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d;Ip=[165.204.84.17];Helo=[SATLEXMB04.amd.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT018.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB2716 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: A4C9F6001987 X-Stat-Signature: x7hz4s6uudrdprfyo4jo6xi7esderrna Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=amd.com header.s=selector1 header.b=BLryTInl; spf=pass (imf10.hostedemail.com: domain of Alex.Sierra@amd.com designates 40.107.237.78 as permitted sender) smtp.mailfrom=Alex.Sierra@amd.com; dmarc=pass (policy=quarantine) header.from=amd.com X-HE-Tag: 1634058790-225302 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Device Coherent type uses device memory that is coherently accesible by the CPU. This could be shown as SP (special purpose) memory range at the BIOS-e820 memory enumeration. If no SP memory is supported in system, this could be faked by setting CONFIG_EFI_FAKE_MEMMAP. Currently, test_hmm only supports two different SP ranges of at least 256MB size. This could be specified in the kernel parameter variable efi_fake_mem. Ex. Two SP ranges of 1GB starting at 0x100000000 & 0x140000000 physical address. Ex. efi_fake_mem=1G@0x100000000:0x40000,1G@0x140000000:0x40000 Signed-off-by: Alex Sierra --- lib/test_hmm.c | 195 ++++++++++++++++++++++++++++++++------------ lib/test_hmm_uapi.h | 16 +++- 2 files changed, 157 insertions(+), 54 deletions(-) diff --git a/lib/test_hmm.c b/lib/test_hmm.c index 70a9be0efa00..b349dd920f04 100644 --- a/lib/test_hmm.c +++ b/lib/test_hmm.c @@ -469,6 +469,7 @@ static int dmirror_allocate_chunk(struct dmirror_device *mdevice, unsigned long pfn_first; unsigned long pfn_last; void *ptr; + int ret = -ENOMEM; devmem = kzalloc(sizeof(*devmem), GFP_KERNEL); if (!devmem) @@ -551,7 +552,7 @@ static int dmirror_allocate_chunk(struct dmirror_device *mdevice, } spin_unlock(&mdevice->lock); - return true; + return 0; err_release: mutex_unlock(&mdevice->devmem_lock); @@ -560,7 +561,7 @@ static int dmirror_allocate_chunk(struct dmirror_device *mdevice, err_devmem: kfree(devmem); - return false; + return ret; } static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) @@ -569,13 +570,14 @@ static struct page *dmirror_devmem_alloc_page(struct dmirror_device *mdevice) struct page *rpage; /* - * This is a fake device so we alloc real system memory to store - * our device memory. + * For ZONE_DEVICE private type, this is a fake device so we alloc real + * system memory to store our device memory. + * For ZONE_DEVICE coherent type we use the actual dpage to store the data + * and ignore rpage. */ rpage = alloc_page(GFP_HIGHUSER); if (!rpage) return NULL; - spin_lock(&mdevice->lock); if (mdevice->free_pages) { @@ -603,7 +605,7 @@ static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args, struct dmirror *dmirror) { struct dmirror_device *mdevice = dmirror->mdevice; - const unsigned long *src = args->src; + unsigned long *src = args->src; unsigned long *dst = args->dst; unsigned long addr; @@ -621,12 +623,17 @@ static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args, * unallocated pte_none() or read-only zero page. */ spage = migrate_pfn_to_page(*src); + if (spage && is_zone_device_page(spage)) + pr_err("page already in device spage pfn: 0x%lx\n", + page_to_pfn(spage)); + BUG_ON(spage && is_zone_device_page(spage)); dpage = dmirror_devmem_alloc_page(mdevice); if (!dpage) continue; - rpage = dpage->zone_device_data; + rpage = is_device_private_page(dpage) ? dpage->zone_device_data : + dpage; if (spage) copy_highpage(rpage, spage); else @@ -638,8 +645,10 @@ static void dmirror_migrate_alloc_and_copy(struct migrate_vma *args, * the simulated device memory and that page holds the pointer * to the mirror. */ + rpage = dpage->zone_device_data; rpage->zone_device_data = dmirror; - + pr_debug("migrating from sys to dev pfn src: 0x%lx pfn dst: 0x%lx\n", + page_to_pfn(spage), page_to_pfn(dpage)); *dst = migrate_pfn(page_to_pfn(dpage)) | MIGRATE_PFN_LOCKED; if ((*src & MIGRATE_PFN_WRITE) || @@ -673,10 +682,13 @@ static int dmirror_migrate_finalize_and_map(struct migrate_vma *args, continue; /* - * Store the page that holds the data so the page table - * doesn't have to deal with ZONE_DEVICE private pages. + * For ZONE_DEVICE private pages we store the page that + * holds the data so the page table doesn't have to deal it. + * For ZONE_DEVICE coherent pages we store the actual page, since + * the CPU has coherent access to the page. */ - entry = dpage->zone_device_data; + entry = is_device_private_page(dpage) ? dpage->zone_device_data : + dpage; if (*dst & MIGRATE_PFN_WRITE) entry = xa_tag_pointer(entry, DPT_XA_TAG_WRITE); entry = xa_store(&dmirror->pt, pfn, entry, GFP_ATOMIC); @@ -690,7 +702,110 @@ static int dmirror_migrate_finalize_and_map(struct migrate_vma *args, return 0; } -static int dmirror_migrate(struct dmirror *dmirror, +static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, + struct dmirror *dmirror) +{ + unsigned long *src = args->src; + unsigned long *dst = args->dst; + unsigned long start = args->start; + unsigned long end = args->end; + unsigned long addr; + + for (addr = start; addr < end; addr += PAGE_SIZE, + src++, dst++) { + struct page *dpage, *spage; + + spage = migrate_pfn_to_page(*src); + if (!spage || !(*src & MIGRATE_PFN_MIGRATE)) + continue; + + BUG_ON(!is_device_page(spage)); + spage = is_device_private_page(spage) ? spage->zone_device_data: + spage; + dpage = alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr); + if (!dpage) + continue; + pr_debug("migrating from dev to sys pfn src: 0x%lx pfn dst: 0x%lx\n", + page_to_pfn(spage), page_to_pfn(dpage)); + + lock_page(dpage); + xa_erase(&dmirror->pt, addr >> PAGE_SHIFT); + copy_highpage(dpage, spage); + *dst = migrate_pfn(page_to_pfn(dpage)) | MIGRATE_PFN_LOCKED; + if (*src & MIGRATE_PFN_WRITE) + *dst |= MIGRATE_PFN_WRITE; + } + return 0; +} + +static int dmirror_migrate_to_system(struct dmirror *dmirror, + struct hmm_dmirror_cmd *cmd) +{ + unsigned long start, end, addr; + unsigned long size = cmd->npages << PAGE_SHIFT; + struct mm_struct *mm = dmirror->notifier.mm; + struct vm_area_struct *vma; + unsigned long src_pfns[64]; + unsigned long dst_pfns[64]; + struct migrate_vma args; + unsigned long next; + int ret; + + start = cmd->addr; + end = start + size; + if (end < start) + return -EINVAL; + + /* Since the mm is for the mirrored process, get a reference first. */ + if (!mmget_not_zero(mm)) + return -EINVAL; + + mmap_read_lock(mm); + for (addr = start; addr < end; addr = next) { + vma = find_vma(mm, addr); + if (!vma || addr < vma->vm_start || + !(vma->vm_flags & VM_READ)) { + ret = -EINVAL; + goto out; + } + next = min(end, addr + (ARRAY_SIZE(src_pfns) << PAGE_SHIFT)); + if (next > vma->vm_end) + next = vma->vm_end; + + args.vma = vma; + args.src = src_pfns; + args.dst = dst_pfns; + args.start = addr; + args.end = next; + args.pgmap_owner = dmirror->mdevice; + args.flags = (dmirror->mdevice->zone_device_type == + HMM_DMIRROR_MEMORY_DEVICE_PRIVATE) ? + MIGRATE_VMA_SELECT_DEVICE_PRIVATE : + MIGRATE_VMA_SELECT_DEVICE_COHERENT; + + ret = migrate_vma_setup(&args); + if (ret) + goto out; + + pr_debug("Migrating from device mem to sys mem\n"); + dmirror_devmem_fault_alloc_and_copy(&args, dmirror); + + migrate_vma_pages(&args); + migrate_vma_finalize(&args); + } + mmap_read_unlock(mm); + mmput(mm); + + return ret; + +out: + mmap_read_unlock(mm); + mmput(mm); + return ret; +} + + +static int dmirror_migrate_to_device(struct dmirror *dmirror, struct hmm_dmirror_cmd *cmd) { unsigned long start, end, addr; @@ -736,6 +851,7 @@ static int dmirror_migrate(struct dmirror *dmirror, if (ret) goto out; + pr_debug("Migrating from sys mem to device mem\n"); dmirror_migrate_alloc_and_copy(&args, dmirror); migrate_vma_pages(&args); dmirror_migrate_finalize_and_map(&args, dmirror); @@ -744,7 +860,7 @@ static int dmirror_migrate(struct dmirror *dmirror, mmap_read_unlock(mm); mmput(mm); - /* Return the migrated data for verification. */ + /* Return the migrated data for verification. only for pages in device zone */ ret = dmirror_bounce_init(&bounce, start, size); if (ret) return ret; @@ -758,6 +874,7 @@ static int dmirror_migrate(struct dmirror *dmirror, } cmd->cpages = bounce.cpages; dmirror_bounce_fini(&bounce); + return ret; out: @@ -781,9 +898,15 @@ static void dmirror_mkentry(struct dmirror *dmirror, struct hmm_range *range, } page = hmm_pfn_to_page(entry); - if (is_device_private_page(page)) { - /* Is the page migrated to this device or some other? */ - if (dmirror->mdevice == dmirror_page_to_device(page)) + if (is_device_page(page)) { + /* Is page ZONE_DEVICE coherent? */ + if (!is_device_private_page(page)) + *perm = HMM_DMIRROR_PROT_DEV_COHERENT; + /* + * Is page ZONE_DEVICE private migrated to + * this device or some other? + */ + else if (dmirror->mdevice == dmirror_page_to_device(page)) *perm = HMM_DMIRROR_PROT_DEV_PRIVATE_LOCAL; else *perm = HMM_DMIRROR_PROT_DEV_PRIVATE_REMOTE; @@ -983,8 +1106,12 @@ static long dmirror_fops_unlocked_ioctl(struct file *filp, ret = dmirror_write(dmirror, &cmd); break; - case HMM_DMIRROR_MIGRATE: - ret = dmirror_migrate(dmirror, &cmd); + case HMM_DMIRROR_MIGRATE_TO_DEV: + ret = dmirror_migrate_to_device(dmirror, &cmd); + break; + + case HMM_DMIRROR_MIGRATE_TO_SYS: + ret = dmirror_migrate_to_system(dmirror, &cmd); break; case HMM_DMIRROR_SNAPSHOT: @@ -1030,38 +1157,6 @@ static void dmirror_devmem_free(struct page *page) spin_unlock(&mdevice->lock); } -static vm_fault_t dmirror_devmem_fault_alloc_and_copy(struct migrate_vma *args, - struct dmirror *dmirror) -{ - const unsigned long *src = args->src; - unsigned long *dst = args->dst; - unsigned long start = args->start; - unsigned long end = args->end; - unsigned long addr; - - for (addr = start; addr < end; addr += PAGE_SIZE, - src++, dst++) { - struct page *dpage, *spage; - - spage = migrate_pfn_to_page(*src); - if (!spage || !(*src & MIGRATE_PFN_MIGRATE)) - continue; - spage = spage->zone_device_data; - - dpage = alloc_page_vma(GFP_HIGHUSER_MOVABLE, args->vma, addr); - if (!dpage) - continue; - - lock_page(dpage); - xa_erase(&dmirror->pt, addr >> PAGE_SHIFT); - copy_highpage(dpage, spage); - *dst = migrate_pfn(page_to_pfn(dpage)) | MIGRATE_PFN_LOCKED; - if (*src & MIGRATE_PFN_WRITE) - *dst |= MIGRATE_PFN_WRITE; - } - return 0; -} - static vm_fault_t dmirror_devmem_fault(struct vm_fault *vmf) { struct migrate_vma args; diff --git a/lib/test_hmm_uapi.h b/lib/test_hmm_uapi.h index f86754be64fd..13cec485328d 100644 --- a/lib/test_hmm_uapi.h +++ b/lib/test_hmm_uapi.h @@ -17,8 +17,12 @@ * @addr: (in) user address the device will read/write * @ptr: (in) user address where device data is copied to/from * @npages: (in) number of pages to read/write + * @alloc_to_devmem: (in) desired allocation destination during migration. + * True if allocation is to device memory. + * False if allocation is to system memory. * @cpages: (out) number of pages copied * @faults: (out) number of device page faults seen + * @zone_device_type: (out) zone device memory type */ struct hmm_dmirror_cmd { __u64 addr; @@ -26,15 +30,16 @@ struct hmm_dmirror_cmd { __u64 npages; __u64 cpages; __u64 faults; - __u64 zone_device_type; + __u32 zone_device_type; }; /* Expose the address space of the calling process through hmm device file */ #define HMM_DMIRROR_READ _IOWR('H', 0x00, struct hmm_dmirror_cmd) #define HMM_DMIRROR_WRITE _IOWR('H', 0x01, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_MIGRATE _IOWR('H', 0x02, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x03, struct hmm_dmirror_cmd) -#define HMM_DMIRROR_GET_MEM_DEV_TYPE _IOWR('H', 0x04, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_MIGRATE_TO_DEV _IOWR('H', 0x02, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_MIGRATE_TO_SYS _IOWR('H', 0x03, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_SNAPSHOT _IOWR('H', 0x04, struct hmm_dmirror_cmd) +#define HMM_DMIRROR_GET_MEM_DEV_TYPE _IOWR('H', 0x05, struct hmm_dmirror_cmd) /* * Values returned in hmm_dmirror_cmd.ptr for HMM_DMIRROR_SNAPSHOT. @@ -49,6 +54,8 @@ struct hmm_dmirror_cmd { * device the ioctl() is made * HMM_DMIRROR_PROT_DEV_PRIVATE_REMOTE: Migrated device private page on some * other device + * HMM_DMIRROR_PROT_DEV_COHERENT: Migrate device coherent page on the device + * the ioctl() is made */ enum { HMM_DMIRROR_PROT_ERROR = 0xFF, @@ -60,6 +67,7 @@ enum { HMM_DMIRROR_PROT_ZERO = 0x10, HMM_DMIRROR_PROT_DEV_PRIVATE_LOCAL = 0x20, HMM_DMIRROR_PROT_DEV_PRIVATE_REMOTE = 0x30, + HMM_DMIRROR_PROT_DEV_COHERENT = 0x40, }; enum {