From patchwork Tue Apr 6 01:45:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12183939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MENTIONS_GIT_HOSTING,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 79D67C433ED for ; Tue, 6 Apr 2021 01:47:21 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 273CF61396 for ; Tue, 6 Apr 2021 01:47:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 273CF61396 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5889D6E1E6; Tue, 6 Apr 2021 01:47:00 +0000 (UTC) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2080.outbound.protection.outlook.com [40.107.94.80]) by gabe.freedesktop.org (Postfix) with ESMTPS id A0C7B6E1CD; Tue, 6 Apr 2021 01:46:56 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Kysp83WIFyYpxFWSfucCBc6AoIAConNmFXExIykK/gdq3Xw8dF4vprJ0nnxWEahDKXNbr5HnBowlWZ6ey6+8zcHrlTw0nbyNdpHBU8ZnqJpDShSNKVzacfJPQpd54lLU6hki2jC2JWAfJeTg9kpsYrNqW7/b3Y2FtIGvH+hYGOc4X+soSYXSihTcH8nzQtNW0rV62yLli4i5Ss7SPilEKEZOGZwsuibEk/TkUmAtzm+Toq2jtTmgPxvqt0qzngErYRFQeoXWsq7d9t5kJxIF5mK0qx3vIdHpPuYEO0WKD8tQuPoWDigbKOsOxlcTdFtGHN+oL5RKhBKZ3SHVM9RQGg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nmucnBKXFb3DEWVhMo+66Uo4GKYaj7v3+cH+UWFvbV8=; b=FwakUp/5CP6QHxQiDvQWN6z4uoLdd4YDEMGRanqjLMScyO4IForiQ+B8cIoNIzfu4QeZGNfBCeT/F5o4j06WcznN7oh0KpXcfdhz0wXpp6r3yAnFTMWJLI9Z32rnBCn/Ks3WtFbXayS5VelsvvzIV/QuWcxCU4Ey+DnKUSk4TZwNQwWECXLjULNw7BOLSijyziQwtyCzcd7C/MPS2VEEHIVrB2Uw+pxjaz0Et09ASQAmn1e6NKEOn7Ct1HOu6lzgMx5aHdmN+Rgrrqdy/JI+2cS3L6IfTgWC5r6aoYOeUh0MuGyasxJw0Ch8+/RhZYLBMs8s2+r5eTnJOp2Vy7Iwcg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=nmucnBKXFb3DEWVhMo+66Uo4GKYaj7v3+cH+UWFvbV8=; b=2QcnvjYDtX9AiqarI9BqEDosVI2cGAwfhqPKRQJrdx3chjEPKI5PWPu5FPDTDhJuE87eQsCHHCdOoOdMu/CAmBypLf1zpacfdd2I0Zr2p3jvusHozYt9w96A2UE4cC3YQl0YZEyPfik399XHLt/i5KH7h2D3L9lcFYMomOAO+tA= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3773.namprd12.prod.outlook.com (2603:10b6:208:164::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.29; Tue, 6 Apr 2021 01:46:53 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%7]) with mapi id 15.20.3933.039; Tue, 6 Apr 2021 01:46:53 +0000 From: Felix Kuehling To: amd-gfx@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: [PATCH 00/34] Add HMM-based SVM memory manager to KFD v4 Date: Mon, 5 Apr 2021 21:45:55 -0400 Message-Id: <20210406014629.25141-1-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.1 X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YT1PR01CA0071.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:2d::10) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YT1PR01CA0071.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b01:2d::10) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3999.32 via Frontend Transport; Tue, 6 Apr 2021 01:46:52 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 272c8535-254a-41dd-e71b-08d8f89dda53 X-MS-TrafficTypeDiagnostic: MN2PR12MB3773: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1923; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: cVh2TBNCZWOLRlPWO+z1m7v9iU58PwN3iTT9d/BvY5Ij8BprLfbkMoCRE0lcAwcz/EFc3L/HYzhFj5/BQLlxEwEFdkXdRVQv//0SQ+a5ZX/P1sdhUdC9AR24zkpF3bVW9vMfDnkOi8v87/vv7bigQiLjV7zzQBCrDvCBcIaEEhUliONaor9ycx0Uul8aBWyNSfUn4CrHfSlhDpogUa+HQZ5C0qdM3ayCzce/ZpQTmnuZq+htIo1DiWjj4hXEwas9Viu0seL5qSej2888/t7bAwIPgxtnwklLh4JN2u8P7J6b4klYeWBc14JfDoeQmaR1Paqet687hXKUMU26U7gk8WC0U+xa8MEGafwl0V+qw/Djcgs2a1kkHfpLAAetacDLQiOU0aqcgZ6+rDev2CmyehOX4VFOOyjFmmjb1meL5aSGFtZ26MaOntqMepYvTxT3CewgUd3qaoTURfuM5Ga4Xa88nTf2zpd5NumSHwybJ3W0UiUi8coE4+zZdNnRRhWHtCMB9CdyBJArQayfKvu9jsaFrOvjf1v45ckJPWviohvvmxIrZWmjXs2Hxt/Ld6UHLIUOtTjQMZNDfCA3FK/QG+3r0mb9iQMHrjBqd9TT9fC6wOmztKjpMhDb1UAenvzr9Z0zOAW28QSFSCeDBIl3Lr/QntZpmUJwShAzSl+DCFB9BFsvRpzVuZQdut6h6fCXu8+4Z4oNGLbyuof9HaFmfSbnLu3dL7PmjTE5qxlgmJ0WrFCMUs9KT0rEMgmpY6N+Fs758HIL3/jZCPdUgSsKHg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(376002)(39860400002)(136003)(346002)(478600001)(5660300002)(6486002)(66574015)(316002)(52116002)(66946007)(1076003)(36756003)(2906002)(16526019)(186003)(26005)(8676002)(8936002)(86362001)(66556008)(66476007)(54906003)(966005)(4326008)(956004)(2616005)(38100700001)(83380400001)(7696005)(38350700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: =?utf-8?q?SeD4qfEKs8iEFTJGATvZqlyS/ipTvY?= =?utf-8?q?wuNysh5pvgSpjx3I/1dLcb9WAp54cFvDqu2yMLvADhl1aN0Br/PtKDFp0ldfV1A3U?= =?utf-8?q?RrfCMM84aaNxLLOk3a+2G77/MwDbYbws7VuRBXGqPp/8IuAzQiybhATa+jIVoIbjv?= =?utf-8?q?FtFBAAJS9y7eEx4jID75mJ6TcfuxP1g/ZitvFdwjlF13GSi0WG00aE4+ZstPWEI+u?= =?utf-8?q?k4jIUBfKsx99wH9mlketxEKXYhMNqqeBszW0ujLbeVh/jnxzLu4BwSk9diI4ENdzb?= =?utf-8?q?yPxZOFgk3UMjdCFkvYp5B6YOvzTygj9paS051Z66AzvkOgOTw6nVIxKnoMLSOtNUP?= =?utf-8?q?SAFf9JTrwh51BfvVcNZsDIlMwD+LL7WZZwobwcoQ2GLobDm40TVm8J5mVfzB9cM/w?= =?utf-8?q?04fhjq320XAECjy56p7JP8U3HfAuiuiARk2ZUD7FoEP18Vl2mz1H00mDUF8RM11qu?= =?utf-8?q?qxPevmDW5QIftjAE+z7tUWJEsjhvU1dpgRgUb37SDQngc5MWmAfJ074ixUt+58EFR?= =?utf-8?q?9LohVbKKd5/eP7W/ax+/5J6Ec2bMvbMZqWZbSSozTUPphcD2EnSox151NejmV8iQ+?= =?utf-8?q?Xh+qQDxVLcGih/lBog+TH4/akcaHK7m1oQO0lOshcc0U22pJhQMK+OzREvzP3pl7G?= =?utf-8?q?gsLGqNr1ENAN5mbfjLhln4uUfIKgozvTTc++drV40Ga+CEZDOjY64aNIgEi57+ZNn?= =?utf-8?q?0M31LnnawWDKp2qbdybO4LqufoOnqeC/OZ8hrnK/VmZY+JdBMSANf3Vu8obon0OKe?= =?utf-8?q?g6VnrVyL9ZA9FSsIq/47Dw+ep6dtlM1jyN1OiWNKel8qBK0OMCVuRGdOuWkpMUVA0?= =?utf-8?q?fjqFkZ2bNKSj63eVlWnta3qDAxZYOInvY2GsUeMZva07w7nH1A9zvQWXehLbO+BpZ?= =?utf-8?q?C4lfp78lq2qxIHK3yVV4o9b+IDhixzQNum9l2kpMa7e6FuWpxWdY+MSIGrXOAXuZ+?= =?utf-8?q?i4wBvcC5AU0RSUMR9WecgiDvoI67taGpEcMW/x2MPBfN3l+iUSp39kjBS0It8VSZS?= =?utf-8?q?VF65v6Ij7vWmDmSH9/zLr53ChlgKS+PI9uVU6CkkLEGf7DMr+9/JGrx9pv8GoMv76?= =?utf-8?q?+0Ehw78FUsWxTMutjIG8/xJThxPo2RteO/feHGEmIFrTxXFqS0V62/OLKunhZSk+Z?= =?utf-8?q?ys01OlF1SfPfteVpWvRLzQ2utl977WvD7hgGVMF58RuBC79gp5gOJZ5p7yrj56XEQ?= =?utf-8?q?FOtLTvxakIrzfC42Fkljbh6PowedeYK7OeFPpKlP9BVkXwcGv51hYzzsnmkThf/p1?= =?utf-8?q?FaeFzGGOFDHW4M?= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 272c8535-254a-41dd-e71b-08d8f89dda53 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 06 Apr 2021 01:46:53.1292 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Z4VxE12UCTJyx87u2cNMQAguKjIdEZTonxtGvpHJR1NRkKkKVODG/efncKuZWvMDHxv4V+9/hI6lvQbBI7Wp7Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3773 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jason Gunthorpe , =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Rebased on upstream. Dropped already upstream patch "drm/amdgpu: reserve fence slot to update page table". Added more fixes: - Fixed kernel test robot warnings about static functions - Fixed a kernel test robot warning about an unused variable - Fixed a kernel test robot warning about select DEVICE_PRIVATE. Using "depends on" now. (see patch 34) - Proportionally longer timeout for hmm_range_fault on large address ranges (see patch 6) - Fixed PTE flags for XGMI mappings on Arcturus and Aldebaran (see patch 17) - Fixed range-list cleanup on process termination to avoid BUGs from dangling interval notifiers (see patch 16) - Fixed SVM range locking and interval notifier sequence update (see patch 8 and related tweaks in patches 10, 11, 21) Added my Reviewed-by to all patches primarily authored by Philip and Alex. I believe this patch series is nearly ready to go. This series and the corresponding ROCm Thunk and KFDTest changes are also available on gitub and patchwork. Link: https://github.com/RadeonOpenCompute/ROCK-Kernel-Driver/tree/fxkamd/hmm-wip Link: https://github.com/RadeonOpenCompute/ROCT-Thunk-Interface/tree/fxkamd/hmm-wip Link: https://patchwork.freedesktop.org/series/85563/ CC: Jérôme Glisse CC: Jason Gunthorpe Alex Sierra (9): drm/amdkfd: helper to convert gpu id and idx drm/amdkfd: add xnack enabled flag to kfd_process drm/amdkfd: add ioctl to configure and query xnack retries drm/amdgpu: enable 48-bit IH timestamp counter drm/amdkfd: SVM API call to restore page tables drm/amdkfd: add svm_bo reference for eviction fence drm/amdgpu: add param bit flag to create SVM BOs drm/amdgpu: svm bo enable_signal call condition drm/amdgpu: add svm_bo eviction to enable_signal cb Felix Kuehling (13): drm/amdkfd: map svm range to GPUs drm/amdkfd: svm range eviction and restore drm/amdgpu: Enable retry faults unconditionally on Aldebaran drm/amdkfd: validate vram svm range from TTM drm/amdkfd: HMM migrate ram to vram drm/amdkfd: HMM migrate vram to ram drm/amdkfd: invalidate tables on page retry fault drm/amdkfd: page table restore through svm API drm/amdkfd: add svm_bo eviction mechanism support drm/amdkfd: refine migration policy with xnack on drm/amdkfd: add svm range validate timestamp drm/amdkfd: multiple gpu migrate vram to vram drm/amdkfd: Add CONFIG_HSA_AMD_SVM Philip Yang (12): drm/amdkfd: add svm ioctl API drm/amdkfd: register svm range drm/amdkfd: add svm ioctl GET_ATTR op drm/amdgpu: add common HMM get pages function drm/amdkfd: support larger svm range allocation drm/amdkfd: validate svm range system memory drm/amdkfd: deregister svm range drm/amdgpu: export vm update mapping interface drm/amdkfd: register HMM device private zone drm/amdkfd: support xgmi same hive mapping drm/amdkfd: copy memory through gart table drm/amdkfd: Add SVM API support capability bits drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 4 +- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c | 16 +- .../gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 13 +- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 3 +- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 86 + drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 7 + drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 4 + drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 90 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 38 +- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 11 + drivers/gpu/drm/amd/amdgpu/gfxhub_v1_0.c | 8 +- drivers/gpu/drm/amd/amdgpu/mmhub_v1_7.c | 6 +- drivers/gpu/drm/amd/amdgpu/vega10_ih.c | 1 + drivers/gpu/drm/amd/amdkfd/Kconfig | 13 + drivers/gpu/drm/amd/amdkfd/Makefile | 5 + drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 64 + drivers/gpu/drm/amd/amdkfd/kfd_device.c | 4 + .../amd/amdkfd/kfd_device_queue_manager_v9.c | 13 +- drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 4 + drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 922 ++++++ drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 64 + drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 36 + drivers/gpu/drm/amd/amdkfd/kfd_process.c | 82 + drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2906 +++++++++++++++++ drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 205 ++ drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 6 + drivers/gpu/drm/amd/amdkfd/kfd_topology.h | 10 +- include/uapi/linux/kfd_ioctl.h | 171 +- 28 files changed, 4686 insertions(+), 106 deletions(-) create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_migrate.h create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_svm.c create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_svm.h