From patchwork Mon Mar 22 10:58:17 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154471 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95B12C433E0 for ; Mon, 22 Mar 2021 11:07:35 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5D2DC61879 for ; Mon, 22 Mar 2021 11:07:35 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5D2DC61879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3973189870; Mon, 22 Mar 2021 11:07:34 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2073.outbound.protection.outlook.com [40.107.237.73]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3ADCD89913; Mon, 22 Mar 2021 11:07:33 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=gsRu0+h+wXnjJdDUTdCp2A5hfE6IpHL6NxIo7BRr+dxRQuOALIe1kO9+R9/IAqmnZzzUGGg6B25oJ6Mvx13o2FkaaYc5BI6nkQqiQoFd09lHKHXcUmstVAfYDqeR6koIZZBE9DJQ/2aaj+NKui13oFkdZFKSCr4q2E5l18v6YdrTTT4kgZPrWbfSKJZrUGRZG/GFRrbrck/ZXNZ7BTl9gi+RPTvH6Xbqj+jiDq5kwh/FyRCRHgNjwHZYsgXbAYusPdJT6MGHM4qHyAVEFohI90U45Z4Bbou15xZFczhuCx6QPfqm50E2PW+NpfIyXk5Opu+lDseiT3lhhYD0BNYBNA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cEkGnQ/dUrzy0vrtXpEDUBVM61SU5hsYR5KSzxWkbKY=; b=OJFK3ch27+0VJsyId9tnsGslE0FzkAD/Dwxg5qxReoLOfQflWp+hvp+Q0o3lJieuTvHj5HOo3KRGBFxiBRzDYm55w/bumUD42vsY9+Gy/L4hv+d9OqmvITgAjtz8/tQth2p65KGdedWdgTSMzao8S4pR1pJBF7U28H+Sb21vwevqHEjFC03nogvPcv2Lm5zLWeaVgF12SURrqFXshXg/mvmPWPI2EqNCabh1RzBhdloRizEdqSjjbENdhTyNRjE7SYNNzcZRkB/RExEbC4f3DuaKKJ1AEpHXfQxviajZsjtGc56rJeHLnUvp9J8xpWKZDj1JTteCDFGgwLlmueC3GQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cEkGnQ/dUrzy0vrtXpEDUBVM61SU5hsYR5KSzxWkbKY=; b=XUbHDExY91XHM2QnZ3+0n3eh3VrSPmgRCK71LEV/YDC2W1QMnh37GezSK+45V/p+RLvZLIyJdq0lEM6a+0PJmDWEA+Vw0clFwhfjUZUiu3U5fi9IY7vdBzlcq3jZej3y/Ku2gJ1A2PnJ5cm/Vwdw61/AOlpJZtGrrdBfpqAutIE= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3935.namprd12.prod.outlook.com (2603:10b6:208:168::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:30 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:30 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 01/44] drm/amdgpu: replace per_device_list by array Date: Mon, 22 Mar 2021 06:58:17 -0400 Message-Id: <20210322105900.14068-2-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:29 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 95de28c3-4a07-4714-d26d-08d8ed22af31 X-MS-TrafficTypeDiagnostic: MN2PR12MB3935: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:546; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: H43X1hMiwhmk87vY0tUR933LZFJcKDZXxeyhp+CefHFsFfPj90NFkd+eBOn6DZJ10Wi1Wz9TrlKjF/S/mZU6nwVubyswZ0+D50kksmiZ52u2jLF4bKOfBGmbChKicFuNyLE/hSXme/3+I+LcXO8l/nTWfNXu91BcKhM+kCq/f0BfucaYk9Zvmv2Kp+4+Ii+WQ3GSRllFY4zdAZF+g7gEU2jfSZiVV9BJwmX8l2jQHGS4yuxrL+IfwHD2MRKo4ANf4jufoP/S08nnVI7UIBRZCHTc371D9BGO6aZNVEd9uTlaV7KdCckNK00QO9L5pD8B9swqyaL2wj4TMp/4sCO+RKGw+HL9gZOyqrPlwFgaskw5zujpipp4b47kCxjpYdX8GmlmHv4r1BfqZYWdbXP98zACl6vMIBe0qZDVfhH5UNEpjjypUfX92JsKeorVsVfXAypdFeKraxkxRPR3px7C3UEdimbwcGrJv9iUCz1nM6WVkKsfVpZnR+iOq5BVmtFEZyPoxOKaKAWHyZtyLOwLPa7Lsi+qXC0+AIoZkAEw7GakWZpnlG80KztwrbND/wv96Ied82/gB0CACr0k+oRjbxW8nwT2PLyf7pzon3IsllQEENo/bvZMvfjkQrxuqAgEFti8NkbvpNEid9RWYy+tx/disImgvj7ra61kHQ6lFrk= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(136003)(366004)(396003)(346002)(376002)(5660300002)(30864003)(8936002)(450100002)(6486002)(478600001)(4326008)(83380400001)(1076003)(86362001)(8676002)(956004)(2906002)(36756003)(7696005)(26005)(16526019)(52116002)(186003)(316002)(66946007)(66556008)(6666004)(66476007)(2616005)(38100700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: O9EDsRKtRNloU2lEcyo18QbpdJJLp1yyL8Qm6ufJuz3Cf89xyXwYniMt6BFPEhrxgCpQYxd1Lwi4q2SzjHg6LftRxfOtbDrPrAdsBQ2Qok/aSnDz55BD25dpfwF02FYut5HrlRiXpvMk8LZ0jZNaZlMTGcgbRrwASh3QHsRxoz9ij3HushFzpoPmDD8eHEdwZwSBnZLG1tyWmDdjFafewVyD7ugXU/30Vcyi0NO6gblxFGqN6ZSMPLi73C3/JNcyR67vCmptryP6MNtPHZwiC/sXkDEF2B0Zfr8x8lzTxpQq3U7ntjm8qk/i3nkLBd9FeZtE8lQR78/6iYYND3aQAXkvH6AwYEVD4926rnVCU92BeZd1ewKx223V4yOPVOiXLehU2jyia2M3Cn2qUGd0VKx6Q+2+QJ3EbJ/UW+AZ3g2X7h9ccL7Ifm2BAS+0UM1oMYid4UB7ObRctXhloGioTmcdihkvUat9UTgA7AbB3d44jA6CfsuKsqkvhBuI+AoKKa1yaPmjFVPJNBgBxPONKUJOEUwSHIr0Lmd+P7H43lqfnwepoNeks6W711Pj8BXAimZvSSFnZWXMnPMjvXaaDx7ZN554vAM5Q9JFGn3KPxjiMheQfPToS5/EQ6z2MzOEvBpX2hS+xHCxnYuyNcgQd+3Gnr5c8ND8ikTEunL2W5vrYToFfgBIeJc5+CJiD4VUyPT+OAhIuaNxfIfHDYt9Qt3e2MOdip2QgukrH2X2+hanQwVnmyozleN4HG5lY2oaLQ81p+SAUoWxCsNSEa/BsCj3Vc05pwUaewzta9bUvvtOoSbmI4zkyI5vF3MA3ySuoxpK7TLA/9Hio+ShtePoOnWLs8c9nmI9N7o82SSLaNUSTLy/D/Kyx0FJt8xit68qHctEWVOqkkouzQL53frtPbhPLK8aP2RfjJIP7SE2eM0iB25NN/Y8x7DJFGheTLweglBXX2NfJCwEQ5tP2RmvJDPf2QRT6mgk/el98fi4vH1s8/NkQ9h2r8kYTO/hBJBaQTH1aeKBU6KZqUW1mIHSmqmAOCOWMv4H7qB0DMYA2y2sA5u0sGi8S//Sw3pKXshyWPGikmY08uElM6EigK35d+QiqLTe0VEp4H+aV4OxWi11AS9SZ+/5oaQIzmRnoOcUE28KdTsOHYqmOuNxHlAYMjjUFG/Nf0KYXxy4xDru9GfZzEPOLEXwkW4yIbNk4n7yestNpUFu/ciJZaXtXyEijRikjstxs7V55lLJ0hl/D4Cwj0TQrn3QGIKFkGskemL/uu4NdSo3CfMv3y4GgiaAREX1eKq0nLLmrI6njKmKrk6haHU+JHdGwII1iWRmcsO9 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 95de28c3-4a07-4714-d26d-08d8ed22af31 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:29.9383 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: vPz58wn6B4wLuq7VNePcZrsh04H+4jNrKleb1ed3ZZEljK5AHz1M4Xya/kMA40ttqrsVgPiqIkdcTgeyQfcI8w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3935 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra Remove per_device_list from kfd_process and replace it with a kfd_process_device pointers array of MAX_GPU_INSTANCES size. This helps to manage the kfd_process_devices binded to a specific kfd_process. Also, functions used by kfd_chardev to iterate over the list were removed, since they are not valid anymore. Instead, it was replaced by a local loop iterating the array. Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling Reviewed-by: Jonathan Kim --- drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 116 ++++++++---------- drivers/gpu/drm/amd/amdkfd/kfd_iommu.c | 8 +- drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 20 +-- drivers/gpu/drm/amd/amdkfd/kfd_process.c | 108 ++++++++-------- .../amd/amdkfd/kfd_process_queue_manager.c | 6 +- 5 files changed, 111 insertions(+), 147 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c index 6802c616e10e..43de260b2230 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -870,52 +870,47 @@ static int kfd_ioctl_get_process_apertures(struct file *filp, { struct kfd_ioctl_get_process_apertures_args *args = data; struct kfd_process_device_apertures *pAperture; - struct kfd_process_device *pdd; + int i; dev_dbg(kfd_device, "get apertures for PASID 0x%x", p->pasid); args->num_of_nodes = 0; mutex_lock(&p->mutex); + /* Run over all pdd of the process */ + for (i = 0; i < p->n_pdds; i++) { + struct kfd_process_device *pdd = p->pdds[i]; + + pAperture = + &args->process_apertures[args->num_of_nodes]; + pAperture->gpu_id = pdd->dev->id; + pAperture->lds_base = pdd->lds_base; + pAperture->lds_limit = pdd->lds_limit; + pAperture->gpuvm_base = pdd->gpuvm_base; + pAperture->gpuvm_limit = pdd->gpuvm_limit; + pAperture->scratch_base = pdd->scratch_base; + pAperture->scratch_limit = pdd->scratch_limit; - /*if the process-device list isn't empty*/ - if (kfd_has_process_device_data(p)) { - /* Run over all pdd of the process */ - pdd = kfd_get_first_process_device_data(p); - do { - pAperture = - &args->process_apertures[args->num_of_nodes]; - pAperture->gpu_id = pdd->dev->id; - pAperture->lds_base = pdd->lds_base; - pAperture->lds_limit = pdd->lds_limit; - pAperture->gpuvm_base = pdd->gpuvm_base; - pAperture->gpuvm_limit = pdd->gpuvm_limit; - pAperture->scratch_base = pdd->scratch_base; - pAperture->scratch_limit = pdd->scratch_limit; - - dev_dbg(kfd_device, - "node id %u\n", args->num_of_nodes); - dev_dbg(kfd_device, - "gpu id %u\n", pdd->dev->id); - dev_dbg(kfd_device, - "lds_base %llX\n", pdd->lds_base); - dev_dbg(kfd_device, - "lds_limit %llX\n", pdd->lds_limit); - dev_dbg(kfd_device, - "gpuvm_base %llX\n", pdd->gpuvm_base); - dev_dbg(kfd_device, - "gpuvm_limit %llX\n", pdd->gpuvm_limit); - dev_dbg(kfd_device, - "scratch_base %llX\n", pdd->scratch_base); - dev_dbg(kfd_device, - "scratch_limit %llX\n", pdd->scratch_limit); - - args->num_of_nodes++; - - pdd = kfd_get_next_process_device_data(p, pdd); - } while (pdd && (args->num_of_nodes < NUM_OF_SUPPORTED_GPUS)); - } + dev_dbg(kfd_device, + "node id %u\n", args->num_of_nodes); + dev_dbg(kfd_device, + "gpu id %u\n", pdd->dev->id); + dev_dbg(kfd_device, + "lds_base %llX\n", pdd->lds_base); + dev_dbg(kfd_device, + "lds_limit %llX\n", pdd->lds_limit); + dev_dbg(kfd_device, + "gpuvm_base %llX\n", pdd->gpuvm_base); + dev_dbg(kfd_device, + "gpuvm_limit %llX\n", pdd->gpuvm_limit); + dev_dbg(kfd_device, + "scratch_base %llX\n", pdd->scratch_base); + dev_dbg(kfd_device, + "scratch_limit %llX\n", pdd->scratch_limit); + if (++args->num_of_nodes >= NUM_OF_SUPPORTED_GPUS) + break; + } mutex_unlock(&p->mutex); return 0; @@ -926,9 +921,8 @@ static int kfd_ioctl_get_process_apertures_new(struct file *filp, { struct kfd_ioctl_get_process_apertures_new_args *args = data; struct kfd_process_device_apertures *pa; - struct kfd_process_device *pdd; - uint32_t nodes = 0; int ret; + int i; dev_dbg(kfd_device, "get apertures for PASID 0x%x", p->pasid); @@ -937,17 +931,7 @@ static int kfd_ioctl_get_process_apertures_new(struct file *filp, * sufficient memory */ mutex_lock(&p->mutex); - - if (!kfd_has_process_device_data(p)) - goto out_unlock; - - /* Run over all pdd of the process */ - pdd = kfd_get_first_process_device_data(p); - do { - args->num_of_nodes++; - pdd = kfd_get_next_process_device_data(p, pdd); - } while (pdd); - + args->num_of_nodes = p->n_pdds; goto out_unlock; } @@ -962,22 +946,23 @@ static int kfd_ioctl_get_process_apertures_new(struct file *filp, mutex_lock(&p->mutex); - if (!kfd_has_process_device_data(p)) { + if (!p->n_pdds) { args->num_of_nodes = 0; kfree(pa); goto out_unlock; } /* Run over all pdd of the process */ - pdd = kfd_get_first_process_device_data(p); - do { - pa[nodes].gpu_id = pdd->dev->id; - pa[nodes].lds_base = pdd->lds_base; - pa[nodes].lds_limit = pdd->lds_limit; - pa[nodes].gpuvm_base = pdd->gpuvm_base; - pa[nodes].gpuvm_limit = pdd->gpuvm_limit; - pa[nodes].scratch_base = pdd->scratch_base; - pa[nodes].scratch_limit = pdd->scratch_limit; + for (i = 0; i < min(p->n_pdds, args->num_of_nodes); i++) { + struct kfd_process_device *pdd = p->pdds[i]; + + pa[i].gpu_id = pdd->dev->id; + pa[i].lds_base = pdd->lds_base; + pa[i].lds_limit = pdd->lds_limit; + pa[i].gpuvm_base = pdd->gpuvm_base; + pa[i].gpuvm_limit = pdd->gpuvm_limit; + pa[i].scratch_base = pdd->scratch_base; + pa[i].scratch_limit = pdd->scratch_limit; dev_dbg(kfd_device, "gpu id %u\n", pdd->dev->id); @@ -993,17 +978,14 @@ static int kfd_ioctl_get_process_apertures_new(struct file *filp, "scratch_base %llX\n", pdd->scratch_base); dev_dbg(kfd_device, "scratch_limit %llX\n", pdd->scratch_limit); - nodes++; - - pdd = kfd_get_next_process_device_data(p, pdd); - } while (pdd && (nodes < args->num_of_nodes)); + } mutex_unlock(&p->mutex); - args->num_of_nodes = nodes; + args->num_of_nodes = i; ret = copy_to_user( (void __user *)args->kfd_process_device_apertures_ptr, pa, - (nodes * sizeof(struct kfd_process_device_apertures))); + (i * sizeof(struct kfd_process_device_apertures))); kfree(pa); return ret ? -EFAULT : 0; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_iommu.c b/drivers/gpu/drm/amd/amdkfd/kfd_iommu.c index 9318936aa805..5a1f2433632b 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_iommu.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_iommu.c @@ -135,11 +135,11 @@ int kfd_iommu_bind_process_to_device(struct kfd_process_device *pdd) */ void kfd_iommu_unbind_process(struct kfd_process *p) { - struct kfd_process_device *pdd; + int i; - list_for_each_entry(pdd, &p->per_device_data, per_device_list) - if (pdd->bound == PDD_BOUND) - amd_iommu_unbind_pasid(pdd->dev->pdev, p->pasid); + for (i = 0; i < p->n_pdds; i++) + if (p->pdds[i]->bound == PDD_BOUND) + amd_iommu_unbind_pasid(p->pdds[i]->dev->pdev, p->pasid); } /* Callback for process shutdown invoked by the IOMMU driver */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index d8c8b5ff449a..33e56db14327 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -45,6 +45,7 @@ #include #include "amd_shared.h" +#include "amdgpu.h" #define KFD_MAX_RING_ENTRY_SIZE 8 @@ -649,12 +650,6 @@ enum kfd_pdd_bound { /* Data that is per-process-per device. */ struct kfd_process_device { - /* - * List of all per-device data for a process. - * Starts from kfd_process.per_device_data. - */ - struct list_head per_device_list; - /* The device that owns this data. */ struct kfd_dev *dev; @@ -771,10 +766,11 @@ struct kfd_process { u32 pasid; /* - * List of kfd_process_device structures, + * Array of kfd_process_device pointers, * one for each device the process is using. */ - struct list_head per_device_data; + struct kfd_process_device *pdds[MAX_GPU_INSTANCE]; + uint32_t n_pdds; struct process_queue_manager pqm; @@ -872,14 +868,6 @@ void *kfd_process_device_translate_handle(struct kfd_process_device *p, void kfd_process_device_remove_obj_handle(struct kfd_process_device *pdd, int handle); -/* Process device data iterator */ -struct kfd_process_device *kfd_get_first_process_device_data( - struct kfd_process *p); -struct kfd_process_device *kfd_get_next_process_device_data( - struct kfd_process *p, - struct kfd_process_device *pdd); -bool kfd_has_process_device_data(struct kfd_process *p); - /* PASIDs */ int kfd_pasid_init(void); void kfd_pasid_exit(void); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index f5237997fa18..d4241d29ea94 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -505,7 +505,7 @@ static int kfd_sysfs_create_file(struct kfd_process *p, struct attribute *attr, static int kfd_procfs_add_sysfs_stats(struct kfd_process *p) { int ret = 0; - struct kfd_process_device *pdd; + int i; char stats_dir_filename[MAX_SYSFS_FILENAME_LEN]; if (!p) @@ -520,7 +520,8 @@ static int kfd_procfs_add_sysfs_stats(struct kfd_process *p) * - proc//stats_/evicted_ms * - proc//stats_/cu_occupancy */ - list_for_each_entry(pdd, &p->per_device_data, per_device_list) { + for (i = 0; i < p->n_pdds; i++) { + struct kfd_process_device *pdd = p->pdds[i]; struct kobject *kobj_stats; snprintf(stats_dir_filename, MAX_SYSFS_FILENAME_LEN, @@ -571,7 +572,7 @@ static int kfd_procfs_add_sysfs_stats(struct kfd_process *p) static int kfd_procfs_add_sysfs_files(struct kfd_process *p) { int ret = 0; - struct kfd_process_device *pdd; + int i; if (!p) return -EINVAL; @@ -584,7 +585,9 @@ static int kfd_procfs_add_sysfs_files(struct kfd_process *p) * - proc//vram_ * - proc//sdma_ */ - list_for_each_entry(pdd, &p->per_device_data, per_device_list) { + for (i = 0; i < p->n_pdds; i++) { + struct kfd_process_device *pdd = p->pdds[i]; + snprintf(pdd->vram_filename, MAX_SYSFS_FILENAME_LEN, "vram_%u", pdd->dev->id); ret = kfd_sysfs_create_file(p, &pdd->attr_vram, pdd->vram_filename); @@ -881,21 +884,23 @@ void kfd_unref_process(struct kfd_process *p) kref_put(&p->ref, kfd_process_ref_release); } + static void kfd_process_device_free_bos(struct kfd_process_device *pdd) { struct kfd_process *p = pdd->process; void *mem; int id; + int i; /* * Remove all handles from idr and release appropriate * local memory object */ idr_for_each_entry(&pdd->alloc_idr, mem, id) { - struct kfd_process_device *peer_pdd; - list_for_each_entry(peer_pdd, &p->per_device_data, - per_device_list) { + for (i = 0; i < p->n_pdds; i++) { + struct kfd_process_device *peer_pdd = p->pdds[i]; + if (!peer_pdd->vm) continue; amdgpu_amdkfd_gpuvm_unmap_memory_from_gpu( @@ -909,18 +914,19 @@ static void kfd_process_device_free_bos(struct kfd_process_device *pdd) static void kfd_process_free_outstanding_kfd_bos(struct kfd_process *p) { - struct kfd_process_device *pdd; + int i; - list_for_each_entry(pdd, &p->per_device_data, per_device_list) - kfd_process_device_free_bos(pdd); + for (i = 0; i < p->n_pdds; i++) + kfd_process_device_free_bos(p->pdds[i]); } static void kfd_process_destroy_pdds(struct kfd_process *p) { - struct kfd_process_device *pdd, *temp; + int i; + + for (i = 0; i < p->n_pdds; i++) { + struct kfd_process_device *pdd = p->pdds[i]; - list_for_each_entry_safe(pdd, temp, &p->per_device_data, - per_device_list) { pr_debug("Releasing pdd (topology id %d) for process (pasid 0x%x)\n", pdd->dev->id, p->pasid); @@ -933,8 +939,6 @@ static void kfd_process_destroy_pdds(struct kfd_process *p) amdgpu_amdkfd_gpuvm_destroy_process_vm( pdd->dev->kgd, pdd->vm); - list_del(&pdd->per_device_list); - if (pdd->qpd.cwsr_kaddr && !pdd->qpd.cwsr_base) free_pages((unsigned long)pdd->qpd.cwsr_kaddr, get_order(KFD_CWSR_TBA_TMA_SIZE)); @@ -955,7 +959,9 @@ static void kfd_process_destroy_pdds(struct kfd_process *p) } kfree(pdd); + p->pdds[i] = NULL; } + p->n_pdds = 0; } /* No process locking is needed in this function, because the process @@ -967,7 +973,7 @@ static void kfd_process_wq_release(struct work_struct *work) { struct kfd_process *p = container_of(work, struct kfd_process, release_work); - struct kfd_process_device *pdd; + int i; /* Remove the procfs files */ if (p->kobj) { @@ -976,7 +982,9 @@ static void kfd_process_wq_release(struct work_struct *work) kobject_put(p->kobj_queues); p->kobj_queues = NULL; - list_for_each_entry(pdd, &p->per_device_data, per_device_list) { + for (i = 0; i < p->n_pdds; i++) { + struct kfd_process_device *pdd = p->pdds[i]; + sysfs_remove_file(p->kobj, &pdd->attr_vram); sysfs_remove_file(p->kobj, &pdd->attr_sdma); sysfs_remove_file(p->kobj, &pdd->attr_evict); @@ -1036,7 +1044,7 @@ static void kfd_process_notifier_release(struct mmu_notifier *mn, struct mm_struct *mm) { struct kfd_process *p; - struct kfd_process_device *pdd = NULL; + int i; /* * The kfd_process structure can not be free because the @@ -1060,8 +1068,8 @@ static void kfd_process_notifier_release(struct mmu_notifier *mn, * pdd is in debug mode, we should first force unregistration, * then we will be able to destroy the queues */ - list_for_each_entry(pdd, &p->per_device_data, per_device_list) { - struct kfd_dev *dev = pdd->dev; + for (i = 0; i < p->n_pdds; i++) { + struct kfd_dev *dev = p->pdds[i]->dev; mutex_lock(kfd_get_dbgmgr_mutex()); if (dev && dev->dbgmgr && dev->dbgmgr->pasid == p->pasid) { @@ -1098,11 +1106,11 @@ static const struct mmu_notifier_ops kfd_process_mmu_notifier_ops = { static int kfd_process_init_cwsr_apu(struct kfd_process *p, struct file *filep) { unsigned long offset; - struct kfd_process_device *pdd; + int i; - list_for_each_entry(pdd, &p->per_device_data, per_device_list) { - struct kfd_dev *dev = pdd->dev; - struct qcm_process_device *qpd = &pdd->qpd; + for (i = 0; i < p->n_pdds; i++) { + struct kfd_dev *dev = p->pdds[i]->dev; + struct qcm_process_device *qpd = &p->pdds[i]->qpd; if (!dev->cwsr_enabled || qpd->cwsr_kaddr || qpd->cwsr_base) continue; @@ -1199,7 +1207,7 @@ static struct kfd_process *create_process(const struct task_struct *thread) mutex_init(&process->mutex); process->mm = thread->mm; process->lead_thread = thread->group_leader; - INIT_LIST_HEAD(&process->per_device_data); + process->n_pdds = 0; INIT_DELAYED_WORK(&process->eviction_work, evict_process_worker); INIT_DELAYED_WORK(&process->restore_work, restore_process_worker); process->last_restore_timestamp = get_jiffies_64(); @@ -1290,11 +1298,11 @@ static int init_doorbell_bitmap(struct qcm_process_device *qpd, struct kfd_process_device *kfd_get_process_device_data(struct kfd_dev *dev, struct kfd_process *p) { - struct kfd_process_device *pdd = NULL; + int i; - list_for_each_entry(pdd, &p->per_device_data, per_device_list) - if (pdd->dev == dev) - return pdd; + for (i = 0; i < p->n_pdds; i++) + if (p->pdds[i]->dev == dev) + return p->pdds[i]; return NULL; } @@ -1304,6 +1312,8 @@ struct kfd_process_device *kfd_create_process_device_data(struct kfd_dev *dev, { struct kfd_process_device *pdd = NULL; + if (WARN_ON_ONCE(p->n_pdds >= MAX_GPU_INSTANCE)) + return NULL; pdd = kzalloc(sizeof(*pdd), GFP_KERNEL); if (!pdd) return NULL; @@ -1332,7 +1342,7 @@ struct kfd_process_device *kfd_create_process_device_data(struct kfd_dev *dev, pdd->vram_usage = 0; pdd->sdma_past_activity_counter = 0; atomic64_set(&pdd->evict_duration_counter, 0); - list_add(&pdd->per_device_list, &p->per_device_data); + p->pdds[p->n_pdds++] = pdd; /* Init idr used for memory handle translation */ idr_init(&pdd->alloc_idr); @@ -1464,28 +1474,6 @@ struct kfd_process_device *kfd_bind_process_to_device(struct kfd_dev *dev, return ERR_PTR(err); } -struct kfd_process_device *kfd_get_first_process_device_data( - struct kfd_process *p) -{ - return list_first_entry(&p->per_device_data, - struct kfd_process_device, - per_device_list); -} - -struct kfd_process_device *kfd_get_next_process_device_data( - struct kfd_process *p, - struct kfd_process_device *pdd) -{ - if (list_is_last(&pdd->per_device_list, &p->per_device_data)) - return NULL; - return list_next_entry(pdd, per_device_list); -} - -bool kfd_has_process_device_data(struct kfd_process *p) -{ - return !(list_empty(&p->per_device_data)); -} - /* Create specific handle mapped to mem from process local memory idr * Assumes that the process lock is held. */ @@ -1561,11 +1549,13 @@ struct kfd_process *kfd_lookup_process_by_mm(const struct mm_struct *mm) */ int kfd_process_evict_queues(struct kfd_process *p) { - struct kfd_process_device *pdd; int r = 0; + int i; unsigned int n_evicted = 0; - list_for_each_entry(pdd, &p->per_device_data, per_device_list) { + for (i = 0; i < p->n_pdds; i++) { + struct kfd_process_device *pdd = p->pdds[i]; + r = pdd->dev->dqm->ops.evict_process_queues(pdd->dev->dqm, &pdd->qpd); if (r) { @@ -1581,7 +1571,9 @@ int kfd_process_evict_queues(struct kfd_process *p) /* To keep state consistent, roll back partial eviction by * restoring queues */ - list_for_each_entry(pdd, &p->per_device_data, per_device_list) { + for (i = 0; i < p->n_pdds; i++) { + struct kfd_process_device *pdd = p->pdds[i]; + if (n_evicted == 0) break; if (pdd->dev->dqm->ops.restore_process_queues(pdd->dev->dqm, @@ -1597,10 +1589,12 @@ int kfd_process_evict_queues(struct kfd_process *p) /* kfd_process_restore_queues - Restore all user queues of a process */ int kfd_process_restore_queues(struct kfd_process *p) { - struct kfd_process_device *pdd; int r, ret = 0; + int i; + + for (i = 0; i < p->n_pdds; i++) { + struct kfd_process_device *pdd = p->pdds[i]; - list_for_each_entry(pdd, &p->per_device_data, per_device_list) { r = pdd->dev->dqm->ops.restore_process_queues(pdd->dev->dqm, &pdd->qpd); if (r) { diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c index eb1635ac8988..95a6c36cea4c 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c @@ -126,10 +126,10 @@ int pqm_set_gws(struct process_queue_manager *pqm, unsigned int qid, void kfd_process_dequeue_from_all_devices(struct kfd_process *p) { - struct kfd_process_device *pdd; + int i; - list_for_each_entry(pdd, &p->per_device_data, per_device_list) - kfd_process_dequeue_from_device(pdd); + for (i = 0; i < p->n_pdds; i++) + kfd_process_dequeue_from_device(p->pdds[i]); } int pqm_init(struct process_queue_manager *pqm, struct kfd_process *p) From patchwork Mon Mar 22 10:58:18 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154499 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2CDDC433E1 for ; Mon, 22 Mar 2021 11:08:20 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 79A5E61879 for ; Mon, 22 Mar 2021 11:08:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 79A5E61879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B34006E0F5; Mon, 22 Mar 2021 11:07:51 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2073.outbound.protection.outlook.com [40.107.237.73]) by gabe.freedesktop.org (Postfix) with ESMTPS id C2A6389870; Mon, 22 Mar 2021 11:07:32 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=HjB0RLe9KSoH2zPw+wFc1XfOF7GyJhvcksKKedcFzcmo2jKgslBOXTnHVLKLgTD8bm6EFEKLOy8F4uyeyQXdxwX0Gd+MCQmDypz9Xl3SvDP0NkmuyjV7YmVGPsiSfcXh4QJWV1Nhk0Jzh6BLL2n6qv/bA7u8SJjYW+2imwSuu7FDheYW4G5Znd99tsuZ6fcmJ5LSsQ3QxPGbDgYzsAhjcwgrrk+Yn9UwYVwoPknZMYfJVEbC6vj9RvqVviw6JUwbvdaiT4cgdkjq1N5KzLae5FcUWTlVvcSyZDOVEvobb6iLQBi1rHPVpnx0qi+vr8pA0MGA667QVUhOgGMlsdE7gQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DWWi/VnipnSDWK3fAmbYzAeXH/H0Ojj6KZberR1F9zI=; b=VlEIUAzUbAzEjFJoq+xb3khfoKmvW2ciPnwikfEatlmXJ5ZlIPXnoSZkpCm6hNjLOZ+0pfre6nsPhr8QUddwOIq9I+F2JZbLXjrAijH8TxbQIT1Rv8hGbkQCLkBbK6brgqmHcQu5ds9ijX2gpK5fOPXo1pXjPY5IFYZixyevh7X5Ky9YB8hQ1+IrNiFy7dwNgwKMA0xENfWTu3pka4HvG6Eff+oHO5uaAkLfSAm+j/BZ/35etOfunmAkJiUAMg84rHFPPsKYQVWoWz2EsNUbjbaekK18dMH9xCYbTgGixZhP01UphkjiFWWF/m1vdTC2wVrRIWzPBx88KXSDGCKUkQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DWWi/VnipnSDWK3fAmbYzAeXH/H0Ojj6KZberR1F9zI=; b=GwEkNMcqYOz6h2nq/V0A8zLh9TNIty1ROCsg6a6UPcwMadCtjrgpbzO35c7/WXxgP/dTQ+3xOP6CgpLkC3JfX3fuse8mcCln4L7k3vVbmIvlBtb7WxdnNYlY/yPNlHPFghyigeiERUS1/DccD6TwostEk25RHMH7Dshv4565SWM= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3935.namprd12.prod.outlook.com (2603:10b6:208:168::31) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:30 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:30 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 02/44] drm/amdkfd: helper to convert gpu id and idx Date: Mon, 22 Mar 2021 06:58:18 -0400 Message-Id: <20210322105900.14068-3-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:30 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 8246a4ea-3fe2-4460-87c3-08d8ed22af85 X-MS-TrafficTypeDiagnostic: MN2PR12MB3935: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3513; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IuM0WuU6oaEbRyiwrUI4t+MJFV1XeZ4U/vhNuAhmSehruF6fRqGUR7B9q0GRgGQyx2/p2I3PW+9+XYrIO0rOqJlHT/yC60jiOusgb9sGZ7UOHz0fAqURAp1kC4pNfaNx5CNUzFgTabO0VDU1pZl1+tMtEljWVjyr0MxgWRTjptprDBoGEyMILmRjS6Iraqu1r1QZAouX+JLvg5MEvrX5u4JpGIZ8aZgzkLIsmQwaawvnTjh1b5ZYkBQFTQrTPtNm+OzpNqgckY3LEQ5zBFBXwZhvtYiNLJKe6naRkQWfHImIIkbXkwGb6JKSaednjh2ky0fgbYfMwlkDOdtGyan2Id2l+GL2FLL3Nmykx0+W7YIAduIu6+uVF/nXAKm2Vuv9L/yn2NbrQ9VMEe0J5b1wAG4+YZYhBwMbMpnWSRyX4fEm0qQ34ARgdxBzD41ezUDDFKWj6IapV6GakExcIcp6SY2WuP3RUg8/C0ePaOJM/bFlMWQSrHNNKJ+b1sKU+EU+qCfZaM5VE17w6VUvLdIXmIvXEeDHKVObeZwhuvWLpjSMEEj6Fmgs9uehEPbzcUsaSf8yxl5bHObMvPuLWW4XE8MwMUWaq+SWA8q9AvMPfirsM+k2Azw4KO3HnzGz21SfxwVpmPHpteyVye6xpSorB4MYxQlYzPTDLnNRe5CN3Rg= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(39860400002)(136003)(366004)(396003)(346002)(376002)(5660300002)(8936002)(450100002)(6486002)(478600001)(4326008)(83380400001)(1076003)(86362001)(8676002)(956004)(2906002)(36756003)(7696005)(26005)(16526019)(52116002)(186003)(316002)(66946007)(66556008)(6666004)(66476007)(2616005)(38100700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: hcFHI0A4WDU4KV+iPxSp3tvn3SfWPkSL5x9F2i5aBctjHJb3q4ltDs24Skf3r5rUnKZeASAIRHK/t71TYWF0BVNvvQQ6NVUMH31X9K57RJSKRI/CW8swN3MeHNnhKdFGBdVdbp71aO53iBmjo0p5duB+pLNgGYs9JuZPHb3LO9K9TO/IYYT/ylHAGqqO+Mn04Zeij2O1ARDrjV3CFGHKSGviX5D1j5QV9kVejzdKGbajYOVQctku5aT2SAYuCBUYz/LYHJOuVhQ6oOltx4xr6o0EMHch6234LJ2uc1DNIYA//xwE2zRiOjyKpetttpBiZQ/D9QsvnIhLKRn0Mx9eOg1foBXBGtslgjhH3abuWFwA9A0fjXSdcxYyJULxLZdkalgejN1bszgn1x3gNkq2UlTPHccSFC/FdrEi68Dz/+HH87QqneY56jgnXeWRxdhTSLc7PAPEkYa+jvaZGmgIviIkMMrV6e3ud/DewrNG0IqjShwg3AiwwP95AHzBowdE/ymFUNa7+l0lbxICoHhrGtgKuVCeTxr9EOBlKtbxnjUMayTcgm3xFAUcQDcsn7AO61HgQ/rn8fOPKMB6xboeQx/ml0BMqy+Xqm1Af+BBAkINoo+thYkYNiXz07WCXbe2y1aAXyxqDDxVzqInW84ZRzXT6Vrja7Z1of0BYuAzwW6zv523vCnJXd/gGBs4XN/1xu1yCikRL3hIxpwLMWaZwiNrGHV0mz/8Q2VJeHMURVPJdd+wt6So5l+vFCmPQEvEGEuv26jONjnUyi1YGHROTXy/iD7BiUQWyGR33vlwFC24poBHYHxKAz+4tSSjIeWNB7fgyck5yHFtKYjLm0zeFjPZEvYt2TYK86ZCPnDAYwQtBy/JSVVbewunCDxM66dRsdl1ojrjGMho/CFHf3FcuzcL/c75e4TbA9maFP3Y/Yl9RXE0U2kfWZgL6Acd743uGY32vo5xyuqN9Okghcm6H2yZdNYs3IaJ23gFCGF+cDGD+DBPseRJHYMdzoY5surf60wNmhrVyvjipo6hpDcPd5iiAM4fNv2LD3BC457vEbLiXCitpwp6mdVNk2KBorZGZBvOxVnuDlkOey+Z6+Po2GYpTboCLNTHa0bcIg9aqOTDMpNiFUlNBgHvCa//hQvZ/UJ4ZgAJC7mqNiWWft5gyeA5zDJx53Ol7ejhWzl7mFH25z3LEGqnAvpqAp+ioUgECjtrM1m6xgTdlNG9uoDOpTZ3BZOWjI6NMCMgh+zDBncvdGqH952Ag9MtoTINciMdkD1QlZU7u2CzW/Y2H1GP88ABvbe8Dgr3cH6Lk7zFEZ5Cyb1gXr4pjQxSSlVLuO4G X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8246a4ea-3fe2-4460-87c3-08d8ed22af85 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:30.4510 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: +fY9aiNOVK/KCi0KxqrNECIJxP/Uav1AZEABoDJ2oK8fK//uqPlIVGxBvL7+vVS52XVEedMJeJsCbf8J/mxhtQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3935 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra svm range uses gpu bitmap to store which GPU svm range maps to. Application pass driver gpu id to specify GPU, the helper is needed to convert gpu id to gpu bitmap idx. Access through kfd_process_device pointers array from kfd_process. Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 5 ++++ drivers/gpu/drm/amd/amdkfd/kfd_process.c | 30 ++++++++++++++++++++++++ 2 files changed, 35 insertions(+) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index 33e56db14327..99b4624ef4c7 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -842,6 +842,11 @@ struct kfd_process *kfd_create_process(struct file *filep); struct kfd_process *kfd_get_process(const struct task_struct *); struct kfd_process *kfd_lookup_process_by_pasid(u32 pasid); struct kfd_process *kfd_lookup_process_by_mm(const struct mm_struct *mm); +int kfd_process_gpuid_from_gpuidx(struct kfd_process *p, + uint32_t gpu_idx, uint32_t *gpuid); +int kfd_process_gpuidx_from_gpuid(struct kfd_process *p, uint32_t gpu_id); +int kfd_process_device_from_gpuidx(struct kfd_process *p, + uint32_t gpu_idx, struct kfd_dev **gpu); void kfd_unref_process(struct kfd_process *p); int kfd_process_evict_queues(struct kfd_process *p); int kfd_process_restore_queues(struct kfd_process *p); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index d4241d29ea94..028ebb0deddd 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -1607,6 +1607,36 @@ int kfd_process_restore_queues(struct kfd_process *p) return ret; } +int kfd_process_gpuid_from_gpuidx(struct kfd_process *p, + uint32_t gpu_idx, uint32_t *gpuid) +{ + if (gpu_idx < p->n_pdds) { + *gpuid = p->pdds[gpu_idx]->dev->id; + return 0; + } + return -EINVAL; +} + +int kfd_process_gpuidx_from_gpuid(struct kfd_process *p, uint32_t gpu_id) +{ + int i; + + for (i = 0; i < p->n_pdds; i++) + if (p->pdds[i] && gpu_id == p->pdds[i]->dev->id) + return i; + return -EINVAL; +} + +int kfd_process_device_from_gpuidx(struct kfd_process *p, + uint32_t gpu_idx, struct kfd_dev **gpu) +{ + if (gpu_idx < p->n_pdds) { + *gpu = p->pdds[gpu_idx]->dev; + return 0; + } + return -EINVAL; +} + static void evict_process_worker(struct work_struct *work) { int ret; From patchwork Mon Mar 22 10:58:19 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154493 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 244F6C433E4 for ; Mon, 22 Mar 2021 11:08:16 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E0A6A61879 for ; Mon, 22 Mar 2021 11:08:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E0A6A61879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C0A066E0F8; Mon, 22 Mar 2021 11:07:51 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2041.outbound.protection.outlook.com [40.107.237.41]) by gabe.freedesktop.org (Postfix) with ESMTPS id 93C5A8999C; Mon, 22 Mar 2021 11:07:34 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=RVO4mY1B3qNe3/HRQXrohFLelbKHiK/vQx+Bsu3dU3QmYpgpAlXKucjWRlp2LeSByp5jISIP6uc/EzRAgtItxgz6uAV83O6A48dDf+pZYyeYrAIVTc1ADyzDupVPg2RMQcDwuFu09yPVQbZQ3ZjdduUuq3/ha7Z6dN3RS3Yaq7Pfw5IYP3RU9v1jTaJs/nSm043y8eF6GjmzdzJEoa20/UVDpzUAKvEjlEe5LwQcs7dwzqO3UW45g5oRJh16qmgB31YpeW0l3hrtYLZMCZhPXnJoRt42eQ2v+aeGF6IpnGO2xs7sw6qe3V656tLrXcYNrSX4Dd2ZQY332/me4uCtXQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Y19EvBHzBPkAb69RdQzjf++FzipztJlyHJIJl9TLg+Q=; b=FWqezcMuRN3i6TLPyBRoZjFDszYJKASNwXpOfumacoehS+mskxqIKeRAY9z8/4+4Xov8nXh6aVdHymZ+Qk+sgtrke90PAOnNoRtlu9k6V/AzK5a8CwAyGmxRCJRiTw2m4IuemKZMTGXwyReiEeHOFcamInqSaFp2sGtwmecdGAn80rbUmwX+sAgcFdUT4TksKTbxeE8qiD3cK/sf7wcmrL/LblxGT3oiXHGzMK/AdXKfEDoQN5TJTZ/4OJujzyRAXCmr7/mMfSPJ7ofOEIAHfelKQv5vRDWtpPrLc08q94w0YzF4tppJoOKUmrK79AEd4VCLvVwPogRWH3qT2lHsMA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Y19EvBHzBPkAb69RdQzjf++FzipztJlyHJIJl9TLg+Q=; b=jMvpWTlX+Dln771SkZenZSTZagLIMmuhf/q0hZjdRcRR4i0bH0MUf5pRBqu9jbIUtVYiTaUH6JOmfV2zfirJELz8u1zNmJuWsEMIrz+b/qgjgEMDykd9OBGcwtotHdBMx2A5D++93utHpjM7DdFLS7NTJFi27WJEALOWVcaPAxs= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:31 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:31 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 03/44] drm/amdkfd: add svm ioctl API Date: Mon, 22 Mar 2021 06:58:19 -0400 Message-Id: <20210322105900.14068-4-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:30 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 9626b566-8190-4322-4acd-08d8ed22afed X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: NeyeiLIkAfvzF+v6arho85dh7B4huJJDTJmybFCFpPBkkgX/QGUV8d1ShygwG/TK9HiIg2w4nS38r7tkJnG871lbTmZ76Rz7UJdc7yAuM2JLmGRnITYs5fkfhkBgSuCLrtL28C6gxfquJc99lfW1iVr35lgzjiTxUHXOWjndbPjopO+AUDI0Q6INIyaKVz5CSUK42L9u13kpK9NPfGBOctuY9/sUBVN20RJvS13zRbHrB2s1nIiYt3HTNGIaKHuQR6P2Q+4Ie0tsG1gQgatM1cyGVAfyGjqqOGVAlfCSaEELsHd9LXO4JHYJ/PYlwEtmIVkEaVICAhxNEpLEYKKXZzGG6+bEFUm4TDjRmoom3yeRAODruct7BvzL8d7P27Q39nlvB4joUdkvqL4S85uXQS0MioYJViNzLVFirf4vMCF2EAPbQYg/xDc4DMBx0wmDI90PXZ6T1Itc7i90YzSEbGch6Q1eoNvCUrxd3AMUYbp3Q7rlZr0EaqOc1Q8A6rQde1SB8zghjLx4wsgjAAtb74Rk99EAcpDlkYEskZEsKRndf9Pw9v7uUGcfW25sxM/4BgX7raZSiDHLsZ0B71BUBxZMuij7LKz8B4Tvm2KyWFnHrj3UP81RwwgLJgrx31XQlnIVhQ+rP10NYjeJbocou3M63eAwqxxEdN0RGSEGhWc= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(54906003)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 7NYOIK8cGTN/kUPA0rXZZjSYXR1iMxQtV9ZTpcHGus4Eg870rqvmz1Ei2T0BThGxV90g334cjWE779cr4/isWB5wCJ/mH5uLLxXLowuIxXgdVf7bhNhZPs1KR+EJ3xiwoguVO2UVuJKeGecxIqyht37niU1H2pDanTENKfiKA373inSj7TwMqdv5o2vvev3I8uEaVepVolViMFGdW04+aytzQjYNyUJNH+jvm2x66kJIdPoKjiYlj+/+fXCDzrJ4+B2iHXy2zIwiraHJp/ZW3Mi+f6XWVAvFVIflxnMCmyt3RY7rzT8cLbgiAVWjV80lzpr/vbcaI3Oug1DqxIjwn8dG9JQp5x1nCP8WrLbsFaCGgiB/SrvuH9YccoHzYu2cL3/lZp0Of0CURcKncqqpWS3LVNH3J5UYES5vf1X5D7H31n32zlnaT74sjYKrYTjr3LvHC0h353Zh2MlI7DYt4DIbm+FuC1U4xss3+GSZ4ZI8GCmtdl7oaLU9tNJTh0lizkcgrpmIv97dK5Yg0fGpOodxi73ZwAVOUpunLTBF0BfY4QP6ptWk3dVTDj5OMo0KJ5KtfcADYz2qJMfUT2YVw0kfwPz6IODvbZwxqfOA/JtP9in5oePFZV9uVKxH4TqlDQxLbH0KE51z9yRypTsUJbM7ZI0G2hIZQPWk+44/EuXxvRduHUO6JY8mQfs+IrFpbSzxDCHBH7BTEUaleqY3WV2DiSD7bq5K4e3Rkwpm2NrW7NeGwWKJka+qJ1lpY/h1Ws2Yjy2U5FrFdRqm8TcEg7y8p5SmNjoB/t73AqMig3vZSekr8s4u4RT0dNR/okoXOaqb2Dcb+F5sukDIwjIC2rJ8d7mIt7TAbyHtM1uUGWMOwHjRtLQCH3f5EmKqz6sPMKGh6H7wvijMNlm6gbGRpF35mO9DQStaySMCsWR89IVFyzlhi4Gbu9hPZuJ4JMf1jWkQGzWGASPcGsn5RrMKRV3yH53LKhSEIli9C8yoLjRhGOkwr3fAZHpOtyUKOgZcjMsGXjgDRDX49P00uRV++fO9bsctOlYEMEdhFIgdd+Hq0lD1ksmDYpxG+ifK7u/9UgCtZOmQ1XEKqg+fHmDp/MnOgYYCtPdsJ9ieLIdl9CP6FkFiAlkEqDJgH/IYlAvtgPmayZnAu4O8kdPOqNYoP3ly/2t3N5Z9DESF5cwI6Vblobg4nvzIZanNifp76yx5lDbRjNRAh8VVF469suCiVkRYr63cT7FODY1ddvpGCi0sWBhVjpZPNXGeJuoY6owqsYuECJHw0VCs1tWYCZpDSHa+7m1Y8KTqgjm2bnGhPFygIfMJA0QB23S5ZGPF27g5 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 9626b566-8190-4322-4acd-08d8ed22afed X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:31.0812 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Sp9oy5j6rFVZ9UYsWjCZ+fObMfLiRUTowjUcb4RaLMv4s25gW03tqteKkyyHpQnoriPsFeek/6t0TnOcbhJWWg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang Add svm (shared virtual memory) ioctl data structure and API definition. The svm ioctl API is designed to be extensible in the future. All operations are provided by a single IOCTL to preserve ioctl number space. The arguments structure ends with a variable size array of attributes that can be used to set or get one or multiple attributes. Signed-off-by: Philip Yang Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 12 ++ drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c | 4 + drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 2 + drivers/gpu/drm/amd/amdkfd/kfd_process.c | 1 + include/uapi/linux/kfd_ioctl.h | 130 ++++++++++++++++++- 5 files changed, 147 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c index 43de260b2230..dbc824cc6b32 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -1742,6 +1742,16 @@ static int kfd_ioctl_smi_events(struct file *filep, return kfd_smi_event_open(dev, &args->anon_fd); } +static int kfd_ioctl_svm(struct file *filep, struct kfd_process *p, void *data) +{ + int r = 0; + + if (p->svm_disabled) + return -EPERM; + + return r; +} + #define AMDKFD_IOCTL_DEF(ioctl, _func, _flags) \ [_IOC_NR(ioctl)] = {.cmd = ioctl, .func = _func, .flags = _flags, \ .cmd_drv = 0, .name = #ioctl} @@ -1840,6 +1850,8 @@ static const struct amdkfd_ioctl_desc amdkfd_ioctls[] = { AMDKFD_IOCTL_DEF(AMDKFD_IOC_SMI_EVENTS, kfd_ioctl_smi_events, 0), + + AMDKFD_IOCTL_DEF(AMDKFD_IOC_SVM, kfd_ioctl_svm, 0), }; #define AMDKFD_CORE_IOCTL_COUNT ARRAY_SIZE(amdkfd_ioctls) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c index a2c9063076cc..52da1a3b2c7a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_flat_memory.c @@ -405,6 +405,10 @@ int kfd_init_apertures(struct kfd_process *process) case CHIP_POLARIS12: case CHIP_VEGAM: kfd_init_apertures_vi(pdd, id); + /* VI GPUs cannot support SVM with only + * 40 bits of virtual address space. + */ + process->svm_disabled |= true; break; case CHIP_VEGA10: case CHIP_VEGA12: diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index 99b4624ef4c7..18fc2ccd1a77 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -809,6 +809,8 @@ struct kfd_process { struct kobject *kobj; struct kobject *kobj_queues; struct attribute attr_pasid; + + bool svm_disabled; }; #define KFD_PROCESS_TABLE_SIZE 5 /* bits: 32 entries */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index 028ebb0deddd..89e7c125d334 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -1208,6 +1208,7 @@ static struct kfd_process *create_process(const struct task_struct *thread) process->mm = thread->mm; process->lead_thread = thread->group_leader; process->n_pdds = 0; + process->svm_disabled = false; INIT_DELAYED_WORK(&process->eviction_work, evict_process_worker); INIT_DELAYED_WORK(&process->restore_work, restore_process_worker); process->last_restore_timestamp = get_jiffies_64(); diff --git a/include/uapi/linux/kfd_ioctl.h b/include/uapi/linux/kfd_ioctl.h index bf5e7d7846dd..247b57baa94f 100644 --- a/include/uapi/linux/kfd_ioctl.h +++ b/include/uapi/linux/kfd_ioctl.h @@ -30,9 +30,10 @@ * - 1.1 - initial version * - 1.3 - Add SMI events support * - 1.4 - Indicate new SRAM EDC bit in device properties + * - 1.5 - Add SVM API */ #define KFD_IOCTL_MAJOR_VERSION 1 -#define KFD_IOCTL_MINOR_VERSION 4 +#define KFD_IOCTL_MINOR_VERSION 5 struct kfd_ioctl_get_version_args { __u32 major_version; /* from KFD */ @@ -473,6 +474,129 @@ enum kfd_mmio_remap { KFD_MMIO_REMAP_HDP_REG_FLUSH_CNTL = 4, }; +/* Guarantee host access to memory */ +#define KFD_IOCTL_SVM_FLAG_HOST_ACCESS 0x00000001 +/* Fine grained coherency between all devices with access */ +#define KFD_IOCTL_SVM_FLAG_COHERENT 0x00000002 +/* Use any GPU in same hive as preferred device */ +#define KFD_IOCTL_SVM_FLAG_HIVE_LOCAL 0x00000004 +/* GPUs only read, allows replication */ +#define KFD_IOCTL_SVM_FLAG_GPU_RO 0x00000008 +/* Allow execution on GPU */ +#define KFD_IOCTL_SVM_FLAG_GPU_EXEC 0x00000010 +/* GPUs mostly read, may allow similar optimizations as RO, but writes fault */ +#define KFD_IOCTL_SVM_FLAG_GPU_READ_MOSTLY 0x00000020 + +/** + * kfd_ioctl_svm_op - SVM ioctl operations + * + * @KFD_IOCTL_SVM_OP_SET_ATTR: Modify one or more attributes + * @KFD_IOCTL_SVM_OP_GET_ATTR: Query one or more attributes + */ +enum kfd_ioctl_svm_op { + KFD_IOCTL_SVM_OP_SET_ATTR, + KFD_IOCTL_SVM_OP_GET_ATTR +}; + +/** kfd_ioctl_svm_location - Enum for preferred and prefetch locations + * + * GPU IDs are used to specify GPUs as preferred and prefetch locations. + * Below definitions are used for system memory or for leaving the preferred + * location unspecified. + */ +enum kfd_ioctl_svm_location { + KFD_IOCTL_SVM_LOCATION_SYSMEM = 0, + KFD_IOCTL_SVM_LOCATION_UNDEFINED = 0xffffffff +}; + +/** + * kfd_ioctl_svm_attr_type - SVM attribute types + * + * @KFD_IOCTL_SVM_ATTR_PREFERRED_LOC: gpuid of the preferred location, 0 for + * system memory + * @KFD_IOCTL_SVM_ATTR_PREFETCH_LOC: gpuid of the prefetch location, 0 for + * system memory. Setting this triggers an + * immediate prefetch (migration). + * @KFD_IOCTL_SVM_ATTR_ACCESS: + * @KFD_IOCTL_SVM_ATTR_ACCESS_IN_PLACE: + * @KFD_IOCTL_SVM_ATTR_NO_ACCESS: specify memory access for the gpuid given + * by the attribute value + * @KFD_IOCTL_SVM_ATTR_SET_FLAGS: bitmask of flags to set (see + * KFD_IOCTL_SVM_FLAG_...) + * @KFD_IOCTL_SVM_ATTR_CLR_FLAGS: bitmask of flags to clear + * @KFD_IOCTL_SVM_ATTR_GRANULARITY: migration granularity + * (log2 num pages) + */ +enum kfd_ioctl_svm_attr_type { + KFD_IOCTL_SVM_ATTR_PREFERRED_LOC, + KFD_IOCTL_SVM_ATTR_PREFETCH_LOC, + KFD_IOCTL_SVM_ATTR_ACCESS, + KFD_IOCTL_SVM_ATTR_ACCESS_IN_PLACE, + KFD_IOCTL_SVM_ATTR_NO_ACCESS, + KFD_IOCTL_SVM_ATTR_SET_FLAGS, + KFD_IOCTL_SVM_ATTR_CLR_FLAGS, + KFD_IOCTL_SVM_ATTR_GRANULARITY +}; + +/** + * kfd_ioctl_svm_attribute - Attributes as pairs of type and value + * + * The meaning of the @value depends on the attribute type. + * + * @type: attribute type (see enum @kfd_ioctl_svm_attr_type) + * @value: attribute value + */ +struct kfd_ioctl_svm_attribute { + __u32 type; + __u32 value; +}; + +/** + * kfd_ioctl_svm_args - Arguments for SVM ioctl + * + * @op specifies the operation to perform (see enum + * @kfd_ioctl_svm_op). @start_addr and @size are common for all + * operations. + * + * A variable number of attributes can be given in @attrs. + * @nattr specifies the number of attributes. New attributes can be + * added in the future without breaking the ABI. If unknown attributes + * are given, the function returns -EINVAL. + * + * @KFD_IOCTL_SVM_OP_SET_ATTR sets attributes for a virtual address + * range. It may overlap existing virtual address ranges. If it does, + * the existing ranges will be split such that the attribute changes + * only apply to the specified address range. + * + * @KFD_IOCTL_SVM_OP_GET_ATTR returns the intersection of attributes + * over all memory in the given range and returns the result as the + * attribute value. If different pages have different preferred or + * prefetch locations, 0xffffffff will be returned for + * @KFD_IOCTL_SVM_ATTR_PREFERRED_LOC or + * @KFD_IOCTL_SVM_ATTR_PREFETCH_LOC resepctively. For + * @KFD_IOCTL_SVM_ATTR_SET_FLAGS, flags of all pages will be + * aggregated by bitwise AND. The minimum migration granularity + * throughout the range will be returned for + * @KFD_IOCTL_SVM_ATTR_GRANULARITY. + * + * Querying of accessibility attributes works by initializing the + * attribute type to @KFD_IOCTL_SVM_ATTR_ACCESS and the value to the + * GPUID being queried. Multiple attributes can be given to allow + * querying multiple GPUIDs. The ioctl function overwrites the + * attribute type to indicate the access for the specified GPU. + * + * @KFD_IOCTL_SVM_ATTR_CLR_FLAGS is invalid for + * @KFD_IOCTL_SVM_OP_GET_ATTR. + */ +struct kfd_ioctl_svm_args { + __u64 start_addr; + __u64 size; + __u32 op; + __u32 nattr; + /* Variable length array of attributes */ + struct kfd_ioctl_svm_attribute attrs[0]; +}; + #define AMDKFD_IOCTL_BASE 'K' #define AMDKFD_IO(nr) _IO(AMDKFD_IOCTL_BASE, nr) #define AMDKFD_IOR(nr, type) _IOR(AMDKFD_IOCTL_BASE, nr, type) @@ -573,7 +697,9 @@ enum kfd_mmio_remap { #define AMDKFD_IOC_SMI_EVENTS \ AMDKFD_IOWR(0x1F, struct kfd_ioctl_smi_events_args) +#define AMDKFD_IOC_SVM AMDKFD_IOWR(0x20, struct kfd_ioctl_svm_args) + #define AMDKFD_COMMAND_START 0x01 -#define AMDKFD_COMMAND_END 0x20 +#define AMDKFD_COMMAND_END 0x21 #endif From patchwork Mon Mar 22 10:58:20 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154483 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7114DC433E0 for ; Mon, 22 Mar 2021 11:08:03 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3360F61879 for ; Mon, 22 Mar 2021 11:08:03 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3360F61879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E0D6589BFC; Mon, 22 Mar 2021 11:07:38 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2041.outbound.protection.outlook.com [40.107.237.41]) by gabe.freedesktop.org (Postfix) with ESMTPS id D55F189ACD; Mon, 22 Mar 2021 11:07:34 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=kbLpcQvS0fd+jrjpkWZ/666zQNoGxL3v6DeXpQqHmkfhD6tSCO08a+xmQ/kTZJ+Qk3/UlKEG8C7tQqn08NpaYjlcp4x/tY3pVItOqs8X8lIeSbGfWWzoTQnLeB/cnDkWeMYoDnVenGyFnJY2qA1P7kCyt7EsLdWvwLCOA5YDUXQ6RftynLJYob1jdcc8LOBR4BWLvG9fPHDkKWVJT83UHv+c68Tyaa6gXQK4cF0w6jQcFq0s04EIEBydpcz4AcOyblGW6Ogkq6dowdoVNGZ//s3XHSrykE9mzAuQqY3fqXb/oSw5zfJbiFoIa5PVGpsUt2MUjUQOkCO5Gz/Ln7L6QA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6P+S7e5pWq2G0004iqIsHPXP0LcGymQONM5u8wfXEfY=; b=epLSGYU4X5uhplcqPancby/nKwmF6Gqqz4vX9hfha8+62e53rZE4Mx2VXX88HNpoTZEvy245rvTSBtbHHs+PMm8oCFblD5Cr1uPuPJTyhjzjRf/Ggdttq1JcX886XTIFWkknjGyCBbzP4ZVPkbiVDTNjSjk1pYB/IXnJia4lqPAnj+Bylj9hodQQnthHggImq8Ey2pgdVNfb7xeHPy9Uv09LEHCZzHCfYsZzkTG6pZJW0bOqA7B0dwaRqb77DM3URG+qEBzrCM6q58sWI1Jfq+wnrNI/vELYdPF/VAeoKhg6Nzx5G3xX3QXHyHhRXOVgV3pzm1Q7cUiTxFbPZjlJag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6P+S7e5pWq2G0004iqIsHPXP0LcGymQONM5u8wfXEfY=; b=hIgMIVl2Q8hCoVR519cXa5Ru6ZYMRm4PWvxGOH+rt+u3QAZnsjwxzEngqzFh53QEOJ63+uBJLbr4354ltUGoRZLWA3jGsj5A3BSJpuqEXrQvZf56FESDiVWmGr7k09fXoO26V/WK6UpiTj9iHFmaUbf/U+ARHGpPzpAusw7GIJ8= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:31 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:31 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 04/44] drm/amdkfd: register svm range Date: Mon, 22 Mar 2021 06:58:20 -0400 Message-Id: <20210322105900.14068-5-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:31 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: dfc6b50c-a888-4a39-880a-08d8ed22b03b X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8882; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: pgNoQzeR1PpPjI4lvJeCBaTPu54Wpjs8qqheqcHsBAMEOoVZQLSpEIrEUHIwEbnnh8rVtbCM3OPpccwC4uTeAgLa3djo/8zxK10M5q/zSDYeFF1W+9k7RlM430hEM+wOiPWy0jLnSEux+49nLe+/Jz8AepUeL0zIfBuWxBaSZbgudRuxPA3+WwOr/rGKzUhHDgOI0y1TaYv6//trEcFooTIJEPFhs3GVsxYqSf3BFynvJuq9rYbWcWQ5hPZ4L2JjeJhIU1qdAoaxfDBJ1ilZCmPjn3QKTMLVWH0bwdGNibL4sMomjcwz3g0B21jVOcWXtOuttHpjebBuerwHdyTrGWRtVoBTLG2wykgyre7wDeeUZIbJKqYquNDEPL9Z/OqLAc55hKDvtbKx8cm+zQ0FDBe0FiuYVXoekDLo8C/7Gx1hWypZRI89i3NRUt0tF/KOfIffjdJ57CadBXgrzL2Dt8fKtClW+Vse6tmDcoczSU1m9iTEkSecDlkmWcR3RKJyUuvAQ+4cpJMOtYpuN3OXATkXXTHCNwQeUoBYGKO58eUhc3eb3hk59VBwV4FpiNz9mxLK35B79AzJw3ccZmspzwHrgU8czc4AGug2T9EJzERSUV21HT1O+k4APo+qrrWmaAvsbp2WwvUG6PYwQf0PebndR9JYJMwLj9xwwz21w3E= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(54906003)(6486002)(1076003)(36756003)(30864003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 2sqndPyhZEiDb6oaIVXQUbmvdw0QF5jBwq4ZhKrkmQKMGzCmQS/wJXL0lRNI2rTqkWUakgrf+3DyeB0/etXvUpywcsip13xeitk/xEMCy2guXD6zh4YDffTrIa2fSf/ryhLcJbFK0I+97X/uCdU+Vie8TO4DTGUXN3dVtyPIQRJFxkXK12fxa3/v0k8c4T30MPOPEJR6OvThMYMw9szKlTJUU+P7WdXgsGDyvfROaOgvcX6QKDs/hVbfwbV+WzONKGVWpKU19vfhnN0jMrJ2DeNxI9/M8J9Nu34uZMQh+kQ8T4vlvveaa27rlrd2S/OxtL5jB0Vfux05ydn0bs4teRmGboh/W+j6P9YPbnkIN179sf/TfqAvl6k+S+1yc7HRClRPn3Y/rRmeSnZIGFyrpi5PErdPsXlug7evwiSs5R2hawxfKkOuhg8oGQCeg75JQ0roWxxiW5qPkwqcQTpyWjEgUZQBVrL3cpISvKo3I1iilFMv0qgXNMSZGDYr2xwguYVdv/2sc543fxyJd++tU8aWrlVP4WJ54VPzptIz4QqwtuLqlK9AnEMIfS03dXrwq4w9ijF+eKHzl7o1q9tnk0jC0gsH09/L+azF7ahgPGM0ihmTp/MS0mpud2t1jbWl6mNP/apeYmR6K87i+ed54cjvscd1h3yNigxMFsJ0U61rqs71P8zmmYTBv31jnSx0yuLZDoa099BVRbDGp0ZWC9u7726jGx/bdg3nFeD7pR5WNLd1RyddtPY9LS9NlpfTqyigCOM8jkmthWPN8fxagpRbmYioIZReskZjb9VDTq0v0KJ1WqqhPPsuJdO9LYXoPGqw5J9yJypULS0Zj2lKEAOx0LMpCDX+7opi5HwODoqqH5h7SXutNwO5XLZW4P6+EifOzJa1BEKOaowNtg3FkzlleXTXJ0GMtVvP915PwiMa5N4i9GWFJUd1XwG1pbVgQlapO/233lwtpfHxoszp6hkFbR87S0TnM5QiMIm1ckYgWm+Zio8uK2QIA9dl+aL2/Jp6ArkxqCJeKnKQhdcb2RUQa5jnFEGYM+8iIBIoWF9R5fFG9AF8G7RI97OowlV656L8lbAY+8JFS7Ae2/DyclVx9ZOLLfUcOXFCI4iDx0MBoB2M7dDr2J/IQMw6FnApFkhp6H3Wg3pEn7n46zu1SAVjANVhGECg31/4nRvLskydnSqF2hD5Nm/3JTsIF6mTBcQNpy/LRI//Ulzw4TGK58uPQ0er0KcRVjOH8WvbWpcVMzIeNpnrQGAHZQe4xPsqcR+P1hzwS+Tq/D/1Q36fr6JVc4Y7pJ8M7VAQK2VURHgF6U9NYbGq6IXKg7oawh8U X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: dfc6b50c-a888-4a39-880a-08d8ed22b03b X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:31.6806 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: z9G6w2qTb1+TQx9uaGjUZc2S6woOidY9Us3IL99SuwHPjuB/KdANbmsHes/C6Uo5ElOL6oLMTZwx+ZrwAjr7iw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang svm range structure stores the range start address, size, attributes, flags, prefetch location and gpu bitmap which indicates which GPU this range maps to. Same virtual address is shared by CPU and GPUs. Process has svm range list which uses both interval tree and list to store all svm ranges registered by the process. Interval tree is used by GPU vm fault handler and CPU page fault handler to get svm range structure from the specific address. List is used to scan all ranges in eviction restore work. No overlap range interval [start, last] exist in svms object interval tree. If process registers new range which has overlap with old range, the old range split into 2 ranges depending on the overlap happens at head or tail part of old range. Apply attributes preferred location, prefetch location, mapping flags, migration granularity to svm range, store mapping gpu index into bitmap. Signed-off-by: Philip Yang Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/Makefile | 3 +- drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 17 + drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 8 + drivers/gpu/drm/amd/amdkfd/kfd_process.c | 9 + drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 791 +++++++++++++++++++++++ drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 87 +++ 6 files changed, 914 insertions(+), 1 deletion(-) create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_svm.c create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_svm.h diff --git a/drivers/gpu/drm/amd/amdkfd/Makefile b/drivers/gpu/drm/amd/amdkfd/Makefile index e1e4115dcf78..387ce0217d35 100644 --- a/drivers/gpu/drm/amd/amdkfd/Makefile +++ b/drivers/gpu/drm/amd/amdkfd/Makefile @@ -54,7 +54,8 @@ AMDKFD_FILES := $(AMDKFD_PATH)/kfd_module.o \ $(AMDKFD_PATH)/kfd_dbgdev.o \ $(AMDKFD_PATH)/kfd_dbgmgr.o \ $(AMDKFD_PATH)/kfd_smi_events.o \ - $(AMDKFD_PATH)/kfd_crat.o + $(AMDKFD_PATH)/kfd_crat.o \ + $(AMDKFD_PATH)/kfd_svm.o ifneq ($(CONFIG_AMD_IOMMU_V2),) AMDKFD_FILES += $(AMDKFD_PATH)/kfd_iommu.o diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c index dbc824cc6b32..9511826ac8ae 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -38,6 +38,7 @@ #include "kfd_priv.h" #include "kfd_device_queue_manager.h" #include "kfd_dbgmgr.h" +#include "kfd_svm.h" #include "amdgpu_amdkfd.h" #include "kfd_smi_events.h" @@ -1744,11 +1745,27 @@ static int kfd_ioctl_smi_events(struct file *filep, static int kfd_ioctl_svm(struct file *filep, struct kfd_process *p, void *data) { + struct kfd_ioctl_svm_args *args = data; int r = 0; if (p->svm_disabled) return -EPERM; + pr_debug("start 0x%llx size 0x%llx op 0x%x nattr 0x%x\n", + args->start_addr, args->size, args->op, args->nattr); + + if ((args->start_addr & ~PAGE_MASK) || (args->size & ~PAGE_MASK)) + return -EINVAL; + if (!args->start_addr || !args->size) + return -EINVAL; + + mutex_lock(&p->mutex); + + r = svm_ioctl(p, args->op, args->start_addr, args->size, args->nattr, + args->attrs); + + mutex_unlock(&p->mutex); + return r; } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index 18fc2ccd1a77..f547e1282d69 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -731,6 +731,12 @@ struct kfd_process_device { #define qpd_to_pdd(x) container_of(x, struct kfd_process_device, qpd) +struct svm_range_list { + struct mutex lock; + struct rb_root_cached objects; + struct list_head list; +}; + /* Process data */ struct kfd_process { /* @@ -810,6 +816,8 @@ struct kfd_process { struct kobject *kobj_queues; struct attribute attr_pasid; + /* shared virtual memory registered by this process */ + struct svm_range_list svms; bool svm_disabled; }; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index 89e7c125d334..4d7a67141190 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -35,6 +35,7 @@ #include #include "amdgpu_amdkfd.h" #include "amdgpu.h" +#include "kfd_svm.h" struct mm_struct; @@ -42,6 +43,7 @@ struct mm_struct; #include "kfd_device_queue_manager.h" #include "kfd_dbgmgr.h" #include "kfd_iommu.h" +#include "kfd_svm.h" /* * List of struct kfd_process (field kfd_process). @@ -1003,6 +1005,7 @@ static void kfd_process_wq_release(struct work_struct *work) kfd_iommu_unbind_process(p); kfd_process_free_outstanding_kfd_bos(p); + svm_range_list_fini(p); kfd_process_destroy_pdds(p); dma_fence_put(p->ef); @@ -1228,6 +1231,10 @@ static struct kfd_process *create_process(const struct task_struct *thread) if (err != 0) goto err_init_apertures; + err = svm_range_list_init(process); + if (err) + goto err_init_svm_range_list; + /* alloc_notifier needs to find the process in the hash table */ hash_add_rcu(kfd_processes_table, &process->kfd_processes, (uintptr_t)process->mm); @@ -1250,6 +1257,8 @@ static struct kfd_process *create_process(const struct task_struct *thread) err_register_notifier: hash_del_rcu(&process->kfd_processes); + svm_range_list_fini(process); +err_init_svm_range_list: kfd_process_free_outstanding_kfd_bos(process); kfd_process_destroy_pdds(process); err_init_apertures: diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c new file mode 100644 index 000000000000..22f942bb2b0c --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -0,0 +1,791 @@ +/* + * Copyright 2020 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include +#include "amdgpu_sync.h" +#include "amdgpu_object.h" +#include "amdgpu_vm.h" +#include "amdgpu_mn.h" +#include "kfd_priv.h" +#include "kfd_svm.h" + +/** + * svm_range_unlink - unlink svm_range from lists and interval tree + * @prange: svm range structure to be removed + * + * Remove the svm range from svms interval tree and link list + * + * Context: The caller must hold svms->lock + */ +static void svm_range_unlink(struct svm_range *prange) +{ + pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx]\n", prange->svms, + prange, prange->start, prange->last); + + list_del(&prange->list); + if (prange->it_node.start != 0 && prange->it_node.last != 0) + interval_tree_remove(&prange->it_node, &prange->svms->objects); +} + +/** + * svm_range_add_to_svms - add svm range to svms + * @prange: svm range structure to be added + * + * Add the svm range to svms interval tree and link list + * + * Context: The caller must hold svms->lock + */ +static void svm_range_add_to_svms(struct svm_range *prange) +{ + pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx]\n", prange->svms, + prange, prange->start, prange->last); + + list_add_tail(&prange->list, &prange->svms->list); + prange->it_node.start = prange->start; + prange->it_node.last = prange->last; + interval_tree_insert(&prange->it_node, &prange->svms->objects); +} + +static void svm_range_free(struct svm_range *prange) +{ + pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx]\n", prange->svms, prange, + prange->start, prange->last); + + kvfree(prange->pages_addr); + kfree(prange); +} + +static void +svm_range_set_default_attributes(int32_t *location, int32_t *prefetch_loc, + uint8_t *granularity, uint32_t *flags) +{ + *location = KFD_IOCTL_SVM_LOCATION_UNDEFINED; + *prefetch_loc = KFD_IOCTL_SVM_LOCATION_UNDEFINED; + *granularity = 9; + *flags = + KFD_IOCTL_SVM_FLAG_HOST_ACCESS | KFD_IOCTL_SVM_FLAG_COHERENT; +} + +static struct +svm_range *svm_range_new(struct svm_range_list *svms, uint64_t start, + uint64_t last) +{ + uint64_t size = last - start + 1; + struct svm_range *prange; + + prange = kzalloc(sizeof(*prange), GFP_KERNEL); + if (!prange) + return NULL; + prange->npages = size; + prange->svms = svms; + prange->start = start; + prange->last = last; + INIT_LIST_HEAD(&prange->list); + INIT_LIST_HEAD(&prange->update_list); + INIT_LIST_HEAD(&prange->remove_list); + INIT_LIST_HEAD(&prange->insert_list); + svm_range_set_default_attributes(&prange->preferred_loc, + &prange->prefetch_loc, + &prange->granularity, &prange->flags); + + pr_debug("svms 0x%p [0x%llx 0x%llx]\n", svms, start, last); + + return prange; +} + +static int +svm_range_check_attr(struct kfd_process *p, + uint32_t nattr, struct kfd_ioctl_svm_attribute *attrs) +{ + uint32_t i; + int gpuidx; + + for (i = 0; i < nattr; i++) { + switch (attrs[i].type) { + case KFD_IOCTL_SVM_ATTR_PREFERRED_LOC: + if (attrs[i].value != KFD_IOCTL_SVM_LOCATION_SYSMEM && + attrs[i].value != KFD_IOCTL_SVM_LOCATION_UNDEFINED && + kfd_process_gpuidx_from_gpuid(p, + attrs[i].value) < 0) { + pr_debug("no GPU 0x%x found\n", attrs[i].value); + return -EINVAL; + } + break; + case KFD_IOCTL_SVM_ATTR_PREFETCH_LOC: + if (attrs[i].value != KFD_IOCTL_SVM_LOCATION_SYSMEM && + kfd_process_gpuidx_from_gpuid(p, + attrs[i].value) < 0) { + pr_debug("no GPU 0x%x found\n", attrs[i].value); + return -EINVAL; + } + break; + case KFD_IOCTL_SVM_ATTR_ACCESS: + case KFD_IOCTL_SVM_ATTR_ACCESS_IN_PLACE: + case KFD_IOCTL_SVM_ATTR_NO_ACCESS: + gpuidx = kfd_process_gpuidx_from_gpuid(p, + attrs[i].value); + if (gpuidx < 0) { + pr_debug("no GPU 0x%x found\n", attrs[i].value); + return -EINVAL; + } + break; + case KFD_IOCTL_SVM_ATTR_SET_FLAGS: + break; + case KFD_IOCTL_SVM_ATTR_CLR_FLAGS: + break; + case KFD_IOCTL_SVM_ATTR_GRANULARITY: + break; + default: + pr_debug("unknown attr type 0x%x\n", attrs[i].type); + return -EINVAL; + } + } + + return 0; +} + +static void +svm_range_apply_attrs(struct kfd_process *p, struct svm_range *prange, + uint32_t nattr, struct kfd_ioctl_svm_attribute *attrs) +{ + uint32_t i; + int gpuidx; + + for (i = 0; i < nattr; i++) { + switch (attrs[i].type) { + case KFD_IOCTL_SVM_ATTR_PREFERRED_LOC: + prange->preferred_loc = attrs[i].value; + break; + case KFD_IOCTL_SVM_ATTR_PREFETCH_LOC: + prange->prefetch_loc = attrs[i].value; + break; + case KFD_IOCTL_SVM_ATTR_ACCESS: + case KFD_IOCTL_SVM_ATTR_ACCESS_IN_PLACE: + case KFD_IOCTL_SVM_ATTR_NO_ACCESS: + gpuidx = kfd_process_gpuidx_from_gpuid(p, + attrs[i].value); + if (attrs[i].type == KFD_IOCTL_SVM_ATTR_NO_ACCESS) { + bitmap_clear(prange->bitmap_access, gpuidx, 1); + bitmap_clear(prange->bitmap_aip, gpuidx, 1); + } else if (attrs[i].type == KFD_IOCTL_SVM_ATTR_ACCESS) { + bitmap_set(prange->bitmap_access, gpuidx, 1); + bitmap_clear(prange->bitmap_aip, gpuidx, 1); + } else { + bitmap_clear(prange->bitmap_access, gpuidx, 1); + bitmap_set(prange->bitmap_aip, gpuidx, 1); + } + break; + case KFD_IOCTL_SVM_ATTR_SET_FLAGS: + prange->flags |= attrs[i].value; + break; + case KFD_IOCTL_SVM_ATTR_CLR_FLAGS: + prange->flags &= ~attrs[i].value; + break; + case KFD_IOCTL_SVM_ATTR_GRANULARITY: + prange->granularity = attrs[i].value; + break; + default: + WARN_ONCE(1, "svm_range_check_attrs wasn't called?"); + } + } +} + +/** + * svm_range_debug_dump - print all range information from svms + * @svms: svm range list header + * + * debug output svm range start, end, pages_addr, prefetch location from svms + * interval tree and link list + * + * Context: The caller must hold svms->lock + */ +static void svm_range_debug_dump(struct svm_range_list *svms) +{ + struct interval_tree_node *node; + struct svm_range *prange; + + pr_debug("dump svms 0x%p list\n", svms); + pr_debug("range\tstart\tpage\tend\t\tpages_addr\tlocation\n"); + + list_for_each_entry(prange, &svms->list, list) { + pr_debug("0x%p 0x%lx\t0x%llx\t0x%llx\t0x%lx\t0x%x\n", + prange, prange->start, prange->npages, + prange->start + prange->npages - 1, + prange->pages_addr ? *prange->pages_addr : 0, + prange->actual_loc); + } + + pr_debug("dump svms 0x%p interval tree\n", svms); + pr_debug("range\tstart\tpage\tend\t\tpages_addr\tlocation\n"); + node = interval_tree_iter_first(&svms->objects, 0, ~0ULL); + while (node) { + prange = container_of(node, struct svm_range, it_node); + pr_debug("0x%p 0x%lx\t0x%llx\t0x%llx\t0x%lx\t0x%x\n", + prange, prange->start, prange->npages, + prange->start + prange->npages - 1, + prange->pages_addr ? *prange->pages_addr : 0, + prange->actual_loc); + node = interval_tree_iter_next(node, 0, ~0ULL); + } +} + +static bool +svm_range_is_same_attrs(struct svm_range *old, struct svm_range *new) +{ + return (old->prefetch_loc == new->prefetch_loc && + old->flags == new->flags && + old->granularity == new->granularity); +} + +static int +svm_range_split_pages(struct svm_range *new, struct svm_range *old, + uint64_t start, uint64_t last) +{ + unsigned long old_start; + unsigned long *pages_addr; + uint64_t d; + + old_start = old->start; + new->pages_addr = kvmalloc_array(new->npages, + sizeof(*new->pages_addr), + GFP_KERNEL | __GFP_ZERO); + if (!new->pages_addr) + return -ENOMEM; + + d = new->start - old_start; + memcpy(new->pages_addr, old->pages_addr + d, + new->npages * sizeof(*new->pages_addr)); + + old->npages = last - start + 1; + old->start = start; + old->last = last; + + pages_addr = kvmalloc_array(old->npages, sizeof(*pages_addr), + GFP_KERNEL); + if (!pages_addr) { + kvfree(new->pages_addr); + return -ENOMEM; + } + + d = start - old_start; + memcpy(pages_addr, old->pages_addr + d, + old->npages * sizeof(*pages_addr)); + + kvfree(old->pages_addr); + old->pages_addr = pages_addr; + + return 0; +} + +/** + * svm_range_split_adjust - split range and adjust + * + * @new: new range + * @old: the old range + * @start: the old range adjust to start address in pages + * @last: the old range adjust to last address in pages + * + * Copy system memory pages, pages_addr or vram mm_nodes in old range to new + * range from new_start up to size new->npages, the remaining old range is from + * start to last + * + * Return: + * 0 - OK, -ENOMEM - out of memory + */ +static int +svm_range_split_adjust(struct svm_range *new, struct svm_range *old, + uint64_t start, uint64_t last) +{ + int r = -EINVAL; + + pr_debug("svms 0x%p new 0x%lx old [0x%lx 0x%lx] => [0x%llx 0x%llx]\n", + new->svms, new->start, old->start, old->last, start, last); + + if (new->start < old->start || + new->last > old->last) { + WARN_ONCE(1, "invalid new range start or last\n"); + return -EINVAL; + } + + if (old->pages_addr) + r = svm_range_split_pages(new, old, start, last); + else + WARN_ONCE(1, "split adjust invalid pages_addr and nodes\n"); + if (r) + return r; + + old->npages = last - start + 1; + old->start = start; + old->last = last; + new->flags = old->flags; + new->preferred_loc = old->preferred_loc; + new->prefetch_loc = old->prefetch_loc; + new->actual_loc = old->actual_loc; + new->granularity = old->granularity; + bitmap_copy(new->bitmap_access, old->bitmap_access, MAX_GPU_INSTANCE); + bitmap_copy(new->bitmap_aip, old->bitmap_aip, MAX_GPU_INSTANCE); + + return 0; +} + +/** + * svm_range_split - split a range in 2 ranges + * + * @prange: the svm range to split + * @start: the remaining range start address in pages + * @last: the remaining range last address in pages + * @new: the result new range generated + * + * Two cases only: + * case 1: if start == prange->start + * prange ==> prange[start, last] + * new range [last + 1, prange->last] + * + * case 2: if last == prange->last + * prange ==> prange[start, last] + * new range [prange->start, start - 1] + * + * Context: Caller hold svms->rw_sem as write mode + * + * Return: + * 0 - OK, -ENOMEM - out of memory, -EINVAL - invalid start, last + */ +static int +svm_range_split(struct svm_range *prange, uint64_t start, uint64_t last, + struct svm_range **new) +{ + uint64_t old_start = prange->start; + uint64_t old_last = prange->last; + struct svm_range_list *svms; + int r = 0; + + pr_debug("svms 0x%p [0x%llx 0x%llx] to [0x%llx 0x%llx]\n", prange->svms, + old_start, old_last, start, last); + + if (old_start != start && old_last != last) + return -EINVAL; + if (start < old_start || last > old_last) + return -EINVAL; + + svms = prange->svms; + if (old_start == start) + *new = svm_range_new(svms, last + 1, old_last); + else + *new = svm_range_new(svms, old_start, start - 1); + if (!*new) + return -ENOMEM; + + r = svm_range_split_adjust(*new, prange, start, last); + if (r) { + pr_debug("failed %d split [0x%llx 0x%llx] to [0x%llx 0x%llx]\n", + r, old_start, old_last, start, last); + svm_range_free(*new); + *new = NULL; + } + + return r; +} + +static int +svm_range_split_tail(struct svm_range *prange, struct svm_range *new, + uint64_t new_last, struct list_head *insert_list) +{ + struct svm_range *tail; + int r = svm_range_split(prange, prange->start, new_last, &tail); + + if (!r) + list_add(&tail->insert_list, insert_list); + return r; +} + +static int +svm_range_split_head(struct svm_range *prange, struct svm_range *new, + uint64_t new_start, struct list_head *insert_list) +{ + struct svm_range *head; + int r = svm_range_split(prange, new_start, prange->last, &head); + + if (!r) + list_add(&head->insert_list, insert_list); + return r; +} + +struct svm_range *svm_range_clone(struct svm_range *old) +{ + struct svm_range *new; + + new = svm_range_new(old->svms, old->start, old->last); + if (!new) + return NULL; + + if (old->pages_addr) { + new->pages_addr = kvmalloc_array(new->npages, + sizeof(*new->pages_addr), + GFP_KERNEL); + if (!new->pages_addr) { + kfree(new); + return NULL; + } + memcpy(new->pages_addr, old->pages_addr, + old->npages * sizeof(*old->pages_addr)); + } + + new->flags = old->flags; + new->preferred_loc = old->preferred_loc; + new->prefetch_loc = old->prefetch_loc; + new->actual_loc = old->actual_loc; + new->granularity = old->granularity; + bitmap_copy(new->bitmap_access, old->bitmap_access, MAX_GPU_INSTANCE); + bitmap_copy(new->bitmap_aip, old->bitmap_aip, MAX_GPU_INSTANCE); + + return new; +} + +/** + * svm_range_handle_overlap - split overlap ranges + * @svms: svm range list header + * @new: range added with this attributes + * @start: range added start address, in pages + * @last: range last address, in pages + * @update_list: output, the ranges attributes are updated. For set_attr, this + * will do validation and map to GPUs. For unmap, this will be + * removed and unmap from GPUs + * @insert_list: output, the ranges will be inserted into svms, attributes are + * not changes. For set_attr, this will add into svms. + * @remove_list:output, the ranges will be removed from svms + * @left: the remaining range after overlap, For set_attr, this will be added + * as new range. + * + * Total have 5 overlap cases. + * + * This function handles overlap of an address interval with existing + * struct svm_ranges for applying new attributes. This may require + * splitting existing struct svm_ranges. All changes should be applied to + * the range_list and interval tree transactionally. If any split operation + * fails, the entire update fails. Therefore the existing overlapping + * svm_ranges are cloned and the original svm_ranges left unchanged. If the + * transaction succeeds, the modified clones are added and the originals + * freed. Otherwise the clones are removed and the old svm_ranges remain. + * + * Context: The caller must hold svms->lock + */ +static int +svm_range_handle_overlap(struct svm_range_list *svms, struct svm_range *new, + unsigned long start, unsigned long last, + struct list_head *update_list, + struct list_head *insert_list, + struct list_head *remove_list, + unsigned long *left) +{ + struct interval_tree_node *node; + struct svm_range *prange; + struct svm_range *tmp; + int r = 0; + + INIT_LIST_HEAD(update_list); + INIT_LIST_HEAD(insert_list); + INIT_LIST_HEAD(remove_list); + + node = interval_tree_iter_first(&svms->objects, start, last); + while (node) { + struct interval_tree_node *next; + struct svm_range *old; + unsigned long next_start; + + pr_debug("found overlap node [0x%lx 0x%lx]\n", node->start, + node->last); + + old = container_of(node, struct svm_range, it_node); + next = interval_tree_iter_next(node, start, last); + next_start = min(node->last, last) + 1; + + if (node->start < start || node->last > last) { + /* node intersects the updated range, clone+split it */ + prange = svm_range_clone(old); + if (!prange) { + r = -ENOMEM; + goto out; + } + + list_add(&old->remove_list, remove_list); + list_add(&prange->insert_list, insert_list); + + if (node->start < start) { + pr_debug("change old range start\n"); + r = svm_range_split_head(prange, new, start, + insert_list); + if (r) + goto out; + } + if (node->last > last) { + pr_debug("change old range last\n"); + r = svm_range_split_tail(prange, new, last, + insert_list); + if (r) + goto out; + } + } else { + /* The node is contained within start..last, + * just update it + */ + prange = old; + } + + if (!svm_range_is_same_attrs(prange, new)) + list_add(&prange->update_list, update_list); + + /* insert a new node if needed */ + if (node->start > start) { + prange = svm_range_new(prange->svms, start, + node->start - 1); + if (!prange) { + r = -ENOMEM; + goto out; + } + + list_add(&prange->insert_list, insert_list); + list_add(&prange->update_list, update_list); + } + + node = next; + start = next_start; + } + + if (left && start <= last) + *left = last - start + 1; + +out: + if (r) + list_for_each_entry_safe(prange, tmp, insert_list, insert_list) + svm_range_free(prange); + + return r; +} + +void svm_range_list_fini(struct kfd_process *p) +{ + pr_debug("pasid 0x%x svms 0x%p\n", p->pasid, &p->svms); +} + +int svm_range_list_init(struct kfd_process *p) +{ + struct svm_range_list *svms = &p->svms; + + svms->objects = RB_ROOT_CACHED; + mutex_init(&svms->lock); + INIT_LIST_HEAD(&svms->list); + + return 0; +} + +/** + * svm_range_is_valid - check if virtual address range is valid + * @mm: current process mm_struct + * @start: range start address, in pages + * @size: range size, in pages + * + * Valid virtual address range means it belongs to one or more VMAs + * + * Context: Process context + * + * Return: + * true - valid svm range + * false - invalid svm range + */ +static bool +svm_range_is_valid(struct mm_struct *mm, uint64_t start, uint64_t size) +{ + const unsigned long device_vma = VM_IO | VM_PFNMAP | VM_MIXEDMAP; + struct vm_area_struct *vma; + unsigned long end; + + start <<= PAGE_SHIFT; + end = start + (size << PAGE_SHIFT); + + do { + vma = find_vma(mm, start); + if (!vma || start < vma->vm_start || + (vma->vm_flags & device_vma)) + return false; + start = min(end, vma->vm_end); + } while (start < end); + + return true; +} + +/** + * svm_range_add - add svm range and handle overlap + * @p: the range add to this process svms + * @start: page size aligned + * @size: page size aligned + * @nattr: number of attributes + * @attrs: array of attributes + * @update_list: output, the ranges need validate and update GPU mapping + * @insert_list: output, the ranges need insert to svms + * @remove_list: output, the ranges are replaced and need remove from svms + * + * Check if the virtual address range has overlap with the registered ranges, + * split the overlapped range, copy and adjust pages address and vram nodes in + * old and new ranges. + * + * Context: Process context, caller must hold svms->lock + * + * Return: + * 0 - OK, otherwise error code + */ +static int +svm_range_add(struct kfd_process *p, uint64_t start, uint64_t size, + uint32_t nattr, struct kfd_ioctl_svm_attribute *attrs, + struct list_head *update_list, struct list_head *insert_list, + struct list_head *remove_list) +{ + uint64_t last = start + size - 1UL; + struct svm_range_list *svms; + struct svm_range new = {0}; + struct svm_range *prange; + unsigned long left = 0; + int r = 0; + + pr_debug("svms 0x%p [0x%llx 0x%llx]\n", &p->svms, start, last); + + svm_range_apply_attrs(p, &new, nattr, attrs); + + svms = &p->svms; + + r = svm_range_handle_overlap(svms, &new, start, last, update_list, + insert_list, remove_list, &left); + if (r) + return r; + + if (left) { + prange = svm_range_new(svms, last - left + 1, last); + list_add(&prange->insert_list, insert_list); + list_add(&prange->update_list, update_list); + } + + return 0; +} + +static int +svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, + uint32_t nattr, struct kfd_ioctl_svm_attribute *attrs) +{ + struct amdkfd_process_info *process_info = p->kgd_process_info; + struct mm_struct *mm = current->mm; + struct list_head update_list; + struct list_head insert_list; + struct list_head remove_list; + struct svm_range_list *svms; + struct svm_range *prange; + struct svm_range *next; + int r = 0; + + pr_debug("pasid 0x%x svms 0x%p [0x%llx 0x%llx] pages 0x%llx\n", + p->pasid, &p->svms, start, start + size - 1, size); + + r = svm_range_check_attr(p, nattr, attrs); + if (r) + return r; + + svms = &p->svms; + + mutex_lock(&process_info->lock); + + mmap_write_lock(mm); + + if (!svm_range_is_valid(mm, start, size)) { + pr_debug("invalid range\n"); + r = -EFAULT; + mmap_write_unlock(mm); + goto out; + } + + mutex_lock(&svms->lock); + + /* Add new range and split existing ranges as needed */ + r = svm_range_add(p, start, size, nattr, attrs, &update_list, + &insert_list, &remove_list); + if (r) { + mutex_unlock(&svms->lock); + mmap_write_unlock(mm); + goto out; + } + /* Apply changes as a transaction */ + list_for_each_entry_safe(prange, next, &insert_list, insert_list) { + svm_range_add_to_svms(prange); + } + list_for_each_entry(prange, &update_list, update_list) { + svm_range_apply_attrs(p, prange, nattr, attrs); + /* TODO: unmap ranges from GPU that lost access */ + } + list_for_each_entry_safe(prange, next, &remove_list, + remove_list) { + pr_debug("unlink old 0x%p prange 0x%p [0x%lx 0x%lx]\n", + prange->svms, prange, prange->start, + prange->last); + svm_range_unlink(prange); + svm_range_free(prange); + } + + mmap_write_downgrade(mm); + /* Trigger migrations and revalidate and map to GPUs as needed. If + * this fails we may be left with partially completed actions. There + * is no clean way of rolling back to the previous state in such a + * case because the rollback wouldn't be guaranteed to work either. + */ + list_for_each_entry(prange, &update_list, update_list) { + /* TODO */ + } + + svm_range_debug_dump(svms); + + mutex_unlock(&svms->lock); + mmap_read_unlock(mm); +out: + mutex_unlock(&process_info->lock); + + pr_debug("pasid 0x%x svms 0x%p [0x%llx 0x%llx] done, r=%d\n", p->pasid, + &p->svms, start, start + size - 1, r); + + return r; +} + +int +svm_ioctl(struct kfd_process *p, enum kfd_ioctl_svm_op op, uint64_t start, + uint64_t size, uint32_t nattrs, struct kfd_ioctl_svm_attribute *attrs) +{ + int r; + + start >>= PAGE_SHIFT; + size >>= PAGE_SHIFT; + + switch (op) { + case KFD_IOCTL_SVM_OP_SET_ATTR: + r = svm_range_set_attr(p, start, size, nattrs, attrs); + break; + default: + r = EINVAL; + break; + } + + return r; +} diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h new file mode 100644 index 000000000000..c46cdad9d1a3 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -0,0 +1,87 @@ +/* + * Copyright 2020 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ + +#ifndef KFD_SVM_H_ +#define KFD_SVM_H_ + +#include +#include +#include +#include +#include +#include "amdgpu.h" +#include "kfd_priv.h" + +/** + * struct svm_range - shared virtual memory range + * + * @svms: list of svm ranges, structure defined in kfd_process + * @start: range start address in pages + * @last: range last address in pages + * @it_node: node [start, last] stored in interval tree, start, last are page + * aligned, page size is (last - start + 1) + * @list: link list node, used to scan all ranges of svms + * @update_list:link list node used to add to update_list + * @remove_list:link list node used to add to remove list + * @insert_list:link list node used to add to insert list + * @npages: number of pages + * @pages_addr: list of system memory physical page address + * @flags: flags defined as KFD_IOCTL_SVM_FLAG_* + * @perferred_loc: perferred location, 0 for CPU, or GPU id + * @perfetch_loc: last prefetch location, 0 for CPU, or GPU id + * @actual_loc: the actual location, 0 for CPU, or GPU id + * @granularity:migration granularity, log2 num pages + * @bitmap_access: index bitmap of GPUs which can access the range + * @bitmap_aip: index bitmap of GPUs which can access the range in place + * + * Data structure for virtual memory range shared by CPU and GPUs, it can be + * allocated from system memory ram or device vram, and migrate from ram to vram + * or from vram to ram. + */ +struct svm_range { + struct svm_range_list *svms; + unsigned long start; + unsigned long last; + struct interval_tree_node it_node; + struct list_head list; + struct list_head update_list; + struct list_head remove_list; + struct list_head insert_list; + uint64_t npages; + unsigned long *pages_addr; + uint32_t flags; + uint32_t preferred_loc; + uint32_t prefetch_loc; + uint32_t actual_loc; + uint8_t granularity; + DECLARE_BITMAP(bitmap_access, MAX_GPU_INSTANCE); + DECLARE_BITMAP(bitmap_aip, MAX_GPU_INSTANCE); +}; + +int svm_range_list_init(struct kfd_process *p); +void svm_range_list_fini(struct kfd_process *p); +int svm_ioctl(struct kfd_process *p, enum kfd_ioctl_svm_op op, uint64_t start, + uint64_t size, uint32_t nattrs, + struct kfd_ioctl_svm_attribute *attrs); + +#endif /* KFD_SVM_H_ */ From patchwork Mon Mar 22 10:58:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9B763C433C1 for ; Mon, 22 Mar 2021 11:08:14 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5DD4C61879 for ; Mon, 22 Mar 2021 11:08:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5DD4C61879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B32FF6E0EB; Mon, 22 Mar 2021 11:07:51 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2041.outbound.protection.outlook.com [40.107.237.41]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6175A89913; Mon, 22 Mar 2021 11:07:34 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hO4ZkuK8zW1a7p1pomtsdTwrMbxip30E1/tFAfBumzcr7iVUjWtroKU+N8GmRtAJwFUEAV6LX5WXy+R2I/2w+mpQMF41o5IHnwxn1mqyAxBN9TaTy0hN+8alIHfwclUROrWuCA5rXhjSRviTZkpLe4B62YKeg8kc44jxqp/xmTW/RO3/339J7mYv26zo0EhA2wlmWRWFnpUZv/tPuFf9YYfSOyhgC4/oJGGinqI03ehyZNAj0FGMXVWjzmXNIeSWKNfnaIvDJw65wd1FvtGCqw+12hRcB51SCQ4/11t0u1A+0hdZqpuPFpB6iG49kJD/nGl0fvqJZ96GhAqxfdN1/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8cqQogQFUFI9BGZRrMAVqWmQffzi1fIhustsKCYW2mk=; b=RIhz08kxo6BVW5TIHC9RKGNtta62c6xbMIhlNaJqye2+gUv6Qw6GlhjklDuPrfPfojD9Cdl2djG+eUuUiOnf07ONYWBwrQRuQKg11peMJEMPVV9zKldOaGlfiKGWJYbqU09v6ohegKMY9pR0hLg+G7PPWTN9KCscrzsFNYEEaExHP0dKBGPW8rlRxFx2T0Zz2fWThIL6KjST4mq0vW0BdlG5EImtc2q/iS5vERW1zMM6gfyVdFEz6kJ5LxM0RJ6iebZDkf3kOJnD8ljjts4ZSjKuuMzurnknnZ2Yk5VDJVgQWrmmyFDelwih0OUxeC/PshwMC1NRiiET2jcWvIvaDw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8cqQogQFUFI9BGZRrMAVqWmQffzi1fIhustsKCYW2mk=; b=ByN2GCJH9teq+yeNDtRoDzmXSyX3PjJmTCfGUZjeVGFvvMwui5myRTzAqlkvNjbohTkRWmGS6yEcrH0e+gTnIatsAv9LOEvh1+JZbxRAkJZZG0o4uRws3nyRcj2d429qd4BGMXjNvwUJLuhPh959KiJtNRl7j13wVey7/rHz0z8= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:32 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:32 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 05/44] drm/amdkfd: add svm ioctl GET_ATTR op Date: Mon, 22 Mar 2021 06:58:21 -0400 Message-Id: <20210322105900.14068-6-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:31 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 628f10ee-257d-42c2-a238-08d8ed22b09a X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:626; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: KfWlrj0zMPIwwq9OVI6Jj9TGlsyaXZ/8XYdqmnNxdcLtwgipW4696IK8txXGj5A0mpVNf0hi/b+hk3a5llKAQdq3vpk6oIWMG/0Z2pvpIp1SBW01mWqtUCs8oy9jhzUPxorFKRbaXp81cw7QSDs0JRtdJf23eFWItTnnY+6F7M53iXHqYJtcXqUfnAcIM6/MRjYa2lDim4dmUauhaa8xZ8Py2K2czUWv2bxzxdOC7xxmced3sbT+B6FIIGBl1NjN7j2mtoJuJIgR1MqhwNzbKcIMcxrPbgNbhhVziB2aJf/AnoxU4GhxaBl+z/njGOZqRt3LFep4/o88vRRsSLZK7VGAYCOowYtOm/z9xeNBxeXI0WFhlnOS8b177+Poypcga1cNQI2U/Py4JQUYdhPKr3xUVSy5ZAnKzW5X2edJma3nJjSvuFhY0Du3XXYqIU2l1Ro/GEhFC4KNbVjJeaipmR0JD/XtahrXLEaDiMc9oNzCp8fW9f5XmvW4EFNNrrZ9TMYPsxbiSrFjvzmeWHqz9d9wJLevzTfJ7e8VEGIkofH10+tC9S5ZnAmynUktIyutM9pjbpDT8pLky8Ee39CY/Q1iDZ1d+Tkr23IVx052xm1h8aXxG6mSL7DCXsmIAXnYUiCKpi+uDmxVtn3JClJgFPOtuSlYoqXa7xTIeuDpay8= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(54906003)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: gohQfFUlW0N58mV0idS2bsmOcIb0FF63xZaDH9xMug2AMxeucmMzyWQ25b4Evov4Oy0jixJ7WprFL2sePshQ18mkTOoJZrFf/yupt+APVIKCcI50vEHNtb3/S2lmla2pw8zj1d8I56tsEkqigx9ZwIH4Ve77brC+Rcc3RA+6OmARE32H910crXcfPtKgDhGCYrrILZA10RbJct8aCK0Cg/6QFv5pXwRM1P8Lc8jDrWZxMdV2/wkNGeasPuSo3AQ4zFZz+RUfABtCID0Qr1+vMUgQ52fOXgsRBWnCEmrDkp6FD7/NV/VfzLwfcXK6VHPgJOZapTgJFg82QMJrmjbkKahDalnskwJ109ei9LFpT2sh5i2y26vI3ozds5zR0nzp+eal801MyJljSraNkg2M3xCv8JfTArKRsjvE6hxDdKFXLQLHZcBKNwdPrBjv60g2JkPXHnn9YtujWqZMZZCf9Zv5KT0CZhpvSYS79fxm03saObWPgy5bFJdPvUcjJSQhtszmcZJoI4Y8w4GDL6kMnXoAH30s2sV1stbIS1rCKm8jtk8jP+5/aSnFSQB+UkBWkzieX67sdFxZZ7iCQLhUgPqYu/DaQC7EDswCtBSMruUadadU0fkshlY6BtDOMjkrJOVQtcQp4D8GqbPmiACtCG/cPiqFwjhPsFCrwByt9SZAhAirhj8xVtQLl/zeCucb8GKcU6yy/2HTttd39Ws//LKgjh2K39XdDmVFHn1KFXrFGAUE9wBCrtwYVgtf5ocd/8V737eNWW8SNJ8UDJKMEOb/nggabir8T7KVKL6Tw2dfZGt7mUMD32DsKLyHySjDWMz6/6xoZBfY6Ez2TIO9MREDyy0sEtOFGDwVrxbd3PhktJQPUeekrokj9p9NuTn83HvPouxnsWDD/XRnsVECFsGgtrtXY8reUYJgdZzWLBYMRS8/dmA9NvLRDZco8mdIzvXTj5TEOo73fTSP71CiGJNiV7wnpOzQs0cBfnHE9sXs76/4MZmRaFS4RrxgL6xv7YnJuVMxJA576YHI0rQdwWyOsq4E4a1JENOhnj88t7fn2QzY3hMc8O9kHIYyEPs4Jg/VYeIuFrOuxvj5ricFcqLgpz8wEqhXBo7HkuKMpIb+Q/c6uPhoRTyDnSCc4Dre5YYSeDp7pYSjvIxAo51LLGI2RCKC//bQxRz4AkwqX/n9jwGyl08WwagWBbNWBZMOUBzEtkE0tNHp3Ni9jD5dcgG4bHHe6XA+pyfgS5adoVK+YTP4I9hH0zE5vNHua2JrErSvVwGP9iwgsDVF+2KQNu58BM4lML+TdzD7accOcumIIFaNB4aoyIhsqHwjkzZB X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 628f10ee-257d-42c2-a238-08d8ed22b09a X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:32.2760 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 2lrauoga0+oQQFnWg0V8zpAp+mCyGi4MTDMY+dUUN6+7T7VDOtUdI9ADLxaJZZfLVyYbxRorj+PpNt9nP8Qzxg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang Get the intersection of attributes over all memory in the given range Signed-off-by: Philip Yang Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 164 +++++++++++++++++++++++++++ 1 file changed, 164 insertions(+) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 22f942bb2b0c..e57103a9025e 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -769,6 +769,167 @@ svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, return r; } +static int +svm_range_get_attr(struct kfd_process *p, uint64_t start, uint64_t size, + uint32_t nattr, struct kfd_ioctl_svm_attribute *attrs) +{ + DECLARE_BITMAP(bitmap_access, MAX_GPU_INSTANCE); + DECLARE_BITMAP(bitmap_aip, MAX_GPU_INSTANCE); + bool get_preferred_loc = false; + bool get_prefetch_loc = false; + bool get_granularity = false; + bool get_accessible = false; + bool get_flags = false; + uint64_t last = start + size - 1UL; + struct mm_struct *mm = current->mm; + uint8_t granularity = 0xff; + struct interval_tree_node *node; + struct svm_range_list *svms; + struct svm_range *prange; + uint32_t prefetch_loc = KFD_IOCTL_SVM_LOCATION_UNDEFINED; + uint32_t location = KFD_IOCTL_SVM_LOCATION_UNDEFINED; + uint32_t flags = 0xffffffff; + int gpuidx; + uint32_t i; + + pr_debug("svms 0x%p [0x%llx 0x%llx] nattr 0x%x\n", &p->svms, start, + start + size - 1, nattr); + + mmap_read_lock(mm); + if (!svm_range_is_valid(mm, start, size)) { + pr_debug("invalid range\n"); + mmap_read_unlock(mm); + return -EINVAL; + } + mmap_read_unlock(mm); + + for (i = 0; i < nattr; i++) { + switch (attrs[i].type) { + case KFD_IOCTL_SVM_ATTR_PREFERRED_LOC: + get_preferred_loc = true; + break; + case KFD_IOCTL_SVM_ATTR_PREFETCH_LOC: + get_prefetch_loc = true; + break; + case KFD_IOCTL_SVM_ATTR_ACCESS: + get_accessible = true; + break; + case KFD_IOCTL_SVM_ATTR_SET_FLAGS: + get_flags = true; + break; + case KFD_IOCTL_SVM_ATTR_GRANULARITY: + get_granularity = true; + break; + case KFD_IOCTL_SVM_ATTR_CLR_FLAGS: + case KFD_IOCTL_SVM_ATTR_ACCESS_IN_PLACE: + case KFD_IOCTL_SVM_ATTR_NO_ACCESS: + fallthrough; + default: + pr_debug("get invalid attr type 0x%x\n", attrs[i].type); + return -EINVAL; + } + } + + svms = &p->svms; + + mutex_lock(&svms->lock); + + node = interval_tree_iter_first(&svms->objects, start, last); + if (!node) { + pr_debug("range attrs not found return default values\n"); + svm_range_set_default_attributes(&location, &prefetch_loc, + &granularity, &flags); + /* TODO: Automatically create SVM ranges and map them on + * GPU page faults + if (p->xnack_enabled) + bitmap_fill(bitmap_access, MAX_GPU_INSTANCE); + */ + + goto fill_values; + } + bitmap_fill(bitmap_access, MAX_GPU_INSTANCE); + bitmap_fill(bitmap_aip, MAX_GPU_INSTANCE); + + while (node) { + struct interval_tree_node *next; + + prange = container_of(node, struct svm_range, it_node); + next = interval_tree_iter_next(node, start, last); + + if (get_preferred_loc) { + if (prange->preferred_loc == + KFD_IOCTL_SVM_LOCATION_UNDEFINED || + (location != KFD_IOCTL_SVM_LOCATION_UNDEFINED && + location != prange->preferred_loc)) { + location = KFD_IOCTL_SVM_LOCATION_UNDEFINED; + get_preferred_loc = false; + } else { + location = prange->preferred_loc; + } + } + if (get_prefetch_loc) { + if (prange->prefetch_loc == + KFD_IOCTL_SVM_LOCATION_UNDEFINED || + (prefetch_loc != KFD_IOCTL_SVM_LOCATION_UNDEFINED && + prefetch_loc != prange->prefetch_loc)) { + prefetch_loc = KFD_IOCTL_SVM_LOCATION_UNDEFINED; + get_prefetch_loc = false; + } else { + prefetch_loc = prange->prefetch_loc; + } + } + if (get_accessible) { + bitmap_and(bitmap_access, bitmap_access, + prange->bitmap_access, MAX_GPU_INSTANCE); + bitmap_and(bitmap_aip, bitmap_aip, + prange->bitmap_aip, MAX_GPU_INSTANCE); + } + if (get_flags) + flags &= prange->flags; + + if (get_granularity && prange->granularity < granularity) + granularity = prange->granularity; + + node = next; + } +fill_values: + mutex_unlock(&svms->lock); + + for (i = 0; i < nattr; i++) { + switch (attrs[i].type) { + case KFD_IOCTL_SVM_ATTR_PREFERRED_LOC: + attrs[i].value = location; + break; + case KFD_IOCTL_SVM_ATTR_PREFETCH_LOC: + attrs[i].value = prefetch_loc; + break; + case KFD_IOCTL_SVM_ATTR_ACCESS: + gpuidx = kfd_process_gpuidx_from_gpuid(p, + attrs[i].value); + if (gpuidx < 0) { + pr_debug("invalid gpuid %x\n", attrs[i].value); + return -EINVAL; + } + if (test_bit(gpuidx, bitmap_access)) + attrs[i].type = KFD_IOCTL_SVM_ATTR_ACCESS; + else if (test_bit(gpuidx, bitmap_aip)) + attrs[i].type = + KFD_IOCTL_SVM_ATTR_ACCESS_IN_PLACE; + else + attrs[i].type = KFD_IOCTL_SVM_ATTR_NO_ACCESS; + break; + case KFD_IOCTL_SVM_ATTR_SET_FLAGS: + attrs[i].value = flags; + break; + case KFD_IOCTL_SVM_ATTR_GRANULARITY: + attrs[i].value = (uint32_t)granularity; + break; + } + } + + return 0; +} + int svm_ioctl(struct kfd_process *p, enum kfd_ioctl_svm_op op, uint64_t start, uint64_t size, uint32_t nattrs, struct kfd_ioctl_svm_attribute *attrs) @@ -782,6 +943,9 @@ svm_ioctl(struct kfd_process *p, enum kfd_ioctl_svm_op op, uint64_t start, case KFD_IOCTL_SVM_OP_SET_ATTR: r = svm_range_set_attr(p, start, size, nattrs, attrs); break; + case KFD_IOCTL_SVM_OP_GET_ATTR: + r = svm_range_get_attr(p, start, size, nattrs, attrs); + break; default: r = EINVAL; break; From patchwork Mon Mar 22 10:58:22 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B636FC433E0 for ; Mon, 22 Mar 2021 11:07:53 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7EAC961931 for ; Mon, 22 Mar 2021 11:07:53 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7EAC961931 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B5B1689EA9; Mon, 22 Mar 2021 11:07:38 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2041.outbound.protection.outlook.com [40.107.237.41]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2258089BAE; Mon, 22 Mar 2021 11:07:35 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=STJ9YiH+7yisbmCUdJ03oRJzoygwH47upaZDAOq6xLsVNGE2Tj8hFrHyJ3jwKh3lv1yACw5vrJxknQsXyT/9/3iq4iwT7zJnqcEzeaem7DkQwecWGfA4XvaK1INjLibo7AiYM87utI0pyM/S+oBNAjeBdi+mRToyevssMGwa84yngHO+oD86+nIVSuvMSUfmZ3GpijfTEqqrCJgdkx9K+mwOkPITuLW7TByR/HivodFPiEMKuwVof+RPR/YWeIiPiYoXNDSjTHxfoI4e85gNlbpC3WIHmtviTgFf+gzmRtrjajtIet6gLXwX1+Hta+CPHK6BjQGB0npBq0ydQzAtbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=t4ng/7eaV6wm4SzX/uBpO+zk7VUjtJ+htMsJmamqLQo=; b=YfYKuV+J8lu3M2qrcgqarxBHBQ4NfxcFCM344EE+Ob8qv7HAFDW2eIcJ6ErTTRLI3+QvKJzHtPPwYv0/4gRrWeq2DRdHEtOV4/4OUu8PQ3XKLAKlAMHXPYkfEBiln+dOuXf25wesTlCD1z53bp4TQ4HHpWQCkA/shNwUvfFaLoqR4jNQQmgutFkHM6RXZTBubsqSJy8vVm1KD4Pj3A0Vu5gdj0smBAXXohsQRZVFfimzaNjopkgOO2Qjhz/h3QpfzSxd444m4gFAJHUMRt7hW6k4rfmu3bH+13q78dKcmcnEEVg/16kh5aKt8lDBItuHBnzlx1ZW/NXhiDTApsFEZw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=t4ng/7eaV6wm4SzX/uBpO+zk7VUjtJ+htMsJmamqLQo=; b=faJtgcZKAVOPaRWin3W8UleihuO7LGG6nN6lPWj0mvcbheVxjKDXy0YvveOWx7zOHpypl2DwnAYZTG8InQEOAiEQthaXqqp5jUdsnqwi14Jw5V7cM68CmJD/Wt90zwjFN5n5i2WNqaEmk8csyoI+NhobqSes5/NSveNUIM7Z6Ow= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:33 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:33 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 06/44] drm/amdgpu: add common HMM get pages function Date: Mon, 22 Mar 2021 06:58:22 -0400 Message-Id: <20210322105900.14068-7-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:32 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: f128e5f4-b81d-4c0f-f328-08d8ed22b0ee X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: q9dwFnBMuBGmidk4MmX9K8wvc7oO/cUIla7QWIHpQcECzP2MIF9qIWBaGxmGkQBGg1EUeeSs/tlsk75b2pKWz3/CgcSfo3uFZk2zEzPUOzcEP0lagm2iAAN87k8QROj2DBujCrWMOMcU4qKh8hKtJw+OPVUqkR1MeKVzQN6eGb5HZmypMkYxyeS04wZL3TCwokPD8DHNhrziq8WR4tumjPSetv4R6hUz+0lA63l7KDnQRwDDDJj8PLr38P4YKIfuV2VXem123W1xymiCyPnIw4PHRgvqqzBa92gBywURNnPRJcQlUFjUS7RiKQU3fr31MT1liy7sjyoprlzQYFwEU9gtxVlnZbHr0nWxFzwtcidKIcwuY0amv5Zv8VzlUDyx2L/wMGWcD1jxVCX/qeG+/Ha0Z4kMAOgqePCa3e+zOTr0NSa7HfD25By/xO1WznAJ9RohMcUizRuXSy8OotvGN3nMoAGbfVVCcrR4kFwQ7qAxu/+gme6M8tj6lQEygO0NZt62W0tOMaenOVXBTRF+/etAxy7w0XoC0wj8JDetv7XrE8WXzUaQXImjh/ZPTaEC2TK4eHeTd0ERW6mn5Etsv/BQ/wHO7QkqdESKhUfolLwqv4LV8PbPYZ91l+sQeBDUIyvgkVbtZ9kNRyU7twzFHp+fao6v0ColHyhhLmQUCi4= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: ASAqsS51eVDEQKY0B9thZCpN6reNQdExliCDF0CYyLfbiC6WgYHSRiuNxLVu0XPh26Fi6ACFmjVYSeuCOkMFTX6gFbl0GtulB5rYsoFgAfhnB0kE1Exqs9DepmI0vAfXSBR94K4KfIFBfYzlbOb4cJLZx1qdrXVYQaWJQEaZpGj9k7QYcB/5/IL4usWCW0qqJ9xTrcoA6P3ERJ9PLVd9Mj14wyg5pfNUzayn0FlSuJ1IMi4ZlPPFghtIDv18dOxdYsXcDy8Ums1pVPY8PMGM1t3Os1UfYy2HlHXqppt3CAoHcqvozRSJZMoMmnOPV9EXCsmNztwgAVq7VP8xT1YZzNNqaV50tSeVrSeVbVw596MJk7uVT+OEVVbSFV5tbmh3GKbf8mfaOOvs/AKu20XdUcAQsw0WErDKPYxAMlLgC8ty0HuhDGCgiTcAnfvSsBgFva84Zxw/d2DbzQfM++8F/jsGuOmKiTqq3ammwfUmLPlEVfAubunJbD6HoSe1PRB05eUWgdCVpUQUFxKCxlLiaozLKGD7EwrgL4gNM4T4CXLDV5/5JF6CN8vLCexZauQ49BQSKrOuvC299L0La7QbNouCtdLZrj8QR7L5XAK/5zUOgasRZEn3x7+WY/M2pOmjp5wiRvyO1PXOjm1YCy3UCTothZU6rBIH2greE/PJ0LYNGYBry8pIDJze9komW/m5pl/grDDaXGjXL2RosLrJ1br/JjZrFTB5chiZcbOS5xl5Mx5njCfSIUOyPAyDktzMZw93CEptWyl4mTcYeEjZkTBAjUkKNUVkRBdeB7Dh9OOmoSV3rFYGwEGaqUoFxOmh0Ov7VG3nInhk/6Sor3dU7C1SHz+cBgFoDLZncl0ZupaUxCyW23uRA+jXHMzBc84mZlpU5lIexIxHxOamRQUBnJ+i0QH4pTnI0WwaUledZp5q9N+mALD6jI2xk0rCGtgx55fhirSCos4lm7JGRPaIPwa4iOGSp0eaYP/91lzg3C3CgocwoJdEwUtd+yAMgLcxJpJMfH2gINcXnhKyNaKMK99QqoW6bDM+b2sa4RD12TR4ncHXvKA6YEQ+lLN8qMxbzf0dN96bAfncQXImzq6WEv9DDeKosnJZDlSyUvxsea8xQH3C77eK7FhDrVEs49zq4c62SbrgjBdDJWzh+7od2gPLqvsWKcZvUyEhUsk2zY8wvQC+Ygf74XCcBkK36jGMowdCMpTPn0SIkumqTVcQRQRZZDrIvgeAiIF9USWcH2jQxiBrIE5/rnCDdxW3nxmoL9sJjLtNhaeNRMO71DE+V99tJ1E+/oaSfxxwmk5PvS8FCuzO1wdLNTZ8XprTDmY7 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: f128e5f4-b81d-4c0f-f328-08d8ed22b0ee X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:32.8465 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: U9AR5KDzG+eTh4Kiu7GSs0pmSD2BkXUlDQAzTbrp98wSkKhbt4Tj/JOlvkiTtks2Tit0UZrv2tF0iniv2eoeUQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang Move the HMM get pages function from amdgpu_ttm and to amdgpu_mn. This common function will be used by new svm APIs. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c | 83 +++++++++++++++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h | 7 +++ drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 76 +++------------------- 3 files changed, 100 insertions(+), 66 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c index 828b5167ff12..997da4237a10 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.c @@ -155,3 +155,86 @@ void amdgpu_mn_unregister(struct amdgpu_bo *bo) mmu_interval_notifier_remove(&bo->notifier); bo->notifier.mm = NULL; } + +int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, + struct mm_struct *mm, struct page **pages, + uint64_t start, uint64_t npages, + struct hmm_range **phmm_range, bool readonly, + bool mmap_locked) +{ + struct hmm_range *hmm_range; + unsigned long timeout; + unsigned long i; + unsigned long *pfns; + int r = 0; + + hmm_range = kzalloc(sizeof(*hmm_range), GFP_KERNEL); + if (unlikely(!hmm_range)) + return -ENOMEM; + + pfns = kvmalloc_array(npages, sizeof(*pfns), GFP_KERNEL); + if (unlikely(!pfns)) { + r = -ENOMEM; + goto out_free_range; + } + + hmm_range->notifier = notifier; + hmm_range->default_flags = HMM_PFN_REQ_FAULT; + if (!readonly) + hmm_range->default_flags |= HMM_PFN_REQ_WRITE; + hmm_range->hmm_pfns = pfns; + hmm_range->start = start; + hmm_range->end = start + npages * PAGE_SIZE; + timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); + +retry: + hmm_range->notifier_seq = mmu_interval_read_begin(notifier); + + if (likely(!mmap_locked)) + mmap_read_lock(mm); + + r = hmm_range_fault(hmm_range); + + if (likely(!mmap_locked)) + mmap_read_unlock(mm); + if (unlikely(r)) { + /* + * FIXME: This timeout should encompass the retry from + * mmu_interval_read_retry() as well. + */ + if (r == -EBUSY && !time_after(jiffies, timeout)) + goto retry; + goto out_free_pfns; + } + + /* + * Due to default_flags, all pages are HMM_PFN_VALID or + * hmm_range_fault() fails. FIXME: The pages cannot be touched outside + * the notifier_lock, and mmu_interval_read_retry() must be done first. + */ + for (i = 0; pages && i < npages; i++) + pages[i] = hmm_pfn_to_page(pfns[i]); + + *phmm_range = hmm_range; + + return 0; + +out_free_pfns: + kvfree(pfns); +out_free_range: + kfree(hmm_range); + + return r; +} + +int amdgpu_hmm_range_get_pages_done(struct hmm_range *hmm_range) +{ + int r; + + r = mmu_interval_read_retry(hmm_range->notifier, + hmm_range->notifier_seq); + kvfree(hmm_range->hmm_pfns); + kfree(hmm_range); + + return r; +} diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h index a292238f75eb..7f7d37a457c3 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_mn.h @@ -30,6 +30,13 @@ #include #include +int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, + struct mm_struct *mm, struct page **pages, + uint64_t start, uint64_t npages, + struct hmm_range **phmm_range, bool readonly, + bool mmap_locked); +int amdgpu_hmm_range_get_pages_done(struct hmm_range *hmm_range); + #if defined(CONFIG_HMM_MIRROR) int amdgpu_mn_register(struct amdgpu_bo *bo, unsigned long addr); void amdgpu_mn_unregister(struct amdgpu_bo *bo); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c index 2bafbd78ba4b..aca5a29f6d2a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c @@ -32,7 +32,6 @@ #include #include -#include #include #include #include @@ -692,10 +691,8 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) struct amdgpu_ttm_tt *gtt = (void *)ttm; unsigned long start = gtt->userptr; struct vm_area_struct *vma; - struct hmm_range *range; - unsigned long timeout; struct mm_struct *mm; - unsigned long i; + bool readonly; int r = 0; mm = bo->notifier.mm; @@ -711,76 +708,26 @@ int amdgpu_ttm_tt_get_user_pages(struct amdgpu_bo *bo, struct page **pages) if (!mmget_not_zero(mm)) /* Happens during process shutdown */ return -ESRCH; - range = kzalloc(sizeof(*range), GFP_KERNEL); - if (unlikely(!range)) { - r = -ENOMEM; - goto out; - } - range->notifier = &bo->notifier; - range->start = bo->notifier.interval_tree.start; - range->end = bo->notifier.interval_tree.last + 1; - range->default_flags = HMM_PFN_REQ_FAULT; - if (!amdgpu_ttm_tt_is_readonly(ttm)) - range->default_flags |= HMM_PFN_REQ_WRITE; - - range->hmm_pfns = kvmalloc_array(ttm->num_pages, - sizeof(*range->hmm_pfns), GFP_KERNEL); - if (unlikely(!range->hmm_pfns)) { - r = -ENOMEM; - goto out_free_ranges; - } - mmap_read_lock(mm); vma = find_vma(mm, start); + mmap_read_unlock(mm); if (unlikely(!vma || start < vma->vm_start)) { r = -EFAULT; - goto out_unlock; + goto out_putmm; } if (unlikely((gtt->userflags & AMDGPU_GEM_USERPTR_ANONONLY) && vma->vm_file)) { r = -EPERM; - goto out_unlock; - } - mmap_read_unlock(mm); - timeout = jiffies + msecs_to_jiffies(HMM_RANGE_DEFAULT_TIMEOUT); - -retry: - range->notifier_seq = mmu_interval_read_begin(&bo->notifier); - - mmap_read_lock(mm); - r = hmm_range_fault(range); - mmap_read_unlock(mm); - if (unlikely(r)) { - /* - * FIXME: This timeout should encompass the retry from - * mmu_interval_read_retry() as well. - */ - if (r == -EBUSY && !time_after(jiffies, timeout)) - goto retry; - goto out_free_pfns; + goto out_putmm; } - /* - * Due to default_flags, all pages are HMM_PFN_VALID or - * hmm_range_fault() fails. FIXME: The pages cannot be touched outside - * the notifier_lock, and mmu_interval_read_retry() must be done first. - */ - for (i = 0; i < ttm->num_pages; i++) - pages[i] = hmm_pfn_to_page(range->hmm_pfns[i]); - - gtt->range = range; + readonly = amdgpu_ttm_tt_is_readonly(ttm); + r = amdgpu_hmm_range_get_pages(&bo->notifier, mm, pages, start, + ttm->num_pages, >t->range, readonly, + false); +out_putmm: mmput(mm); - return 0; - -out_unlock: - mmap_read_unlock(mm); -out_free_pfns: - kvfree(range->hmm_pfns); -out_free_ranges: - kfree(range); -out: - mmput(mm); return r; } @@ -809,10 +756,7 @@ bool amdgpu_ttm_tt_get_user_pages_done(struct ttm_tt *ttm) * FIXME: Must always hold notifier_lock for this, and must * not ignore the return code. */ - r = mmu_interval_read_retry(gtt->range->notifier, - gtt->range->notifier_seq); - kvfree(gtt->range->hmm_pfns); - kfree(gtt->range); + r = amdgpu_hmm_range_get_pages_done(gtt->range); gtt->range = NULL; } From patchwork Mon Mar 22 10:58:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154475 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 90090C433E2 for ; Mon, 22 Mar 2021 11:07:48 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 54B4E61972 for ; Mon, 22 Mar 2021 11:07:48 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 54B4E61972 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 942AA89C52; Mon, 22 Mar 2021 11:07:38 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2041.outbound.protection.outlook.com [40.107.237.41]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5050889ACD; Mon, 22 Mar 2021 11:07:35 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nKFOtZHw1tr+3e4yYKvSbZCziz0nrKdOub2g1YFCfDRcnsP/AlcHizi/ffMIsy0i3RhDKTXKCBxrDruJZLcZR2dbYq0kqKPXcr1OEY6VkFm6lQ7bIRSyWfRcLg+bBQy/ix0tgXx3XfyuPul4q1XC+w6l5yzK4ghc3Q4EQmhhBfshqdTcLuU+jDq8uzb8LKnuxVLinHmcuglfqB2ddX4KnMvrlX4giDCNUMkGKzFHKstVuj+d4xy8f4F/xgep5WXE2rB+pf9N7wJBnfD9bbqeTt763w5evmbRHDm6exTnWvtVaAH7HpTtp/icIJesbl0EJ1ahNwqbemACuPz0fNMOJQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AKYW9UxUahbksmhRthUo0cfZQGRrkvNkqdTw7fkHdIg=; b=ElY044hBkkz6Ax08YHfLqpdRkMKK7ak4Z4iX8SnkExYy/6uOHz/u0TK3Hoo1DYywCpbHvTD+nj6V5lIa++ujWztbhCMZ6ehs6BWyuuZHNyTv/CpRz5uub97y1AljI29pS4HPuF+u/+eRE6YjIVfl+CUOpz2q1QZTRGwOnhBUyA2MWDF+U2TTj5G5IOoTHjBrG6OkejmE7B1NytAV5azLYOiZoOHFIKBa8ylzQwdXI3AcC+bZzqPWAbGU8yU22JqqrXTs5r/g9hYOW/Eo9sdZeg+F6jxdZptUmX9pVnE2GnUhsISUpAE8DW6bNor+1Exm8rlZyo9IKwUdEH9HgD9mHA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AKYW9UxUahbksmhRthUo0cfZQGRrkvNkqdTw7fkHdIg=; b=oPbORVEYLbWE/5MTE88aL4Mfp5kjZ60zk6b/cWmPW9IQbpQLfH2l7LbJtPaCraKYbLtQo1EPyfIbXPpJvZcDi1MIcBkMM3ar6/7eRps8YVhfQ9JKaVoLV+TpVVC1vdaGALRZRNTIn5K3MLqIPxzqkOKO7b5ktNT35SMwlz3Dlq0= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:33 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:33 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 07/44] drm/amdkfd: validate svm range system memory Date: Mon, 22 Mar 2021 06:58:23 -0400 Message-Id: <20210322105900.14068-8-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:33 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 5019bc38-b5b3-48cc-7b2f-08d8ed22b144 X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: +X1DZqh2WfHNFAxC0fcLfh56zypq1IJ4XQhC2mdxuKXXWXodlXCLESi37Pn/GH2bSvyEQu6POZIf+T8pyG+1BFmImUPvD5YjyzzHRbywhAHRhoVhDu9JHy1wTdCOsmSR9DUOA65ybM0jO4loQq0n+W9LdZo4GAnfpo1IBRmO/DvAJeGXKjnklQHdXJ2lC0z+5c3X35SoHfS7J0gmYWgFtsf+rQE2LjDGNA1XKP10MEyFOXv3mpgTTbdMNmcEjwgFiNKNOVo6zshTsGdy4+TtrPT1YM1VlUrkZQRv896/0ypWt7m7ujfai3gk/LcVC8MqfMNvAmVCBYe0DHbvFcXkMRimQm82hUVkjVF/Tp798X3uzxCOJqr7KS6EdDgsmE67bbyWCJpVr4VoOEcE3FNCrnYlLN02goueLSmdzb8F80Kw8RdSd1hyhKSG/0UeV6YLI4Dr/UDGidrugPnZwl3I7wTGBCPRFGNu1H/FfOGdLFtRBoVDR2B+Lgi4ihbZ3u7zU1Cpgs/4TXoneLGPyHbrD1LKB76ZXv3nF++dRKMglLX0525pKuWgektX29U02jBYgnJ8SZfRTcNBi56/zld+qiJ9iX4zLXe8tp7UIVWLdX02bCM9nzRO9iPHC+C2wQTxrsxU0N1GP2dD6/jA5q7pmREcEsUU0H1+h9/Yc6RwkTQ= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(15650500001)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: LUmf/b3whSwgswuamESQX+ihSddp2ZVCCilItpmdCqf9sVBVE61t7CEVqIhNOeiM8ZE+muAD0IXKSRAmiKlsOwbnmFwEX9tD2jmROXTGK1PR4p2Ux3oo+avC/MBikcRpqCsWY2Alt1pmggxXxHgAliwzF1xrPqO+r1LLKfFTk1MIYQkTP/pvengbsSr9QEFMBeza8/+OF/ozpNctUQ9fg1V59AHth500QfucsrqnrTK0cNC35QUz9RWwxrbB3/UYJqVDP1GX+3U+B+7iyVkQDMUP1pskbUaZe7rNLXQxE5VmeFA+St7Y+umPntrrWw6xTuvGSLL6gpLxbbvpZxGaFkMdLSYikYBRhF+WVtAXMWGOzRcEgcvn61G8vnickpF+/+0s3BU7ZmtEUPEc0IEphbFaQITmo9epagK4cDs0OcqwYGzCoZYgo8GbEi88+zEUYwTObsw5ZBu7gelVqsdQp9zc+OIjY8uwfvb+l+UamaM1xxiA5lHONS+zwTld9lHx7BgXBcSi68iguQwyDWSnX2h4PCq9CXF9rshOqyD6Dtku4JuZot54mPTEOsKKY/axgPIbvQLhZ8Jnal8boHo2JgyS86RfndmQoBXdKSb+fDnJa70wWJzTDzk80c6bGFO6E5YpzhbyCiMuL4mMtrDofeJ8TAjf5Tiu+Ux4ziaF6FGZTyYqEjhFTcl8FpDX1Iy60kb+/kKEFeW70B4QLEZ1e22n0wcKbbVqy6+Ti3j7nAKuJvpayB+jnD46gg6LTiD4uBzVyOXYG/vgRCy0pYsBPBTX3GFlFsj7bkyLRt3PCKo5YjFp2RK0g4Ua8OEx9bcg1DuuFi+yXCB8HbIdNqilMlbVQoRSNUwjppaMXUcviqMZ4Ql8Lc8KRB3Md8bVPlgUrKoZh8mXUhy5GxR+rRmy+OUzhu/MhQuIol7yUAia0yOmX4AKsoYiNLjEw6Mdq5IxgB5WQyI3uWGjOGS7uF1UqraX4dq0zlYnSEHN4rWmHxH7tqjBQWnUMwyVccrG8e0KmUZcwhfai8wKmN3KlV/KOGW53p9oeC83XhmvW/qEPocij0SMC2qcoG0M6UbK//G/r288b7jxh37YWU7sPrnrFn45LQlp+A8syIE4u2DLA/yIxVrGfO5ePwzzh73yAk44UAJ3cAKUhcdhrj33xJHxpiHL27cP2+Eu818DJ8ryl+6l/+iLciHE6wBlpdty9OHge4627LO+EkVzas7+TBVdB9WGpYhVPmv7Qq/sEOsW1NdpOJtVJMSQ4kVckqn0porDB1xXZQ5NitWIKkrVBYnL+knyNlwT5vr3VkyGhOwzBK21IPsoGVj1yJuPuZeBGPND X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5019bc38-b5b3-48cc-7b2f-08d8ed22b144 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:33.3901 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: OI/ndT2W8PP9UWjGgbcGhIpPbnUYOqgF2aTQ44FE3+LGNR8jNymS8+W7zdHi1DZjs1KNDgvWEUr/FbeT/oOCxQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang Use HMM to get system memory pages address, which will be used to map to GPUs or migrate to vram. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 103 ++++++++++++++++++++++++++- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 4 ++ 2 files changed, 106 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index e57103a9025e..6024caf7373f 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -28,6 +28,15 @@ #include "kfd_priv.h" #include "kfd_svm.h" +static bool +svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, + const struct mmu_notifier_range *range, + unsigned long cur_seq); + +static const struct mmu_interval_notifier_ops svm_range_mn_ops = { + .invalidate = svm_range_cpu_invalidate_pagetables, +}; + /** * svm_range_unlink - unlink svm_range from lists and interval tree * @prange: svm range structure to be removed @@ -46,6 +55,18 @@ static void svm_range_unlink(struct svm_range *prange) interval_tree_remove(&prange->it_node, &prange->svms->objects); } +static void +svm_range_add_notifier_locked(struct mm_struct *mm, struct svm_range *prange) +{ + pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx]\n", prange->svms, + prange, prange->start, prange->last); + + mmu_interval_notifier_insert_locked(&prange->notifier, mm, + prange->start << PAGE_SHIFT, + prange->npages << PAGE_SHIFT, + &svm_range_mn_ops); +} + /** * svm_range_add_to_svms - add svm range to svms * @prange: svm range structure to be added @@ -65,6 +86,18 @@ static void svm_range_add_to_svms(struct svm_range *prange) interval_tree_insert(&prange->it_node, &prange->svms->objects); } +static void svm_range_remove_notifier(struct svm_range *prange) +{ + pr_debug("remove notifier svms 0x%p prange 0x%p [0x%lx 0x%lx]\n", + prange->svms, prange, + prange->notifier.interval_tree.start >> PAGE_SHIFT, + prange->notifier.interval_tree.last >> PAGE_SHIFT); + + if (prange->notifier.interval_tree.start != 0 && + prange->notifier.interval_tree.last != 0) + mmu_interval_notifier_remove(&prange->notifier); +} + static void svm_range_free(struct svm_range *prange) { pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx]\n", prange->svms, prange, @@ -112,6 +145,56 @@ svm_range *svm_range_new(struct svm_range_list *svms, uint64_t start, return prange; } +/** + * svm_range_validate_ram - get system memory pages of svm range + * + * @mm: the mm_struct of process + * @prange: the range struct + * + * After mapping system memory to GPU, system memory maybe invalidated anytime + * during application running, we use HMM callback to sync GPU with CPU page + * table update, so we don't need use lock to prevent CPU invalidation and check + * hmm_range_get_pages_done return value. + * + * Return: + * 0 - OK, otherwise error code + */ +static int +svm_range_validate_ram(struct mm_struct *mm, struct svm_range *prange) +{ + int r; + + r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL, + prange->start << PAGE_SHIFT, + prange->npages, &prange->hmm_range, + false, true); + if (r) { + pr_debug("failed %d to get svm range pages\n", r); + return r; + } + + kvfree(prange->pages_addr); + prange->pages_addr = prange->hmm_range->hmm_pfns; + prange->hmm_range->hmm_pfns = NULL; + + amdgpu_hmm_range_get_pages_done(prange->hmm_range); + prange->hmm_range = NULL; + + return 0; +} + +static int +svm_range_validate(struct mm_struct *mm, struct svm_range *prange) +{ + int r = 0; + + pr_debug("actual loc 0x%x\n", prange->actual_loc); + + r = svm_range_validate_ram(mm, prange); + + return r; +} + static int svm_range_check_attr(struct kfd_process *p, uint32_t nattr, struct kfd_ioctl_svm_attribute *attrs) @@ -581,6 +664,18 @@ svm_range_handle_overlap(struct svm_range_list *svms, struct svm_range *new, return r; } +/** + * svm_range_cpu_invalidate_pagetables - interval notifier callback + * + */ +static bool +svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, + const struct mmu_notifier_range *range, + unsigned long cur_seq) +{ + return true; +} + void svm_range_list_fini(struct kfd_process *p) { pr_debug("pasid 0x%x svms 0x%p\n", p->pasid, &p->svms); @@ -732,6 +827,7 @@ svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, /* Apply changes as a transaction */ list_for_each_entry_safe(prange, next, &insert_list, insert_list) { svm_range_add_to_svms(prange); + svm_range_add_notifier_locked(mm, prange); } list_for_each_entry(prange, &update_list, update_list) { svm_range_apply_attrs(p, prange, nattr, attrs); @@ -743,6 +839,7 @@ svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, prange->svms, prange, prange->start, prange->last); svm_range_unlink(prange); + svm_range_remove_notifier(prange); svm_range_free(prange); } @@ -753,7 +850,11 @@ svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, * case because the rollback wouldn't be guaranteed to work either. */ list_for_each_entry(prange, &update_list, update_list) { - /* TODO */ + r = svm_range_validate(mm, prange); + if (r) { + pr_debug("failed %d to validate svm range\n", r); + break; + } } svm_range_debug_dump(svms); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index c46cdad9d1a3..f35a178c607f 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -44,6 +44,7 @@ * @update_list:link list node used to add to update_list * @remove_list:link list node used to add to remove list * @insert_list:link list node used to add to insert list + * @hmm_range: hmm range structure used by hmm_range_fault to get system pages * @npages: number of pages * @pages_addr: list of system memory physical page address * @flags: flags defined as KFD_IOCTL_SVM_FLAG_* @@ -51,6 +52,7 @@ * @perfetch_loc: last prefetch location, 0 for CPU, or GPU id * @actual_loc: the actual location, 0 for CPU, or GPU id * @granularity:migration granularity, log2 num pages + * @notifier: register mmu interval notifier * @bitmap_access: index bitmap of GPUs which can access the range * @bitmap_aip: index bitmap of GPUs which can access the range in place * @@ -67,6 +69,7 @@ struct svm_range { struct list_head update_list; struct list_head remove_list; struct list_head insert_list; + struct hmm_range *hmm_range; uint64_t npages; unsigned long *pages_addr; uint32_t flags; @@ -74,6 +77,7 @@ struct svm_range { uint32_t prefetch_loc; uint32_t actual_loc; uint8_t granularity; + struct mmu_interval_notifier notifier; DECLARE_BITMAP(bitmap_access, MAX_GPU_INSTANCE); DECLARE_BITMAP(bitmap_aip, MAX_GPU_INSTANCE); }; From patchwork Mon Mar 22 10:58:24 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154485 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C0A85C433C1 for ; Mon, 22 Mar 2021 11:08:06 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 67D2A61931 for ; Mon, 22 Mar 2021 11:08:06 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 67D2A61931 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9E2F589F19; Mon, 22 Mar 2021 11:07:40 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2041.outbound.protection.outlook.com [40.107.237.41]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7E1FC89BAE; Mon, 22 Mar 2021 11:07:35 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ead8+lu943YuTMTngJfmgCVKFSBwrUGYI7NlSV4F6np4H6prF0jQYb5RE4cI5u6lPPJ630RBeuRMgTpM8nvrLINLqHyE3xmb6L6O8IlvJrN++9FrR3TnxkHMBftqSJJtmkrFEpepgNIhxuZ9LJf/dKR6B6mpuBK30bt7BIW0E+rIzWVVZJtAZWN/7yLN1/ahs+mQoDJxlNH1cBOWiAMpionJMaN58sKg0qZjRGAvUfljSS3IBem07lvy+V/hdaqk5Pq4yrWKl9u3tKo4thmpUf2ek6OPoiWE7hfPvUDMYi7wtEX6NnDbkMma/3NU2VwmNtMrBbw8e19fT/fHT/PRsQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=epi4MJAZ/bhbWerSJ8ZE/JFgIX+o9R2KTuuUA0TdN9U=; b=bNLKTmlELbkO/RjuNc2Qn1v5Y8VuVNJQTLdUVQwK9Uwq/GU104IRuzPVdbWNba61ip0ky0yK6wIt4R5gUelMb0530JkOtidZmteyR28XTOni2rprpNToEOFqMP5bN8E0IWG59eQsxJ3dLlS3MeqVUnASdc33lpzpFP/6aTR8AOblBVTvbkB6y2eJAbvgpZBryhapl8cgTxy/xbHH9FhKCKgXn+xTCDASpq8v86BYUOFh3iEQGTtgg9hHk92nnOf5xzkk/Vy8WrbcpVQ90wA5afedPz1/vzVZYSlD+dtnguw1MVIky3XWMRkv0DpWKLVOEklNqKuRjOLjfavDGgol7g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=epi4MJAZ/bhbWerSJ8ZE/JFgIX+o9R2KTuuUA0TdN9U=; b=zS2i/02EWUrLXPc6rcaj5OmM99wRxxYY7rs3svYFcCeq8hh4LQ+wBwlKEYOsUWXCb/WMBz808n8RX66O0yWCV5EwLeHV4V0+m18THK3qnELUiDpmsSB0nUXiEPLcV8Pcd8S7LLfmH16AFJWMNXn3673NsVv8m1Vi3eYgzlkMhkQ= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:34 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:34 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 08/44] drm/amdkfd: deregister svm range Date: Mon, 22 Mar 2021 06:58:24 -0400 Message-Id: <20210322105900.14068-9-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:33 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 68c72243-fd66-4327-e2bd-08d8ed22b197 X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6430; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tAhmJ8yfuD2nUlWpo8bvdqnVDQW0vk9a8rM5JxG3PiyO6rHFy79Onz7cowzhQNFj21yGc5CXQGfYrSjYwnzMJQu3Sd9DRcMVpO04KrtifQ71veM0o1G29GZyUXeRjq+ZSUxfFu7IAiYT0B7d7QRJe7HGU+5l0Iamfoe7B/6DmpCGGhiYb8ECQkjoH3i7k1TnaZg0SqSDcTJ2Hylzvx+eZwUhxdocDCNqWidv0y4YcgRphWHGmoKUHcrxgo2UzLrOFT4KIoxRpQJUwm81JVPJsAh6q6Fdcp/nqQDVIIDFOJrxekILE06FDKodsHScUPOd+50cbZQoAiGNu0zfXmbU2w0vRjayitGeGFWHTnh1ANdeFtWuH1tBDkY3OJwUPpksKoumEZuQXXj1UYiomRDMoHPkxGEhlMQMScfNK6R3thuMTokLMj2LXG2BNy+91MyBEy9eR1blA8yx8V+Rki41jpo1W0NroYQB5Tq4FZ3inUj0yMCVsXA4iKg/esv/qt2nzfH2I0wZg91VXjH2MAIT5Dljhlq7zAxIzmvl3FtYea4LyarRWqFXyG/hgRt5HQ/zepvTgy1KY+d4Y6o6jnbTN4a7ILyI9GnISK5+HlQq84CfC9F4H8KpTvgMBWJf4ZtnbY5QmxFzsiZ7t0p7TCNiv0xocYEzPPdc5ir0RjAbX3k= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(54906003)(6486002)(1076003)(36756003)(30864003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: VFHPDizAZ8f8wntBPazoHZ+NeGYlGkf3c8QQ6lJT/RaYKr3mvDG8ziibc2/vxsb+8zGcBJKKYxxNlbNEVfGYiSZ1iPY6vA1FvF5BSxioP/awIo1waoZuvEp2vFnRMwUQPhwPFJwTlk6abncTCWL8ekd6BID2nhIWqYAquhg6KKoZLFwcuC1ATEy9ih6Gv8Nw5j9qQvK4zknE52XzLr2Ufnz/uW3Y+7Ijtpk4XgskBVshXYGIn0JwNoYHsTvxV+7SiNDIpcK6OS7gq0cK+j1bcfByhmt76ZQsRLAjcYyDXl4Usl69rOEqarfp/VEsY2IQqZmwqzLmwbOIVh1kYiLSMAtRf2BY+vkQopwc+dH5iTA9YhsywsORWr1ceKL1vkQATJzoIxXRH+Zb+Z3HpcddlxkkkDSfkyifd2FLlxEmrW/J0X0ZeXj3s7B8BqU1/pFZy49n3bHjKkt8KiF7l5gAolXjqhzORPcC+jBV3ry46KISm//9ILlK//IDY9XD2BtHpS0p1mY4Qu1tyDd0wLgul4OUdLVIPcIe1QiBZTcTC5EmNpyQbDo2bqNCKMpu2mWk/ACA6TcOC4XvKl2ozza4oA8LSumYKI2Ss8axztq4TSLOoocxtjGA+g1MH9z/TKchoKffRVasTXleE/qauLDgig4LDnVzrC1AjWBj8jwHn/smtEEEnmD5/vfG//Eg+bqtEF2ST/+wlGr8bpB683K5didW35m4fsPdxtHni+iak8ub0GbpfBzqc2+TqTda4/XijcfOHsBlZB4icZwqaCavOpUA4MpOe8kk0+0917kYKDVjTLSyroK0JJDCdnOcqkb0nLU/1vGeCWSa07hPiK28Bg8EexWmcm8S7CEoLF16nL14egzLys+w23KH/OE18d17Y0GTqi8wJ1kNSCrb6HF+vmFrpK4pyDcrGEhsCcPddgRoO7B0xmrQlHuU7KpfQY6vcDSzbjjI/JDKxMRN/9u5OHHQJK06RoPRWkj7SGDKf+lSfbRaQza1GsGO+64GsH5+ZzzhNU7H2X+TzgLeh1sJwYBHZPanOGHHoDBSkj+0JeyyKc701Jh/OoIzgMnrK99PPP2wuX1kjF3K9gpfsUS8fH6+8+/fKrwG61ohAzAgAhUae1OaTHtMHZK0/f80Eugi2NdkZaV3yFGuJKt8A+HE7sl9vccectNrb4YJljGRejyegjlNsR101gxFIRrhrdYE5U6F9bFgiKS5JhyAPViW6jM+jYArptcrwp0tqMI5RFkYHHroeMJu20vv3ylSjAuiXopm5amt4bSGmIHSc8/EvX9PEb2z3GEob2DhQ7gx5mg1P6BKqf3ywMjEzb4BkJA6 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 68c72243-fd66-4327-e2bd-08d8ed22b197 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:33.8999 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 4/QV96p+95aqSoPcseUP2GNtfr5EoFav7Cxp0PJ+7ohts04M76T+tvInZzZKkUmOGyeKieWRDMYtf6hF/W4IOw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang When application explicitly call unmap or unmap from mmput when application exit, driver will receive MMU_NOTIFY_UNMAP event to remove svm range from process svms object tree and list first, unmap from GPUs (in the following patch). Split the svm ranges to handle partial unmapping of svm ranges. To avoid deadlocks, updating MMU notifiers, range lists and interval trees is done in a deferred worker. New child ranges are attached to their parent range's child_list until the worker can update the svm_range_list. svm_range_set_attr flushes deferred work and takes the mmap_write_lock to guarantee that it has an up-to-date svm_range_list. Signed-off-by: Philip Yang Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 3 + drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 285 +++++++++++++++++++++++++- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 18 ++ 3 files changed, 305 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index f547e1282d69..4101f5341ec9 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -735,6 +735,9 @@ struct svm_range_list { struct mutex lock; struct rb_root_cached objects; struct list_head list; + struct work_struct deferred_list_work; + struct list_head deferred_range_list; + spinlock_t deferred_list_lock; }; /* Process data */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 6024caf7373f..e23171ac866a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -136,6 +136,8 @@ svm_range *svm_range_new(struct svm_range_list *svms, uint64_t start, INIT_LIST_HEAD(&prange->update_list); INIT_LIST_HEAD(&prange->remove_list); INIT_LIST_HEAD(&prange->insert_list); + INIT_LIST_HEAD(&prange->deferred_list); + INIT_LIST_HEAD(&prange->child_list); svm_range_set_default_attributes(&prange->preferred_loc, &prange->prefetch_loc, &prange->granularity, &prange->flags); @@ -512,6 +514,41 @@ svm_range_split_head(struct svm_range *prange, struct svm_range *new, return r; } +void svm_range_add_child(struct svm_range *prange, struct mm_struct *mm, + struct svm_range *pchild, enum svm_work_list_ops op) +{ + pr_debug("add child 0x%p [0x%lx 0x%lx] to prange 0x%p child list %d\n", + pchild, pchild->start, pchild->last, prange, op); + + pchild->work_item.mm = mm; + pchild->work_item.op = op; + list_add_tail(&pchild->child_list, &prange->child_list); +} + +/** + * svm_range_list_lock_and_flush_work - flush pending deferred work + * + * @svms: the svm range list + * @mm: the mm structure + * + * Context: Returns with mmap write lock held, pending deferred work flushed + * + */ +static void +svm_range_list_lock_and_flush_work(struct svm_range_list *svms, + struct mm_struct *mm) +{ +retry_flush_work: + flush_work(&svms->deferred_list_work); + mmap_write_lock(mm); + + if (list_empty(&svms->deferred_range_list)) + return; + mmap_write_unlock(mm); + pr_debug("retry flush\n"); + goto retry_flush_work; +} + struct svm_range *svm_range_clone(struct svm_range *old) { struct svm_range *new; @@ -664,21 +701,264 @@ svm_range_handle_overlap(struct svm_range_list *svms, struct svm_range *new, return r; } +static void +svm_range_update_notifier_and_interval_tree(struct mm_struct *mm, + struct svm_range *prange) +{ + unsigned long start; + unsigned long last; + + start = prange->notifier.interval_tree.start >> PAGE_SHIFT; + last = prange->notifier.interval_tree.last >> PAGE_SHIFT; + + if (prange->start == start && prange->last == last) + return; + + pr_debug("up notifier 0x%p prange 0x%p [0x%lx 0x%lx] [0x%lx 0x%lx]\n", + prange->svms, prange, start, last, prange->start, + prange->last); + + if (start != 0 && last != 0) { + interval_tree_remove(&prange->it_node, &prange->svms->objects); + svm_range_remove_notifier(prange); + } + prange->it_node.start = prange->start; + prange->it_node.last = prange->last; + + interval_tree_insert(&prange->it_node, &prange->svms->objects); + svm_range_add_notifier_locked(mm, prange); +} + +static void +svm_range_handle_list_op(struct svm_range_list *svms, struct svm_range *prange) +{ + struct mm_struct *mm = prange->work_item.mm; + + switch (prange->work_item.op) { + case SVM_OP_NULL: + pr_debug("NULL OP 0x%p prange 0x%p [0x%lx 0x%lx]\n", + svms, prange, prange->start, prange->last); + break; + case SVM_OP_UNMAP_RANGE: + pr_debug("remove 0x%p prange 0x%p [0x%lx 0x%lx]\n", + svms, prange, prange->start, prange->last); + svm_range_unlink(prange); + svm_range_remove_notifier(prange); + svm_range_free(prange); + break; + case SVM_OP_UPDATE_RANGE_NOTIFIER: + pr_debug("update notifier 0x%p prange 0x%p [0x%lx 0x%lx]\n", + svms, prange, prange->start, prange->last); + svm_range_update_notifier_and_interval_tree(mm, prange); + break; + case SVM_OP_ADD_RANGE: + pr_debug("add 0x%p prange 0x%p [0x%lx 0x%lx]\n", svms, prange, + prange->start, prange->last); + svm_range_add_to_svms(prange); + svm_range_add_notifier_locked(mm, prange); + break; + default: + WARN_ONCE(1, "Unknown prange 0x%p work op %d\n", prange, + prange->work_item.op); + } +} + +static void svm_range_deferred_list_work(struct work_struct *work) +{ + struct svm_range_list *svms; + struct svm_range *prange; + struct mm_struct *mm; + + svms = container_of(work, struct svm_range_list, deferred_list_work); + pr_debug("enter svms 0x%p\n", svms); + + spin_lock(&svms->deferred_list_lock); + while (!list_empty(&svms->deferred_range_list)) { + prange = list_first_entry(&svms->deferred_range_list, + struct svm_range, deferred_list); + spin_unlock(&svms->deferred_list_lock); + pr_debug("prange 0x%p [0x%lx 0x%lx] op %d\n", prange, + prange->start, prange->last, prange->work_item.op); + + mm = prange->work_item.mm; + mmap_write_lock(mm); + mutex_lock(&svms->lock); + + /* Remove from deferred_list must be inside mmap write lock, + * otherwise, svm_range_list_lock_and_flush_work may hold mmap + * write lock, and continue because deferred_list is empty, then + * deferred_list handle is blocked by mmap write lock. + */ + spin_lock(&svms->deferred_list_lock); + list_del_init(&prange->deferred_list); + spin_unlock(&svms->deferred_list_lock); + + while (!list_empty(&prange->child_list)) { + struct svm_range *pchild; + + pchild = list_first_entry(&prange->child_list, + struct svm_range, child_list); + pr_debug("child prange 0x%p op %d\n", pchild, + pchild->work_item.op); + list_del_init(&pchild->child_list); + svm_range_handle_list_op(svms, pchild); + } + + svm_range_handle_list_op(svms, prange); + mutex_unlock(&svms->lock); + mmap_write_unlock(mm); + + spin_lock(&svms->deferred_list_lock); + } + spin_unlock(&svms->deferred_list_lock); + + pr_debug("exit svms 0x%p\n", svms); +} + +void +svm_range_add_list_work(struct svm_range_list *svms, struct svm_range *prange, + struct mm_struct *mm, enum svm_work_list_ops op) +{ + spin_lock(&svms->deferred_list_lock); + /* if prange is on the deferred list */ + if (!list_empty(&prange->deferred_list)) { + pr_debug("update exist prange 0x%p work op %d\n", prange, op); + WARN_ONCE(prange->work_item.mm != mm, "unmatch mm\n"); + if (op != SVM_OP_NULL && + prange->work_item.op != SVM_OP_UNMAP_RANGE) + prange->work_item.op = op; + } else { + prange->work_item.op = op; + prange->work_item.mm = mm; + list_add_tail(&prange->deferred_list, + &prange->svms->deferred_range_list); + pr_debug("add prange 0x%p [0x%lx 0x%lx] to work list op %d\n", + prange, prange->start, prange->last, op); + } + spin_unlock(&svms->deferred_list_lock); +} + +void schedule_deferred_list_work(struct svm_range_list *svms) +{ + spin_lock(&svms->deferred_list_lock); + if (!list_empty(&svms->deferred_range_list)) + schedule_work(&svms->deferred_list_work); + spin_unlock(&svms->deferred_list_lock); +} + +static void +svm_range_unmap_split(struct mm_struct *mm, struct svm_range *parent, + struct svm_range *prange, unsigned long start, + unsigned long last) +{ + struct svm_range *head; + struct svm_range *tail; + + if (prange->work_item.op == SVM_OP_UNMAP_RANGE) { + pr_debug("prange 0x%p [0x%lx 0x%lx] is already freed\n", prange, + prange->start, prange->last); + return; + } + if (start > prange->last || last < prange->start) + return; + + head = tail = prange; + if (start > prange->start) + svm_range_split(prange, prange->start, start - 1, &tail); + if (last < tail->last) + svm_range_split(tail, last + 1, tail->last, &head); + + if (head != prange && tail != prange) { + svm_range_add_child(parent, mm, head, SVM_OP_UNMAP_RANGE); + svm_range_add_child(parent, mm, tail, SVM_OP_ADD_RANGE); + } else if (tail != prange) { + svm_range_add_child(parent, mm, tail, SVM_OP_UNMAP_RANGE); + } else if (head != prange) { + svm_range_add_child(parent, mm, head, SVM_OP_UNMAP_RANGE); + } else if (parent != prange) { + prange->work_item.op = SVM_OP_UNMAP_RANGE; + } +} + +static void +svm_range_unmap_from_cpu(struct mm_struct *mm, struct svm_range *prange, + unsigned long start, unsigned long last) +{ + struct svm_range_list *svms; + struct svm_range *pchild; + struct kfd_process *p; + bool unmap_parent; + + p = kfd_lookup_process_by_mm(mm); + if (!p) + return; + svms = &p->svms; + + pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx] [0x%lx 0x%lx]\n", svms, + prange, prange->start, prange->last, start, last); + + unmap_parent = start <= prange->start && last >= prange->last; + + list_for_each_entry(pchild, &prange->child_list, child_list) + svm_range_unmap_split(mm, prange, pchild, start, last); + svm_range_unmap_split(mm, prange, prange, start, last); + + if (unmap_parent) + svm_range_add_list_work(svms, prange, mm, SVM_OP_UNMAP_RANGE); + else + svm_range_add_list_work(svms, prange, mm, + SVM_OP_UPDATE_RANGE_NOTIFIER); + schedule_deferred_list_work(svms); + + kfd_unref_process(p); +} + /** * svm_range_cpu_invalidate_pagetables - interval notifier callback * + * MMU range unmap notifier to remove svm ranges */ static bool svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, const struct mmu_notifier_range *range, unsigned long cur_seq) { + struct svm_range *prange; + unsigned long start; + unsigned long last; + + if (range->event == MMU_NOTIFY_RELEASE) + return true; + + start = mni->interval_tree.start; + last = mni->interval_tree.last; + start = (start > range->start ? start : range->start) >> PAGE_SHIFT; + last = (last < (range->end - 1) ? last : range->end - 1) >> PAGE_SHIFT; + pr_debug("[0x%lx 0x%lx] range[0x%lx 0x%lx] notifier[0x%lx 0x%lx] %d\n", + start, last, range->start >> PAGE_SHIFT, + (range->end - 1) >> PAGE_SHIFT, + mni->interval_tree.start >> PAGE_SHIFT, + mni->interval_tree.last >> PAGE_SHIFT, range->event); + + prange = container_of(mni, struct svm_range, notifier); + + switch (range->event) { + case MMU_NOTIFY_UNMAP: + svm_range_unmap_from_cpu(mni->mm, prange, start, last); + break; + default: + break; + } + return true; } void svm_range_list_fini(struct kfd_process *p) { pr_debug("pasid 0x%x svms 0x%p\n", p->pasid, &p->svms); + + /* Ensure list work is finished before process is destroyed */ + flush_work(&p->svms.deferred_list_work); } int svm_range_list_init(struct kfd_process *p) @@ -688,6 +968,9 @@ int svm_range_list_init(struct kfd_process *p) svms->objects = RB_ROOT_CACHED; mutex_init(&svms->lock); INIT_LIST_HEAD(&svms->list); + INIT_WORK(&svms->deferred_list_work, svm_range_deferred_list_work); + INIT_LIST_HEAD(&svms->deferred_range_list); + spin_lock_init(&svms->deferred_list_lock); return 0; } @@ -805,7 +1088,7 @@ svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, mutex_lock(&process_info->lock); - mmap_write_lock(mm); + svm_range_list_lock_and_flush_work(svms, mm); if (!svm_range_is_valid(mm, start, size)) { pr_debug("invalid range\n"); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index f35a178c607f..f9b81245b94a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -32,6 +32,18 @@ #include "amdgpu.h" #include "kfd_priv.h" +enum svm_work_list_ops { + SVM_OP_NULL, + SVM_OP_UNMAP_RANGE, + SVM_OP_UPDATE_RANGE_NOTIFIER, + SVM_OP_ADD_RANGE +}; + +struct svm_work_list_item { + enum svm_work_list_ops op; + struct mm_struct *mm; +}; + /** * struct svm_range - shared virtual memory range * @@ -53,6 +65,9 @@ * @actual_loc: the actual location, 0 for CPU, or GPU id * @granularity:migration granularity, log2 num pages * @notifier: register mmu interval notifier + * @work_item: deferred work item information + * @deferred_list: list header used to add range to deferred list + * @child_list: list header for split ranges which are not added to svms yet * @bitmap_access: index bitmap of GPUs which can access the range * @bitmap_aip: index bitmap of GPUs which can access the range in place * @@ -78,6 +93,9 @@ struct svm_range { uint32_t actual_loc; uint8_t granularity; struct mmu_interval_notifier notifier; + struct svm_work_list_item work_item; + struct list_head deferred_list; + struct list_head child_list; DECLARE_BITMAP(bitmap_access, MAX_GPU_INSTANCE); DECLARE_BITMAP(bitmap_aip, MAX_GPU_INSTANCE); }; From patchwork Mon Mar 22 10:58:25 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 40001C433E0 for ; Mon, 22 Mar 2021 11:08:08 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0CA7761931 for ; Mon, 22 Mar 2021 11:08:08 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0CA7761931 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 2E66C89EBD; Mon, 22 Mar 2021 11:07:39 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2041.outbound.protection.outlook.com [40.107.237.41]) by gabe.freedesktop.org (Postfix) with ESMTPS id AC58189B27; Mon, 22 Mar 2021 11:07:35 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KEbxWds7zrWiXmpydv88jh6ak5zqvNQpmQULTTmwX0GA3MJ6PHc1LsJLT/Sh/GrU1YXBc1OZSnohlKBlPewyrT18Dh0B+cdFzD8KcUI9qE/COAc10sRjnbS3gQuBgOt1TnZ+0/VGMicFE6nO6ajVfX1OAfxLChnYmMXm1O6+HmR7+NDXrd6GVAIgHBO4qgpicT0hEnnubKnmkQbwBZswlS+1t0gpxFnc5oikzNc+VHm8Lboj6aczVmf6GpgCLyYUbWjLQeXneylvdoyinqaeSgpvIDrOK7Egu+8pu9++mXogE4YAiCoQ49+VRO3PjOmYlQrGIQNBonY1474YZE8Bhg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4mrhwpruagGf+8gNs25WWLkvR1NYEmptQMB/RQRodxE=; b=j9k+E7LF8nzVCq/WIii6hL4q63p6IckDQs/0DLCFzVZ04mDqyseaGJiXeRZbR8QDwOmT3G9c3ijFkycy39jxx851RLqv/Y93qDyDNqR7as3EDgulS1Cth+TLa4cWRZpA3g7vosJRYsCBzjQPXgBwegh4Q1NuA1rAX0sjfH9Tb7T5KYXf6JZFufO/Wby51RUKnUVc/lBadkhkQA0cSQgJO8+zqPtuXezFdFXi1dDnAzHLcoBa+vwEQDfhUVnzSy+LXqJmeR1d8C9MvOhvL2NkoJEWo3dZv20FK2Ao9Q/pueyvnfa9BkByVCVFi8vkdo52d5PbnCE/R4TA4f5r5zZPmw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=4mrhwpruagGf+8gNs25WWLkvR1NYEmptQMB/RQRodxE=; b=kuWhI4aaDbGKVaiCZPtcwX9w3LuPao24zEwBu7IUmX09MDeTin9gEeDa6wbdUBmoHEuwQHEiF8td4ebOcHOq2S5dgiEXIOkEHqPNtydNGh6F0k+BIMnCeYqunCk2jCpP7WVksXDp480rPsm+IJ+LAtqvHEFfbZvhNrluRtuBCSc= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:34 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:34 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 09/44] drm/amdgpu: export vm update mapping interface Date: Mon, 22 Mar 2021 06:58:25 -0400 Message-Id: <20210322105900.14068-10-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:34 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 86310b36-b0c7-4ce4-a4e0-08d8ed22b1e7 X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4125; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 8CFTKJQTf2UfTiOfSQBEg12ALyQO/fCPUjmJv1dJ2cp15jL6iBuW9wsFvUL890ZF0eDU72eV5C7jxVcey8wegdlJqdK6953FAxhUFs/1Hk3aEbDr0Wi4o0OTrELFZZun+9fdhSajUVk0BmQXHGTUsIQC7vpdy9SCx6U511UWUu08Lgw0FCRLx0CR4zbYFJ2vSEDeJ6Uw0P3XuhakCtWwlMJYm8jqr2pJT5MdsmjwVnJXAqIUa+4/+TVQ5rKg/aFzVFVPEWsz6Vs4fJPrCfHINf6Yp2mPldDYM1NpnR8ljPUHw6zOLI2c3jGQEEP5rhTwXrp+p3Y1hLDu550kxFY3mMWxaxsQiJkGMhpr2sk9uB0Ur+AY+HiIv3yEtbMETZiSuwjLuwxEgWGoKIEu12BuNwtvVW27bZuKN6knkEHA3VpQiIaF1cksWCs+x/w5j1kQVooBfLomqcarmYbeBXyj/xxpoDsKyHihahAyhOOmJuKtn11qy3QRBDH/oC6zTgC4YS6r5i6hNivtMi8TqPKWyJyXPxakPC4gsLVTIx7SAVM37QzFgpIocZRUY/IqTP3mkna+jJsoBwIOJMAISTEnsTPo0ZazkRPnmNiF+yJkjVs0i0cqSwNzl1sBK3sBL16TmswE/WB99yiJ7McsO5kALU6RZ1WxLeucUlxR89z2yvo= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(15650500001)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: O2bm3Ybu01RQesJUv/afYfIM9f0cBSSKAitzJ3zV0hOfnoU8ZZo4e21cnBJLclX+G3drWCRNDA3aqlPo5LCp8P2MraTPneT0h3+uhiXbxIr6Bk4IeNLZwRjWrlnV0Lu6EekZeBwSLrHYtiJcnen7c8EwSNRES8v1En+UaX9jzEGlGdHjsGoZdLiCRBOXtgZGxPPkBVAZgCk72NluoJSbdzS9eDYZGdP5Pe15VJjT5A1PkdMmTvD9KpxIxqVJvag6PkEpJNtvNiysyzdPH+ctDqB4uxlOaRZrZrLLWjfDsEDQg3HICZcNCeUq6GwoSs6+jMTDlAzwvCDHTDjmFacvuU2XpRlvbU7QC1uhtBURlubvs2A11AfTQAMrP8zubxCHI63+4NBaTOPH8nl5hA3m/Qal5/FcSerSCc/lt9yTYYecw1wQBtoP7dutyK0NNCAOBrM8F1jSvzhVJ3YRLWwdocqEWCdRqlB7EBRQQWwxeJn2JlxTMSnQotO85j0scFknkw58GAE7RGRfP6x3Nt3KY5HfeDLH6Hst4QaFP5LgyyvWMixGnKM+u+Va53bOIIbEmvEXp8YO6O+onOnaLAuXYRmABB9kTdqngoMVh19i8wP2wzHcb1DsE7XtJQQ4wXEmxeeNu2KO9cRTA74AnUKS4Z3N0Z9setQzkBzCcN/hj0jq5cOE5EtremiPjkneQCfExMMmNKBct/tumbNzs/r8HVZeWwKMonB3Qr9braDV9AEz2xa7G57auBjgmD1B1YGmtnUGHK2vbQuAShShV8+Vx4iguOnQxZw3/qC4orPeTQV85zA8CHQrvjlYVdtHPqdm86VgrzgI3gulTXp+ENSdpUB7HxalE9BjtLjR2ssHBf6RogZAY557Zm7yhQyRxeGnSS8Ps7QXpZdsGG6utRFJS2uUAJLhZMSxujmdbSCfvOrTJYF24uGkffOgA1eOrptxVrU0bRXokxwIzk1E9gOISC3G9z0MLE4rMNCtw8CRQ7b8qsdlNKMZGBX1wuS/Guy8tcJ+KogDYibQkHVF9+n6BnW7V2uKbhTMvsVaoRED7JkpXeTEi6Wiw2PTcZ5OyrEul95AzHjVuLWxZq5Fd7AWdnzOgTUU8mCGIKkclT5SPptbHY+jF7bbWjfnT5tVGZrjxBmEQDQqvnOmdOhiNH2yRCdEriPX9GEp9cuoy07TDexD7tY7y7aOJeESunLBrr4LWwCOx2FTulrzqDVTpNwQajQQ9eBe9fMtITWxYSMG5izCuyKT1kiajtaelKT7u9ENUJnnfyccGCVz0W30bMdf2fhogOc7wXH1zs15qzNK5JfChlhXBmsS2IsOitbycKij X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 86310b36-b0c7-4ce4-a4e0-08d8ed22b1e7 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:34.4265 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 5oCLs0JG3OQZ2JRK0+sNRXqzAZVWGk0ZCp6KEg4Tc7raEqPzdjGNZviXq73qG9OYEESvUPJTSrUgFNl3FDcKUw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang It will be used by kfd to map svm range to GPU, because svm range does not have amdgpu_bo and bo_va, cannot use amdgpu_bo_update interface, use amdgpu vm update interface directly. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 18 +++++++++--------- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h | 11 +++++++++++ 2 files changed, 20 insertions(+), 9 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index 9268db1172bd..0e9ae5f91c7c 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -1592,15 +1592,15 @@ static int amdgpu_vm_update_ptes(struct amdgpu_vm_update_params *params, * Returns: * 0 for success, -EINVAL for failure. */ -static int amdgpu_vm_bo_update_mapping(struct amdgpu_device *adev, - struct amdgpu_device *bo_adev, - struct amdgpu_vm *vm, bool immediate, - bool unlocked, struct dma_resv *resv, - uint64_t start, uint64_t last, - uint64_t flags, uint64_t offset, - struct drm_mm_node *nodes, - dma_addr_t *pages_addr, - struct dma_fence **fence) +int amdgpu_vm_bo_update_mapping(struct amdgpu_device *adev, + struct amdgpu_device *bo_adev, + struct amdgpu_vm *vm, bool immediate, + bool unlocked, struct dma_resv *resv, + uint64_t start, uint64_t last, + uint64_t flags, uint64_t offset, + struct drm_mm_node *nodes, + dma_addr_t *pages_addr, + struct dma_fence **fence) { struct amdgpu_vm_update_params params; enum amdgpu_sync_mode sync_mode; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h index 976a12e5a8b9..848e175e99ff 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h @@ -366,6 +366,8 @@ struct amdgpu_vm_manager { spinlock_t pasid_lock; }; +struct amdgpu_bo_va_mapping; + #define amdgpu_vm_copy_pte(adev, ib, pe, src, count) ((adev)->vm_manager.vm_pte_funcs->copy_pte((ib), (pe), (src), (count))) #define amdgpu_vm_write_pte(adev, ib, pe, value, count, incr) ((adev)->vm_manager.vm_pte_funcs->write_pte((ib), (pe), (value), (count), (incr))) #define amdgpu_vm_set_pte_pde(adev, ib, pe, addr, count, incr, flags) ((adev)->vm_manager.vm_pte_funcs->set_pte_pde((ib), (pe), (addr), (count), (incr), (flags))) @@ -397,6 +399,15 @@ int amdgpu_vm_clear_freed(struct amdgpu_device *adev, struct dma_fence **fence); int amdgpu_vm_handle_moved(struct amdgpu_device *adev, struct amdgpu_vm *vm); +int amdgpu_vm_bo_update_mapping(struct amdgpu_device *adev, + struct amdgpu_device *bo_adev, + struct amdgpu_vm *vm, bool immediate, + bool unlocked, struct dma_resv *resv, + uint64_t start, uint64_t last, + uint64_t flags, uint64_t offset, + struct drm_mm_node *nodes, + dma_addr_t *pages_addr, + struct dma_fence **fence); int amdgpu_vm_bo_update(struct amdgpu_device *adev, struct amdgpu_bo_va *bo_va, bool clear); From patchwork Mon Mar 22 10:58:26 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154481 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B859CC433DB for ; Mon, 22 Mar 2021 11:08:01 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 7CF9E61879 for ; Mon, 22 Mar 2021 11:08:01 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7CF9E61879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 747C889EF7; Mon, 22 Mar 2021 11:07:39 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2058.outbound.protection.outlook.com [40.107.237.58]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3EEC489BF5; Mon, 22 Mar 2021 11:07:37 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lSionmR/b6fjNfKK0EuNmiZQZCdQ54aOMtAK9MZmtuYAMaANNpdhqR0pkXH4FccGjGJqAKxhK+E+ZRUhAIlnSGDHxzBzrCuDzrZ91ZVs4GmzRIEe54hEL2w/Dnakv8AERjlGcEU8/xkqZmkHYsKY7I63UbDp712P5gRgHBqLVOrqz3f2xvA6fQa4ADlb4En6IkvkLnjNbx2iwogTOhmKYjl1C7B+XaqhpEa0SBKrKXwuqyD/Y4Azhwe7AdenPDGom4dFqZXKE+IOh6BTTeShYNl3irlQunDcV3UPxdcOb2IKLXeG40oY53p7afykQMB64ha5mhYbnoNoaqnnO6ekWQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WGe2Ikg/vsojRLYLsLsboCdpRGRawCN8SVJFs/lkgUA=; b=fCvUgnYwZpUpLgnkvh1lKdyEnIvTZtcTpdlNhb28OGuYoUz326XJD76k8ZuHkE8W1630kOc4DH2tZmK1kjS/aalSuY7CWPAK5DownOS0u0daDp2oXEG0OWIw6hIfnnTk97OFVdbqzNo2W/dPNC1cymZ5YcwrX3Hz4B3y7U+WOxB7ZNhE8URx0O2tkE+tsdkcIe810YNcRMXiT4UePcsvkqZZXdpeluqA2bbdSzZxQRd6wh/23OE7CCkcX3ZEz/s7C720sY/KJ8otUIsKcYgCiPfoNg5UavJupEve4MDJ5JQ8w8RIsuDGeMPzLb80Gw3CfJ5LyJrvC2k+HQAoCLWk9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WGe2Ikg/vsojRLYLsLsboCdpRGRawCN8SVJFs/lkgUA=; b=PwSjZJ0e6Z+EF/UAL9RoDfkRxO8beporIO4LbD8wSsxKs9AyP5hERRKiqm8U+2ijP/rTQpizaisO2EQMWBkCdtlNBv7+UDRNlU5JW9iLbhFpxIWiJLzDgPTaWM/84cNR+MG2hAyHC0v+T2bp9fZ9yZ2FtCYVXeBLicYY+PIXmkU= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:35 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:35 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 10/44] drm/amdkfd: map svm range to GPUs Date: Mon, 22 Mar 2021 06:58:26 -0400 Message-Id: <20210322105900.14068-11-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:34 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 8c908fc7-9595-4092-bf14-08d8ed22b246 X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:270; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: ySeFgDiEGkH5CJiCOPbPbixBAdKOYsgy91GNbPU1sM2OPZBVqCWqsy1ACaDLrVuVQKVncEopey/L3RUf+nkJmdDFCizm742OUhk0sg5HM5a/Ii21u36TKU1JNF8OeVZqojFjJJ6CPU4SpCbZloC13GjNYvZMC8kmm8ntfIoZntvN7GlciN/gtXRdzV5msMAUZkmPbYCCEoxipfKSzl9p6XvJt68ChPD7b183fjyvoo1pYFhVMginY376KRzq1JWS9aXrH36PvbFm8S5k2el6a6tiE0RTaf7sdQyS8U8mO6xaWEPossPdyGc+InsXtVZMFzLufsfEujnQpsTs2DImVWihShp84E7miwjL96gqjRJPMm8LI/5CZZp7FtKkVxbJmIbNp74CIgxiCRuwGVgABZkRnT1MeAuHuNx2PZtf7XFfx7xfQpce85ZakPzgmb4egKt7pPf2oaVCxGxsSDJ9ElIXLiBoVOv8kuK9eGfNkTDkdy4KBwLMV+hUcy06cMMcJ/bCvkv0Uw7cDXt/L2iZM2SJNbbPbPHci1TEkGIrJHJ5QPDm0InaSOd0+M0LF6oWmoAPws3XEeqIYIvOQBueud/6QC+jq7mNzGlJ1lSyRW4TIg2erPP46vSorYjPldkUZc6UHxbEdLX/8UITkqb2yQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(54906003)(6486002)(1076003)(36756003)(30864003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: E1RcMzrled4HDzPSkPi0LXKs1/+FgDVydawoBdRKa0jUwQH66BE7H5WkLetfToH0X8Jja/O8RScRwAK//tUfuYCbBojNa7UgTCEvYKalT1wEgiuzGosIaYYRgieDdTSSJwJlbkjoc6OJg/3cBYCKDN8DQ3B3dMl4zjjUzWI22xYxLI3uw9dQVk2hazmgqLOTDmUy3xEfE9MGdA/6+rzWlkaWi58Np9mN6SUh8E8/7muPeRQotGhvCW1bQr4kQ4ijjtZX5OrdcNnR6CbvehOVy0X8A/VLalf6gGbd0c8k0H8rF13F2Rv+ZU4V2aXZYkCQLsvPytM114aFcB24BKeg1RbGd4RAlW8ndR7B2whU7c0VWn8xlBxjoQlcfELwcsBWu3LbA1S6o3E7wRkgOVWZd9h7QBwifyZU/R73g+IfAidvUbC9GuSMFYNPiBcFe47s6Ibl4NrLDIdx90qj69lppDvJOJGE4D3Dbqw+SCVy1FCGNpkrXsZl0aWPrgWIZSMNa/o6iZ8WtsSTqhX4zdPxzSPAiwvxqKtJbyv+gQnsQEp9JPsdMjHrBMwFA5eBYcpCyB0sNLGwG1KQ9UdkdMfzO4jWitHTGcL+JsFEmCpDfG7gfV305kuKOuQI4/Ax3+kFuFex/o1yf5DRIF1xwEMjh4jnTw3EPUCW2H6SoNcdTdwyflj8IDUoIzaz4h/qFmL7kaFLrqnsm8gzV+4mYY+x5I/J7iREq0575hL/sD1E4Ckpf7+DbDvxPYRx7gKUd9VNhcORRfho4KTCznKqef2fTtpPw658L4OS6v63+P9wZFeaOtF5zzkB4J5mpCWnHUoJyw+XsYIrD9+Vziaq2atQQ69bEF7kqA8nzmiOj702V3VpCdU/7YCSugZZASw3N+NB7x+yv/gWgH8r1E/9V6Bb9WwPIcnQUk3ZiJ9/Mfw7JWb02aOFNJRaK6QG+HCJA49UBz0AQTqhWtgHfKza/hGs62hXqcP2lVza7BQi2tNTVYw+hVjzfcYCDs5+NIvB/OK3WXEbR/kXVeiJVfjYEQxn4n8ZJD7HJxpKxcm4cNMFj/F9vGLpcOY4Frw897hCECyNggobUq4jZxLWoDMxuyAtxC24VGHNNEMqY4yZIHgdZkGdixHUS+O5A/hrzqz83rkTrCU1yNUIA60iv3m1yj3RqRR3IUJZz6ot8EuMs3XezqQqY1wjPTS4NPX2GdxAGS5u4/ltVsetJuoCDw6DXbHyM7kfCqpjAHOiReT1K+HqRRIib8syPmifh+ziYa2S6RDeYaXwQehhNmDtfKgNwVmAwcNwi3K7+Q6Rawc/ETtp/LI+zxikTaumqdQWelXCjdtK X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8c908fc7-9595-4092-bf14-08d8ed22b246 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:35.1563 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: mO3wNF4rbQIhwIYa4Ye0MV/IOu4NzZeF1WYBTIXfd28N3v27ClNsQNVMsA6bZYAINabt0nLYVz9oN31/c9bSEg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Use amdgpu_vm_bo_update_mapping to update GPU page table to map or unmap svm range system memory pages address to GPUs. Signed-off-by: Philip Yang Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 395 +++++++++++++++++++++++++-- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 4 + 2 files changed, 374 insertions(+), 25 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index e23171ac866a..1244ba380292 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -98,11 +98,99 @@ static void svm_range_remove_notifier(struct svm_range *prange) mmu_interval_notifier_remove(&prange->notifier); } +static int +svm_range_dma_map(struct device *dev, dma_addr_t **dma_addr, + unsigned long *pages_addr, uint64_t npages) +{ + enum dma_data_direction dir = DMA_BIDIRECTIONAL; + dma_addr_t *addr = *dma_addr; + struct page *page; + int i, r; + + if (!pages_addr) + return 0; + + if (!addr) { + addr = kvmalloc_array(npages, sizeof(*addr), + GFP_KERNEL | __GFP_ZERO); + if (!addr) + return -ENOMEM; + *dma_addr = addr; + } + + for (i = 0; i < npages; i++) { + if (WARN_ONCE(addr[i] && !dma_mapping_error(dev, addr[i]), + "leaking dma mapping\n")) + dma_unmap_page(dev, addr[i], PAGE_SIZE, dir); + + page = hmm_pfn_to_page(pages_addr[i]); + addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir); + r = dma_mapping_error(dev, addr[i]); + if (r) { + pr_debug("failed %d dma_map_page\n", r); + return r; + } + pr_debug("dma mapping 0x%llx for page addr 0x%lx\n", + addr[i] >> PAGE_SHIFT, page_to_pfn(page)); + } + return 0; +} + +void svm_range_dma_unmap(struct device *dev, dma_addr_t *dma_addr, + unsigned long offset, unsigned long npages) +{ + enum dma_data_direction dir = DMA_BIDIRECTIONAL; + int i; + + if (!dma_addr) + return; + + for (i = offset; i < offset + npages; i++) { + if (!dma_addr[i] || dma_mapping_error(dev, dma_addr[i])) + continue; + pr_debug("dma unmapping 0x%llx\n", dma_addr[i] >> PAGE_SHIFT); + dma_unmap_page(dev, dma_addr[i], PAGE_SIZE, dir); + dma_addr[i] = 0; + } +} + +static void svm_range_free_dma_mappings(struct svm_range *prange) +{ + DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE); + struct kfd_dev *kfd_dev; + dma_addr_t *dma_addr; + struct device *dev; + struct kfd_process *p; + uint32_t gpuidx; + int r; + + p = container_of(prange->svms, struct kfd_process, svms); + bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, + MAX_GPU_INSTANCE); + + for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { + dma_addr = prange->dma_addr[gpuidx]; + if (!dma_addr) + continue; + + r = kfd_process_device_from_gpuidx(p, gpuidx, &kfd_dev); + if (r) { + pr_debug("failed to find device idx %d\n", gpuidx); + return; + } + dev = &kfd_dev->pdev->dev; + svm_range_dma_unmap(dev, dma_addr, 0, prange->npages); + kvfree(dma_addr); + prange->dma_addr[gpuidx] = NULL; + } +} + static void svm_range_free(struct svm_range *prange) { pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx]\n", prange->svms, prange, prange->start, prange->last); + svm_range_free_dma_mappings(prange); kvfree(prange->pages_addr); kfree(prange); } @@ -342,41 +430,62 @@ svm_range_is_same_attrs(struct svm_range *old, struct svm_range *new) } static int -svm_range_split_pages(struct svm_range *new, struct svm_range *old, - uint64_t start, uint64_t last) +svm_range_split_array(void *ppnew, void *ppold, size_t size, + uint64_t old_start, uint64_t old_n, + uint64_t new_start, uint64_t new_n) { - unsigned long old_start; - unsigned long *pages_addr; + unsigned char *new, *old, *pold; uint64_t d; - old_start = old->start; - new->pages_addr = kvmalloc_array(new->npages, - sizeof(*new->pages_addr), - GFP_KERNEL | __GFP_ZERO); - if (!new->pages_addr) - return -ENOMEM; + if (!ppold) + return 0; + pold = *(unsigned char **)ppold; + if (!pold) + return 0; - d = new->start - old_start; - memcpy(new->pages_addr, old->pages_addr + d, - new->npages * sizeof(*new->pages_addr)); + new = kvmalloc_array(new_n, size, GFP_KERNEL); + if (!new) + return -ENOMEM; - old->npages = last - start + 1; - old->start = start; - old->last = last; + d = (new_start - old_start) * size; + memcpy(new, pold + d, new_n * size); - pages_addr = kvmalloc_array(old->npages, sizeof(*pages_addr), - GFP_KERNEL); - if (!pages_addr) { - kvfree(new->pages_addr); + old = kvmalloc_array(old_n, size, GFP_KERNEL); + if (!old) { + kvfree(new); return -ENOMEM; } - d = start - old_start; - memcpy(pages_addr, old->pages_addr + d, - old->npages * sizeof(*pages_addr)); + d = (new_start == old_start) ? new_n * size : 0; + memcpy(old, pold + d, old_n * size); - kvfree(old->pages_addr); - old->pages_addr = pages_addr; + kvfree(pold); + *(void **)ppold = old; + *(void **)ppnew = new; + + return 0; +} + +static int +svm_range_split_pages(struct svm_range *new, struct svm_range *old, + uint64_t start, uint64_t last) +{ + uint64_t npages = last - start + 1; + int i, r; + + for (i = 0; i < MAX_GPU_INSTANCE; i++) { + r = svm_range_split_array(&new->dma_addr[i], &old->dma_addr[i], + sizeof(*old->dma_addr[i]), old->start, + npages, new->start, new->npages); + if (r) + return r; + } + + r = svm_range_split_array(&new->pages_addr, &old->pages_addr, + sizeof(*old->pages_addr), old->start, npages, + new->start, new->npages); + if (r) + return r; return 0; } @@ -525,6 +634,234 @@ void svm_range_add_child(struct svm_range *prange, struct mm_struct *mm, list_add_tail(&pchild->child_list, &prange->child_list); } +static uint64_t +svm_range_get_pte_flags(struct amdgpu_device *adev, struct svm_range *prange) +{ + uint32_t flags = prange->flags; + uint32_t mapping_flags; + uint64_t pte_flags; + + pte_flags = AMDGPU_PTE_VALID; + pte_flags |= AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED; + + mapping_flags = AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE; + + if (flags & KFD_IOCTL_SVM_FLAG_GPU_RO) + mapping_flags &= ~AMDGPU_VM_PAGE_WRITEABLE; + if (flags & KFD_IOCTL_SVM_FLAG_GPU_EXEC) + mapping_flags |= AMDGPU_VM_PAGE_EXECUTABLE; + if (flags & KFD_IOCTL_SVM_FLAG_COHERENT) + mapping_flags |= AMDGPU_VM_MTYPE_UC; + else + mapping_flags |= AMDGPU_VM_MTYPE_NC; + + /* TODO: add CHIP_ARCTURUS new flags for vram mapping */ + + pte_flags |= amdgpu_gem_va_map_flags(adev, mapping_flags); + + /* Apply ASIC specific mapping flags */ + amdgpu_gmc_get_vm_pte(adev, &prange->mapping, &pte_flags); + + pr_debug("PTE flags 0x%llx\n", pte_flags); + + return pte_flags; +} + +static int +svm_range_unmap_from_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, + uint64_t start, uint64_t last, + struct dma_fence **fence) +{ + uint64_t init_pte_value = 0; + + pr_debug("[0x%llx 0x%llx]\n", start, last); + + return amdgpu_vm_bo_update_mapping(adev, adev, vm, false, true, NULL, + start, last, init_pte_value, 0, + NULL, NULL, fence); +} + +static int +svm_range_unmap_from_gpus(struct svm_range *prange, unsigned long start, + unsigned long last) +{ + DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE); + struct kfd_process_device *pdd; + struct dma_fence *fence = NULL; + struct amdgpu_device *adev; + struct kfd_process *p; + struct kfd_dev *dev; + uint32_t gpuidx; + int r = 0; + + bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, + MAX_GPU_INSTANCE); + p = container_of(prange->svms, struct kfd_process, svms); + + for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { + pr_debug("unmap from gpu idx 0x%x\n", gpuidx); + r = kfd_process_device_from_gpuidx(p, gpuidx, &dev); + if (r) { + pr_debug("failed to find device idx %d\n", gpuidx); + return -EINVAL; + } + + pdd = kfd_get_process_device_data(dev, p); + if (!pdd) + return -EINVAL; + + adev = (struct amdgpu_device *)dev->kgd; + + r = svm_range_unmap_from_gpu(adev, pdd->vm, start, last, + &fence); + if (r) + break; + + if (fence) { + r = dma_fence_wait(fence, false); + dma_fence_put(fence); + fence = NULL; + if (r) + break; + } + + svm_range_dma_unmap(adev->dev, prange->dma_addr[gpuidx], + start - prange->start, + last - start + 1); + + amdgpu_amdkfd_flush_gpu_tlb_pasid((struct kgd_dev *)adev, + p->pasid); + } + + return r; +} + +static int svm_range_bo_validate(void *param, struct amdgpu_bo *bo) +{ + struct ttm_operation_ctx ctx = { false, false }; + + amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_VRAM); + + return ttm_bo_validate(&bo->tbo, &bo->placement, &ctx); +} + +static int +svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, + struct svm_range *prange, dma_addr_t *pages_addr, + bool reserve_vm, struct dma_fence **fence) +{ + struct amdgpu_bo *root; + uint64_t pte_flags; + int r = 0; + + pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, + prange->last); + + if (reserve_vm) { + root = amdgpu_bo_ref(vm->root.base.bo); + r = amdgpu_bo_reserve(root, true); + if (r) { + pr_debug("failed %d to reserve root bo\n", r); + amdgpu_bo_unref(&root); + goto out; + } + r = amdgpu_vm_validate_pt_bos(adev, vm, svm_range_bo_validate, + NULL); + if (r) { + pr_debug("failed %d validate pt bos\n", r); + goto unreserve_out; + } + } + + prange->mapping.start = prange->start; + prange->mapping.last = prange->last; + prange->mapping.offset = 0; + pte_flags = svm_range_get_pte_flags(adev, prange); + prange->mapping.flags = pte_flags; + + r = amdgpu_vm_bo_update_mapping(adev, adev, vm, false, false, NULL, + prange->mapping.start, + prange->mapping.last, pte_flags, + prange->mapping.offset, NULL, + pages_addr, &vm->last_update); + if (r) { + pr_debug("failed %d to map to gpu 0x%lx\n", r, prange->start); + goto unreserve_out; + } + + + r = amdgpu_vm_update_pdes(adev, vm, false); + if (r) { + pr_debug("failed %d to update directories 0x%lx\n", r, + prange->start); + goto unreserve_out; + } + + if (fence) + *fence = dma_fence_get(vm->last_update); + +unreserve_out: + if (reserve_vm) { + amdgpu_bo_unreserve(root); + amdgpu_bo_unref(&root); + } + +out: + return r; +} + +static int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) +{ + DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE); + struct kfd_process_device *pdd; + struct amdgpu_device *adev; + struct kfd_process *p; + struct kfd_dev *dev; + struct dma_fence *fence = NULL; + uint32_t gpuidx; + int r = 0; + + bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, + MAX_GPU_INSTANCE); + p = container_of(prange->svms, struct kfd_process, svms); + + for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { + r = kfd_process_device_from_gpuidx(p, gpuidx, &dev); + if (r) { + pr_debug("failed to find device idx %d\n", gpuidx); + return -EINVAL; + } + + pdd = kfd_bind_process_to_device(dev, p); + if (IS_ERR(pdd)) + return -EINVAL; + adev = (struct amdgpu_device *)dev->kgd; + + r = svm_range_dma_map(adev->dev, &prange->dma_addr[gpuidx], + prange->pages_addr, prange->npages); + if (r) + break; + + r = svm_range_map_to_gpu(adev, pdd->vm, prange, + prange->dma_addr[gpuidx], reserve_vm, + &fence); + if (r) + break; + + if (fence) { + r = dma_fence_wait(fence, false); + dma_fence_put(fence); + fence = NULL; + if (r) { + pr_debug("failed %d to dma fence wait\n", r); + break; + } + } + } + + return r; +} + /** * svm_range_list_lock_and_flush_work - flush pending deferred work * @@ -897,6 +1234,8 @@ svm_range_unmap_from_cpu(struct mm_struct *mm, struct svm_range *prange, pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx] [0x%lx 0x%lx]\n", svms, prange, prange->start, prange->last, start, last); + svm_range_unmap_from_gpus(prange, start, last); + unmap_parent = start <= prange->start && last >= prange->last; list_for_each_entry(pchild, &prange->child_list, child_list) @@ -1138,6 +1477,12 @@ svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, pr_debug("failed %d to validate svm range\n", r); break; } + + r = svm_range_map_to_gpus(prange, true); + if (r) { + pr_debug("failed %d to map svm range\n", r); + break; + } } svm_range_debug_dump(svms); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index f9b81245b94a..596c881f7674 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -57,8 +57,10 @@ struct svm_work_list_item { * @remove_list:link list node used to add to remove list * @insert_list:link list node used to add to insert list * @hmm_range: hmm range structure used by hmm_range_fault to get system pages + * @mapping: bo_va mapping structure to create and update GPU page table * @npages: number of pages * @pages_addr: list of system memory physical page address + * @dma_addr: dma mapping address on each GPU for system memory physical page * @flags: flags defined as KFD_IOCTL_SVM_FLAG_* * @perferred_loc: perferred location, 0 for CPU, or GPU id * @perfetch_loc: last prefetch location, 0 for CPU, or GPU id @@ -85,8 +87,10 @@ struct svm_range { struct list_head remove_list; struct list_head insert_list; struct hmm_range *hmm_range; + struct amdgpu_bo_va_mapping mapping; uint64_t npages; unsigned long *pages_addr; + dma_addr_t *dma_addr[MAX_GPU_INSTANCE]; uint32_t flags; uint32_t preferred_loc; uint32_t prefetch_loc; From patchwork Mon Mar 22 10:58:27 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154525 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 56AB5C433DB for ; Mon, 22 Mar 2021 11:08:51 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1D41561919 for ; Mon, 22 Mar 2021 11:08:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1D41561919 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D85666E40F; Mon, 22 Mar 2021 11:07:58 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2058.outbound.protection.outlook.com [40.107.237.58]) by gabe.freedesktop.org (Postfix) with ESMTPS id 80B3A89C52; Mon, 22 Mar 2021 11:07:37 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=B9kd4JdA81Z0/etsNrJI35WIvCI4wfe/7t4apbSn3hCtvG97XPf8jyD7CzWkR769ku5/boEn84qhUY/I0lUL9ttdzRhpSsG2WDesg9ZXLoBV/1btpVRszyYjB7GlLICaghnir537xZSIC6Ofk7sOl0aVerw5Ow6yqYDbqx1XfL8FCjrehjYgIcajKdJ6mQ+/M6Xc3QTlvPWG+PJrDM5PX0oSrH+Xk5zHqyspvTKQFavgeIiRWCzge8PPW41usybVvpwnzJo+G+rNsquIQtC6vJHzuGpo2IAsraKLEiN4IFuVi+2w2eIJLE3SqSLoNFVmMZz+Q2J1hyQqWTD6P0H52w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LzrRUwlZRwRXoYlwkSZ919/grqACVrnSHCZhFyCSFRU=; b=mCMX1TcP9N+fJx8hwqiJa5QrZGvcA8TrVjRf7bAYCd44d+PhtVSgG5KtKmxJ/zRnSzP1GOvPTe6vQkHZvUtUuYJxUrbL/EeKS13m84j/QUseFSNAf664IM15FTs02PGT8qXImQ0rfLyoPvKssTotmjnHLML7G3xkKdMUN3h1mPqZd1mLF5bPMU4AFBuPnacIChQH/w22L0sUZqt9zYxJkRcCLRFSws2zvGb1gjQAiFSKtLuhulECpNzXso1/59p6YMEO90xocBEojeERMV06cVlegbmzUmxSlRzRIUDZfZbe4IYlBRFjuTi0ko9kI4R3b17hMQ1gzgy3rZfiNqPyYg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=LzrRUwlZRwRXoYlwkSZ919/grqACVrnSHCZhFyCSFRU=; b=ptxwkF97ayFhvs3kAWmUM5Uz2iU4PE4liZwR+fd2AnIEqNN+Qr8ujq7yzSVhobIo4OahVjWzgHyoLsK0Sagv3ogmiHGrOnrxYh9cdwPK61/oVuSyegPALlF+WFdcy6j+efT/c5LAWbCr9a/ksGLiQoGXh/ls8Ik4mJ/7sZo8xjs= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:35 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:35 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 11/44] drm/amdkfd: svm range eviction and restore Date: Mon, 22 Mar 2021 06:58:27 -0400 Message-Id: <20210322105900.14068-12-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:35 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 91164246-2608-4576-64a2-08d8ed22b2a5 X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6108; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OoEdR5jnHdXNEdyX2QJeBW5nNVHehFq4+CAKU9Ut2tSs/jtQ6IA+4MexYkHSwX1ga8CeA38EVHuI3OJSlQBYtQEKRnc80XTvfB3s/9Rr1hixFASVX48KKR4FKixx65dWjpBeBy37T5V0N95S+fALiipcUFeGG3+U5lHc399uC0LSvhMdArdaUMStqsn52lX7StlS8j7psVlEDGFvoCAFyD43SOMxt+/FmLAz4iYHgP2rQGKRfCfonqxqDQ/8q2qNgaDojaw1dIBiLQd37dPleEk1Hx/u9+CrVeijIXzbPihsvK9Op1mbhQ5YWh5Zn333cc4ydAZdmOZu0+POlbKehgo7a9XflBU8Kj8vZ6HgCA1UvESFe/GBNHOeF9BCtLBuQU48jYf4UH8GNfwv7RtsxEOZlkhC/rreukbCJBiaSuFb1xjScaLXgNT1alDlFLTrVvtreo4zDTwELAbF3kDuXnGSfQidkwO+5IIZfSNr/ci+oAzWgnbDl9032uYipIziH53kea9aQsxRNpX/lV0p9sAV8E8arS3QpLTCwnSknTxGyD7BzXZxZVJivuZyTHolrJ0sLvsU+cQIAJTKxXXpI3GdyQW91iCbB6dk41kWesxj14hSx69zJAQsmKlchGlbxKi7uoSi1U+CrhPRZAVVWA== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: VBDCokWRhdc9PoNx0of/u3ncP4+yP+liSrYtQWny+BLuz9IKaHdGKB4XvFwSjyl4MfHJLaT2/ir4olqO/63R8RRutNBgXDZO3e/18FuLLydJm5JyV9iq481kghiaiYMDRSRzRSMiIKtiQBsLKlRu8KfbDoO8m8PZUDtgEc5UekhWhxCnAChqaa1+Hqb5edJO9akR6XC6BWnFhIrujqUo38tlBkzB1/4yn7EugCmhz+kYuMSIs5l7Xla6cy5lcmO28MlYY9wlNZ7xD6O10cRhZH7lTj9z6Oj6dsE/WFdieikmTg2fPRIaWt236vICnicY0xBgZSURhWWLAgeg2Kwiz6+kvHlpm5KuIei/1DLD2/sB7rsWC8fqifCkfw/j+zbPHnzbsJmNEUaq4N13z5GXIaKoTqvHLuFTP0vVgmJpwHqKeSx+9oXHJomoEmctkgFN9K0ZwG3iL3/3y7zL1lYcb/URiqzD7jqcCuIuAARTem0udtX48yTkbLNCE/mEIZgREL1H/SuIcGLY9sHUUvgf6SRp9mBcYZIRwC6E6A/Ci+HPs1P/VFzw4BKA0BcgWxyVvGnNzGZQhqT2boJbiHFbEtI3Z3DhDL8RkcDIpidnjG6ZOI6VhoxzUyz7Kkd+y+Nu0+ODueIvnW+55uA33OcwK8yT4GJZnKtDT2xwUAQMsNG0yI0La41ljC8mFO8mJXlI8FeHsvVt43fshVddQbaaP+Bf1vpaghUbVrMxheTO+MKzdXcnukSdl9RdHlAj623HzZocWJ5zimzAMM5nNPOlLuDJgt50E7EXaNVkbeQOaF7Az6717obG/zyqYulXuyIggbvrrkTreYDQ8eByxAVKJSPFzJg5NXqt2YxrwpWp+vhqin9/BM0YJNck0xDnRnyQ6FA4xhSxc8yLj53TMjdUDq6ubG9o9LGsMFdEIGevsVq2Ua3yFDVTyWVVD6rVgZ/u5SQf8A4ZoyGmzfXfvHGMzoEey2ziFwhQhrHZ3HtyAWsA/GG50b5PlltME6G5u7bHJQNA/rYh88XiSn6F+NpI8OwcbTNv6QqRw0QCrI/2NC8mVLni5B/G/5Adq0/XZgERye/9woVfIbykkXOPHRlUxZQkJBVQnA5tLAT/RnmjzA0QWMXOO53Su37mAlGtlynG1q6Jg/zuCH+xjDDEK0dsALEWJ3T2/1D3eF7B1G3xsagaTCQGNyUuyQr6fjcas/epvkA6273aZ/puRgjdFDxjN4vib3ME92ARSnbXPWpHtWODohyCITsxMoodMmXhI/Rkk1rHrtBj6CFDhX0MSQ7H3I1TSO4YI67frla0IdshejQ2xuYB+F57qA8QYCjjCtEW X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 91164246-2608-4576-64a2-08d8ed22b2a5 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:35.7010 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: rN7HGT/wst0qv2jkOiiO0RRP3YG2yp36jWkb1JiooFkwdz/jQ2tWm5us09g06TJy+BVXsO7DNlImD83cB8+mPg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" HMM interval notifier callback notify CPU page table will be updated, stop process queues if the updated address belongs to svm range registered in process svms objects tree. Scheduled restore work to update GPU page table using new pages address in the updated svm range. The restore worker flushes any deferred work to make sure it restores an up-to-date svm_range_list. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 2 + drivers/gpu/drm/amd/amdkfd/kfd_process.c | 1 + drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 145 ++++++++++++++++++++++- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 2 + 4 files changed, 149 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index 4101f5341ec9..7d70af26b5c7 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -738,6 +738,8 @@ struct svm_range_list { struct work_struct deferred_list_work; struct list_head deferred_range_list; spinlock_t deferred_list_lock; + atomic_t evicted_ranges; + struct delayed_work restore_work; }; /* Process data */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index 4d7a67141190..321895d7555a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -1064,6 +1064,7 @@ static void kfd_process_notifier_release(struct mmu_notifier *mn, cancel_delayed_work_sync(&p->eviction_work); cancel_delayed_work_sync(&p->restore_work); + cancel_delayed_work_sync(&p->svms.restore_work); mutex_lock(&p->mutex); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 1244ba380292..55828d6fbea9 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -21,6 +21,7 @@ */ #include +#include #include "amdgpu_sync.h" #include "amdgpu_object.h" #include "amdgpu_vm.h" @@ -28,6 +29,8 @@ #include "kfd_priv.h" #include "kfd_svm.h" +#define AMDGPU_SVM_RANGE_RESTORE_DELAY_MS 1 + static bool svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, const struct mmu_notifier_range *range, @@ -226,6 +229,7 @@ svm_range *svm_range_new(struct svm_range_list *svms, uint64_t start, INIT_LIST_HEAD(&prange->insert_list); INIT_LIST_HEAD(&prange->deferred_list); INIT_LIST_HEAD(&prange->child_list); + atomic_set(&prange->invalid, 0); svm_range_set_default_attributes(&prange->preferred_loc, &prange->prefetch_loc, &prange->granularity, &prange->flags); @@ -282,6 +286,9 @@ svm_range_validate(struct mm_struct *mm, struct svm_range *prange) r = svm_range_validate_ram(mm, prange); + pr_debug("svms 0x%p [0x%lx 0x%lx] ret %d invalid %d\n", prange->svms, + prange->start, prange->last, r, atomic_read(&prange->invalid)); + return r; } @@ -886,6 +893,134 @@ svm_range_list_lock_and_flush_work(struct svm_range_list *svms, goto retry_flush_work; } +static void svm_range_restore_work(struct work_struct *work) +{ + struct delayed_work *dwork = to_delayed_work(work); + struct amdkfd_process_info *process_info; + struct svm_range_list *svms; + struct svm_range *prange; + struct kfd_process *p; + struct mm_struct *mm; + int evicted_ranges; + int invalid; + int r; + + svms = container_of(dwork, struct svm_range_list, restore_work); + evicted_ranges = atomic_read(&svms->evicted_ranges); + if (!evicted_ranges) + return; + + pr_debug("restore svm ranges\n"); + + /* kfd_process_notifier_release destroys this worker thread. So during + * the lifetime of this thread, kfd_process and mm will be valid. + */ + p = container_of(svms, struct kfd_process, svms); + process_info = p->kgd_process_info; + mm = p->mm; + if (!mm) + return; + + mutex_lock(&process_info->lock); + svm_range_list_lock_and_flush_work(svms, mm); + mutex_lock(&svms->lock); + + list_for_each_entry(prange, &svms->list, list) { + invalid = atomic_read(&prange->invalid); + if (!invalid) + continue; + + pr_debug("restoring svms 0x%p prange 0x%p [0x%lx %lx] inv %d\n", + prange->svms, prange, prange->start, prange->last, + invalid); + + r = svm_range_validate(mm, prange); + if (r) { + pr_debug("failed %d to validate [0x%lx 0x%lx]\n", r, + prange->start, prange->last); + + goto unlock_out; + } + + r = svm_range_map_to_gpus(prange, true); + if (r) { + pr_debug("failed %d to map 0x%lx to gpu\n", r, + prange->start); + goto unlock_out; + } + + if (atomic_cmpxchg(&prange->invalid, invalid, 0) != invalid) + goto unlock_out; + } + + if (atomic_cmpxchg(&svms->evicted_ranges, evicted_ranges, 0) != + evicted_ranges) + goto unlock_out; + + evicted_ranges = 0; + + r = kgd2kfd_resume_mm(mm); + if (r) { + /* No recovery from this failure. Probably the CP is + * hanging. No point trying again. + */ + pr_debug("failed %d to resume KFD\n", r); + } + + pr_debug("restore svm ranges successfully\n"); + +unlock_out: + mutex_unlock(&svms->lock); + mmap_write_unlock(mm); + mutex_unlock(&process_info->lock); + + /* If validation failed, reschedule another attempt */ + if (evicted_ranges) { + pr_debug("reschedule to restore svm range\n"); + schedule_delayed_work(&svms->restore_work, + msecs_to_jiffies(AMDGPU_SVM_RANGE_RESTORE_DELAY_MS)); + } +} + +/** + * svm_range_evict - evict svm range + * + * Stop all queues of the process to ensure GPU doesn't access the memory, then + * return to let CPU evict the buffer and proceed CPU pagetable update. + * + * Don't need use lock to sync cpu pagetable invalidation with GPU execution. + * If invalidation happens while restore work is running, restore work will + * restart to ensure to get the latest CPU pages mapping to GPU, then start + * the queues. + */ +static int +svm_range_evict(struct svm_range *prange, struct mm_struct *mm, + unsigned long start, unsigned long last) +{ + struct svm_range_list *svms = prange->svms; + int invalid, evicted_ranges; + int r = 0; + + invalid = atomic_inc_return(&prange->invalid); + evicted_ranges = atomic_inc_return(&svms->evicted_ranges); + if (evicted_ranges != 1) + return r; + + pr_debug("evicting svms 0x%p range [0x%lx 0x%lx]\n", + prange->svms, prange->start, prange->last); + + /* First eviction, stop the queues */ + r = kgd2kfd_quiesce_mm(mm); + if (r) + pr_debug("failed to quiesce KFD\n"); + + pr_debug("schedule to restore svm %p ranges\n", svms); + schedule_delayed_work(&svms->restore_work, + msecs_to_jiffies(AMDGPU_SVM_RANGE_RESTORE_DELAY_MS)); + + return r; +} + struct svm_range *svm_range_clone(struct svm_range *old) { struct svm_range *new; @@ -1256,6 +1391,11 @@ svm_range_unmap_from_cpu(struct mm_struct *mm, struct svm_range *prange, * svm_range_cpu_invalidate_pagetables - interval notifier callback * * MMU range unmap notifier to remove svm ranges + * + * If GPU vm fault retry is not enabled, evict the svm range, then restore + * work will update GPU mapping. + * If GPU vm fault retry is enabled, unmap the svm range from GPU, vm fault + * will update GPU mapping. */ static bool svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, @@ -1286,9 +1426,10 @@ svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, svm_range_unmap_from_cpu(mni->mm, prange, start, last); break; default: + mmu_interval_set_seq(mni, cur_seq); + svm_range_evict(prange, mni->mm, start, last); break; } - return true; } @@ -1307,6 +1448,8 @@ int svm_range_list_init(struct kfd_process *p) svms->objects = RB_ROOT_CACHED; mutex_init(&svms->lock); INIT_LIST_HEAD(&svms->list); + atomic_set(&svms->evicted_ranges, 0); + INIT_DELAYED_WORK(&svms->restore_work, svm_range_restore_work); INIT_WORK(&svms->deferred_list_work, svm_range_deferred_list_work); INIT_LIST_HEAD(&svms->deferred_range_list); spin_lock_init(&svms->deferred_list_lock); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 596c881f7674..2a76e89dec04 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -66,6 +66,7 @@ struct svm_work_list_item { * @perfetch_loc: last prefetch location, 0 for CPU, or GPU id * @actual_loc: the actual location, 0 for CPU, or GPU id * @granularity:migration granularity, log2 num pages + * @invalid: not 0 means cpu page table is invalidated * @notifier: register mmu interval notifier * @work_item: deferred work item information * @deferred_list: list header used to add range to deferred list @@ -96,6 +97,7 @@ struct svm_range { uint32_t prefetch_loc; uint32_t actual_loc; uint8_t granularity; + atomic_t invalid; struct mmu_interval_notifier notifier; struct svm_work_list_item work_item; struct list_head deferred_list; From patchwork Mon Mar 22 10:58:28 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154477 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8BEB4C433E0 for ; Mon, 22 Mar 2021 11:07:50 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4DA116196F for ; Mon, 22 Mar 2021 11:07:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4DA116196F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5183D89ED6; Mon, 22 Mar 2021 11:07:39 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2058.outbound.protection.outlook.com [40.107.237.58]) by gabe.freedesktop.org (Postfix) with ESMTPS id C279B89BF5; Mon, 22 Mar 2021 11:07:37 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=K+3guLpkUT8/jrih/lYX+PQk5xV1FcIy4KzyCVZ09TBOH9BTS6s7VWY15dCkzg1vJ6JSnsgmsdZCuywUkLNva7fk60dIqKj1aerwuB+LP7VYmU4N79dpqoQf2+r0Hj2hEY4xk1+ac1dUHVds68hx/dc/MruSr+ocyoFYediyB4on4009+F2XpwJRdqBVHHyihwK/9CRm6VpqKCKS5uqXA0a/FyUsCGGH9yy4mXWQqZsnHt67HMfmyerflAUtsDlwhrANBoGTVNUCxuI/iiFkfwQ5QmLPtzTtbL0EnsHUN2IZwIi2HOoy+BCd34SJ56ZKU3RLFgG7qu/JZZWwJZokdQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kN5YAv8bz3xTUe1pbWt/af+RIwUCwjeiVHoVzVkc1nE=; b=fAHVQ1sC8lSYNkyNk4mXXpyrsmnJDf4rUudh0RWoK0PQeiRAAtSjA+M4XrS5jqTa0KQIvV39E7cNhtvVOieH/ep41/UqeAY3uCYu6reZghwls0RkaGh2TeqREu0ejEHj142BaqAfkKIQgLhsYVsme4iQFSCTmo+fAVfQybW9X4Jgn5GAjp6ivIKgSSDY7afND6OAeOm9YFcaVJjHNW4KySgYCmCTvA4YXn6ectC8Y9L54tdzG/V3JLryQvEfv9bmtmZJvWNdiCYo6TqjyqJqw9yItEeo+uCO/mQ6YPX4UJ1ATYxIq7n9kop6jjqtZRmTE4WRU+yiM3khBOvplyuxQw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kN5YAv8bz3xTUe1pbWt/af+RIwUCwjeiVHoVzVkc1nE=; b=pJ0m6MUqrdUeFQtSeYFOBlB4IAOMGy62s1QkIvJuHbloAC2RAtd55gHBAHnbBrGa758R+JGtJZHOI9nsnqy498nNnybHuZ5DeWwaaa0CEqBYHaAd6QAumBI05/KRWLR4sAdPnZihN01uTUJRlTxi1D+vpbetWqQ83/LhNtAVy9U= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:36 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:36 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 12/44] drm/amdkfd: add xnack enabled flag to kfd_process Date: Mon, 22 Mar 2021 06:58:28 -0400 Message-Id: <20210322105900.14068-13-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:35 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: b538c24e-2d30-4c43-511a-08d8ed22b2fa X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IuiEOCXBR+mhP6IU17RgoFWx3SF3NgXsxtvrlmJi9KLiSWPhBY1WXM59S0ugMrgm5u/qAS28J4ubGV6EnN2XlSu9z/b4wesDiQfQH9gpt8th8bwMHqC590jcKV4NJ/mHoKqBpHfaYpgY+ruRcgYNJNfenvUC1/3xi7pB+Bgt/vYvx9Zl8VuKxByB9v5qDNsqeZ/aMp1N+eZKKQKPxEjM6IWxdTZklMdiq9+BPY5iHTm4XPALMi4FS+0wJmEF7ZIOC3J7SPOLzQPgWGeMNFnLXiVqpfB7aiG79YBno5Vcjnb1VUig0HRMhokaosVo0/6N5UP/3aMvSv7fsR0nV7TAjxLShv5BZ+PZZyLtMRa1yHfL3xTRNfPIHHoUscI6g3iYSanMkYT4b//HeMeO3V7ItjzBuYN2datyKwhbkfuddQdBRjsKwycs83UBAOXS+pEXZCSlO5bCbkCsJjnFhJSaXa3e0l8l+/LBuUzQLTu4KBWwdK1HtYb+6/pfdKj2ACTVjIR37VhcL+oJApOvguiEX3iS8brRI9aKEQ12ypBfaNq26LGzodTGhU10lPBeSiYMtnBJdG+PR6uKn7gDtj0NbBDT7lWC3B7YehrZBzlFDkCj959b3vFH9Lh+JhlYYs7tlKEk+b1AiDu0AVIP0I8U0JENrD/0o9rvW8SXFJBsseY= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: J3sD/Vqs2B20l3DfVQJoKlMKHmllcMBvrMo24CuqR6EA76YB/esiLqzCfSQlKYtYFcRj3uLRG8IfAJDbDR3S17WYwAYbEKjrbCc/zuk9Gvnk11uTgpwDALyr+GbNO88VLL5W1IbUGiAKDKHXzndHoFWIcoKkwZnH0PYSogctH9kGCOZwZWlvRLyJCyDNTwr28zyfyPXLJOQkXBtl96AsyeGwvvE6fHvM2zVjsfebNiPcBKnLMfrUZX+228TJD0JgAFncIWHRLLJhTpOF+/FzfySrw2R6713/ylZGlpC1YLGEthfxrnrKbxOm87pncXjI4Wo7d+GmcmGIRNZX57rlBhBWzoE4TkkOZ7SWjFLpMkKBpDMbdaCBaAJK6eOBZUu6bEeAH9F+m0Qy/SbxfES4AYBNzRnV7L0N9bGncGJKxgXB3g1a8JBO1jH6NAfXKIRd9nXPrMLBwM2JY+rq3vpl39jvvygPr8KdUH8opptb/202yzYE4+ilfo95kP8hBEcP7qnPn1/VILvHMV1smKbSCByCRMJXdyRj1+jJ8kEFbfjwYT8BJuVGMK2NFwkgnr25Q8vR0ThZvDiAXPiWpInZKiFVOFaLhWhxrH9zbMyhTVKsYJi1FB5atzpMKf1/qxpqjOqKPFDB12+TRcC+qFzBsIlODIyswFntUNMgdIQSTlHXNqilyZ8i5qfF5nHPlDk/APoeLhfS3qnbqYgFI24quGgvCjzecHgfhXQAKV4t3dLXldesmrlkPTTJeb7RBGeq2nHir77aOerxRk0ipcmOUuz/qRCRDKojkUXId7wksyfBUDIw0r8e7radaBafQ3+HYAa57BBY+Ta1CQi+1BHaSnocwTXEnl1Jn0GvG83ir9FUOQVzr7OIhPJVFli0lnwLDRRXJI5lskWaWgx0inUINsSSqGC6HIyLqUa5OuFc5aKXdE6DBW2fdOTPHa47juXJNzgDEnzVqKMvpNoFa8XuI+6CIClsJvCjxh1C8dur86JcdL4ylzo/Lmx7AIx0UuP8QDScFLqAMlCwvZpUDhL7GLZbUUDf283yLp4oy7dWySOXKyd664mPIZgFnkvZFPnIfsJLgp0x2VShnVG1ObeS1V1WWxxWPagQdLtCqdVj5CAivY/yw8IW5+doZ9p45rnSobWZxV/MlgsJiOc0u2Y1wiBflwS2tjYjWXNdzLRm3FzgmgNh63gFTn0GUogdwUzALGgEZqzKDsVbxReoRLA9ZNMi8L15bi5P1cSqBeWxI2W0h4jBMyjkQ3BARv/NjOuz6Fb0lRhq3Q+kFf4o2Z0Lw9TVu4CVe7WXmoSXsKlVnS4RVZqVHhwvjNIloDWBzU0J X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: b538c24e-2d30-4c43-511a-08d8ed22b2fa X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:36.1759 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Nungjb8eOIRtnHy2iqXymzLPmc1AI8SJ3zBybtDbPyBv+LdDnd+uuj3046QoP1StvhmfEI1r18JQpnwILtexFw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra This flag is useful at cpu invalidation page table decision. Between select queue eviction or page fault. Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 4 +++ drivers/gpu/drm/amd/amdkfd/kfd_process.c | 36 ++++++++++++++++++++++++ 2 files changed, 40 insertions(+) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index 7d70af26b5c7..d66430740e52 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -824,6 +824,8 @@ struct kfd_process { /* shared virtual memory registered by this process */ struct svm_range_list svms; bool svm_disabled; + + bool xnack_enabled; }; #define KFD_PROCESS_TABLE_SIZE 5 /* bits: 32 entries */ @@ -877,6 +879,8 @@ struct kfd_process_device *kfd_get_process_device_data(struct kfd_dev *dev, struct kfd_process_device *kfd_create_process_device_data(struct kfd_dev *dev, struct kfd_process *p); +bool kfd_process_xnack_supported(struct kfd_process *p); + int kfd_reserved_mem_mmap(struct kfd_dev *dev, struct kfd_process *process, struct vm_area_struct *vma); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index 321895d7555a..c8479f6bd68c 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -1193,6 +1193,39 @@ void kfd_process_set_trap_handler(struct qcm_process_device *qpd, } } +bool kfd_process_xnack_supported(struct kfd_process *p) +{ + int i; + + /* On most GFXv9 GPUs, the retry mode in the SQ must match the + * boot time retry setting. Mixing processes with different + * XNACK/retry settings can hang the GPU. + * + * Different GPUs can have different noretry settings depending + * on HW bugs or limitations. We need to find at least one + * XNACK mode for this process that's compatible with all GPUs. + * Fortunately GPUs with retry enabled (noretry=0) can run code + * built for XNACK-off. On GFXv9 it may perform slower. + * + * Therefore applications built for XNACK-off can always be + * supported and will be our fallback if any GPU does not + * support retry. + */ + for (i = 0; i < p->n_pdds; i++) { + struct kfd_dev *dev = p->pdds[i]->dev; + + /* Only consider GFXv9 and higher GPUs. Older GPUs don't + * support the SVM APIs and don't need to be considered + * for the XNACK mode selection. + */ + if (dev->device_info->asic_family >= CHIP_VEGA10 && + dev->noretry) + return false; + } + + return true; +} + /* * On return the kfd_process is fully operational and will be freed when the * mm is released @@ -1232,6 +1265,9 @@ static struct kfd_process *create_process(const struct task_struct *thread) if (err != 0) goto err_init_apertures; + /* Check XNACK support after PDDs are created in kfd_init_apertures */ + process->xnack_enabled = kfd_process_xnack_supported(process); + err = svm_range_list_init(process); if (err) goto err_init_svm_range_list; From patchwork Mon Mar 22 10:58:29 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0EB64C433E5 for ; Mon, 22 Mar 2021 11:08:19 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C496061931 for ; Mon, 22 Mar 2021 11:08:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C496061931 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5818D6E0CC; Mon, 22 Mar 2021 11:07:51 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2053.outbound.protection.outlook.com [40.107.237.53]) by gabe.freedesktop.org (Postfix) with ESMTPS id C818389EB1; Mon, 22 Mar 2021 11:07:38 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=nFqQLNdOITBeUS8/1EjwVRx2Ndq+i9tmGDsKnj2e7r3nMlCTjVV5nxR12drgJfEog754uQgNSKpxjZk6+3Dr2nOtMk6a3QV12y7Auu+5qcwv+CPiD6cSxnFs6Q1kLOnJap1EurhtwepidPXHCFY9RuK3Z6psM83LtetFa1g5KFELlJ/IyTAkkjEMilNZOrK24faG0312ANE+nImRly6zqyDm+g+2NJdbgNVVcsVtYh0G3zmwIzjinS9L6mP9wzLIj/ppAaXN6Pnq3FstXT5EmlUbYwQixdGRi/vZ5JaniiwsRxS3FNNXiu15xGK327W24hdiIIIW0r2r8ahNPk5TKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7arCvRa/0V2Nc4Fqyd+wz49NawQlux4qsKns1D5belg=; b=k6pKBfGH0/FlXr5OaJNhy5v6b1Ye3PdCpHL4gPvwdFpat8UiYBu3eNIvgc2FzCveUduUWBx0cXRa38/IoRGdx1NmQtp1xeGUqb4Gb0Mzf1ylxgCVIkM4UfefORA06R0aMUsdix+yasSiMcqhOPPxnqgOx51+2qXG6mitvz7p1RCsKGyX0C0Cuc/4kbQPjrWLVt93XNKnf2PzthA1RiSTVekacSMKEVq3uY7EvLVj7/w8VLMqj/4GYL+amFapSuDhjdsh9K5LLxANZhvU1E4l7uiB/dDGrVnv2O+O3uQsSO7NyhzDb9x4RJMHqaR1MCOvsfYfXMcCe06dYpJtKmbz0g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=7arCvRa/0V2Nc4Fqyd+wz49NawQlux4qsKns1D5belg=; b=lx0yjtji/2YHpCoAAnXXfkTBYO+B3IpYKGY+qHFzTDC+z8z2jtM+sAy6S484tYHZrXo+mbzfbkUm7w/PKF3UepIJeOkgkha8eoQTob3NZ+b1eW7M6XMPRdP3BxytTy4rd2B/ks6PTVC7bwqDC3Y/Uiw5o/Ei0skewxlyQl1ypXU= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:37 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:37 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 13/44] drm/amdkfd: add ioctl to configure and query xnack retries Date: Mon, 22 Mar 2021 06:58:29 -0400 Message-Id: <20210322105900.14068-14-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:36 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 5d4e24f4-70bc-42b5-5057-08d8ed22b33d X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6nFQfppPZH20OIx2wLX2+EjBSJ7xQRpvnWU+8sgbTXV3vdzEt2BgSmZlutBtJ7YT7SdXoRIDnzxwe5r7Lj1/m7WSftZvyzaACA5O2sF99AOU4QITyfvFMyq6SNbGHggv04/Ov4z3YWJSiFu9PYa/tGQLuYq1IcKeDCFn0ysARabddtgwonGFM6peud2Xu202cbR92mwjy/yyj1eF6/1HmDFTAJrFOZgf/qyxu4r6wImqO/fS0XPOPk8rfIiA3VNNAv7vzNIKXV6e+6Z7USf1qiU8XSEvdyuBIK6VURpbqud78MN11oU1um8+/MWSMe6DpG7wN/AZdTUfOARzJCLqChEo+KnKl1yqLxH4Ybzc1K5jR3tjK/Vc24K0Gem5k1U8v4YNzeLMdS37fFDY/YOwidTvP/s/GmTs/BgNQ3eET31e5813NLRdQJ1jq0wwobXMbHIDt18m6wJft5FxDrdYHTgouYLlZvS8wM/1omHT7ETraj5Nw3s6IX2SEUWLrUgolYSq+5HUEF9BEDuQpaokICZdCZT1d089Y+EtDHZXUO5d8UMW4xWGerdnOBisb800t1RdxOf66ZTE4GcbDG1dUmIGwbLppn+uXJeoVAt/fRIHUyF+OpRuy43yGgz7KsqkpBmlyObcOMvuZfBGyLAvNg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: igCJjMAU7+MrQ7AUyN3askbMxIxj6gFWEJN2YsXssiaWNNyt/dkNpTX7zY7rjr2n4KRmpRtbtktO15DAFsQwB6vMcmXPp9TyTYXbV8Tq0qLmURlPTphoGQN8abS9peNGrZHwUvq9Fm38bZnBx3zLAfFJDx9e5lJo0mXBKtN4jyT7NY7ggPeWe4wi4t4xsv/oCHsySxEPFFt2QbKrnHohy0jhmUludJ888vkKLJopOmz/+mJXPtk9Ieq6TDv6usBIwxFT4g7MxLGEHg5/uDh6KziOGwK06CWFr5AzlyXhrV3b9Rl20QdeviiFrMejLibcXJcqeS3WemZQE2yAaVEqoOOytVsaBCjsap1IIPyjHSlvYK+8IZ6G2qskL9twxQLsaTpnffVsDXg5R5VpjrQ51lhva2CAlZW4yvHVp4kLRwc7Wy1Giw5hw6jIfkLKZHaLg2c5/iyVwojo0XBZkgK+Y7so1irpzMDr0IMURbLYvIc2JcEft+3SXo6FTLDATHYYbn2/X7KLg4vx8aI2lfqzsSKtzR3UF6POyufBsd65NTKAf5Q3Cj+VHVKzxKIAVzaGY0KYUv1LOFvnCNR4U5kWKp3FFz0AP1S8OY1q7MM+jpv/0tiAXy9gEoYbqorDUJc2a3MyG/HyXpPA43s0T0dFYMe0q+2C5p2IvNCtvZtUaqzEMyMbZ6nROIXsBPmz4xEWSSj8bm2/QAL82rzrR/J3pt4UtqZFgm23h/sCMDVZGt7pWGhNn185bPIcRsUU7GyGLJRyBPSwePFQTOyVfgmN2xp5/U57ZGpqCMF5z+yhXUemp+s0BScpjcx2hh31llLOVvqviE0UDf3MnVDHjf9TyIGqximc2iXwx5IZBICdRVpZB7yCWzIbb4Kqnpo3Vn80hxmtZEgoAJtpV3vIefcTJf7rwEyITvOqswW3RG6TRnYQHy1hYkwdsPMpRSWLTnwgFfz6apMDdBx2o96dabzOR7FRSusB546r3Q+HUq7TcYYDNx5FemmOHVkuJFHqXtuSC7YEP8M2LpaEBQFOS2g3R83LKsuwwqBve1e9uiIwyQdFQSXPkr/ezfMJKplk7v4EE3tRm8KOEeCfVbeRZ7YGU62QK0ydsob/T3nC1tvkvBbDyU1UqsFe3U3GKu9ORCZ312ZUIW/zpqV+Wl+D2JUV27jxeQnVnt8Llq+4WPnE6mskvIXQEtpWSUvaQo4YDPy4PlLSRsbQh8hx0XtV+9AhYWqGZN5PSgkFhfUQE3o2ReJ2VFLItk7DxnNKxby0iM9sUCF5ZG+RsEtfwOhyjEAJ1EbDirBNXqZpXP9grbbck0VrFZCmFJms5VOIHRskscIt X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5d4e24f4-70bc-42b5-5057-08d8ed22b33d X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:36.7304 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 8tAGyVT4MGNmbv7wnMPtS7G7ICFgv4hD2qdNSB901QBe9vHdMDnd1Y/oGbTutVeTtaQQPoxd33/f9vLlvQMk1w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra Xnack retries are used for page fault recovery. Some AMD chip families support continuously retry while page table entries are invalid. The driver must handle the page fault interrupt and fill in a valid entry for the GPU to continue. This ioctl allows to enable/disable XNACK retries per KFD process. Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_chardev.c | 28 +++++++++++++++ include/uapi/linux/kfd_ioctl.h | 43 +++++++++++++++++++++++- 2 files changed, 70 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c index 9511826ac8ae..63eee7ef3355 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_chardev.c @@ -1743,6 +1743,31 @@ static int kfd_ioctl_smi_events(struct file *filep, return kfd_smi_event_open(dev, &args->anon_fd); } +static int kfd_ioctl_set_xnack_mode(struct file *filep, + struct kfd_process *p, void *data) +{ + struct kfd_ioctl_set_xnack_mode_args *args = data; + int r = 0; + + mutex_lock(&p->mutex); + if (args->xnack_enabled >= 0) { + if (!list_empty(&p->pqm.queues)) { + pr_debug("Process has user queues running\n"); + mutex_unlock(&p->mutex); + return -EBUSY; + } + if (args->xnack_enabled && !kfd_process_xnack_supported(p)) + r = -EPERM; + else + p->xnack_enabled = args->xnack_enabled; + } else { + args->xnack_enabled = p->xnack_enabled; + } + mutex_unlock(&p->mutex); + + return r; +} + static int kfd_ioctl_svm(struct file *filep, struct kfd_process *p, void *data) { struct kfd_ioctl_svm_args *args = data; @@ -1869,6 +1894,9 @@ static const struct amdkfd_ioctl_desc amdkfd_ioctls[] = { kfd_ioctl_smi_events, 0), AMDKFD_IOCTL_DEF(AMDKFD_IOC_SVM, kfd_ioctl_svm, 0), + + AMDKFD_IOCTL_DEF(AMDKFD_IOC_SET_XNACK_MODE, + kfd_ioctl_set_xnack_mode, 0), }; #define AMDKFD_CORE_IOCTL_COUNT ARRAY_SIZE(amdkfd_ioctls) diff --git a/include/uapi/linux/kfd_ioctl.h b/include/uapi/linux/kfd_ioctl.h index 247b57baa94f..3cb5b5dd9f77 100644 --- a/include/uapi/linux/kfd_ioctl.h +++ b/include/uapi/linux/kfd_ioctl.h @@ -597,6 +597,44 @@ struct kfd_ioctl_svm_args { struct kfd_ioctl_svm_attribute attrs[0]; }; +/** + * kfd_ioctl_set_xnack_mode_args - Arguments for set_xnack_mode + * + * @xnack_enabled: [in/out] Whether to enable XNACK mode for this process + * + * @xnack_enabled indicates whether recoverable page faults should be + * enabled for the current process. 0 means disabled, positive means + * enabled, negative means leave unchanged. If enabled, virtual address + * translations on GFXv9 and later AMD GPUs can return XNACK and retry + * the access until a valid PTE is available. This is used to implement + * device page faults. + * + * On output, @xnack_enabled returns the (new) current mode (0 or + * positive). Therefore, a negative input value can be used to query + * the current mode without changing it. + * + * The XNACK mode fundamentally changes the way SVM managed memory works + * in the driver, with subtle effects on application performance and + * functionality. + * + * Enabling XNACK mode requires shader programs to be compiled + * differently. Furthermore, not all GPUs support changing the mode + * per-process. Therefore changing the mode is only allowed while no + * user mode queues exist in the process. This ensure that no shader + * code is running that may be compiled for the wrong mode. And GPUs + * that cannot change to the requested mode will prevent the XNACK + * mode from occurring. All GPUs used by the process must be in the + * same XNACK mode. + * + * GFXv8 or older GPUs do not support 48 bit virtual addresses or SVM. + * Therefore those GPUs are not considered for the XNACK mode switch. + * + * Return: 0 on success, -errno on failure + */ +struct kfd_ioctl_set_xnack_mode_args { + __s32 xnack_enabled; +}; + #define AMDKFD_IOCTL_BASE 'K' #define AMDKFD_IO(nr) _IO(AMDKFD_IOCTL_BASE, nr) #define AMDKFD_IOR(nr, type) _IOR(AMDKFD_IOCTL_BASE, nr, type) @@ -699,7 +737,10 @@ struct kfd_ioctl_svm_args { #define AMDKFD_IOC_SVM AMDKFD_IOWR(0x20, struct kfd_ioctl_svm_args) +#define AMDKFD_IOC_SET_XNACK_MODE \ + AMDKFD_IOWR(0x21, struct kfd_ioctl_set_xnack_mode_args) + #define AMDKFD_COMMAND_START 0x01 -#define AMDKFD_COMMAND_END 0x21 +#define AMDKFD_COMMAND_END 0x22 #endif From patchwork Mon Mar 22 10:58:30 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5384CC433E0 for ; Mon, 22 Mar 2021 11:08:26 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1ADDA6196F for ; Mon, 22 Mar 2021 11:08:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1ADDA6196F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B19E489FC3; Mon, 22 Mar 2021 11:07:53 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2053.outbound.protection.outlook.com [40.107.237.53]) by gabe.freedesktop.org (Postfix) with ESMTPS id 02E5089EB4; Mon, 22 Mar 2021 11:07:39 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=EtRCt+wlZJvLYYOgvH/te3oEpWHbqngWAeEJCt2sVsimTmhRMb0zPqKxFh43owxV9y6lItV70Cc2HREMtBF+4SnTE+IrXwoSrxwGswT6Qp6xaea6KYejBkmGLmrxedHbectLCj50508kv6GOrd6emFKBJMH9JGCqAIyqeKLq+NOeT6PzKGpXKkjtCeyz6WQgTqpieoXK2BU5INhE7x/J8nF7kO0w7XzP5hCIPQXo36qpJkG1KOPP0ZslozYpRnR9/sjX1wRut5Dobtj9meHQMGmZiKz+0h8+88XSOWID+x0QMZTPvK4JoX6O2sVleAiDUtSK/HqmmlpH1Nj4tARepQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mACx5Cqe/3Man3BKcrkGl5gcwrFXhOoREcT9169rPXA=; b=ZOxeIRfnx8jhmH/A0Zb1gqB5l2aVDf57QsHid5I+DnVCIPD+ci4bDwypEtJIkB0oI3Q7rxB0F/JgZ4jzU8lEozFl3tJwebMqtfmDxTGJeDisaPOT5kwVe4k040ZPBmYOHF+N/5NQ1haViq0eRxkkt866vo9Lr8K4qilKsUqL6j9NsIwJ4hUlPekCRDdlTReBMqfimMwgi546KPX1GMK5PrIyuM3+pgzCdyYOiqwwmNUcrvB0bBpHbDMN/gbUjx0qhlTXF+vUT/dYdAUC3GDlvpJG4w+KO0NzrHtrpOber9VlByAbY28qExkZixctETSEyvUL5psxyFnJH1JlJSjh9g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=mACx5Cqe/3Man3BKcrkGl5gcwrFXhOoREcT9169rPXA=; b=btzn5JC214g20KsIMIppA7GRuVHCbSLfZRS/fGwkG2eXzobuA53IOK3yiUkAg583kv6NOIkS+X98IbBS3vzQHq21vqWtK1o2lzcKgRgPmd2xMkGwnaeeOZs/GnoWgc88fIyGHfWzPH0HZbataNYq7rhovIPj0YSWoU3zeiL0vm0= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:37 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:37 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 14/44] drm/amdkfd: register HMM device private zone Date: Mon, 22 Mar 2021 06:58:30 -0400 Message-Id: <20210322105900.14068-15-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:36 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 41f19750-44b0-4115-05ab-08d8ed22b391 X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 7f1Yly11KNkjejkB6AMYKNO5CdrwqTN6VtUceys9iWOjwClMW0psKAb75B0AErtHOXprtJHMdvDfvr8bKGqTQtUwvqw/dRPA8qcpQF9effQoXscQkOdP1SkPaIJ+ZEPDFdxBPtnLnCGKz9K50Ej01fYFcQdklYw5ZmBV0IrY6Hb5k+l5pmCcqjBoifC2S+L9WJTjDytnawFopZKu/pgnKjabat8/xrdhH2cmdRcuj877xpeSM0Sfl8mgL1Oko3HpyszPYoWpPm6j393bl9Oh5aoFamhHdTYEJcVai8QQ6vpME62+jxQjANU6rLdd0OtlB+Pyn/q1DIWztqlCdkYGGPf32ZpDbcGeftUHK7LGXjd8peBrrpoq36u/s6oGUkVP0zF1Qu8iABzwDrA6+gTGbMeQtqFbPhLg2x9KZvPTnNVaPAk6jCIEK6BSzJL8KltOc7B7g1bnPqyyYqnFLnpSQB888aWGqX3sJ7KeT3szlpyULDVstfEwBWBjOf+cVy4yru+/C0+rh9Z7+RvrU4KJfxoeUr/dvu6FYBN5ywQQEz48t/rb1xM8SUkgWVzGYB4QVWLezHawbW9AVmA4SfHMNuJFYNUdI+fcuSy/yGyy2jWASzJLvIKx3uC4L3uswv30rIEC1bgJatucITbOd2c9fWtT3kfQMfgKTA6jHS7VS2Q= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: s4zGFi/7w/h6n8SSX5AVt+NTUlxZ87hmfAb6tJVy04Jqi9X/DgYAgKHS8HlO+VvKQ5lSrPGmXl+e0OfVGdhOmRRuutVAfUjrHBQ1rDnsCHCGRPFdIJLVD38QpcMI0epvMjbqNTuXX4NEFewkeMpDs6xYXBsCbqHyRkjYrKPpouZ/oseJKZ/SKzm2PG6ypqS+SoRnsyKGJcgebamJTiKU8qv0fXo8IPrHqr9aqA50ZxRW1JB/8vSC/gaFKV31RN88VwoVTFvRIZIEHbD1VXVXp3Fnk1FA2vR9XesIqBxZ1cHIYWUF6m5ZhyrAxZvXqPZstOQv81s+56bLdkXDxuzuwR18/fD1Zy6bp3wnsdn6H6zNilPIGJYN2SnVd3p68uBEEoeLK6B5jOifH3tF6KL2vWEIosTK26m+Ni4eOk2261QTtUr3AHHouLkjmEoNImUWIHprLYtK1X6/nsllYMqmPvsdj6knHR42YTSIWyRfIqIicSnBjqHtSX5JAFFYCA+gKlvnyGRiIx0zQR9kk8JWI1cwX/FJF25JMXcHoTPVXTen3GeQpanSP4B4anErlCmTf6qa5X8gn2DQaNLASM9BJBatIzkzDnx6bBjmouFnVVAmvpRQKEpoWrO/LnsDPUfvVRVUP8csnI/W1QGZZYRQ14W2K2u2MgUVt359y29pr1RcoIVKphFdL6xl5XzIMskJVJX41Jw9qehaS6Nmpp587NfmMIxf0LY9woF2gl2kBgCYSAcP/N1PUhfwA71O2hID/dz5lXdJ+R9q2KzdIszBhMrhFrUGzyVrCM3nUTNcb55dS/P8IJey3nGaGlPwkCO5FN9ZEJFVOVTbo6Kf1870po7pYWPbKzkGbKaHAv3D/vUdE19BQf3fvNPShc9TfM2hqV1OFMdSPbpq+6+5wgrFblAu9sc3DPQzg40VgTeDBP43TVjuq9EXdv+fPuNu+PxRnPGRhP4ATYX2T4sb31o3HqYy4ImIzC+mcxcRYXqDr29eXEIkslqe2aT5OtHEtx1uKriXdpVXG9funDbzG5cm7XV7eJkkoQWf9XQGvHtfizjkK3SnDqtgnFK46JBGXqjUxzrt83OGoxKiZZSf1p7wyJl/2KEsZoImAUahHckg3iBgpSZT5M47PZs7smZvy3DqmmblLLg/hP4146FU7NuvIMadGN2j/wujTHGvABxvdQ5An7GyphnI6drYlHG0BsztmW/wyJdPyKhId4nVuMufrcaz/mRk7mkyIUMhzyvJluqy4DRcKWDZVoi3Jg9nksy4PPMLFJH8o/zgqKJE3rfABoKvVUKzBwYpn6OZukS82LuJIahJXCYWdhBiaxa1ibVi X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 41f19750-44b0-4115-05ab-08d8ed22b391 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:37.2103 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: U9T1wLu3b93zNtzcvUZbuRKDkTiwh2BMFk2ePsnu9CBQkkE0r3aSSgNrEfOB+0czFCsjIFWIYvv21Iey1XOlSA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang Register vram memory as MEMORY_DEVICE_PRIVATE type resource, to allocate vram backing pages for page migration. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c | 4 + drivers/gpu/drm/amd/amdkfd/Kconfig | 1 + drivers/gpu/drm/amd/amdkfd/Makefile | 3 +- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 103 +++++++++++++++++++++ drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 48 ++++++++++ drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 3 + 6 files changed, 161 insertions(+), 1 deletion(-) create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_migrate.c create mode 100644 drivers/gpu/drm/amd/amdkfd/kfd_migrate.h diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c index 5f6696a3c778..eb40b3879a0b 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c @@ -30,6 +30,7 @@ #include #include "amdgpu_xgmi.h" #include +#include "kfd_migrate.h" /* Total memory size in system memory and all GPU VRAM. Used to * estimate worst case amount of memory to reserve for page tables @@ -167,12 +168,15 @@ void amdgpu_amdkfd_device_init(struct amdgpu_device *adev) adev->kfd.init_complete = kgd2kfd_device_init(adev->kfd.dev, adev_to_drm(adev), &gpu_resources); + if (adev->kfd.init_complete) + svm_migrate_init(adev); } } void amdgpu_amdkfd_device_fini(struct amdgpu_device *adev) { if (adev->kfd.dev) { + svm_migrate_fini(adev); kgd2kfd_device_exit(adev->kfd.dev); adev->kfd.dev = NULL; } diff --git a/drivers/gpu/drm/amd/amdkfd/Kconfig b/drivers/gpu/drm/amd/amdkfd/Kconfig index f02c938f75da..7880fc101a3b 100644 --- a/drivers/gpu/drm/amd/amdkfd/Kconfig +++ b/drivers/gpu/drm/amd/amdkfd/Kconfig @@ -8,6 +8,7 @@ config HSA_AMD depends on DRM_AMDGPU && (X86_64 || ARM64 || PPC64) imply AMD_IOMMU_V2 if X86_64 select HMM_MIRROR + select DEVICE_PRIVATE select MMU_NOTIFIER select DRM_AMDGPU_USERPTR help diff --git a/drivers/gpu/drm/amd/amdkfd/Makefile b/drivers/gpu/drm/amd/amdkfd/Makefile index 387ce0217d35..a93301dbc464 100644 --- a/drivers/gpu/drm/amd/amdkfd/Makefile +++ b/drivers/gpu/drm/amd/amdkfd/Makefile @@ -55,7 +55,8 @@ AMDKFD_FILES := $(AMDKFD_PATH)/kfd_module.o \ $(AMDKFD_PATH)/kfd_dbgmgr.o \ $(AMDKFD_PATH)/kfd_smi_events.o \ $(AMDKFD_PATH)/kfd_crat.o \ - $(AMDKFD_PATH)/kfd_svm.o + $(AMDKFD_PATH)/kfd_svm.o \ + $(AMDKFD_PATH)/kfd_migrate.o ifneq ($(CONFIG_AMD_IOMMU_V2),) AMDKFD_FILES += $(AMDKFD_PATH)/kfd_iommu.o diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c new file mode 100644 index 000000000000..4bb39c562665 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -0,0 +1,103 @@ +/* + * Copyright 2020 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include +#include +#include +#include +#include "amdgpu_sync.h" +#include "amdgpu_object.h" +#include "amdgpu_vm.h" +#include "amdgpu_mn.h" +#include "kfd_priv.h" +#include "kfd_svm.h" +#include "kfd_migrate.h" + +static void svm_migrate_page_free(struct page *page) +{ +} + +/** + * svm_migrate_to_ram - CPU page fault handler + * @vmf: CPU vm fault vma, address + * + * Context: vm fault handler, mm->mmap_sem is taken + * + * Return: + * 0 - OK + * VM_FAULT_SIGBUS - notice application to have SIGBUS page fault + */ +static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) +{ + return VM_FAULT_SIGBUS; +} + +static const struct dev_pagemap_ops svm_migrate_pgmap_ops = { + .page_free = svm_migrate_page_free, + .migrate_to_ram = svm_migrate_to_ram, +}; + +int svm_migrate_init(struct amdgpu_device *adev) +{ + struct kfd_dev *kfddev = adev->kfd.dev; + struct dev_pagemap *pgmap; + struct resource *res; + unsigned long size; + void *r; + + /* Page migration works on Vega10 or newer */ + if (kfddev->device_info->asic_family < CHIP_VEGA10) + return -EINVAL; + + pgmap = &kfddev->pgmap; + memset(pgmap, 0, sizeof(*pgmap)); + + /* TODO: register all vram to HMM for now. + * should remove reserved size + */ + size = ALIGN(adev->gmc.real_vram_size, 2ULL << 20); + res = devm_request_free_mem_region(adev->dev, &iomem_resource, size); + if (IS_ERR(res)) + return -ENOMEM; + + pgmap->type = MEMORY_DEVICE_PRIVATE; + pgmap->nr_range = 1; + pgmap->range.start = res->start; + pgmap->range.end = res->end; + pgmap->ops = &svm_migrate_pgmap_ops; + pgmap->owner = adev; + pgmap->flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE; + r = devm_memremap_pages(adev->dev, pgmap); + if (IS_ERR(r)) { + pr_err("failed to register HMM device memory\n"); + return PTR_ERR(r); + } + + pr_info("HMM registered %ldMB device memory\n", size >> 20); + + return 0; +} + +void svm_migrate_fini(struct amdgpu_device *adev) +{ + memunmap_pages(&adev->kfd.dev->pgmap); +} diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h new file mode 100644 index 000000000000..98ab685d3e17 --- /dev/null +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h @@ -0,0 +1,48 @@ +/* + * Copyright 2020 Advanced Micro Devices, Inc. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + */ + +#ifndef KFD_MIGRATE_H_ +#define KFD_MIGRATE_H_ + +#include +#include +#include +#include +#include +#include "kfd_priv.h" +#include "kfd_svm.h" + +#if defined(CONFIG_DEVICE_PRIVATE) +int svm_migrate_init(struct amdgpu_device *adev); +void svm_migrate_fini(struct amdgpu_device *adev); + +#else +static inline int svm_migrate_init(struct amdgpu_device *adev) +{ + DRM_WARN_ONCE("DEVICE_PRIVATE kernel config option is not enabled, " + "add CONFIG_DEVICE_PRIVATE=y in config file to fix\n"); + return -ENODEV; +} +static inline void svm_migrate_fini(struct amdgpu_device *adev) {} +#endif +#endif /* KFD_MIGRATE_H_ */ diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index d66430740e52..755c0517867d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -322,6 +322,9 @@ struct kfd_dev { unsigned int max_doorbell_slices; int noretry; + + /* HMM page migration MEMORY_DEVICE_PRIVATE mapping */ + struct dev_pagemap pgmap; }; enum kfd_mempool { From patchwork Mon Mar 22 10:58:31 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A4CC0C433DB for ; Mon, 22 Mar 2021 11:08:17 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6266461879 for ; Mon, 22 Mar 2021 11:08:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6266461879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A77AD6E0D7; Mon, 22 Mar 2021 11:07:51 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2053.outbound.protection.outlook.com [40.107.237.53]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8367489EFF; Mon, 22 Mar 2021 11:07:39 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=UD8QWedBws2kTTtCxF4MfUkjTL17GBN5JYK0N3wMVJ6W2sIyGHD9XUyG9SzdS5EYJbZ/P3Ln9YX+iChCKKrz9uy4KnohpMWJrC9pKPfxKcSKH9vds7ObBpDCinUmQpRjxP56AyyVsNoRkGZwaK70V6dV6y7VXwS+Koe3Bz+YgDuZc24GmHzLnfphJrKmYDFm7KD8EQ+wvVWVGFQPSIOlzH5+RmNbAZvJZRv0edQSWzNDmGYPx9EgmmK/HATESyqGp67J4hlC7MFBqA5vKiV4IVsUZe67kR9mR53+a1kKSj7TjBw64B8zJz0w0kWbkafe5rMT9+fpJb7abDX2NW7HKw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JpziIVBQK7V3OK+KDPr6ZesEz4SbZR+TZ/Tt50Y9Yeg=; b=ZZqEsV2frWmM4J4u6nRfo9OIot5OErk269aDdgLHfMH0iFhF8bAgr1U8/Gmaf8HDba95rI2isGnKRbVTD9+uAJRVrIPOjAQbVzX2slgj5Y5QfB4HmM7OdjQ245QtRzzcDbnQoafx64VkW7nk9fK5dvmZphnCsgmKMYGI++SzDi0aeccV9/NOkgfLjOe5ZBqlO+EyI2CtrSbnzTQxAlhfSrj5/kPVSsueZCf2MOmQWa/S6xhBVXg8XQgqc7/emzQeHa/W66GYVnVvhwiN1WxIc8UZuwxRa1TotdmS9rpMfybPF7ML05m6EjQfuHb9NL2vaNzgnioU8QHFxhFBUGpQNQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=JpziIVBQK7V3OK+KDPr6ZesEz4SbZR+TZ/Tt50Y9Yeg=; b=DekepuOEBYlmBvHhy3XVhB+uB7U0q9XfBbMFjMmUoZUP974ALxv3l7gPhpBUU46j8setqpTTKLiEVaYkxsdPzqkgq3hQKi4NNlc9NjU3K2Bqgw43hmvp8RV6OcfluFDa/C+7tDfiTHN49+80QyYhjWqYrMUJb/eVTlfpCmQdzSU= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:37 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:37 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 15/44] drm/amdkfd: validate vram svm range from TTM Date: Mon, 22 Mar 2021 06:58:31 -0400 Message-Id: <20210322105900.14068-16-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:37 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: cf9d62c0-3bc7-4ba6-e1e5-08d8ed22b3e4 X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: CHUGAhtZQjM94FSCVQjBVmxTwqYb/s+QzpZttwbtxIHNRCRH+e+aMT9xppHeRSEN1oLGg/FyLt8NoGqU9rpsZx1lihF3fj3uzKeyTmGvcZcmiBfZnanPDjnG+poJDNBV9G3sPZ6mN49/SoAtdgc7+EdHFJ9tDKFXVKbrKJPUtg93qTzcHP5WHI5GZtZLC9xcAnbnknIfLjBJzHUslg6D9dRF6hlHFl46nWS0TrmwYoZnAZd+P+CaqASC9tWfNlIAMsRraBYuXdtQfuxPtvrArOgWGOS9Kx08WFgEJlTr2M3lGtJecYTQflD+XC5lnQj0U60NmywmF+c73nRP9MRzDxeQecxISh4atVzEw9Ki37cXLrSRsFgxFJo3UvqH3pubq8LOoF12skX1gvR1/RHmDNr23DjKSx7sfEBUq2CcnB6O4qj96dRydQlhKE+05WYKFHtuKkMjdg9D0yNJqd/75mmPcaf+mf+L2kKwXFUmU5SFu0CwZ0ZxBfvShO/h91vw6kpULRXojLVd6sJEOejt8g1WdnfTqso0Fib/ZsP3kyrmQeURtvVgGNU9Xp3F8kaRukDp+oDcmwLVqcGIV49OH074ZSdFSf9QI+Jq2hwhYx7XMNXzDQ8XLK/1UKe2lQJNAyzYhWVE3t6TDIWt88mR/Q== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(54906003)(15650500001)(6486002)(1076003)(36756003)(30864003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 7ztzZw1HlfKMbZ0hw+THDukf0B3L1Sim6Dc2Lh8Tr3mgDyyXyH+juU2cDckuveJM/gR1gFi18eyhS0hyYmMK5ZAlXt9CHu1egnQVR54NchdcKxW8gWwJaYB5cK98nMVNjnIz7hMdcSZ2zahf6GXsxD2qyN72ZnxQptYeNn+amSO5nPO/kyOkX/TEwTeZDNE+x6wpRg5MzSxnyMuMBd3X7gePKk5ualTVJ8BM8+S7DN0dERwnv+o2KCUGe7VDNvnarSbUZNwBG2agKO6TYu/XVflltta+lmdxcYwt+q+7R4sTr0ukLEuC78uMXZ/Zj1H2fOGOEp5V4G44irTxhSgX0kt6QUZMQGN1sBkchUXDYXXsaPEySFIucKRQsCAIo2GgYNcIaf+9LMHNU+/zhoxOJbVnToKHf1alrcYuFJXOF1/Q5OHYmEoco9qfS9uyxJyDgQrWUqLLR8VPjitV+5ZCuRF1cqlZCC0tr77ghUTDBsQ2ozjKg9kQvKfRIMxHgKmrvhTKfNzrpmQQUq+p61AwVp73Gt7Z0a0mlj4nFfeQAjKCaf36Yho52+6uo3iiDJcL+ReDnFgaNGnej2TCypwtek1feSwK+cYX3S4MkboqbJlreq7hCDagEhXce9Ylt4TVdsmn9+8q47wxs97hiqxFvLi1YPcLsVD1n/ePFGJfexL8V+HiPwkndg+2UmYYbEYV/FpAOJnHidHJ2cjRCt7hbThd4ariW9+I/YEK1oh/vYkWG3Sr893kXCuxgXLxavRj0CvsF8MBrIzScsIoxMyYylMtT4IWJLkzAkNtxvja7XBFSce1JThLW1Qy5Xg1/29dsuIUov39KhxF/OgtkFw88LmnchSHx7sI31h/sEGwqIcYNKqnvMaXYKWlXVOQ3pZg1ptuDWL7HdQoNKy9Wpx83TNsrk0hCfWYp+NnlFYQQqP4EkYfnMxkTS9oJkpgdXRLzIzuQESEvmRNamdW7NJvA7LfExps9TuGpEapDvgkfs6klirL7vEMNzoFeKVNXGURnSCZgBDhdj8sF/riW0l/gDuLTl5QJRmo7RPu459vGbbjwI3ez2qakIj1hqv6xJ743hRnUytwb7UH9fFXSp+mofCjWkbuClIodvHHdduULmZqDNfkFk0N4/nqA5aEOKw+NmKxW/Ej2Ef1J9e0w3BK+1rQ1yzkNmWERe1A+81CKiTW4Z8AQM6kP0BoLD03aIMchyUYKlZY2QlJa7rJ0ZMJn9h9Gm7b0gITzVmnzKWuv4HwAm8oud0nzHf8lquW+5qx9lftUj2EwFmJyxJulKUwa/Kut1afmbiacLX3ArDcBxMNFsiXprSNVNxdEC45fOjB X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: cf9d62c0-3bc7-4ba6-e1e5-08d8ed22b3e4 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:37.7888 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 70zwLFVhKm3u+s1QMA1Ei/5abK1uNZzDJwIvm0V0BjtpjCUwzAIMylzgIqKDhFqnoqK7Pfha847yJ0vfnPEe9A== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" If svm range perfetch location is not zero, use TTM to alloc amdgpu_bo vram nodes to validate svm range, then map vram nodes to GPUs. Use offset to sub allocate from the same amdgpu_bo to handle overlap vram range while adding new range or unmapping range. svm_bo has ref count to trace the shared ranges. If all ranges of shared amdgpu_bo are migrated to ram, ref count becomes 0, then amdgpu_bo is released, all ranges svm_bo is set to NULL. To migrate range from ram back to vram, allocate the same amdgpu_bo with previous offset if the range has svm_bo. If prange migrate to VRAM, no CPU mapping exist, then process exit will not have unmap callback for this prange to free prange and svm bo. Free outstanding pranges from svms list before process is freed in svm_range_list_fini. Signed-off-by: Philip Yang Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 338 +++++++++++++++++++++++++-- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 36 +++ 2 files changed, 355 insertions(+), 19 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 55828d6fbea9..9b1c5aa86f4a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -44,7 +44,8 @@ static const struct mmu_interval_notifier_ops svm_range_mn_ops = { * svm_range_unlink - unlink svm_range from lists and interval tree * @prange: svm range structure to be removed * - * Remove the svm range from svms interval tree and link list + * Remove the svm_range from the svms and svm_bo lists and the svms + * interval tree. * * Context: The caller must hold svms->lock */ @@ -53,6 +54,12 @@ static void svm_range_unlink(struct svm_range *prange) pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx]\n", prange->svms, prange, prange->start, prange->last); + if (prange->svm_bo) { + spin_lock(&prange->svm_bo->list_lock); + list_del(&prange->svm_bo_list); + spin_unlock(&prange->svm_bo->list_lock); + } + list_del(&prange->list); if (prange->it_node.start != 0 && prange->it_node.last != 0) interval_tree_remove(&prange->it_node, &prange->svms->objects); @@ -193,6 +200,7 @@ static void svm_range_free(struct svm_range *prange) pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx]\n", prange->svms, prange, prange->start, prange->last); + svm_range_vram_node_free(prange); svm_range_free_dma_mappings(prange); kvfree(prange->pages_addr); kfree(prange); @@ -227,9 +235,11 @@ svm_range *svm_range_new(struct svm_range_list *svms, uint64_t start, INIT_LIST_HEAD(&prange->update_list); INIT_LIST_HEAD(&prange->remove_list); INIT_LIST_HEAD(&prange->insert_list); + INIT_LIST_HEAD(&prange->svm_bo_list); INIT_LIST_HEAD(&prange->deferred_list); INIT_LIST_HEAD(&prange->child_list); atomic_set(&prange->invalid, 0); + mutex_init(&prange->lock); svm_range_set_default_attributes(&prange->preferred_loc, &prange->prefetch_loc, &prange->granularity, &prange->flags); @@ -277,14 +287,244 @@ svm_range_validate_ram(struct mm_struct *mm, struct svm_range *prange) return 0; } +static bool svm_bo_ref_unless_zero(struct svm_range_bo *svm_bo) +{ + if (!svm_bo || !kref_get_unless_zero(&svm_bo->kref)) + return false; + + return true; +} + +static struct svm_range_bo *svm_range_bo_ref(struct svm_range_bo *svm_bo) +{ + if (svm_bo) + kref_get(&svm_bo->kref); + + return svm_bo; +} + +static void svm_range_bo_release(struct kref *kref) +{ + struct svm_range_bo *svm_bo; + + svm_bo = container_of(kref, struct svm_range_bo, kref); + spin_lock(&svm_bo->list_lock); + while (!list_empty(&svm_bo->range_list)) { + struct svm_range *prange = + list_first_entry(&svm_bo->range_list, + struct svm_range, svm_bo_list); + /* list_del_init tells a concurrent svm_range_vram_node_new when + * it's safe to reuse the svm_bo pointer and svm_bo_list head. + */ + list_del_init(&prange->svm_bo_list); + spin_unlock(&svm_bo->list_lock); + + pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, + prange->start, prange->last); + mutex_lock(&prange->lock); + prange->svm_bo = NULL; + mutex_unlock(&prange->lock); + + spin_lock(&svm_bo->list_lock); + } + spin_unlock(&svm_bo->list_lock); + + amdgpu_bo_unref(&svm_bo->bo); + kfree(svm_bo); +} + +static void svm_range_bo_unref(struct svm_range_bo *svm_bo) +{ + if (!svm_bo) + return; + + kref_put(&svm_bo->kref, svm_range_bo_release); +} + +static struct svm_range_bo *svm_range_bo_new(void) +{ + struct svm_range_bo *svm_bo; + + svm_bo = kzalloc(sizeof(*svm_bo), GFP_KERNEL); + if (!svm_bo) + return NULL; + + kref_init(&svm_bo->kref); + INIT_LIST_HEAD(&svm_bo->range_list); + spin_lock_init(&svm_bo->list_lock); + + return svm_bo; +} + +int +svm_range_vram_node_new(struct amdgpu_device *adev, struct svm_range *prange, + bool clear) +{ + struct amdkfd_process_info *process_info; + struct amdgpu_bo_param bp; + struct svm_range_bo *svm_bo; + struct amdgpu_bo_user *ubo; + struct amdgpu_bo *bo; + struct kfd_process *p; + int r; + + pr_debug("[0x%lx 0x%lx]\n", prange->start, prange->last); + mutex_lock(&prange->lock); + if (prange->svm_bo) { + if (prange->ttm_res) { + /* We still have a reference, all is well */ + mutex_unlock(&prange->lock); + return 0; + } + if (svm_bo_ref_unless_zero(prange->svm_bo)) { + /* The BO was still around and we got + * a new reference to it + */ + mutex_unlock(&prange->lock); + pr_debug("reuse old bo [0x%lx 0x%lx]\n", + prange->start, prange->last); + + prange->ttm_res = &prange->svm_bo->bo->tbo.mem; + return 0; + } + + mutex_unlock(&prange->lock); + + /* We need a new svm_bo. Spin-loop to wait for concurrent + * svm_range_bo_release to finish removing this range from + * its range list. After this, it is safe to reuse the + * svm_bo pointer and svm_bo_list head. + */ + while (!list_empty_careful(&prange->svm_bo_list)) + ; + + } else { + mutex_unlock(&prange->lock); + } + + svm_bo = svm_range_bo_new(); + if (!svm_bo) { + pr_debug("failed to alloc svm bo\n"); + return -ENOMEM; + } + + memset(&bp, 0, sizeof(bp)); + bp.size = prange->npages * PAGE_SIZE; + bp.byte_align = PAGE_SIZE; + bp.domain = AMDGPU_GEM_DOMAIN_VRAM; + bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS; + bp.flags |= clear ? AMDGPU_GEM_CREATE_VRAM_CLEARED : 0; + bp.type = ttm_bo_type_device; + bp.resv = NULL; + + r = amdgpu_bo_create_user(adev, &bp, &ubo); + if (r) { + pr_debug("failed %d to create bo\n", r); + kfree(svm_bo); + return r; + } + bo = &ubo->bo; + + p = container_of(prange->svms, struct kfd_process, svms); + r = amdgpu_bo_reserve(bo, true); + if (r) { + pr_debug("failed %d to reserve bo\n", r); + goto reserve_bo_failed; + } + + r = dma_resv_reserve_shared(bo->tbo.base.resv, 1); + if (r) { + pr_debug("failed %d to reserve bo\n", r); + amdgpu_bo_unreserve(bo); + goto reserve_bo_failed; + } + process_info = p->kgd_process_info; + amdgpu_bo_fence(bo, &process_info->eviction_fence->base, true); + + amdgpu_bo_unreserve(bo); + + svm_bo->bo = bo; + prange->svm_bo = svm_bo; + prange->ttm_res = &bo->tbo.mem; + prange->offset = 0; + + spin_lock(&svm_bo->list_lock); + list_add(&prange->svm_bo_list, &svm_bo->range_list); + spin_unlock(&svm_bo->list_lock); + + return 0; + +reserve_bo_failed: + kfree(svm_bo); + amdgpu_bo_unref(&bo); + prange->ttm_res = NULL; + + return r; +} + +void svm_range_vram_node_free(struct svm_range *prange) +{ + svm_range_bo_unref(prange->svm_bo); + prange->ttm_res = NULL; +} + +struct amdgpu_device * +svm_range_get_adev_by_id(struct svm_range *prange, uint32_t gpu_id) +{ + struct kfd_process *p; + struct kfd_dev *dev; + int32_t gpu_idx; + int r; + + p = container_of(prange->svms, struct kfd_process, svms); + + gpu_idx = kfd_process_gpuidx_from_gpuid(p, gpu_id); + if (gpu_idx < 0) { + pr_debug("failed to get device by id 0x%x\n", gpu_id); + return NULL; + } + r = kfd_process_device_from_gpuidx(p, gpu_idx, &dev); + if (r < 0) { + pr_debug("failed to get device by idx 0x%x\n", gpu_idx); + return NULL; + } + + return (struct amdgpu_device *)dev->kgd; +} + +static int svm_range_validate_vram(struct svm_range *prange) +{ + struct amdgpu_device *adev; + int r; + + pr_debug("svms 0x%p [0x%lx 0x%lx] actual_loc 0x%x\n", prange->svms, + prange->start, prange->last, prange->actual_loc); + + adev = svm_range_get_adev_by_id(prange, prange->actual_loc); + if (!adev) { + pr_debug("failed to get device by id 0x%x\n", + prange->actual_loc); + return -EINVAL; + } + + r = svm_range_vram_node_new(adev, prange, true); + if (r) + pr_debug("failed %d to alloc vram\n", r); + + return r; +} + static int svm_range_validate(struct mm_struct *mm, struct svm_range *prange) { - int r = 0; + int r; pr_debug("actual loc 0x%x\n", prange->actual_loc); - r = svm_range_validate_ram(mm, prange); + if (!prange->actual_loc) + r = svm_range_validate_ram(mm, prange); + else + r = svm_range_validate_vram(prange); pr_debug("svms 0x%p [0x%lx 0x%lx] ret %d invalid %d\n", prange->svms, prange->start, prange->last, r, atomic_read(&prange->invalid)); @@ -497,6 +737,32 @@ svm_range_split_pages(struct svm_range *new, struct svm_range *old, return 0; } +static int +svm_range_split_nodes(struct svm_range *new, struct svm_range *old, + uint64_t start, uint64_t last) +{ + uint64_t npages = last - start + 1; + + pr_debug("svms 0x%p new prange 0x%p start 0x%lx [0x%llx 0x%llx]\n", + new->svms, new, new->start, start, last); + + if (new->start == old->start) { + new->offset = old->offset; + old->offset += new->npages; + } else { + new->offset = old->offset + npages; + } + + new->svm_bo = svm_range_bo_ref(old->svm_bo); + new->ttm_res = old->ttm_res; + + spin_lock(&new->svm_bo->list_lock); + list_add(&new->svm_bo_list, &new->svm_bo->range_list); + spin_unlock(&new->svm_bo->list_lock); + + return 0; +} + /** * svm_range_split_adjust - split range and adjust * @@ -505,7 +771,7 @@ svm_range_split_pages(struct svm_range *new, struct svm_range *old, * @start: the old range adjust to start address in pages * @last: the old range adjust to last address in pages * - * Copy system memory pages, pages_addr or vram mm_nodes in old range to new + * Copy system memory pages, pages_addr or vram ttm_res in old range to new * range from new_start up to size new->npages, the remaining old range is from * start to last * @@ -529,6 +795,8 @@ svm_range_split_adjust(struct svm_range *new, struct svm_range *old, if (old->pages_addr) r = svm_range_split_pages(new, old, start, last); + else if (old->actual_loc && old->ttm_res) + r = svm_range_split_nodes(new, old, start, last); else WARN_ONCE(1, "split adjust invalid pages_addr and nodes\n"); if (r) @@ -649,7 +917,8 @@ svm_range_get_pte_flags(struct amdgpu_device *adev, struct svm_range *prange) uint64_t pte_flags; pte_flags = AMDGPU_PTE_VALID; - pte_flags |= AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED; + if (!prange->ttm_res) + pte_flags |= AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED; mapping_flags = AMDGPU_VM_PAGE_READABLE | AMDGPU_VM_PAGE_WRITEABLE; @@ -669,7 +938,9 @@ svm_range_get_pte_flags(struct amdgpu_device *adev, struct svm_range *prange) /* Apply ASIC specific mapping flags */ amdgpu_gmc_get_vm_pte(adev, &prange->mapping, &pte_flags); - pr_debug("PTE flags 0x%llx\n", pte_flags); + pr_debug("svms 0x%p [0x%lx 0x%lx] vram %d system %d PTE flags 0x%llx\n", + prange->svms, prange->start, prange->last, + prange->ttm_res ? 1:0, prange->pages_addr ? 1:0, pte_flags); return pte_flags; } @@ -757,7 +1028,9 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, struct svm_range *prange, dma_addr_t *pages_addr, bool reserve_vm, struct dma_fence **fence) { - struct amdgpu_bo *root; + struct ttm_validate_buffer tv[2]; + struct ww_acquire_ctx ticket; + struct list_head list; uint64_t pte_flags; int r = 0; @@ -765,13 +1038,22 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, prange->last); if (reserve_vm) { - root = amdgpu_bo_ref(vm->root.base.bo); - r = amdgpu_bo_reserve(root, true); + INIT_LIST_HEAD(&list); + + tv[0].bo = &vm->root.base.bo->tbo; + tv[0].num_shared = 4; + list_add(&tv[0].head, &list); + if (prange->svm_bo && prange->ttm_res) { + tv[1].bo = &prange->svm_bo->bo->tbo; + tv[1].num_shared = 1; + list_add(&tv[1].head, &list); + } + r = ttm_eu_reserve_buffers(&ticket, &list, true, NULL); if (r) { - pr_debug("failed %d to reserve root bo\n", r); - amdgpu_bo_unref(&root); + pr_debug("failed %d to reserve bo\n", r); goto out; } + r = amdgpu_vm_validate_pt_bos(adev, vm, svm_range_bo_validate, NULL); if (r) { @@ -782,14 +1064,16 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, prange->mapping.start = prange->start; prange->mapping.last = prange->last; - prange->mapping.offset = 0; + prange->mapping.offset = prange->offset; pte_flags = svm_range_get_pte_flags(adev, prange); prange->mapping.flags = pte_flags; r = amdgpu_vm_bo_update_mapping(adev, adev, vm, false, false, NULL, prange->mapping.start, prange->mapping.last, pte_flags, - prange->mapping.offset, NULL, + prange->mapping.offset, + prange->ttm_res ? + prange->ttm_res->mm_node : NULL, pages_addr, &vm->last_update); if (r) { pr_debug("failed %d to map to gpu 0x%lx\n", r, prange->start); @@ -808,11 +1092,8 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, *fence = dma_fence_get(vm->last_update); unreserve_out: - if (reserve_vm) { - amdgpu_bo_unreserve(root); - amdgpu_bo_unref(&root); - } - + if (reserve_vm) + ttm_eu_backoff_reservation(&ticket, &list); out: return r; } @@ -1040,7 +1321,14 @@ struct svm_range *svm_range_clone(struct svm_range *old) memcpy(new->pages_addr, old->pages_addr, old->npages * sizeof(*old->pages_addr)); } - + if (old->svm_bo) { + new->ttm_res = old->ttm_res; + new->offset = old->offset; + new->svm_bo = svm_range_bo_ref(old->svm_bo); + spin_lock(&new->svm_bo->list_lock); + list_add(&new->svm_bo_list, &new->svm_bo->range_list); + spin_unlock(&new->svm_bo->list_lock); + } new->flags = old->flags; new->preferred_loc = old->preferred_loc; new->prefetch_loc = old->prefetch_loc; @@ -1371,12 +1659,16 @@ svm_range_unmap_from_cpu(struct mm_struct *mm, struct svm_range *prange, svm_range_unmap_from_gpus(prange, start, last); + svm_range_lock(prange); + unmap_parent = start <= prange->start && last >= prange->last; list_for_each_entry(pchild, &prange->child_list, child_list) svm_range_unmap_split(mm, prange, pchild, start, last); svm_range_unmap_split(mm, prange, prange, start, last); + svm_range_unlock(prange); + if (unmap_parent) svm_range_add_list_work(svms, prange, mm, SVM_OP_UNMAP_RANGE); else @@ -1435,10 +1727,18 @@ svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, void svm_range_list_fini(struct kfd_process *p) { + struct svm_range *prange; + struct svm_range *next; + pr_debug("pasid 0x%x svms 0x%p\n", p->pasid, &p->svms); /* Ensure list work is finished before process is destroyed */ flush_work(&p->svms.deferred_list_work); + + list_for_each_entry_safe(prange, next, &p->svms.list, list) + svm_range_free(prange); + + pr_debug("pasid 0x%x svms 0x%p done\n", p->pasid, &p->svms); } int svm_range_list_init(struct kfd_process *p) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 2a76e89dec04..8e2b63a134f1 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -32,6 +32,13 @@ #include "amdgpu.h" #include "kfd_priv.h" +struct svm_range_bo { + struct amdgpu_bo *bo; + struct kref kref; + struct list_head range_list; /* all svm ranges shared this bo */ + spinlock_t list_lock; +}; + enum svm_work_list_ops { SVM_OP_NULL, SVM_OP_UNMAP_RANGE, @@ -61,6 +68,12 @@ struct svm_work_list_item { * @npages: number of pages * @pages_addr: list of system memory physical page address * @dma_addr: dma mapping address on each GPU for system memory physical page + * @ttm_res: vram ttm resource map + * @offset: range start offset within mm_nodes + * @svm_bo: struct to manage splited amdgpu_bo + * @svm_bo_list:link list node, to scan all ranges which share same svm_bo + * @lock: protect prange start, last, child_list, svm_bo_list, pages_addr + * @saved_flags:save/restore current PF_MEMALLOC flags * @flags: flags defined as KFD_IOCTL_SVM_FLAG_* * @perferred_loc: perferred location, 0 for CPU, or GPU id * @perfetch_loc: last prefetch location, 0 for CPU, or GPU id @@ -92,6 +105,12 @@ struct svm_range { uint64_t npages; unsigned long *pages_addr; dma_addr_t *dma_addr[MAX_GPU_INSTANCE]; + struct ttm_resource *ttm_res; + uint64_t offset; + struct svm_range_bo *svm_bo; + struct list_head svm_bo_list; + struct mutex lock; + unsigned int saved_flags; uint32_t flags; uint32_t preferred_loc; uint32_t prefetch_loc; @@ -106,10 +125,27 @@ struct svm_range { DECLARE_BITMAP(bitmap_aip, MAX_GPU_INSTANCE); }; +static inline void svm_range_lock(struct svm_range *prange) +{ + mutex_lock(&prange->lock); + prange->saved_flags = memalloc_noreclaim_save(); + +} +static inline void svm_range_unlock(struct svm_range *prange) +{ + memalloc_noreclaim_restore(prange->saved_flags); + mutex_unlock(&prange->lock); +} + int svm_range_list_init(struct kfd_process *p); void svm_range_list_fini(struct kfd_process *p); int svm_ioctl(struct kfd_process *p, enum kfd_ioctl_svm_op op, uint64_t start, uint64_t size, uint32_t nattrs, struct kfd_ioctl_svm_attribute *attrs); +struct amdgpu_device *svm_range_get_adev_by_id(struct svm_range *prange, + uint32_t id); +int svm_range_vram_node_new(struct amdgpu_device *adev, + struct svm_range *prange, bool clear); +void svm_range_vram_node_free(struct svm_range *prange); #endif /* KFD_SVM_H_ */ From patchwork Mon Mar 22 10:58:32 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154509 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 926CBC433C1 for ; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5DC5F61931 for ; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5DC5F61931 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id CCBDA89F49; Mon, 22 Mar 2021 11:07:52 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2053.outbound.protection.outlook.com [40.107.237.53]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4976689ECD; Mon, 22 Mar 2021 11:07:39 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Y8D9f8Yn8l6V+5VRIbqxtGqbEHCIPFeQqBZpxV1JADk991+xzt3mYPobtlu81BLpKhj2bgH4/rDti7nNpGkQSSFgwI8MMe2e0QbDql7F2hn0O20S0ipPsHSCr9x+Vt3eL1bJMiWUN17qCgaiaZa+j4CZFW81LeSWjn1kyRW+2UBOyeOhxKtJ15vprToigQrDLbQfQZO+SgZMnbEFGv3ZePY4ruCHO2eUjLo9tNhu3UUjTfxMWfSz/gDDlSJREDdOxG4/Ly5jk38ClLR8DnpaHo5dMEalVMPIQZu/Gg6TBbz2Isq7hOSUnPgg1JKsr6YrgwKmwJ5od2KrMkuOSG7xlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P1a9lKtIkPZes2B6a1c6oAOIaPDsq1t8O0hVbXfjDMI=; b=DgvN+HsilB4WJyMrfvYq2RDISoWYvhPJDNegNScqYSYDiY4PwVXBxPNjdy+uVwnT58bhQBfvfpS3lN+EBGW4Q1t7UfpAxzkHplIQlw6dNqf9v1OCUf6K0OoTt2xfeDFaOBDK/d0jExxbs2rSnACuzmCndO/KSvowJVq3G8Chlh+/yYUH4WyR8OoFhywzoURJXzN+ww/smip/jaoWsOxez1Bci32sBvztr4T/KhMHMDsAkYcAJorLzWD6CelhziNJZe3O3IauD0jWiI3pKdKuLTuQCVg7z/rlO5rFNZqjGctu6D2V9OlMiI8gMqH3zE99X5Czp4KgOtEV6Q8Fda3mLg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=P1a9lKtIkPZes2B6a1c6oAOIaPDsq1t8O0hVbXfjDMI=; b=feoq9TIwItqc3MwDn8B9gWJxZyoSi9lMz/OaI0gTIyVRBIeAjPrjb4ej6cgihW0XPDd6byOATkPRuhmbJUa2LzJosHHotTtSMns/iSyJLw33cBuLnuW45443h9ucbYQfnHdiAcnzbLGO9melzUYFNDo2MSCVq79yX54o2+t2BR0= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:38 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:38 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 16/44] drm/amdkfd: support xgmi same hive mapping Date: Mon, 22 Mar 2021 06:58:32 -0400 Message-Id: <20210322105900.14068-17-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:37 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 8d19f0c0-42fe-4316-4603-08d8ed22b433 X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:449; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: JQWYRcqzg8bJfLXJC/hZya/LPnM5MKe1K2B/MHQzDTxrLOyObI7bSQjiIb9xa6ZOHv3hme7YISbI7l20zboQI5so7t3RnbgQq9WPOSzdHPzpCWgNDwF5aSkaob+xeFopMyoAo8KLqyU29H8W0UhKUJSGV9MLymJIF0PcBe0y2hDrHb+3ZSlK2TyX2qObXKx08vCt66fBiFLQgffjiVqGQgyZyM0/Dda00kwftsEyMIQCS8pSQnzpTpJUtHrEAVgLmKjIN0Ds7RGfX/ZwkxREXO5x6wyGoPhzwjLYVo1LX56Ork67nhzvuiZG5FwwydBM00EMYQym767YoowdPOPLe01tA/z6pyC/XlzFSPsTlaZa9jCYlfTp/Tr1HaL6c4RPH2qmOMOTpKsRswv1+mG1uOGqVotUCBlY5NUI+zJi+ilFq0+BT3KbRb/X9i6uoxly5vWkilOPhpMYMZDlzoyWhF0KxNJTYh3+RviYib9t0QSkCF5HGKcb1xAnB92RoJtPJqKlPXaMpX8FSTJ27ou1cu8nnasJv4ZeuPjoc5OZNos0i4YBO2ueaKIJDtNs3jb9SbXHI3289ahcqYBeQeJVB+txTbQTISCgZYDtKOJbqq+mwrRe8Rron7R7JknR/snZ8+4Bok6Uy55PPVql0brPbBNkzaeQSOaNdgKw/ZnyGFQ= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 7OY7M8JQcGqRPFlgzmItF+hYsD0Rnhek+1Qm/P3+6eCQbnuOea2VwHq128a3tsjFE1z8GLryrByA+DgDFsC0bxYCMZvBqSORSfozJ/IHaLwtVlcUQIphe/bdWpx/lg2wK7SNz8i4AwP80pwY+Wfqve8CBXAIC1kvb9MUy01xmLD1+SObs4Nm8r+X9FT9t89pYzV815tOXW3Ux++ujLtRFROcw6X/yQkDHGjK1b6kcW91lIYTfsrvnUKJH8rFsi2I9VtSC/YKvIkpyeeAUPmVb/JtGZN6eHlahXixjJILqm+EcQSpNcbF6sToC3sJM2V9L7bYODVpOytp8HlhgWiu9aPbqDKxYLeuHkGUqPTFdLvRhNZmjQMhM1g54wxEs8/70PwkZf7yi+pAIhr/c/cUdvcGzx4N6WAC9sx/IQwfWJKYCnrgM23jxhNAog+wsHgZ/9Q9/7CwrnUvqDC2GTyG6/tUGlvEYM2c8nGcUXxiC/EK7YJ/o2TTkwsI/ZEiXmdRqrT7coABqVvSK9w1IU8a5yxOIljNFDoPROdjhNeqPUS+uBGlkZ/G8b/T3ox0wBSc1qt5sSMI7RNl1W3Ej+zKYSKLJlGxEJtL77qq/XBxJstj832y5roE+I8kJGhAmnXaYMvIb9/Ine5vBt7GWAnDuPU7El2B8iTmsprKG2DcZwaJf3Yjbi77EA7QxFItrTm9v1iECvccE0RPlPYku7u/AujRrR/7lCqrGSMURowynJodW4hjXO0COylVtKLwfKI+5nlhuZjOqKhelJcq1Axj0VwUy5RfsW7Pl74uxvayTmPf6SJzu2lPpu4/107JixGVoxMf7ZLEF7o5JxwzjUdqEI1DQAIwXQpr6ixyTWeyqruwFz+JpMmTNSPZWPJ3/fbcZcKvGoUUNlQrJC8kqPZxXKNvsfDwxGYs/t90IjnIDA+OLnvONiXDLAWspe1+/rHcp9VA97w6dApZc2FNzIEr+MXwTyL4gcnCo2+6pbwXv/4xe0pjcWklKF/Cyr5Cslr+hOH1edRQ/D7sSaKBtlG7qFDw6iQCCVi/fVnoz2qgl6IIWUTU0WmxQDEfw3zN2/bR+PuwVHykJRx3TiEBEYSr+bTi1nDPYGPsPk0IP/yrMvCpOJXfgZ4xnpFdYF0XEXXHs6lbQvGXxOqrP2m/CSeuZ2eqZdG8nF/47c/RREmuMfS7o4YgMR5bQkc2M01AzxfBMPiJqHZkW7eBfPr6QHmF26cXnruy2VDzRWeBE9DDfPMttkXR/aKDr6d6G4O5yxirOMZe6nwd7KltrgerbnaxAii+Am8xIuSsy/lOt4vmAWJHdNAkSYmFlKjHs6mFU2ZE X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8d19f0c0-42fe-4316-4603-08d8ed22b433 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:38.2627 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cFyR+/cB1DZMVHsO6wdogCU/3ffNpf1PmqI2H9mXIiZE3MBteRIgJPz7fChyo9+HrMb1lalPTjss74r0/0ahAQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang amdgpu_gmc_get_vm_pte use bo_va->is_xgmi same hive information to set pte flags to update GPU mapping. Add local structure variable bo_va, and update bo_va.is_xgmi, pass it to mapping->bo_va while mapping to GPU. Assuming xgmi pstate is hi after boot. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 9b1c5aa86f4a..de5777330d23 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -26,6 +26,8 @@ #include "amdgpu_object.h" #include "amdgpu_vm.h" #include "amdgpu_mn.h" +#include "amdgpu.h" +#include "amdgpu_xgmi.h" #include "kfd_priv.h" #include "kfd_svm.h" @@ -1026,10 +1028,12 @@ static int svm_range_bo_validate(void *param, struct amdgpu_bo *bo) static int svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, struct svm_range *prange, dma_addr_t *pages_addr, - bool reserve_vm, struct dma_fence **fence) + bool reserve_vm, struct amdgpu_device *bo_adev, + struct dma_fence **fence) { struct ttm_validate_buffer tv[2]; struct ww_acquire_ctx ticket; + struct amdgpu_bo_va bo_va; struct list_head list; uint64_t pte_flags; int r = 0; @@ -1062,13 +1066,18 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, } } + if (prange->svm_bo && prange->ttm_res) { + bo_va.is_xgmi = amdgpu_xgmi_same_hive(adev, bo_adev); + prange->mapping.bo_va = &bo_va; + } + prange->mapping.start = prange->start; prange->mapping.last = prange->last; prange->mapping.offset = prange->offset; pte_flags = svm_range_get_pte_flags(adev, prange); prange->mapping.flags = pte_flags; - r = amdgpu_vm_bo_update_mapping(adev, adev, vm, false, false, NULL, + r = amdgpu_vm_bo_update_mapping(adev, bo_adev, vm, false, false, NULL, prange->mapping.start, prange->mapping.last, pte_flags, prange->mapping.offset, @@ -1092,6 +1101,7 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, *fence = dma_fence_get(vm->last_update); unreserve_out: + prange->mapping.bo_va = NULL; if (reserve_vm) ttm_eu_backoff_reservation(&ticket, &list); out: @@ -1102,6 +1112,7 @@ static int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) { DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE); struct kfd_process_device *pdd; + struct amdgpu_device *bo_adev; struct amdgpu_device *adev; struct kfd_process *p; struct kfd_dev *dev; @@ -1109,6 +1120,11 @@ static int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) uint32_t gpuidx; int r = 0; + if (prange->svm_bo && prange->ttm_res) + bo_adev = amdgpu_ttm_adev(prange->svm_bo->bo->tbo.bdev); + else + bo_adev = NULL; + bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, MAX_GPU_INSTANCE); p = container_of(prange->svms, struct kfd_process, svms); @@ -1125,6 +1141,12 @@ static int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) return -EINVAL; adev = (struct amdgpu_device *)dev->kgd; + if (bo_adev && adev != bo_adev && + !amdgpu_xgmi_same_hive(adev, bo_adev)) { + pr_debug("cannot map to device idx %d\n", gpuidx); + continue; + } + r = svm_range_dma_map(adev->dev, &prange->dma_addr[gpuidx], prange->pages_addr, prange->npages); if (r) @@ -1132,7 +1154,7 @@ static int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) r = svm_range_map_to_gpu(adev, pdd->vm, prange, prange->dma_addr[gpuidx], reserve_vm, - &fence); + bo_adev, &fence); if (r) break; From patchwork Mon Mar 22 10:58:33 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C1433C433DB for ; Mon, 22 Mar 2021 11:08:13 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CBE0861984 for ; Mon, 22 Mar 2021 11:08:12 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CBE0861984 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E0CE66E05A; Mon, 22 Mar 2021 11:07:50 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2053.outbound.protection.outlook.com [40.107.237.53]) by gabe.freedesktop.org (Postfix) with ESMTPS id 71DD189EFF; Mon, 22 Mar 2021 11:07:40 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=TNvhusVowRr+su2hZE77C4Aja/FwESrej4QcqgorN0X/UVUiVMMoEOMVC/rBgSu6mtYK7rS9n9MvYIilCMs2eyIExHonUcy9K+igcmnORtybTILQ9lA9MA+Tad+0EKBMd076YhwGUDRA0T8U7OD4RsUC5XnXMjkYdPCH+4LoMr1ri6JBZk8UsH7CdCDY3M1Iw9EXe6MQLyaMNJU2WpBk9FCID3zswZnAzZy920ti4keO+Xqjgezeu1t5o8iGCqakYp4ODjJRi21JPVWp3Da8PwVuNAbU7RMwPmm7Y8/sW25ZMCgapVKxZxzl9F7nWL6+bS6s01p2vbkbxNZ3saM3hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HY1dd63Dk6XEAKgs57nBVUzScP7S1cUMNZ4xslA5jXw=; b=Wur+ts0SxBT58h3/NJ1tcFWxc++VKXB5/xdW8111tTL5M30ub3qvCoYZ6Qv2taNZq4TFcMDxNoZIbAOgxuh+mZ5QOXBRyyfkuypPjf5ptNf54QIN8Qg3oaO25X8jCAWJ0Tdmn4iSls6kt3Ws69nhc9GoM/Ddp2V6JeewvDzq51TgmVvx7iHxiUs7CosaNfC0BqFYrIMB/guqjHIG8Mod+pS/sdooUhX6R2Yqqmw1f0fX9DGXQo+4xZNyF2GUO8TTW31cty2O9Wd9I0uOufcEzpB0raL1mKtnzEmvKmfSFv/lUzN/3JGgNDft+B4m+M3WCma5VfeHovPoV0Z24vkJUg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=HY1dd63Dk6XEAKgs57nBVUzScP7S1cUMNZ4xslA5jXw=; b=ZUnJnqWGzBR4f4fNOC+SEFTO43JIIClO8esXcrQyzAZhtiJk+h7kCD5YwhmKK7Gj+COORDw/CeXBW2R8xhyNV2XGhhuWCsiihgcZO0btC61LJM1sTTuEKTUm28BGp9eyMvlo2ikNd+OTlYLi/YRb8APfebkMZzlRDt7quXc8fTY= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:39 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:39 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 17/44] drm/amdkfd: copy memory through gart table Date: Mon, 22 Mar 2021 06:58:33 -0400 Message-Id: <20210322105900.14068-18-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:38 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: b1d97d0a-d944-4182-7304-08d8ed22b480 X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2582; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4MCZT6DQrPmXMjhvzefK14lM+KiEahywc+DzmsFbccIvZL2tXmm2Z9cvc5LWJflcOVco5tN9YXGUsr7So1Hbk0oVPl9HOQQupVQrBnsmgs1oaemCqX6IpVlxg4eMUtnWctJ6MX/2Q7RGe+g+wuO5r5y2m2lzNg71UhmnyVFYcm7THhWiACPFeMLtohdaMTfW+2gUCtAJWSHgkCkYsTGDgyHXDrveYNXzEVZL4zwlSeq2ljb6KGU+psqSkHGJ82G85lPzUwKKGXApELubkdyLU2ZToTQZN0GsWEPXxfgoloo8N4RgyJRvx4REKp+y+rOtaV/zVAF7yLfXSkr2shZyp+YeE5z/aounLXZXqdjQn+swFuxM9c+9Ovgkmgx1WEHhHgBYAa0kdTyqP7nuJPP93Duet5Pt5tlZKLqpglVouZOl+HZRUs5+G27zx9ySuBAHBWngkih8AmpQwSuooVj/2VZ8NkeZGKDJ1w3c0aDR1w1KtD13+EHj1QV8O962tWIZbtB5LWASWyPXWN7PjtZgXfyx/J5Rgsso1OMVMUh4hff1K8WPcQ3n9OP0pKoG4QQjcn/UnQCsda9Q5zyJ56VHn9erbhOz4arzPLsrhaOBV1G1B9kGvAUVDpiBSmszPOi1fsL4ghDj1g+a3480tl7Kdw== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(6486002)(1076003)(36756003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: Fo77Wdl1aoWNk42ZPcyFKlQkoZMVvy1FK1cUJXdAVLv9jKasiIne+QVapgCzVVzTJLL763r0NlQhrYA/2MK6uRut08YW/SKYiL9qON4NBGzpmHwaycn94C75SNffbbrDXg05xlrZn4bS4h3d1Y6HhDVBUUuHeOhoGhprKHogWNrMO3tV4aQ0neBpUhrT1qMnUrppVQe5jabC9jLKHLG7OvTCv/B9IYdpNo+9KrqPE4EMPnp6cZe1L9zG6VZkorBk+xm3jJP/NBkS/xkjsiNBxqrtgEgzYb24Awamd+oJerzy4O1WA+6jOEy7ewxEWJ/h8a1KfMh9MTXX2YV+Ytt4PuR37NkzC2BJZqBIxayb1EoPKDC6fHio7/y8i8g4DqkNpk42FzeknJ2pB+eStc/U6HWtJAK5yCcKQ5EhAwbdilH/MleeLII9mAG1HhkM78YvShALup/LrLazoOhUXqD5Vg0adG05t3kvimmcTdPLDb/a8zeZqsY346yw+iaYFLdWYHBm9VKkp4nrt5VxigG9T+2U/4TBSABDsluW3wFGNS/B9r+bKSKITdoNM5Cb1Q1BDn3yx9QySNJGVEZr39LL+3ivfi1ZfCdsgP8+i+ZAMgJMARO7e4KTL4+mglUHLaAvxczX73sPuY5Hmmq+FhJpXkQUS+OwXAygxVURJjbcsMZxNWK3ribersnfr5cWFPwB/0Lo/1XOQUX7esQEK4ggABzq55zOYqiaJ4+BUvIVPG8cnMqALmP2c2sbgKuuYuckdzFivoyZadb6pCT60tarUgxpDfH5xbNRlWcXx5ySCagxUxvZ6BsgTj0QNjiEWF7pR1UlPjkYi3XN6GSc3P8c8zZT/jjX0aIFZpqgCbU6pq/pOHwdJM6BaNnmybX9j5rl2qhac0lNmJVoVaw3uLyu9yxu2nH73/yZgt1+W0PQMwGgSlOpyv2PJ4FuU20YZA9SzxGIGRk0pGeRuRxGkr6/ztPQTDSnYQiZF9sN5tcf5svHMMzyp/583ho65xgRxM9XZxas5Z818siPk6Tewy9aTPT2AFGSjqegVy2G1osTFAx4BniN4d/oNzTXTvsUTJTgJo38z1E/yz5JyPgSxEgdhrBnfZCdd1DtOyVybowZapVSDcUSSoQDZw2lm3GlrVOzwq2tSokk0700n8HLxzepfgPmGwzbgwFsK3tmlA/Dx8/Y/paJdBU7qZxKG6YoEsdmuUWoMCzfiH7zgQ05enS3l3nS5DI8nhacRLGQxluTJymRp0VPBq/0q2+qLBYIEhAUV+asS1n3/fyRqcJLvireQXIKzrbY+juB5YaPmPoQsirL99ADykYTPnBcBDfDAaRF X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: b1d97d0a-d944-4182-7304-08d8ed22b480 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:38.8501 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: cSmjfeLcllP8/v01iGFG92F4/loLgtCHKFkhnh5Tvm4azhjSOOwcuN0tPYnrV/B3n5I2d0sr/KX2MMpE+nFjTQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang Use sdma linear copy to migrate data between ram and vram. The sdma linear copy command uses kernel buffer function queue to access system memory through gart table. Use reserved gart table window 0 to map system page address, and vram page address is direct mapping. Use the same kernel buffer function to fill in gart table mapping, so this is serialized with memory copy by sdma job submit. We only need wait for the last memory copy sdma fence for larger buffer migration. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 172 +++++++++++++++++++++++ drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 5 + 2 files changed, 177 insertions(+) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 4bb39c562665..2a6824ddae88 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -32,6 +32,178 @@ #include "kfd_svm.h" #include "kfd_migrate.h" +static uint64_t +svm_migrate_direct_mapping_addr(struct amdgpu_device *adev, uint64_t addr) +{ + return addr + amdgpu_ttm_domain_start(adev, TTM_PL_VRAM); +} + +static int +svm_migrate_gart_map(struct amdgpu_ring *ring, uint64_t npages, + dma_addr_t *addr, uint64_t *gart_addr, uint64_t flags) +{ + struct amdgpu_device *adev = ring->adev; + struct amdgpu_job *job; + unsigned int num_dw, num_bytes; + struct dma_fence *fence; + uint64_t src_addr, dst_addr; + uint64_t pte_flags; + void *cpu_addr; + int r; + + /* use gart window 0 */ + *gart_addr = adev->gmc.gart_start; + + num_dw = ALIGN(adev->mman.buffer_funcs->copy_num_dw, 8); + num_bytes = npages * 8; + + r = amdgpu_job_alloc_with_ib(adev, num_dw * 4 + num_bytes, + AMDGPU_IB_POOL_DELAYED, &job); + if (r) + return r; + + src_addr = num_dw * 4; + src_addr += job->ibs[0].gpu_addr; + + dst_addr = amdgpu_bo_gpu_offset(adev->gart.bo); + amdgpu_emit_copy_buffer(adev, &job->ibs[0], src_addr, + dst_addr, num_bytes, false); + + amdgpu_ring_pad_ib(ring, &job->ibs[0]); + WARN_ON(job->ibs[0].length_dw > num_dw); + + pte_flags = AMDGPU_PTE_VALID | AMDGPU_PTE_READABLE; + pte_flags |= AMDGPU_PTE_SYSTEM | AMDGPU_PTE_SNOOPED; + if (!(flags & KFD_IOCTL_SVM_FLAG_GPU_RO)) + pte_flags |= AMDGPU_PTE_WRITEABLE; + pte_flags |= adev->gart.gart_pte_flags; + + cpu_addr = &job->ibs[0].ptr[num_dw]; + + r = amdgpu_gart_map(adev, 0, npages, addr, pte_flags, cpu_addr); + if (r) + goto error_free; + + r = amdgpu_job_submit(job, &adev->mman.entity, + AMDGPU_FENCE_OWNER_UNDEFINED, &fence); + if (r) + goto error_free; + + dma_fence_put(fence); + + return r; + +error_free: + amdgpu_job_free(job); + return r; +} + +/** + * svm_migrate_copy_memory_gart - sdma copy data between ram and vram + * + * @adev: amdgpu device the sdma ring running + * @src: source page address array + * @dst: destination page address array + * @npages: number of pages to copy + * @direction: enum MIGRATION_COPY_DIR + * @mfence: output, sdma fence to signal after sdma is done + * + * ram address uses GART table continuous entries mapping to ram pages, + * vram address uses direct mapping of vram pages, which must have npages + * number of continuous pages. + * GART update and sdma uses same buf copy function ring, sdma is splited to + * multiple GTT_MAX_PAGES transfer, all sdma operations are serialized, wait for + * the last sdma finish fence which is returned to check copy memory is done. + * + * Context: Process context, takes and releases gtt_window_lock + * + * Return: + * 0 - OK, otherwise error code + */ + +static int +svm_migrate_copy_memory_gart(struct amdgpu_device *adev, dma_addr_t *sys, + uint64_t *vram, uint64_t npages, + enum MIGRATION_COPY_DIR direction, + struct dma_fence **mfence) +{ + const uint64_t GTT_MAX_PAGES = AMDGPU_GTT_MAX_TRANSFER_SIZE; + struct amdgpu_ring *ring = adev->mman.buffer_funcs_ring; + uint64_t gart_s, gart_d; + struct dma_fence *next; + uint64_t size; + int r; + + mutex_lock(&adev->mman.gtt_window_lock); + + while (npages) { + size = min(GTT_MAX_PAGES, npages); + + if (direction == FROM_VRAM_TO_RAM) { + gart_s = svm_migrate_direct_mapping_addr(adev, *vram); + r = svm_migrate_gart_map(ring, size, sys, &gart_d, 0); + + } else if (direction == FROM_RAM_TO_VRAM) { + r = svm_migrate_gart_map(ring, size, sys, &gart_s, + KFD_IOCTL_SVM_FLAG_GPU_RO); + gart_d = svm_migrate_direct_mapping_addr(adev, *vram); + } + if (r) { + pr_debug("failed %d to create gart mapping\n", r); + goto out_unlock; + } + + r = amdgpu_copy_buffer(ring, gart_s, gart_d, size * PAGE_SIZE, + NULL, &next, false, true, false); + if (r) { + pr_debug("failed %d to copy memory\n", r); + goto out_unlock; + } + + dma_fence_put(*mfence); + *mfence = next; + npages -= size; + if (npages) { + sys += size; + vram += size; + } + } + +out_unlock: + mutex_unlock(&adev->mman.gtt_window_lock); + + return r; +} + +/** + * svm_migrate_copy_done - wait for memory copy sdma is done + * + * @adev: amdgpu device the sdma memory copy is executing on + * @mfence: migrate fence + * + * Wait for dma fence is signaled, if the copy ssplit into multiple sdma + * operations, this is the last sdma operation fence. + * + * Context: called after svm_migrate_copy_memory + * + * Return: + * 0 - success + * otherwise - error code from dma fence signal + */ +int +svm_migrate_copy_done(struct amdgpu_device *adev, struct dma_fence *mfence) +{ + int r = 0; + + if (mfence) { + r = dma_fence_wait(mfence, false); + dma_fence_put(mfence); + pr_debug("sdma copy memory fence done\n"); + } + + return r; +} + static void svm_migrate_page_free(struct page *page) { } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h index 98ab685d3e17..5db5686fa46a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h @@ -32,6 +32,11 @@ #include "kfd_priv.h" #include "kfd_svm.h" +enum MIGRATION_COPY_DIR { + FROM_RAM_TO_VRAM = 0, + FROM_VRAM_TO_RAM +}; + #if defined(CONFIG_DEVICE_PRIVATE) int svm_migrate_init(struct amdgpu_device *adev); void svm_migrate_fini(struct amdgpu_device *adev); From patchwork Mon Mar 22 10:58:34 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, UNWANTED_LANGUAGE_BODY,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A24D5C433E1 for ; Mon, 22 Mar 2021 11:08:23 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 70F6761879 for ; Mon, 22 Mar 2021 11:08:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 70F6761879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3958689F35; Mon, 22 Mar 2021 11:07:52 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2075.outbound.protection.outlook.com [40.107.237.75]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5DA0F89F31; Mon, 22 Mar 2021 11:07:41 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ijXY6qbaeGqKF/aqaz6GbAEHK8JtnVL1wqGKNqa/ULLt7VJ33ZNrGD0lE7J9vjzX6Aw3Nw10UwRehK3lX6BSnVyBhcMGEx8oF0P71MLVP8dEAcCiXdRkeO8SqsZW8DNkPKYxdbsSsmcZVvyS996Uamqh+PQawVaApaHZLuX4k72Vzb/FrPJDGvAe04pCr2pbUNAH6c6GoTcU6IhX//YwWywuYa5tIkVT7fRQwfc6uEVZ3MHW3uuu/Sz1VXCC8l/l41tdejbFJWh1qBlPtfIZPuaXhj+sEogNAJQbUPShHg90upyiNFx0kA3HTSoUl/Srs1L8HoX9WlimhEdoVcLL4Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=372XeK1TNBR/VkRyM3gexURtcbsx3/Q3DFuH+oe1zVI=; b=V1isjyFzxftQXdCpYvDXGWu04cWfQ21VYcs6YRJhqe6Bq37x5VNFbydRHVROK/sDHT/RZpQUl9/dFbRhSvzI/LHLnwiq+ruVzXq9pQfFw+ga27e/BRjveIE0iTyjqxLHZ6cm/xSaWdOZW0dSurSUjrFtJc3e/Fbthlp205th5JecOKdrpSNU6x08QOspIiZxRmNtOT8yzndrWgIfFj6qYipnSmi0Ec5r7j0cIl7ly8LloWp+yyTbC+ytD5U9y1Rs6p3EUYQ5SJ/8seLnXB5kIxjHpsNunKyVrtD7YlMg6FLzbxuCUrs46Cp5JEIkslK71PdC7ufagi6Ia4d/D/7pLw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=372XeK1TNBR/VkRyM3gexURtcbsx3/Q3DFuH+oe1zVI=; b=vCbpBsggO5VBwRYnYHky5tKgYpzEGyv2F3FxtangnGx4WtLmYSMqks7kSeiW0ajEQXEF5TS2cLfp7KGktFj0Xmd+qhH/UyQOm0CiYL4NZjoPxgLLLKcQYqTNPfcWzv4RXq4TDun3EmfQPAJhVTUMWNmsyAVEd5PeeMnJmlJ2hh0= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:39 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:39 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 18/44] drm/amdkfd: HMM migrate ram to vram Date: Mon, 22 Mar 2021 06:58:34 -0400 Message-Id: <20210322105900.14068-19-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:39 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 3fb0281b-4fff-4e2b-b41b-08d8ed22b4df X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: OiXwQmAe6u/ajBYSIHa56uFgCRJTvi7vCUXeBklqYRj20xMm8iVhw77acY5XJlzJgeW/Bb0RROUb8EVY+IkhWvpH20PO7VRgmBZzB9CFKT8hHN+aIHllL1fCxkSwjvh3yrvzC4Ogbaf1hCf9WYPEX6Xj6nJZNcI9wFO4/8T6rKT1Jt1bJ/MD4tWxp8nfURroSm9HHYEeErS1Xlw6f8KK9aAQLoIz0iUjFZ5eL/7sZip6djQK3yFiaM4EqvCxYvOMdBk9zbNojxrZcapLNFjfZe0WQSFdCHDVtqHh8w3L1xYklCiKOnMKMuqqDs0k8smoWGL/OrCzhTqy2g1A4USibeeTJExuqUdIbdCfg1GhTdp0L3s1tZ5WdCj+QHRey2eQiCYzR3r5drf7P/7tLdchFHvgghenBVXSCsKx3dLAYQ6FM3Js9uqqvkJeWHV7TK4VQgRl6GILbv52Vc5eJzC7V8IHAgLV3PRGHkjU8pyFsVzLNk8uHxGhncfinyyQXHf+SIarhXK9oGYOT5Su80baP78nwXgL8/dxQa7zIEIe9tfT73YqGx12nOLdPbJ+BZDZSxv2t0Ze+rGPb1zzzrHvz1wDofGgeR928H3IMZ738BIzgoRaDUf0MrH3j5mTRp4l4TCB7XMocjAZ7JlcC77G/LbnuKrpUxRErssz61QhM+E= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(6486002)(1076003)(36756003)(30864003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: P2tZwFm5uRv9PjKSoNxICMLn29CAXTqbOlBz4JiN3q8edRYcfJb6rZZdIfyVFlRCtSIa7ZQ3cfL/Jnvmb+2AjQgQvvWpCbSqgXys/W/Fz0weUPwgb6vtnyvkLP9lBvZpgiD9WM/skC+Mlv7eKlI2reVazWiCcH4ocuubtIOlwrFwQhNylRZBZ2VuwNb4cn1NXCKF2ZMsNLPTiiG4YaU4QQKsMBjxlCzkUJZzGuvwCwh5Y8OqZSneyiwS5AkCAROJuBpqjrFhSdi8O5i4ivHGh2/Q3R5DsXv9cWx7SOiKhvv8gk68331fwVIYJgwBbs0oVPL65TXWJBma9eBjupTFyDSrpIts9oRv28Uqgudm0RGOYxyuswSAVILX2NVDTyXn0cbYRpT/iwL2O/ZJiFhF3DLXxh1oRtppaKkHGd8WuOrYVyeCEc/QROJ5X6vIr4XV2OofZJhpp8N0FaKKhhBUHZAuepfZRNOVT//bOxHNv4WpRZMzBKyqpSrrtqiTsBE9Ycc15snDTZLp8iTE+Vlxri6XytaPbxn0OGMDv25drG6+w6L/ZJhRfVbYQ4qSm0rRu0qMIt7V4hAlecvJjKGK0ZQkclSxH87LVv3lwzZLgtzedEW2PvmSdG4z4dZql7I6+Tqm3//NpTDGQNPp3G00+o7g1PA0I6IMDEZOUboCVPreOrWhzI6e5CBAQJqwWYTp+fmxJITDJRm5VzsQXx4ZHmqKNR1TtmgU+As73e0Fl/4ZVUaHvoF4UuQIKkT9YvSQ4SL/AFrlBCYeImckoBj06anqqE5QJXT8Zf9DIAkFi3/I1FOfUfTXfuiaBEaoxiABNnF4s7siIbJFlmRm0jk5M/UZ9qpGYgLPKU1JyOcbqPHzElTpYp1g5IDU8/J8piMgCsvhMxEV9TMwlVT+tGKODRhna4JLHmcb+fYGNHGG2E2Nhjb9Txmszx+8lbiLZnEpz9S4lYv9efsLQvscGJ7LTRwPDOOnj3t31qWyLP1Co9Qp2MaXMx/Spshgdt83kUw5KCtD1gFcxTPjJA/C3Rl4Y4zPCRiYW1rCPPOd9hUCl6dlPh1Rs8QPuMk17iHc9mkviR6gEr+V3w79LhTzmJaBsWm+Bu+xMQvvR8K+6FcNKiIbqOejhBpRZOMYIGksacfiylCz4TPe13O1WLEMth7gYDBld0xQLbW8Njn/sllp7oO7I30wnxztlA3YBaCSxGaklga78wXpQkQZwgWRMY2M1yXVv6z4w7a7EnWRc6b2SxYl+/LMzfca4DA/GDSt+MT4euAXOgI4fPsT+lYBbYwo7xIvfJtuD1FmXwjlq02l8Y0pbMzmOq2L7vqCxYZbddty X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3fb0281b-4fff-4e2b-b41b-08d8ed22b4df X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:39.4236 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: S6fuQDBjBC0nOEPC0s1WfPxrnEHO2X4FQedUrsxADfCDFBvohBn7GMXDU7l8Z0XlcyXlUManRS7yrNYI0i2HKw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Register svm range with same address and size but perferred_location is changed from CPU to GPU or from GPU to CPU, trigger migration the svm range from ram to vram or from vram to ram. If svm range prefetch location is GPU with flags KFD_IOCTL_SVM_FLAG_HOST_ACCESS, validate the svm range on ram first, then migrate it from ram to vram. After migrating to vram is done, CPU access will have cpu page fault, page fault handler migrate it back to ram and resume cpu access. Migration steps: 1. migrate_vma_pages get svm range ram pages, notify the interval is invalidated and unmap from CPU page table, HMM interval notifier callback evict process queues 2. Allocate new pages in vram using TTM 3. Use svm copy memory to sdma copy data from ram to vram 4. migrate_vma_pages copy ram pages structure to vram pages structure 5. migrate_vma_finalize put ram pages to free ram pages and memory 6. Restore work wait for migration is finished, then update GPUs page table mapping to new vram pages, resume process queues If migrate_vma_setup failed to collect all ram pages of range, retry 3 times until success to start migration. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 278 +++++++++++++++++++++++ drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 2 + drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 187 ++++++++++++++- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 4 + 4 files changed, 460 insertions(+), 11 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 2a6824ddae88..668c360be0bb 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -204,6 +204,284 @@ svm_migrate_copy_done(struct amdgpu_device *adev, struct dma_fence *mfence) return r; } +static uint64_t +svm_migrate_node_physical_addr(struct amdgpu_device *adev, + struct drm_mm_node **mm_node, uint64_t *offset) +{ + struct drm_mm_node *node = *mm_node; + uint64_t pos = *offset; + + if (node->start == AMDGPU_BO_INVALID_OFFSET) { + pr_debug("drm node is not validated\n"); + return 0; + } + + pr_debug("vram node start 0x%llx npages 0x%llx\n", node->start, + node->size); + + if (pos >= node->size) { + do { + pos -= node->size; + node++; + } while (pos >= node->size); + + *mm_node = node; + *offset = pos; + } + + return (node->start + pos) << PAGE_SHIFT; +} + +unsigned long +svm_migrate_addr_to_pfn(struct amdgpu_device *adev, unsigned long addr) +{ + return (addr + adev->kfd.dev->pgmap.range.start) >> PAGE_SHIFT; +} + +static void +svm_migrate_get_vram_page(struct svm_range *prange, unsigned long pfn) +{ + struct page *page; + + page = pfn_to_page(pfn); + page->zone_device_data = prange; + get_page(page); + lock_page(page); +} + +static void +svm_migrate_put_vram_page(struct amdgpu_device *adev, unsigned long addr) +{ + struct page *page; + + page = pfn_to_page(svm_migrate_addr_to_pfn(adev, addr)); + unlock_page(page); + put_page(page); +} + + +static int +svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange, + struct migrate_vma *migrate, struct dma_fence **mfence, + dma_addr_t *scratch) +{ + uint64_t npages = migrate->cpages; + struct device *dev = adev->dev; + struct drm_mm_node *node; + dma_addr_t *src; + uint64_t *dst; + uint64_t vram_addr; + uint64_t offset; + uint64_t i, j; + int r = -ENOMEM; + + pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, + prange->last); + + src = scratch; + dst = (uint64_t *)(scratch + npages); + + r = svm_range_vram_node_new(adev, prange, false); + if (r) { + pr_debug("failed %d get 0x%llx pages from vram\n", r, npages); + goto out; + } + + node = prange->ttm_res->mm_node; + offset = prange->offset; + vram_addr = svm_migrate_node_physical_addr(adev, &node, &offset); + if (!vram_addr) { + WARN_ONCE(1, "vram node address is 0\n"); + r = -ENOMEM; + goto out; + } + + for (i = j = 0; i < npages; i++) { + struct page *spage; + + spage = migrate_pfn_to_page(migrate->src[i]); + src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE, DMA_TO_DEVICE); + r = dma_mapping_error(dev, src[i]); + if (r) { + pr_debug("failed %d dma_map_page\n", r); + goto out_free_vram_pages; + } + + pr_debug("dma mapping src to 0x%llx, page_to_pfn 0x%lx\n", + src[i] >> PAGE_SHIFT, page_to_pfn(spage)); + + dst[i] = vram_addr + (j << PAGE_SHIFT); + migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); + svm_migrate_get_vram_page(prange, migrate->dst[i]); + + migrate->dst[i] = migrate_pfn(migrate->dst[i]); + migrate->dst[i] |= MIGRATE_PFN_LOCKED; + + if (j + offset >= node->size - 1 && i < npages - 1) { + r = svm_migrate_copy_memory_gart(adev, src + i - j, + dst + i - j, j + 1, + FROM_RAM_TO_VRAM, + mfence); + if (r) + goto out_free_vram_pages; + + node++; + pr_debug("next node size 0x%llx\n", node->size); + vram_addr = node->start << PAGE_SHIFT; + offset = 0; + j = 0; + } else { + j++; + } + } + + r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j, + FROM_RAM_TO_VRAM, mfence); + +out_free_vram_pages: + if (r) { + pr_debug("failed %d to copy memory to vram\n", r); + while (i--) { + svm_migrate_put_vram_page(adev, dst[i]); + migrate->dst[i] = 0; + } + } + +out: + return r; +} + +static int +svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange, + struct vm_area_struct *vma, uint64_t start, + uint64_t end) +{ + uint64_t npages = (end - start) >> PAGE_SHIFT; + struct dma_fence *mfence = NULL; + struct migrate_vma migrate; + dma_addr_t *scratch; + size_t size; + void *buf; + int r = -ENOMEM; + int retry = 0; + + memset(&migrate, 0, sizeof(migrate)); + migrate.vma = vma; + migrate.start = start; + migrate.end = end; + migrate.flags = MIGRATE_VMA_SELECT_SYSTEM; + migrate.pgmap_owner = adev; + + size = 2 * sizeof(*migrate.src) + sizeof(uint64_t) + sizeof(dma_addr_t); + size *= npages; + buf = kvmalloc(size, GFP_KERNEL | __GFP_ZERO); + if (!buf) + goto out; + + migrate.src = buf; + migrate.dst = migrate.src + npages; + scratch = (dma_addr_t *)(migrate.dst + npages); + +retry: + r = migrate_vma_setup(&migrate); + if (r) { + pr_debug("failed %d prepare migrate svms 0x%p [0x%lx 0x%lx]\n", + r, prange->svms, prange->start, prange->last); + goto out_free; + } + if (migrate.cpages != npages) { + pr_debug("collect 0x%lx/0x%llx pages, retry\n", migrate.cpages, + npages); + migrate_vma_finalize(&migrate); + if (retry++ >= 3) { + r = -ENOMEM; + pr_debug("failed %d migrate svms 0x%p [0x%lx 0x%lx]\n", + r, prange->svms, prange->start, prange->last); + goto out_free; + } + + goto retry; + } + + if (migrate.cpages) { + svm_migrate_copy_to_vram(adev, prange, &migrate, &mfence, + scratch); + migrate_vma_pages(&migrate); + svm_migrate_copy_done(adev, mfence); + migrate_vma_finalize(&migrate); + } + + kvfree(prange->pages_addr); + prange->pages_addr = NULL; + + svm_range_dma_unmap(adev->dev, scratch, 0, npages); + +out_free: + kvfree(buf); +out: + return r; +} + +/** + * svm_migrate_ram_to_vram - migrate svm range from system to device + * @prange: range structure + * @best_loc: the device to migrate to + * + * Context: Process context, caller hold mmap read lock, svms lock, prange lock + * + * Return: + * 0 - OK, otherwise error code + */ +int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc) +{ + unsigned long addr, start, end; + struct vm_area_struct *vma; + struct amdgpu_device *adev; + struct mm_struct *mm; + int r = 0; + + if (prange->actual_loc == best_loc) { + pr_debug("svms 0x%p [0x%lx 0x%lx] already on best_loc 0x%x\n", + prange->svms, prange->start, prange->last, best_loc); + return 0; + } + + adev = svm_range_get_adev_by_id(prange, best_loc); + if (!adev) { + pr_debug("failed to get device by id 0x%x\n", best_loc); + return -ENODEV; + } + + pr_debug("svms 0x%p [0x%lx 0x%lx] to gpu 0x%x\n", prange->svms, + prange->start, prange->last, best_loc); + + start = prange->start << PAGE_SHIFT; + end = (prange->last + 1) << PAGE_SHIFT; + + mm = current->mm; + + for (addr = start; addr < end;) { + unsigned long next; + + vma = find_vma(mm, addr); + if (!vma || addr < vma->vm_start) + break; + + next = min(vma->vm_end, end); + r = svm_migrate_vma_to_vram(adev, prange, vma, addr, next); + if (r) { + pr_debug("failed to migrate\n"); + break; + } + addr = next; + } + + if (!r) + prange->actual_loc = best_loc; + + return r; +} + static void svm_migrate_page_free(struct page *page) { } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h index 5db5686fa46a..ffae5f989909 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h @@ -37,6 +37,8 @@ enum MIGRATION_COPY_DIR { FROM_VRAM_TO_RAM }; +int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc); + #if defined(CONFIG_DEVICE_PRIVATE) int svm_migrate_init(struct amdgpu_device *adev); void svm_migrate_fini(struct amdgpu_device *adev); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index de5777330d23..cf3d2c203007 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -30,6 +30,7 @@ #include "amdgpu_xgmi.h" #include "kfd_priv.h" #include "kfd_svm.h" +#include "kfd_migrate.h" #define AMDGPU_SVM_RANGE_RESTORE_DELAY_MS 1 @@ -241,6 +242,7 @@ svm_range *svm_range_new(struct svm_range_list *svms, uint64_t start, INIT_LIST_HEAD(&prange->deferred_list); INIT_LIST_HEAD(&prange->child_list); atomic_set(&prange->invalid, 0); + mutex_init(&prange->migrate_mutex); mutex_init(&prange->lock); svm_range_set_default_attributes(&prange->preferred_loc, &prange->prefetch_loc, @@ -502,6 +504,11 @@ static int svm_range_validate_vram(struct svm_range *prange) pr_debug("svms 0x%p [0x%lx 0x%lx] actual_loc 0x%x\n", prange->svms, prange->start, prange->last, prange->actual_loc); + if (prange->ttm_res) { + pr_debug("validation skipped after migration\n"); + return 0; + } + adev = svm_range_get_adev_by_id(prange, prange->actual_loc); if (!adev) { pr_debug("failed to get device by id 0x%x\n", @@ -521,7 +528,9 @@ svm_range_validate(struct mm_struct *mm, struct svm_range *prange) { int r; - pr_debug("actual loc 0x%x\n", prange->actual_loc); + pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx] actual loc 0x%x\n", + prange->svms, prange, prange->start, prange->last, + prange->actual_loc); if (!prange->actual_loc) r = svm_range_validate_ram(mm, prange); @@ -1237,28 +1246,36 @@ static void svm_range_restore_work(struct work_struct *work) prange->svms, prange, prange->start, prange->last, invalid); + /* + * If range is migrating, wait for migration is done. + */ + mutex_lock(&prange->migrate_mutex); + r = svm_range_validate(mm, prange); if (r) { pr_debug("failed %d to validate [0x%lx 0x%lx]\n", r, prange->start, prange->last); - goto unlock_out; + goto out_unlock; } r = svm_range_map_to_gpus(prange, true); - if (r) { + if (r) pr_debug("failed %d to map 0x%lx to gpu\n", r, prange->start); - goto unlock_out; - } + +out_unlock: + mutex_unlock(&prange->migrate_mutex); + if (r) + goto out_reschedule; if (atomic_cmpxchg(&prange->invalid, invalid, 0) != invalid) - goto unlock_out; + goto out_reschedule; } if (atomic_cmpxchg(&svms->evicted_ranges, evicted_ranges, 0) != evicted_ranges) - goto unlock_out; + goto out_reschedule; evicted_ranges = 0; @@ -1272,7 +1289,7 @@ static void svm_range_restore_work(struct work_struct *work) pr_debug("restore svm ranges successfully\n"); -unlock_out: +out_reschedule: mutex_unlock(&svms->lock); mmap_write_unlock(mm); mutex_unlock(&process_info->lock); @@ -1575,6 +1592,7 @@ static void svm_range_deferred_list_work(struct work_struct *work) list_del_init(&prange->deferred_list); spin_unlock(&svms->deferred_list_lock); + mutex_lock(&prange->migrate_mutex); while (!list_empty(&prange->child_list)) { struct svm_range *pchild; @@ -1585,6 +1603,7 @@ static void svm_range_deferred_list_work(struct work_struct *work) list_del_init(&pchild->child_list); svm_range_handle_list_op(svms, pchild); } + mutex_unlock(&prange->migrate_mutex); svm_range_handle_list_op(svms, prange); mutex_unlock(&svms->lock); @@ -1867,6 +1886,135 @@ svm_range_add(struct kfd_process *p, uint64_t start, uint64_t size, return 0; } +/* svm_range_best_location - decide the best actual location + * @prange: svm range structure + * + * For xnack off: + * If range map to single GPU, the best acutal location is prefetch loc, which + * can be CPU or GPU. + * + * If range map to multiple GPUs, only if mGPU connection on xgmi same hive, + * the best actual location could be prefetch_loc GPU. If mGPU connection on + * PCIe, the best actual location is always CPU, because GPU cannot access vram + * of other GPUs, assuming PCIe small bar (large bar support is not upstream). + * + * For xnack on: + * The best actual location is prefetch location. If mGPU connection on xgmi + * same hive, range map to multiple GPUs. Otherwise, the range only map to + * actual location GPU. Other GPU access vm fault will trigger migration. + * + * Context: Process context + * + * Return: + * 0 for CPU or GPU id + */ +static uint32_t svm_range_best_location(struct svm_range *prange) +{ + DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE); + uint32_t best_loc = prange->prefetch_loc; + struct amdgpu_device *bo_adev; + struct amdgpu_device *adev; + struct kfd_dev *kfd_dev; + struct kfd_process *p; + uint32_t gpuidx; + + p = container_of(prange->svms, struct kfd_process, svms); + + /* xnack on */ + if (p->xnack_enabled) + goto out; + + /* xnack off */ + if (!best_loc || best_loc == KFD_IOCTL_SVM_LOCATION_UNDEFINED) + goto out; + + bo_adev = svm_range_get_adev_by_id(prange, best_loc); + bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, + MAX_GPU_INSTANCE); + + for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { + kfd_process_device_from_gpuidx(p, gpuidx, &kfd_dev); + adev = (struct amdgpu_device *)kfd_dev->kgd; + + if (adev == bo_adev) + continue; + + if (!amdgpu_xgmi_same_hive(adev, bo_adev)) { + best_loc = 0; + break; + } + } + +out: + pr_debug("xnack %d svms 0x%p [0x%lx 0x%lx] best loc 0x%x\n", + p->xnack_enabled, &p->svms, prange->start, prange->last, + best_loc); + + return best_loc; +} + +/* svm_range_trigger_migration - start page migration if prefetch loc changed + * @mm: current process mm_struct + * @prange: svm range structure + * @migrated: output, true if migration is triggered + * + * If range perfetch_loc is GPU, actual loc is cpu 0, then migrate the range + * from ram to vram. + * If range prefetch_loc is cpu 0, actual loc is GPU, then migrate the range + * from vram to ram. + * + * If GPU vm fault retry is not enabled, migration interact with MMU notifier + * and restore work: + * 1. migrate_vma_setup invalidate pages, MMU notifier callback svm_range_evict + * stops all queues, schedule restore work + * 2. svm_range_restore_work wait for migration is done by + * a. svm_range_validate_vram takes prange->migrate_mutex + * b. svm_range_validate_ram HMM get pages wait for CPU fault handle returns + * 3. restore work update mappings of GPU, resume all queues. + * + * Context: Process context + * + * Return: + * 0 - OK, otherwise - error code of migration + */ +static int +svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange, + bool *migrated) +{ + uint32_t best_loc; + int r = 0; + + *migrated = false; + best_loc = svm_range_best_location(prange); + + if (best_loc == KFD_IOCTL_SVM_LOCATION_UNDEFINED || + best_loc == prange->actual_loc) + return 0; + + if (best_loc && !prange->actual_loc && + !(prange->flags & KFD_IOCTL_SVM_FLAG_HOST_ACCESS)) + return 0; + + if (best_loc) { + if (!prange->actual_loc && !prange->pages_addr) { + pr_debug("host access and prefetch to gpu\n"); + r = svm_range_validate_ram(mm, prange); + if (r) { + pr_debug("failed %d to validate on ram\n", r); + return r; + } + } + + pr_debug("migrate from ram to vram\n"); + r = svm_migrate_ram_to_vram(prange, best_loc); + + if (!r) + *migrated = true; + } + + return r; +} + static int svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, uint32_t nattr, struct kfd_ioctl_svm_attribute *attrs) @@ -1937,17 +2085,34 @@ svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, * case because the rollback wouldn't be guaranteed to work either. */ list_for_each_entry(prange, &update_list, update_list) { + bool migrated; + + mutex_lock(&prange->migrate_mutex); + + r = svm_range_trigger_migration(mm, prange, &migrated); + if (r) + goto out_unlock_range; + + if (migrated) { + pr_debug("restore_work will update mappings of GPUs\n"); + mutex_unlock(&prange->migrate_mutex); + continue; + } + r = svm_range_validate(mm, prange); if (r) { pr_debug("failed %d to validate svm range\n", r); - break; + goto out_unlock_range; } r = svm_range_map_to_gpus(prange, true); - if (r) { + if (r) pr_debug("failed %d to map svm range\n", r); + +out_unlock_range: + mutex_unlock(&prange->migrate_mutex); + if (r) break; - } } svm_range_debug_dump(svms); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 8e2b63a134f1..aa6edc3efcb8 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -55,6 +55,7 @@ struct svm_work_list_item { * struct svm_range - shared virtual memory range * * @svms: list of svm ranges, structure defined in kfd_process + * @migrate_mutex: to serialize range migration, validation and mapping update * @start: range start address in pages * @last: range last address in pages * @it_node: node [start, last] stored in interval tree, start, last are page @@ -93,6 +94,7 @@ struct svm_work_list_item { */ struct svm_range { struct svm_range_list *svms; + struct mutex migrate_mutex; unsigned long start; unsigned long last; struct interval_tree_node it_node; @@ -147,5 +149,7 @@ struct amdgpu_device *svm_range_get_adev_by_id(struct svm_range *prange, int svm_range_vram_node_new(struct amdgpu_device *adev, struct svm_range *prange, bool clear); void svm_range_vram_node_free(struct svm_range *prange); +void svm_range_dma_unmap(struct device *dev, dma_addr_t *dma_addr, + unsigned long offset, unsigned long npages); #endif /* KFD_SVM_H_ */ From patchwork Mon Mar 22 10:58:35 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154527 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.7 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS, UNWANTED_LANGUAGE_BODY,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D88A1C433E0 for ; Mon, 22 Mar 2021 11:08:52 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A1F326191A for ; Mon, 22 Mar 2021 11:08:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A1F326191A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E92306E431; Mon, 22 Mar 2021 11:07:59 +0000 (UTC) Received: from NAM12-BN8-obe.outbound.protection.outlook.com (mail-bn8nam12on2045.outbound.protection.outlook.com [40.107.237.45]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5BFF189FA9; Mon, 22 Mar 2021 11:07:42 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=LCTjyLJnfRDqpgwYRtAcrnhN64QrP5LbzEnx8XxQTREMO/ITKnnKBLyo6ZR6Atnk/nztLIMFpu+jjBPf4m5h1Ah7zH2ZzJ8LwlUGJp7PSWMWmsc221Ya/2PHy98UoA64XJ0eBxuu6s5DbATM0PiD/+FzYoBn201Eg8tqQ7mLFrrL+B/pNB6ayiHVcpGtihv/4EgKPUTXNiaieoyEZynyjGCEdT1OmfGj99FZSHafRKLsgUlek6C2qHXsFzr8+/IvEf/X7VpDlTZWPiPpzjNO44EfcW8qCok3rDaa4oXQjoEK0Sc43P+FCwn/qTGxrVRxTILVcL+1OgLDxb+CVRZH6w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NWVz8MEXoqCDgWsEtrLMap/GakmGyAuF+3zqLhlSoOQ=; b=grqptmbaAX1zyXgfdDEq28hnawzM1L0/dnWUqQsE5hjyxF+OVvwsq/MtTFUu2m/jxt750yOQGcNm9UGRgFbomhFH70Wy1c+LNWFDjrGqC+I21VAE1sGpJhDhrxGLTU9R75+xi8il4WlcITUmZasTeCdugAhUoPT52yDYQbUd1f/yruJ6v+OONE/59YqHbTxBowMJPoQAZcRp9KNQL1x/ayeeuOemW1M07cQiDvsIu2AjONk+g8ovgBYqFQrOHmeKsiHgvq98RWldZVzyCCZy/BJV5Xnir+X469IDA8E+m8mBDHSPE+1Yasrm7BsOxdKe79JyNVpg4IxR4fMZFK0cDw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=NWVz8MEXoqCDgWsEtrLMap/GakmGyAuF+3zqLhlSoOQ=; b=wDuF68WUu0vqOepCifcnPDxag+w6PbxDQ+AaVaXsvZg0Lq0pnTslLQddDD4jpgBByGjTDmXwQeMfzDFiOOTKLtg/fbnI+qZMyEOGWRtojBv4lX66m4Bc4r4MIQR874DHx8bxQYi/qMXh33J3ix0bKdOQqdyld69ZbtiCY+Zz54Y= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB3678.namprd12.prod.outlook.com (2603:10b6:208:158::26) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:40 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:40 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 19/44] drm/amdkfd: HMM migrate vram to ram Date: Mon, 22 Mar 2021 06:58:35 -0400 Message-Id: <20210322105900.14068-20-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:39 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 271652af-5d0c-4555-0502-08d8ed22b534 X-MS-TrafficTypeDiagnostic: MN2PR12MB3678: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:330; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: BlI9Zz1XqiMdd5gSlxI9Rn2FFhjWcD60gWjlM8n00bMg2vCXjIPsIKwS+WO/APsNQVChOzTeZWGnbvWVQyHR1NlC1u+IqJyAvzeldU51btBkGyULoi4JzWB5D5GbnFLJvmIWUK0n2hyeL5jPzUGHZ5vW1b/gleCc3xUmYTpiM4ruAFMuVTAm99j7sqi7OuAdyk9eoCxgWpTmAHCdVc1yGN/EEwbUiLSr4iYSxjFP7FNfvDmqSvnAx9p8BbDuDEoj9NpFLBqY0IRr6/p8G1+3MTEsOnfiblg28w/lEECVO9ViC5duWxwy1VYY0u0I1aSfcQzMRJffbs5+eEUF8Sa0chBJ7YXdhHkosWrAJwhFd1oqZrY0XTONH90SovwvnYkYfulwn34AKv4VAxly6aLh8qcZOrfkqEifFiHX6w+5GB2CfWi1jPHuYBdYA5Mm+wZx33ABLM3dGQr65WXtQbW5ScvM1hzVbvj7JWZclnpxldsAqu2ocuOQ+9WlfajJ4zU4Yzt65gQfaRPdSDcI2jozaOh0e/IX5y7rC2WOYYe37+SSxvtlEQRt7ItpLxKsKeZwO8bsnuni3LeA4b8Y1ulUmI88xQnNcaXp2oMOzewX3UH9cpz84iTupNb/jgOAKNJiEuGz8JQR0HMNTNuou2KVox92woopgrTPgyBdw4MuAFM= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(376002)(136003)(366004)(39860400002)(346002)(52116002)(7696005)(6486002)(1076003)(36756003)(30864003)(38100700001)(6666004)(83380400001)(478600001)(2616005)(956004)(4326008)(450100002)(16526019)(2906002)(186003)(26005)(5660300002)(66556008)(66476007)(8936002)(66946007)(316002)(86362001)(8676002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: wDBysCpiLmMq70vEpnyYsKSBKcZSwpTh05KdMJlHFk7wV3aMd18MMNaJz0YPedfmIgOFfiMyrkgJDbc351vZnTTHlyDIh4RGOKswp5syln3Rwr7eIEQta/eYA2lLTntbZeHzma4E7ZkqRhS0Dsyy5foRnRmYJfgspg7FYxydgbfxmZiexiUr6p5Oc9BEaZzvm6/eYQlCoiNa25HaJydOrqEDo3LXuY5tj5FTi0KD2rYuHb54bXeFor/O7wHLXcRK+LQ/E93f00XYMM1HCcw9o0UcKxKuLx7BUfj8qy34A8MeuidfygPWuVKKGzZBjpr+L3w2jlnx26JvFsZSb5ZztGH9GfDfQkDsHlHaco2u6IdmalD+vxtF6rNw2rinrxi8y4WNpQoQm9CTvmdS0zn8kl/f2wmTpIZi/eFVy7jMRzJm+nd2YE4fuY4CiAG/oZKYiR07hClvAJ4OkUu8ICyew8zwBsitNQF5GHY7IyXSywiW0Uz9YuplWlCDIzJ+W6hyDiET85gLve9RS4LVTx7OIxG0+Ku8Kc7c+eTsRp7666XuwKsePIn7O5UOSq/ILpQKR6RHSe/XgiO74DO6P1w1brj8JoXrqGxlOZbNQp0Mw6l/bJU3FUanoNAhy5iygdw/9XKdurOPUahyMnzDciXfqebxWCHPdWLXKP2OFWT+/MuAhkpTH5j7HZri+y2WxIuQpI8Ii4qMkGo3pwL2RacVTOf/IIgct8W6okD1DdSnQQ4GGOTTGH1PqinxJc93+nS1/GF9nrSkRkhNHzHnqctUUjdVl7fcqPEvt88NznkEWzlhwsokbMsS95jSPxJO12UpRcxMtXfdkO3Kd1+NevCjzo3hYsj0GP90nr1no5OsG9iW+sWALd/6tU8vswaHZQOLvK89NP4oz4dtokV5PJUWtm0tjvJMX5T2/UFROtSXDj56byzIdswklUkRwvXTh8nBKkJQy3GBOGFmiBT9UIyTsDOUAAQjIC/GxGBeX3UwKHWnAK+lvrBviabgEOzqz1jM4nni7twwe4tBId3FVTrZEuh+iZLVAmWStKcPU8/C4fHU5lHKCn8txHG1b2W+sSIQOrp/WbN9ykfWLlE49W66tsgQJsNAyG7ws8afcFoFf2h1t5Kthwu3oxWDMUkiSlw5ZPuenP0ecXUFvIMx/tkNUG6qkwEZvACI3s2GKBvs6MKuNHax7xVNbovrjBMfZY58xpcUGi3LGekpMnQowKlOEV2mztAou7/gDAKzVXzA8ykDtIm9QgVmbSveiEEOj+j44EYJnye7Lc6iLkWlM7bdjs1LKsRtGeZEVnmyhRMHeSYvlpyCk6x+CZkk6WrRW0FN X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 271652af-5d0c-4555-0502-08d8ed22b534 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:39.9463 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: AaotapnSuBKPGKeLj/CYNc65/RkIJncfK+UG5gT14RpNRdqe6/zG0q8I9vegjHz4YGAL2E/lsXE7vRelGm0h9Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB3678 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" If CPU page fault happens, HMM pgmap_ops callback migrate_to_ram start migrate memory from vram to ram in steps: 1. migrate_vma_pages get vram pages, and notify HMM to invalidate the pages, HMM interval notifier callback evict process queues 2. Allocate system memory pages 3. Use svm copy memory to migrate data from vram to ram 4. migrate_vma_pages copy pages structure from vram pages to ram pages 5. Return VM_FAULT_SIGBUS if migration failed, to notify application 6. migrate_vma_finalize put vram pages, page_free callback free vram pages and vram nodes 7. Restore work wait for migration is finished, then update GPU page table mapping to system memory, and resume process queues Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 310 ++++++++++++++++++++++- drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 3 + drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 158 +++++++++++- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 12 + 4 files changed, 473 insertions(+), 10 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 668c360be0bb..1243cf02f872 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -259,6 +259,35 @@ svm_migrate_put_vram_page(struct amdgpu_device *adev, unsigned long addr) put_page(page); } +static unsigned long +svm_migrate_addr(struct amdgpu_device *adev, struct page *page) +{ + unsigned long addr; + + addr = page_to_pfn(page) << PAGE_SHIFT; + return (addr - adev->kfd.dev->pgmap.range.start); +} + +static struct page * +svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long addr) +{ + struct page *page; + + page = alloc_page_vma(GFP_HIGHUSER, vma, addr); + if (page) + lock_page(page); + + return page; +} + +void svm_migrate_put_sys_page(unsigned long addr) +{ + struct page *page; + + page = pfn_to_page(addr >> PAGE_SHIFT); + unlock_page(page); + put_page(page); +} static int svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange, @@ -484,13 +513,222 @@ int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc) static void svm_migrate_page_free(struct page *page) { + /* Keep this function to avoid warning */ +} + +static int +svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, + struct migrate_vma *migrate, struct dma_fence **mfence, + dma_addr_t *scratch) +{ + uint64_t npages = migrate->cpages; + struct device *dev = adev->dev; + uint64_t *src; + dma_addr_t *dst; + struct page *dpage; + uint64_t i = 0, j; + uint64_t addr; + int r = 0; + + pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, + prange->last); + + addr = prange->start << PAGE_SHIFT; + + src = (uint64_t *)(scratch + npages); + dst = scratch; + + prange->pages_addr = kvmalloc_array(npages, sizeof(*prange->pages_addr), + GFP_KERNEL | __GFP_ZERO); + if (!prange->pages_addr) { + r = -ENOMEM; + goto out_oom; + } + + for (i = 0, j = 0; i < npages; i++, j++, addr += PAGE_SIZE) { + struct page *spage; + + spage = migrate_pfn_to_page(migrate->src[i]); + if (!spage) { + pr_debug("failed get spage svms 0x%p [0x%lx 0x%lx]\n", + prange->svms, prange->start, prange->last); + r = -ENOMEM; + goto out_oom; + } + src[i] = svm_migrate_addr(adev, spage); + if (i > 0 && src[i] != src[i - 1] + PAGE_SIZE) { + r = svm_migrate_copy_memory_gart(adev, dst + i - j, + src + i - j, j, + FROM_VRAM_TO_RAM, + mfence); + if (r) + goto out_oom; + j = 0; + } + + dpage = svm_migrate_get_sys_page(migrate->vma, addr); + if (!dpage) { + pr_debug("failed get page svms 0x%p [0x%lx 0x%lx]\n", + prange->svms, prange->start, prange->last); + r = -ENOMEM; + goto out_oom; + } + + dst[i] = dma_map_page(dev, dpage, 0, PAGE_SIZE, DMA_FROM_DEVICE); + r = dma_mapping_error(dev, dst[i]); + if (r) { + pr_debug("failed %d dma_map_page\n", r); + goto out_oom; + } + + pr_debug("dma mapping dst to 0x%llx, page_to_pfn 0x%lx\n", + dst[i] >> PAGE_SHIFT, page_to_pfn(dpage)); + + prange->pages_addr[i] = page_to_pfn(dpage); + + migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)); + migrate->dst[i] |= MIGRATE_PFN_LOCKED; + } + + r = svm_migrate_copy_memory_gart(adev, dst + i - j, src + i - j, j, + FROM_VRAM_TO_RAM, mfence); + +out_oom: + if (r) { + pr_debug("failed %d copy to ram\n", r); + while (i--) { + svm_migrate_put_sys_page(dst[i]); + migrate->dst[i] = 0; + } + } + + return r; +} + +static int +svm_migrate_vma_to_ram(struct amdgpu_device *adev, struct svm_range *prange, + struct vm_area_struct *vma, uint64_t start, uint64_t end) +{ + uint64_t npages = (end - start) >> PAGE_SHIFT; + struct dma_fence *mfence = NULL; + struct migrate_vma migrate; + dma_addr_t *scratch; + size_t size; + void *buf; + int r = -ENOMEM; + + memset(&migrate, 0, sizeof(migrate)); + migrate.vma = vma; + migrate.start = start; + migrate.end = end; + migrate.flags = MIGRATE_VMA_SELECT_DEVICE_PRIVATE; + migrate.pgmap_owner = adev; + + size = 2 * sizeof(*migrate.src) + sizeof(uint64_t) + sizeof(dma_addr_t); + size *= npages; + buf = kvmalloc(size, GFP_KERNEL | __GFP_ZERO); + if (!buf) + goto out; + + migrate.src = buf; + migrate.dst = migrate.src + npages; + scratch = (dma_addr_t *)(migrate.dst + npages); + + r = migrate_vma_setup(&migrate); + if (r) { + pr_debug("failed %d prepare migrate svms 0x%p [0x%lx 0x%lx]\n", + r, prange->svms, prange->start, prange->last); + goto out_free; + } + + pr_debug("cpages %ld\n", migrate.cpages); + + if (migrate.cpages) { + svm_migrate_copy_to_ram(adev, prange, &migrate, &mfence, + scratch); + migrate_vma_pages(&migrate); + svm_migrate_copy_done(adev, mfence); + migrate_vma_finalize(&migrate); + } else { + pr_debug("failed collect migrate device pages [0x%lx 0x%lx]\n", + prange->start, prange->last); + } + + svm_range_dma_unmap(adev->dev, scratch, 0, npages); + +out_free: + kvfree(buf); +out: + return r; +} + +/** + * svm_migrate_vram_to_ram - migrate svm range from device to system + * @prange: range structure + * @mm: process mm, use current->mm if NULL + * + * Context: Process context, caller hold mmap read lock, svms lock, prange lock + * + * Return: + * 0 - OK, otherwise error code + */ +int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm) +{ + struct amdgpu_device *adev; + struct vm_area_struct *vma; + unsigned long addr; + unsigned long start; + unsigned long end; + int r = 0; + + if (!prange->actual_loc) { + pr_debug("[0x%lx 0x%lx] already migrated to ram\n", + prange->start, prange->last); + return 0; + } + + adev = svm_range_get_adev_by_id(prange, prange->actual_loc); + if (!adev) { + pr_debug("failed to get device by id 0x%x\n", + prange->actual_loc); + return -ENODEV; + } + + pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx] from gpu 0x%x to ram\n", + prange->svms, prange, prange->start, prange->last, + prange->actual_loc); + + start = prange->start << PAGE_SHIFT; + end = (prange->last + 1) << PAGE_SHIFT; + + for (addr = start; addr < end;) { + unsigned long next; + + vma = find_vma(mm, addr); + if (!vma || addr < vma->vm_start) + break; + + next = min(vma->vm_end, end); + r = svm_migrate_vma_to_ram(adev, prange, vma, addr, next); + if (r) { + pr_debug("failed %d to migrate\n", r); + break; + } + addr = next; + } + + if (!r) { + svm_range_vram_node_free(prange); + prange->actual_loc = 0; + } + return r; } /** * svm_migrate_to_ram - CPU page fault handler * @vmf: CPU vm fault vma, address * - * Context: vm fault handler, mm->mmap_sem is taken + * Context: vm fault handler, caller holds the mmap read lock * * Return: * 0 - OK @@ -498,7 +736,75 @@ static void svm_migrate_page_free(struct page *page) */ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) { - return VM_FAULT_SIGBUS; + unsigned long addr = vmf->address; + struct list_head update_list; + struct svm_range *pmigrate; + struct vm_area_struct *vma; + struct svm_range *parent; + struct svm_range *prange; + struct svm_range *next; + struct kfd_process *p; + struct mm_struct *mm; + int r = 0; + + vma = vmf->vma; + mm = vma->vm_mm; + + p = kfd_lookup_process_by_mm(vma->vm_mm); + if (!p) { + pr_debug("failed find process at fault address 0x%lx\n", addr); + return VM_FAULT_SIGBUS; + } + addr >>= PAGE_SHIFT; + pr_debug("CPU page fault svms 0x%p address 0x%lx\n", &p->svms, addr); + + mutex_lock(&p->svms.lock); + + prange = svm_range_from_addr(&p->svms, addr, &parent); + if (!prange) { + pr_debug("cannot find svm range at 0x%lx\n", addr); + r = -EFAULT; + goto out; + } + + mutex_lock(&parent->migrate_mutex); + + if (!prange->actual_loc) + goto out_unlock_prange; + + svm_range_lock(parent); + r = svm_range_split_by_granularity(p, mm, addr, parent, prange, + &pmigrate, &update_list); + svm_range_unlock(parent); + if (r) { + pr_debug("failed %d to split range by granularity\n", r); + goto out_unlock_prange; + } + + r = svm_migrate_vram_to_ram(pmigrate, mm); + if (r) + pr_debug("failed %d migrate 0x%p [0x%lx 0x%lx] to ram\n", r, + pmigrate, pmigrate->start, pmigrate->last); + + list_for_each_entry_safe(prange, next, &update_list, update_list) { + enum svm_work_list_ops op; + + op = SVM_OP_UPDATE_RANGE_NOTIFIER; + + svm_range_add_list_work(&p->svms, prange, mm, op); + list_del_init(&prange->update_list); + } + schedule_deferred_list_work(&p->svms); + +out_unlock_prange: + mutex_unlock(&parent->migrate_mutex); +out: + mutex_unlock(&p->svms.lock); + kfd_unref_process(p); + + pr_debug("CPU fault svms 0x%p address 0x%lx done\n", &p->svms, addr); + + return r ? VM_FAULT_SIGBUS : 0; } static const struct dev_pagemap_ops svm_migrate_pgmap_ops = { diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h index ffae5f989909..95fd7b21791f 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h @@ -38,6 +38,9 @@ enum MIGRATION_COPY_DIR { }; int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc); +int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm); +unsigned long +svm_migrate_addr_to_pfn(struct amdgpu_device *adev, unsigned long addr); #if defined(CONFIG_DEVICE_PRIVATE) int svm_migrate_init(struct amdgpu_device *adev); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index cf3d2c203007..69241ed4a377 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -920,6 +920,95 @@ void svm_range_add_child(struct svm_range *prange, struct mm_struct *mm, list_add_tail(&pchild->child_list, &prange->child_list); } +/** + * svm_range_split_by_granularity - collect ranges within granularity boundary + * + * @p: the process with svms list + * @mm: mm structure + * @parent: parent range if prange is from child list + * @prange: prange to split + * @addr: the vm fault address in pages, to split the prange + * @pmigrate: output, the range to be migrated to ram + * @update_list: output, the ranges to update notifier + * + * Collects small ranges that make up one migration granule and splits the first + * and the last range at the granularity boundary + * + * Context: caller hold svms lock + * + * Return: + * 0 - OK, otherwise error code + */ +int +svm_range_split_by_granularity(struct kfd_process *p, struct mm_struct *mm, + unsigned long addr, struct svm_range *parent, + struct svm_range *prange, + struct svm_range **pmigrate, + struct list_head *update_list) +{ + struct svm_range *tail; + struct svm_range *new; + unsigned long start; + unsigned long last; + unsigned long size; + int r = 0; + + /* Align splited range start and size to granularity size, then a single + * PTE will be used for whole range, this reduces the number of PTE + * updated and the L1 TLB space used for translation. + */ + size = 1ULL << prange->granularity; + start = ALIGN_DOWN(addr, size); + last = ALIGN(addr + 1, size) - 1; + INIT_LIST_HEAD(update_list); + INIT_LIST_HEAD(&parent->update_list); + + pr_debug("svms 0x%p split [0x%lx 0x%lx] to [0x%lx 0x%lx] size 0x%lx\n", + prange->svms, prange->start, prange->last, start, last, size); + + if (start > prange->start) { + r = svm_range_split(prange, prange->start, start - 1, &new); + if (r) + return r; + + svm_range_add_child(parent, mm, new, SVM_OP_ADD_RANGE); + + if (parent == prange) { + pr_debug("add to update list prange 0x%p [0x%lx 0x%lx]\n", + parent, parent->start, parent->last); + list_add(&parent->update_list, update_list); + } + } else { + new = prange; + } + + if (last >= new->last) { + pr_debug("entire prange 0x%p [0x%lx 0x%lx] on prange %s list\n", + new, new->start, new->last, + (parent == prange) ? "" : "child"); + goto out_update; + } + + pr_debug("split remaining last 0x%lx [0x%lx 0x%lx] from prange %s\n", + last, new->start, new->last, (parent == new) ? "" : "child"); + r = svm_range_split(new, new->start, last, &tail); + if (r) + return r; + svm_range_add_child(parent, mm, tail, SVM_OP_ADD_RANGE); + +out_update: + /* If parent is not on update list, add it to put into deferred work */ + if (list_empty(&parent->update_list)) { + pr_debug("add to update list parange 0x%p [0x%lx 0x%lx]\n", + prange, parent->start, parent->last); + list_add(&parent->update_list, update_list); + } + + *pmigrate = new; + + return 0; +} + static uint64_t svm_range_get_pte_flags(struct amdgpu_device *adev, struct svm_range *prange) { @@ -1723,12 +1812,19 @@ svm_range_unmap_from_cpu(struct mm_struct *mm, struct svm_range *prange, /** * svm_range_cpu_invalidate_pagetables - interval notifier callback * - * MMU range unmap notifier to remove svm ranges + * If event is MMU_NOTIFY_UNMAP, this is from CPU unmap range, otherwise, it + * is from migration, or CPU page invalidation callback. + * + * For unmap event, unmap range from GPUs, remove prange from svms in a delayed + * work thread, and split prange if only part of prange is unmapped. + * + * For invalidation event, if GPU retry fault is not enabled, evict the queues, + * then schedule svm_range_restore_work to update GPU mapping and resume queues. + * If GPU retry fault is enabled, unmap the svm range from GPU, retry fault will + * update GPU mapping to recover. * - * If GPU vm fault retry is not enabled, evict the svm range, then restore - * work will update GPU mapping. - * If GPU vm fault retry is enabled, unmap the svm range from GPU, vm fault - * will update GPU mapping. + * Context: mmap lock, notifier_invalidate_start lock are held + * for invalidate event, prange lock is held if this is from migration */ static bool svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, @@ -1766,6 +1862,49 @@ svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, return true; } +/** + * svm_range_from_addr - find svm range from fault address + * @svms: svm range list header + * @addr: address to search range interval tree, in pages + * @parent: parent range if range is on child list + * + * Context: The caller must hold svms->lock + * + * Return: the svm_range found or NULL + */ +struct svm_range * +svm_range_from_addr(struct svm_range_list *svms, unsigned long addr, + struct svm_range **parent) +{ + struct interval_tree_node *node; + struct svm_range *prange; + struct svm_range *pchild; + + node = interval_tree_iter_first(&svms->objects, addr, addr); + if (!node) + return NULL; + + prange = container_of(node, struct svm_range, it_node); + pr_debug("address 0x%lx prange [0x%lx 0x%lx] node [0x%lx 0x%lx]\n", + addr, prange->start, prange->last, node->start, node->last); + + if (addr >= prange->start && addr <= prange->last) { + if (parent) + *parent = prange; + return prange; + } + list_for_each_entry(pchild, &prange->child_list, child_list) + if (addr >= pchild->start && addr <= pchild->last) { + pr_debug("found address 0x%lx pchild [0x%lx 0x%lx]\n", + addr, pchild->start, pchild->last); + if (parent) + *parent = prange; + return pchild; + } + + return NULL; +} + void svm_range_list_fini(struct kfd_process *p) { struct svm_range *prange; @@ -2007,11 +2146,14 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange, pr_debug("migrate from ram to vram\n"); r = svm_migrate_ram_to_vram(prange, best_loc); - - if (!r) - *migrated = true; + } else { + pr_debug("migrate from vram to ram\n"); + r = svm_migrate_vram_to_ram(prange, current->mm); } + if (!r) + *migrated = true; + return r; } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index aa6edc3efcb8..e6b737889bb3 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -144,11 +144,23 @@ void svm_range_list_fini(struct kfd_process *p); int svm_ioctl(struct kfd_process *p, enum kfd_ioctl_svm_op op, uint64_t start, uint64_t size, uint32_t nattrs, struct kfd_ioctl_svm_attribute *attrs); +struct svm_range *svm_range_from_addr(struct svm_range_list *svms, + unsigned long addr, + struct svm_range **parent); struct amdgpu_device *svm_range_get_adev_by_id(struct svm_range *prange, uint32_t id); int svm_range_vram_node_new(struct amdgpu_device *adev, struct svm_range *prange, bool clear); void svm_range_vram_node_free(struct svm_range *prange); +int svm_range_split_by_granularity(struct kfd_process *p, struct mm_struct *mm, + unsigned long addr, struct svm_range *parent, + struct svm_range *prange, + struct svm_range **pmigrate, + struct list_head *deferred_update_list); +void svm_range_add_list_work(struct svm_range_list *svms, + struct svm_range *prange, struct mm_struct *mm, + enum svm_work_list_ops op); +void schedule_deferred_list_work(struct svm_range_list *svms); void svm_range_dma_unmap(struct device *dev, dma_addr_t *dma_addr, unsigned long offset, unsigned long npages); From patchwork Mon Mar 22 10:58:36 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154517 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FBD8C433C1 for ; Mon, 22 Mar 2021 11:08:43 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id E9C7C61879 for ; Mon, 22 Mar 2021 11:08:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org E9C7C61879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DD0BB6E22B; Mon, 22 Mar 2021 11:07:56 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2067.outbound.protection.outlook.com [40.107.243.67]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7DD306E0CF; Mon, 22 Mar 2021 11:07:51 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GQNnPPwF8pLJpii3VtqhJvEIBEmzx7/kcC12deL5D2+fAuU7EvTMhguIO19UN7hmc5g2lFfy9Wrbns3N1TPhYBnSoEwdmpRKKdhPlViDEHu9S6ybTCIb0UCeKIc1xDFr8naMn4jhJEYG4kNvjCRKNdqRP9PWwvl6nUMsmJka07uMQ/m/hqsW5QMH3SOCmFo7E/L2iTyWnmV10I1C5yl3kJB5UCQdU3hHbZuJ2Cx1SsyZU9Yr8C/CYF7cklDDnkv8SziusCgqWlMeqcqNwuJnqQEi5EH4RwxakHoZ9R3DtDsoMvvgPms1qSy/w9iGtZJbnP65e042JliIdN4iwF9LXw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QkCBII+VppymZAtYCSdLPQHO6WMl8IAuIZ+94tklk2s=; b=ihmmUsliWZiVMOs8adVyQkLjcJCKRQFmxYP4ZHg4944H1uL2qX+WOBym/3rm/xmAo8Y00iSsNqQgfANiAbrsOVuTuh3Y2BDlRv3G3J1WhoYJuqEF+/ckTkt1ySXyIO+7rf8qBuPCF0HeYpKeL7F5/O9BQAn9jI8febb6Ap7bISMV+idyu4gLHDaMTIkUnRyX1Aaa/jVBlJlBcla3MeWzqTelT4Q0MkHwC+8mJI9DJ3DhNANnoBZIlbpzBJOFVL16MGP/HtW8GbOJ/Xw7EK15eTyg5Q/mtiHx/sll18czfjuMzmZ676gQFBkV02AKkSh/jT5aCySH0IByS4c5/t+Hyg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=QkCBII+VppymZAtYCSdLPQHO6WMl8IAuIZ+94tklk2s=; b=gIM821JoCot7OUG/AKI0kdsp7EXjkAxWTQxFtAkaZiGRGKP0DuIsv/aEs+xzmZzjZvuwfD7oKTppoKn03j1DeEa5XQqXDBEh2gYEwotIqVbkx9iaiNerwWzwCHQYO2ouY4c3MvdJwctWvVs9nJkRajnXQlmzx078gunjqyMTJNE= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:40 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:40 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 20/44] drm/amdkfd: invalidate tables on page retry fault Date: Mon, 22 Mar 2021 06:58:36 -0400 Message-Id: <20210322105900.14068-21-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 43144203-82d1-47a4-38db-08d8ed22b58b X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7219; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Hm9T8HJSNM36eRVeFdmWQVYVtusEIkD+x89tw1IVOmwmkDCBQgTBB+UW8lqmHVk6ffGZuUqnrcMxRmpXPZp3eRijVnC2Ww+kiFE9EzMCmWpmcjQj9V/aMDjUlGCsK7vg8gvN53QKZQmr0t4N62TTddLn482pVu6NCuUakc/kqACWzMATRS7tLRmhHJgbm3IdqjmtC09rV/QJ4FFlu81FvP5uNP8toZ0aRhFJ9QruGcBhvq/IBXJcy1B113/QRQvyDRJL7Zhxd3sHpjuV4pccOw4H79UKpcj7t+/NUyaS+FxTpZ37tP4bfAgESPiq8ncyphsHHTB2Ah1OwGq1jm+GXxl8PLFamFok2Ha01m0fYUSYKCsUGHh11qTXk5Ix6XG8GKHhCiMnsZCcSBfEfCvJkphKErwVbTQZ5vx8Q8cqBnB/DOQ+iBN4a94AqKVTYswwOxO3AXyFXyrwQ/0jphmy6/lwklMXpQMD8xVuoPSEYXYGN/eETacvkzuSa1JzYXKzQMKmqRdTIu+kRScfarS+igUMx1XteIpJjAPhE7itDMlQDWpata9KvAvN9jgQq+YbGG45WDGbGiN3rtAopAbR5jtJv4GukfFnZUBx8HOX/CW6AocQmI+xiDdKbx+ytBZ0QUD27QtKj7QKmFyor8mCcC+LbkjpFWwtLDWdKI7QVj0= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(316002)(2906002)(54906003)(7696005)(1076003)(52116002)(38100700001)(6666004); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: PDDMKzCrLvdL0iIrHeWlRmcUj0C9lOP/vqYHyEcdVXUFZsAXm00ZJpBUajv7S9kMmobJuG3LschyRByz92rvlkjXs+NO2yelk+pnepsNq/++hxC9BZExS3pT3mqtT0SGubT5o9Cxv3lGsXS4hPe8P6fGdO25+ixaxFJ0r9sN6Q7Lw3sFGoJAKvVeGlm1008pJA8DkDxP5D8jRpuBDXhT56uPEx81T8Ejh8TI2u0TXk9P2a64z1q60B+YoGH2RSnnNOeBgb7BJQcSAbIM8cYdncOapHpuCUWL9KvsEuJ3t9LyHR5VLcvOELOupU7IsmmjfNGoylyq/MNgcC+u6cu0czyAWh4uGe4Oaq6H1lDbMy4q2PjK6qG71oDdWuipwb7zP24yvAYroAljwmsVj0BbfL0zk9Kmq+yiU10cy7DLxvRsTcUWTMna2+WV/xR7kYWM7H8AsbkagI2PlWksORJIJFuQ4dfKRE8CPa0R64/a8g1mzpamV+I0IaxCDHh2sukTiXeMrtrBMk/6bD06YSy3aKcPt7XHXggPEbaZjrez6Qsge1aL+eTCkxNBPtAw81T+ENgEZsCkrHm32ynaWzEz8hcECqz5RAq3IZdMju7BV1W8/bFnNNIClKe8MUDLHu5SCe594Cdf8+BH0Y0um+aXo3Z9SG1VCcCDqM1iLvyyfl2jyA8IEO44tqLRhcHgsdbakCSM4OoCGLk5v0hklWWrOXrLxb1Rak8oCItKrLJNjokO9v0hQh+LTyUCMp0Pouc9BuT8Cip63oKeG0r4P+nEiJ8vbo1Ed0mvBs5YdDcfz/HWXB50KUENJkyaPmNMO8zzFYu6qwPD26p5No7vfLvznXFhP7zu9k8a/jZtgFbabCVOj5HvHKbh/vtyz/U31ogzTcvDVpI3rp+hbe47y+8+qPlJyCmLCTPofw6Xp7DVqiWpJghbi6OSPNgguiOfTzv+0i0wHYDjNy8nDHVXZjY+5SYo10Vm0X/JlJyX/ZA5xL4bw5E7cMwA7Xo24CzQ48/IcxPOu7js7KR5z+Hn8KG5v7JaocWeL8cOtOK/YKuTXkTVL79HED6T6q4aw+2RltW2ybAouHR9MN8pwSaJbRHarh/isy3E7p3UDM7JlDiCbTuDSV3h9zLfG5/7O585g/OB8WAO85kgsdRBzfqIE5+9IxaKJ+KHhrz35plUKHjQHugQm/It0+a6hCREUaPQhNZ0GRsbg98093nLCWiqfwMLfQxsOBbM4483Aa38RGxyEODEJgTFDwSXW4WyBrazRZqb4aCMuvIQ0jc2LlY7UD18JOEvzaSERdB1vjSRIGCsIJhYmEOdmmVQawMnLvA27T9h X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 43144203-82d1-47a4-38db-08d8ed22b58b X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:40.5994 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: uwuzD7vOaaCNxyKb1+bvheQoRQHig6EAArBFHbRpgJCuPFCMvoCL6uET4nLmng8QVimxQO9a2d5PHEyxj5lruw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" GPU page tables are invalidated by unmapping prange directly at the mmu notifier, when page fault retry is enabled through amdgpu_noretry global parameter. The restore page table is performed at the page fault handler. If xnack is on, we update GPU mappings after migration to avoid unnecessary GPUVM faults. Signed-off-by: Alex Sierra Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 6 +- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 71 +++++++++++++++++++----- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 4 +- 3 files changed, 64 insertions(+), 17 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 1243cf02f872..8ce3ff56a0ce 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -789,7 +789,11 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) list_for_each_entry_safe(prange, next, &update_list, update_list) { enum svm_work_list_ops op; - op = SVM_OP_UPDATE_RANGE_NOTIFIER; + /* xnack on, update mapping on GPUs with ACCESS_IN_PLACE */ + if (p->xnack_enabled && prange == pmigrate) + op = SVM_OP_UPDATE_RANGE_NOTIFIER_AND_MAP; + else + op = SVM_OP_UPDATE_RANGE_NOTIFIER; svm_range_add_list_work(&p->svms, prange, mm, op); list_del_init(&prange->update_list); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 69241ed4a377..fb8ca844d9bd 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1006,6 +1006,13 @@ svm_range_split_by_granularity(struct kfd_process *p, struct mm_struct *mm, *pmigrate = new; + /* xnack on, update mapping on GPUs with ACCESS_IN_PLACE */ + if (p->xnack_enabled && (*pmigrate)->work_item.op == SVM_OP_ADD_RANGE) { + (*pmigrate)->work_item.op = SVM_OP_ADD_RANGE_AND_MAP; + pr_debug("change prange 0x%p [0x%lx 0x%lx] op %d\n", + *pmigrate, (*pmigrate)->start, (*pmigrate)->last, + SVM_OP_ADD_RANGE_AND_MAP); + } return 0; } @@ -1407,25 +1414,38 @@ svm_range_evict(struct svm_range *prange, struct mm_struct *mm, unsigned long start, unsigned long last) { struct svm_range_list *svms = prange->svms; - int invalid, evicted_ranges; + struct kfd_process *p; int r = 0; - invalid = atomic_inc_return(&prange->invalid); - evicted_ranges = atomic_inc_return(&svms->evicted_ranges); - if (evicted_ranges != 1) - return r; + p = container_of(svms, struct kfd_process, svms); - pr_debug("evicting svms 0x%p range [0x%lx 0x%lx]\n", - prange->svms, prange->start, prange->last); + pr_debug("invalidate svms 0x%p prange [0x%lx 0x%lx] [0x%lx 0x%lx]\n", + svms, prange->start, prange->last, start, last); - /* First eviction, stop the queues */ - r = kgd2kfd_quiesce_mm(mm); - if (r) - pr_debug("failed to quiesce KFD\n"); + if (!p->xnack_enabled) { + int invalid, evicted_ranges; + + invalid = atomic_inc_return(&prange->invalid); + evicted_ranges = atomic_inc_return(&svms->evicted_ranges); + if (evicted_ranges != 1) + return r; - pr_debug("schedule to restore svm %p ranges\n", svms); - schedule_delayed_work(&svms->restore_work, - msecs_to_jiffies(AMDGPU_SVM_RANGE_RESTORE_DELAY_MS)); + pr_debug("evicting svms 0x%p range [0x%lx 0x%lx]\n", + prange->svms, prange->start, prange->last); + + /* First eviction, stop the queues */ + r = kgd2kfd_quiesce_mm(mm); + if (r) + pr_debug("failed to quiesce KFD\n"); + + pr_debug("schedule to restore svm %p ranges\n", svms); + schedule_delayed_work(&svms->restore_work, + msecs_to_jiffies(AMDGPU_SVM_RANGE_RESTORE_DELAY_MS)); + } else { + pr_debug("invalidate unmap svms 0x%p [0x%lx 0x%lx] from GPUs\n", + prange->svms, start, last); + svm_range_unmap_from_gpus(prange, start, last); + } return r; } @@ -1621,6 +1641,7 @@ static void svm_range_handle_list_op(struct svm_range_list *svms, struct svm_range *prange) { struct mm_struct *mm = prange->work_item.mm; + int r; switch (prange->work_item.op) { case SVM_OP_NULL: @@ -1639,12 +1660,32 @@ svm_range_handle_list_op(struct svm_range_list *svms, struct svm_range *prange) svms, prange, prange->start, prange->last); svm_range_update_notifier_and_interval_tree(mm, prange); break; + case SVM_OP_UPDATE_RANGE_NOTIFIER_AND_MAP: + pr_debug("update and map 0x%p prange 0x%p [0x%lx 0x%lx]\n", + svms, prange, prange->start, prange->last); + svm_range_update_notifier_and_interval_tree(mm, prange); + + r = svm_range_map_to_gpus(prange, true); + if (r) + pr_debug("failed %d map 0x%p [0x%lx 0x%lx]\n", + r, svms, prange->start, prange->last); + break; case SVM_OP_ADD_RANGE: pr_debug("add 0x%p prange 0x%p [0x%lx 0x%lx]\n", svms, prange, prange->start, prange->last); svm_range_add_to_svms(prange); svm_range_add_notifier_locked(mm, prange); break; + case SVM_OP_ADD_RANGE_AND_MAP: + pr_debug("add and map 0x%p prange 0x%p [0x%lx 0x%lx]\n", svms, + prange, prange->start, prange->last); + svm_range_add_to_svms(prange); + svm_range_add_notifier_locked(mm, prange); + r = svm_range_map_to_gpus(prange, true); + if (r) + pr_debug("failed %d map 0x%p [0x%lx 0x%lx]\n", + r, svms, prange->start, prange->last); + break; default: WARN_ONCE(1, "Unknown prange 0x%p work op %d\n", prange, prange->work_item.op); @@ -2235,7 +2276,7 @@ svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, if (r) goto out_unlock_range; - if (migrated) { + if (migrated && !p->xnack_enabled) { pr_debug("restore_work will update mappings of GPUs\n"); mutex_unlock(&prange->migrate_mutex); continue; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index e6b737889bb3..3f945a601546 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -43,7 +43,9 @@ enum svm_work_list_ops { SVM_OP_NULL, SVM_OP_UNMAP_RANGE, SVM_OP_UPDATE_RANGE_NOTIFIER, - SVM_OP_ADD_RANGE + SVM_OP_UPDATE_RANGE_NOTIFIER_AND_MAP, + SVM_OP_ADD_RANGE, + SVM_OP_ADD_RANGE_AND_MAP }; struct svm_work_list_item { From patchwork Mon Mar 22 10:58:37 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0B691C433C1 for ; Mon, 22 Mar 2021 11:08:25 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id CC90261879 for ; Mon, 22 Mar 2021 11:08:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org CC90261879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4686989FF7; Mon, 22 Mar 2021 11:07:54 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2067.outbound.protection.outlook.com [40.107.243.67]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0227D89F35; Mon, 22 Mar 2021 11:07:49 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=bFRFZvhTrMfFypg/KkND19823e464aBJNwSKgYSJ3EiPH29LDPhOXygCu04738rFFVBOsjuX8mdwj/9kwhA6y0ZwUMemL2c2V3Xi7ZnfqjOg6TKiNu/1XIIPwJnLK/aOC27X60qE4dLn6xnJm7DrHrnP689gfI2ls5CoZJig1ziYT06/DICdcW3g+UC7WQW2vh9wZPbSciVeMIg76+1PpX4314O9eB6tL5ULbnoBPlLZY1WLK/b3ZokBcvSRFLMAJTABSFFbzkkeubLc4w/nB36hVitHNgPcDZun0bPZel6Dk/OwbPf/T7lLo4+qtXPhMp10BMiE8aUPdLRWuRP2TA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zSOF46vcEJCtp9l6h2ymvJ/xiyqpslj9emq/Uy9HCMI=; b=oeKipmyRWLgRUYTeybjF7JM9QN0fdCV62ORgLnWdJSHlZQBQlmEoyJ06Bit73+8Fy/9mGKurhgnI+UkuNtmPeBupfAXD5a7KyrUeQ3qMF39X7tTQtpSZrABQp1KUMP3dOeofS7gbGPTE8Drv1y534ilJrf45ecA4bkg2XmMQc+K4MxoDgMoM9se/UxnDQ1v0L3NLNv+LjJk0vmSdmPetZN9FNFD3icBQkXkL7o6ztPL+pjaJmp0MSxdSpsdUN5eB1ls81z7YuFNxXms/upLw/fdG0SWVZFODF0Y9o9UP6OgRa+C70yE56e/ifZMB39oX9ZeK9HdWpH8a1RSgWMh3zg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zSOF46vcEJCtp9l6h2ymvJ/xiyqpslj9emq/Uy9HCMI=; b=cLaHC3xXYVcErrml6sndYVvU3prvvr9YBJ3GI/msnidT0NCEIEgEycuiC4cyrht+8Vnj4tY/1MrOKGsxLNKytvKKenVMJYe159lZI0I5gz+MAweYl1Gd1dCxo7dhd4XS354e3hF/ONJKtoH4e9Ugap7laRDoqIEDVY307cmezGU= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:41 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:41 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 21/44] drm/amdgpu: enable 48-bit IH timestamp counter Date: Mon, 22 Mar 2021 06:58:37 -0400 Message-Id: <20210322105900.14068-22-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:40 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 4ce6c393-f90d-4066-3cd8-08d8ed22b5f7 X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1443; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eD7YFa9SeOSCcRUq8r8kya8AsauvfNqp1kny40GXw5SV/5vFT0G0thaboo9Ph5uv58LJYnFzyL7rt/1rBZCpqMQ/NXsaSkpeu/IBa3HMNnsSU3IrIkGw/KvPtgiwkWdUOWKYtmH9HMTtVSUXAaVTp0FYsp3V5Oo5/efV7N643Rwz01IX1vqx8vHNlAIXKnRkArS+vpiBFAk4S6tEwJqFt19MnvNsiIHzb3RLRhrVeNLKMh4UWqt9hBptnnXPJn2TIlRap0xnyu1g9RvRHPSbdYikBl50awLIx7M3zPSQhaXxZAxKdtsC5IFoBp5EOdz1EpS3vwpbeog/swTHvxtG5zBL2SnuKwDEdR6eIZrqaFYnZTtEy8s/dRcIzbVotzRL9BOIIjJlmeTBsROVtY0GnAmce2qic8zSL3BANfcnPz1c6CNfSlWXc31QHWmgE2EzOaBz5gYs0YRmcexmr2jliyWGpy4GiwcdbG1FYLr9ItWorVdCCdUkUoAZOlviO29R6fSxl+Z2vs/pDX5CH+iluuBWsAsoWPH9icOgg9kYX+Py14oOo19McAfApUDM23PwuEk4pQVaoyvcOfs2B5+tdoGBZqLT4n/xpt8p6v2kr9hYVzAxcC5rMh5vSv/GOdChnwRHOJrWx5KrFUOThEY6hGnxmQYlq9Ao2Abyc1zCsZk= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(4744005)(316002)(2906002)(54906003)(7696005)(1076003)(52116002)(38100700001)(6666004); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: clSAHTVRQtauXFgtNt6UxD5Pg0v9I6mR4ocI9jSYRcxrbSJpYsaZySxMkeiTD9ZcDaJypVVGQRwXiVkvuJravN/JomPsEHGIyMNU0du8jVPbk44rwNOJrC7wF0h4nAtRCkDjszMngJ3Yn+zAu4syTjL5Vr9EiiodrJeQicmHHPLYamkxDBt4b4g92GDhVwBfyzqOTd1CNYGpTWHnsdoY3uHkRyqFLPWn2sJ1VbBzxoKVNLCVTlwz7veSqTyHhevFyS/LSNRXdq8hS5ZWV2LlV4V72EP6lns/yLFsHfYnESiPzw3lEWx8Yos315Tew+P0nLebaOZBVNac8x3Wx12jsO/ASGmlZBNjKYo+N7vPmgeGpP/bJ8JK6OTsorigcf/T9QNkUb38MwJ5mCizPYpfmNNZ61Qp6d+YPkOCMxbF2ic7D9BpMylHMsni25hIATZ1CmdqAezWZZ6fa9geYXynrfT3d/r7pl7AF9/akM+Uizrx5fyoHv1yasjbncCqzQyriXW9/dHTmJZjV2zNo0zbuAFI20dZs5XMYzt7bj2Vp9uZwl5Vp7kBtWjkPW2bktFxsui4NeTM0GYuxofyuM/CaYM/trOoV+3duCZX+r5UTr1coVp0qbvU8FO1nxooZ0GgO2dhzjYcn7V2ZiTNTR+4XxVMEgyLSVXCIMnfHkYeh5tcYvEIL1FPJR0q0xdKqVhn2PUURjw5ZJXct7A8DUZxzGCS/bmvXc4jp0hBU4NzxpRLIN4WUlOWNGPLNXAODJWYv52sIJ+x6H5Ir7YNI7I/QONRANmMYFkqPEPN9VK8+KoLpovU83iFVfH6pXQtqXOmle6qBV+TJpbrHgl6vlHuRvEFuIiIRR02+l8O4PFPUK+I+0Pq9ABITcbCFKCKzMIlshwOJGDycE48KkMYXOGEec4g2r2rSkvUfbk8fYT/4UOlKhaKejYAHaPDUr1M75DrBTCw0N66FqxePqZ4Y74pKtSPUdLcY94Ky6itWN8Vb7FOiC98g6KBYcClEoMn4U7AyGYY88c7pojd6MNg3CQbDFLf0rZPrCEaJeIfbMFgWTleqq5BfGY/AR9uyVTvainu+Wznq8K2skTBky1ahZLpQSaddmDFBuBwYVGaHA3ZAVIxNglD58i0u0LdVFhL7nqwk1QT3gJovx3jox1m6PvuVf7p/HdYq28lzCGqVMuuotigBdQgFn1HO738Y1qyB3JeDWnjjCN4oxNeJ6WiUKw4SSlRTYQUj8xJIglzcvsMn/tbccTtmCTefnOOq1yOyCtFuW1i66T8x7hwDFxZe+opjiELwJruJH9ZLkccu260bnjsYoQlq2jEPeNUKX0zFCz0 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4ce6c393-f90d-4066-3cd8-08d8ed22b5f7 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:41.2785 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: bpXhkettOrUwZ4TsMno4fMQpbkAXaZ1/RXmIoQXNCRjhZX+aYmgNGBEuAynWxfFT8Uif5/Yek+dFw7734Kpiyw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra By default this timestamp is 32 bit counter. It gets overflowed in around 10 minutes. Change-Id: I7c46604b0272dcfd1ce24351437c16fe53dca0ab Signed-off-by: Alex Sierra Signed-off-by: Philip Yang --- drivers/gpu/drm/amd/amdgpu/vega10_ih.c | 1 + 1 file changed, 1 insertion(+) diff --git a/drivers/gpu/drm/amd/amdgpu/vega10_ih.c b/drivers/gpu/drm/amd/amdgpu/vega10_ih.c index ca8efa5c6978..2f17c8a57015 100644 --- a/drivers/gpu/drm/amd/amdgpu/vega10_ih.c +++ b/drivers/gpu/drm/amd/amdgpu/vega10_ih.c @@ -104,6 +104,7 @@ static int vega10_ih_toggle_ring_interrupts(struct amdgpu_device *adev, tmp = RREG32(ih_regs->ih_rb_cntl); tmp = REG_SET_FIELD(tmp, IH_RB_CNTL, RB_ENABLE, (enable ? 1 : 0)); + tmp = REG_SET_FIELD(tmp, IH_RB_CNTL, RB_GPU_TS_ENABLE, 1); /* enable_intr field is only valid in ring0 */ if (ih == &adev->irq.ih) tmp = REG_SET_FIELD(tmp, IH_RB_CNTL, ENABLE_INTR, (enable ? 1 : 0)); From patchwork Mon Mar 22 10:58:38 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154529 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C77EEC433DB for ; Mon, 22 Mar 2021 11:08:54 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 91AB46188B for ; Mon, 22 Mar 2021 11:08:54 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 91AB46188B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9E3216E42F; Mon, 22 Mar 2021 11:07:59 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2067.outbound.protection.outlook.com [40.107.243.67]) by gabe.freedesktop.org (Postfix) with ESMTPS id AD8806E0E4; Mon, 22 Mar 2021 11:07:51 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PkCsW0AZEs36lS/7BiEGWEygPMZkxdUOcQ9xvsucB4utMxoV06pf4FenrRh1gy9H5xV61xVI5U8r4FEJNp3a0dpK8JUiRiZCwu14E7FkWwfYj9MrKtkR5UocqFkNn/bNdS/YYofSCcFG25Mk/ljC2ehh8RpqoF9gagS7P8i0qPGt+O30sppikgm9HAYopEoLUqYLnW5E1UGAraFOyDb/+yf2anDUKi0H8C8P6xc1ZMrYhLJ5LnFdyNKMStualmXZEOH+zfmk7U4pzVMJY8PmQb2Egrhym3w7kDtbuChvO+BMRPagEfLwaiIVpgeBUyqsYM0TBJcv5sh5IXU6Cqypvg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tTZX86J8KK+zx4nIO4Nya5+nWInbDDU47GWX04WnALY=; b=f9ahZC+5Msc84jzIKk3a1DAmS+xH1ZoVuOKbHvGkMYJ+mOXs4+T02HqpvvNPc9u+QiPxGpwPQ87V237cQYsLdv1crMCo80y2EAKFMjuCTSdi06/MsX9Qs24aWfWyZ7iCvgUZKw5himp8tB17AN8W98GBLvMQ0/2ERWTSHaUPKKbbdAExaLdjpioaK4T5VkNueTBuYm8HbCNwToK9xpi+bANId8sNiyi/iT1grcbtnQA8Ock4pvJf3LqifsVXn81s8xcQhu3RHtQIArNBLYQWIDD6iodqzMrWw4YckKwtx51MXzrZ3q+Jytst2VxwuPuYbKVMxutyCThNtD+ZFLv+qQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=tTZX86J8KK+zx4nIO4Nya5+nWInbDDU47GWX04WnALY=; b=j6QrQ6TXJyfg/rwUegJ0sp7W5tlupWVYQ7kunQzsvNoqijey0q/maLZF5b/3qVGyMuKuLkK0M3olmKEsxThJepHfFyGi263f9g06GKdGlE7pkZJcAUPYs4RASiXyUS1JHG4bffN4k84cMO7YksHAXdRv3FFzewkL3C7zCf86ir4= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:42 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:42 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 22/44] drm/amdkfd: page table restore through svm API Date: Mon, 22 Mar 2021 06:58:38 -0400 Message-Id: <20210322105900.14068-23-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:41 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 5d8c01f7-b266-4cd2-f22e-08d8ed22b64a X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5236; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: qoTcdT9LaDdlFX6JGLZVgeWfl7AAj0BrD/ZK0Li56glhne20L079eEiXhSCMdWmCUY6MGRRD9+NHMoUbFTJ2WHxH7aP9dkvlQ+9e7ESTVoLyy0SLGmTB/JAd5xQr5datydr5d+Dl4DuNuulhXQ9VNNxuhyRCcibdFy1mj/98ZaCrJrzDj5n6LVH6XLhaU1g5wbSajT0N5JvOHM5FAsFvb+/HIvZphXhQ1itD7/zwbaAKoh24QXg3Xs4/xpghhcAP2xPN9q2OT0d45z058mXUvyYREFmCDzXHZQm8MQxBju3C7cs6ET/LWK8/MiBOBgkcXaAMCg+pktmI25eVJJQ8eTTyeci4WeOsEq9qch6sqPaUxwPaOFKlr98awEBUOXqwi7ZhGTj+v7ipnhrOm64mk6Ay/PiWh1ixTrmBGWEHa3hQAJ1RgjFHkBsaQhuz5YGG7UtjC2Pr3BqnsuEl/Vc0UTQSEC4/MypKXqWjWmcDqrfzTYzomRQmoYmdtxnpDSivM2Tpc6pOzF48Ic12BkICQeMvBS3jCOYhfPEZ874Yrvnb3Ca7/TBzWrKPyD9WBWScK1/H+L4jnPTaV8nglKnzif/iUO0gRnlO89ZtZ2Ac4PD+3Pkzr2yvfzOuK/XXSHN6OGXtZaN3bNvEvBiP6qiPFsOnb+fA2eH4RRLuc2T724Q= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(316002)(2906002)(54906003)(7696005)(1076003)(52116002)(38100700001)(6666004); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: fClwDJLeFMK3afbtIN9+MnhQm9QBWu//osCHRYooaSlD9dUHRYT04UIS2GZAW4jgnQ3w4nq1K9UYAz7zcoOTf3B1LcxoKDFoTTok3yGXk59IyEYMpAyx5tORKKbYwJOyf80+LRA8c8L8RTwE23z5Y6KUT/O4E5sZgBEr3DOjX1SMhDpL1Uom/NSY1MppdCgwHojVN0px6/II4h3Ee5BIB4WgNihngig0a+NwobASVu1C125krraxBPLPjKL9FJl8VJx1fsnIFSHvTDLE2w49wARQTG2gkn8sHilG9Eih913L36hHNDXbelQxRoSJt3N/IuYWgcKuxPYFiAM3TIZ5oDpLYcpUC8zN5rpcTmK8QOusZvdhpg9uLtAGu/5LK20SsFNR3PIolnf+Nk5BQe9TgqapoLnegNSQU1fx36MolQdUsYJ4u1romvm47nRN7sE63y+ubcVIe/1pTkFS3S9NVl5JijNDPAU+fpmcoPEE3CuM0DhkmeL+zqTvw6LdUHN4u+Fj88GjPlv+PNI9n8PER+ZxLNp5AOF11eUBuT8p4mNRp8ROnfh6Sr6/fj356lskvZmFc1MAE2auTyjSxeqdR1sKXiFcmgLOGmZoVucWiHvj7LGiqb3TbgwxPSqKXvcFgu1V4G8pLHJ6KE/AvaTuKPUgeGmtgC9s55lxV7tNGpgNVR5v4SsRJF5aVkLGoAvM+Tse7pwfZ1unZZ2cdTkplAqQxI0weDHqZZ9Gwpr+wwbGDa/tG9pn2nWtJus0bRBhY/q6zh0hJPRsdEIAKJV7jCrQ/BCuXrpeZw0PzIy7aVE1cK1HI5HJfHPzC/uMtmjWBCsmarqD0HbDBAHJFPXHigPaIBdBu4wk1PxDXRr+4VaPnRdygMuaE/9ytisSCd2aZ+0GmDybcnjEQtcfV/f5o1WC6MseGf6Lqrcz+Ssw8p0vURGCE9ivmn8x/3COhUBZv9KrialZFJ6dh6h3EGSpGOHjHn4PYIIZpbo50PSbXW+DKsanRmiidgRboI59FcFlv8AD1hFja9uUgLgOs8gHexIpRzMnUw7DVdWdC84Q57f2gydmWEiVnDGsFn8CCzqaruUvMApwENFeMwqQVSFSYslMYY6HdXf1Ep0buuRiTc6zpxaYwz5Ic1AbdgEpeDaTIhFpZ7IFZ/zwudqT9+w33P5HMJseNmdVYY4AUf7jqN/wRPJwG3JNJbLp0hlCaiea2M/P4Udaga7kRma8v8LHc4N/Hul13l8h9WsyxPcwCA/bQ5cGWE/ZrNYlNfqMkij+6sMJJxKoguDiINz9z6yuc9oyumWnkQNw+CJIKGm8lfL7AHIW5721BlRCZ4Mn4eEq X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 5d8c01f7-b266-4cd2-f22e-08d8ed22b64a X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:41.7683 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Z6JlTyrnbNHhqDrlILN5VxOhiPUz479ovUYGn/YwX4xCTe3LPHqcFu9StbYbug2gmb4cBv4PlJKpPx6vNjjghQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Page table restore implementation in SVM API. This is called from the fault handler at amdgpu_vm. To update page tables through the page fault retry IH. Signed-off-by: Alex Sierra Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 69 ++++++++++++++++++++++++++++ drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 2 + 2 files changed, 71 insertions(+) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index fb8ca844d9bd..c791d91cb45d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1946,6 +1946,75 @@ svm_range_from_addr(struct svm_range_list *svms, unsigned long addr, return NULL; } +int +svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, + uint64_t addr) +{ + int r = 0; + struct mm_struct *mm = NULL; + struct svm_range *prange; + struct svm_range_list *svms; + struct kfd_process *p; + + p = kfd_lookup_process_by_pasid(pasid); + if (!p) { + pr_debug("kfd process not founded pasid 0x%x\n", pasid); + return -ESRCH; + } + svms = &p->svms; + + pr_debug("restoring svms 0x%p fault address 0x%llx\n", svms, addr); + + mm = get_task_mm(p->lead_thread); + if (!mm) { + pr_debug("svms 0x%p failed to get mm\n", svms); + r = -ESRCH; + goto out; + } + + svm_range_list_lock_and_flush_work(svms, mm); + mutex_lock(&svms->lock); + prange = svm_range_from_addr(svms, addr, NULL); + + mmap_write_downgrade(mm); + + if (!prange) { + pr_debug("failed to find prange svms 0x%p address [0x%llx]\n", + svms, addr); + r = -EFAULT; + goto out_unlock_svms; + } + + mutex_lock(&prange->migrate_mutex); + + r = svm_range_validate(mm, prange); + if (r) { + pr_debug("failed %d to validate svms 0x%p [0x%lx 0x%lx]\n", r, + svms, prange->start, prange->last); + + goto out_unlock_range; + } + + pr_debug("restoring svms 0x%p [0x%lx %lx] mapping\n", + svms, prange->start, prange->last); + + r = svm_range_map_to_gpus(prange, true); + if (r) + pr_debug("failed %d to map svms 0x%p [0x%lx 0x%lx] to gpu\n", r, + svms, prange->start, prange->last); + +out_unlock_range: + mutex_unlock(&prange->migrate_mutex); +out_unlock_svms: + mutex_unlock(&svms->lock); + mmap_read_unlock(mm); + mmput(mm); +out: + kfd_unref_process(p); + + return r; +} + void svm_range_list_fini(struct kfd_process *p) { struct svm_range *prange; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 3f945a601546..3aa6f6b97481 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -159,6 +159,8 @@ int svm_range_split_by_granularity(struct kfd_process *p, struct mm_struct *mm, struct svm_range *prange, struct svm_range **pmigrate, struct list_head *deferred_update_list); +int svm_range_restore_pages(struct amdgpu_device *adev, + unsigned int pasid, uint64_t addr); void svm_range_add_list_work(struct svm_range_list *svms, struct svm_range *prange, struct mm_struct *mm, enum svm_work_list_ops op); From patchwork Mon Mar 22 10:58:39 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2FC41C433C1 for ; Mon, 22 Mar 2021 11:08:22 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EE9776196F for ; Mon, 22 Mar 2021 11:08:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EE9776196F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C65AF89FCE; Mon, 22 Mar 2021 11:07:53 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2067.outbound.protection.outlook.com [40.107.243.67]) by gabe.freedesktop.org (Postfix) with ESMTPS id EC8806E084; Mon, 22 Mar 2021 11:07:50 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PVGZ5uIo2P+q/DNavGuUHMWA5lHd6vvXLnFATzC+R27JDCWM5eEHjsTJ9gzpvGPibxN3XGfzJjhwsZMxe86quK8mj8K8zzCWCkU9nJHlfSfM/A7AmVfxjh36HyS1TKZmxc/gAGlIGRDKIFjJTtFJiZPRC0IxDzo7QkHqK36oNVEchtNI5ctzfcPCV0Sjyu3yF2exsNtOMMClx4HtS5R5iOY6fHb72BWry1w4Sdv2aYJG+5pUE4qfNpNw73JH6aYk6tUbjybcnXrIxcvSO5DJnhAIZnewIhtW0HSP/5OGvLWwL4snwMGxbBQrs2hyyk5HNNeXPw2a54ZK8/RRSIj9ow== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=v0X7YfFcFXT3XV07AP0ySRE68VyaQVsgS5LrGQaQzaA=; b=mDRC8QFMB2ilkBQ0t6cax4crXBooFKS4cM2/4XDR+oCvS9Sm4EG4aQM8PFkZUX9zctgQQI7phAHe8P+rnbxsfeXyr6nXKEa9tkG8N65Fk889alXndQmllpooTrvGA2s10pSyrk0CsC3Kvh6AL49UzLqdxZGIgIIauUT72Ut+Lfgl8x+Ey996NSpAgeVvibI8tKqBkPNSHWvM+LWbU4vYryPhYCEMrrmat3yGx+zOH6tiJcn7H6XMiLg8AB1yEizS6iVbvLftMG0ppg5eWG1JzloGT+GiBu377iX+8m6l5fsotAiBaOoSdTppMjUOIgFsjLoZVeIq4gS3IelR37u/mg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=v0X7YfFcFXT3XV07AP0ySRE68VyaQVsgS5LrGQaQzaA=; b=B7JQZsq+YirdvXjDHHXc4dKZD2QgYI/am/lZautcnQyAC4VtrjCx4kX9PkZdyp77RthacnVub/PbCqqwd0eg8sAu4DlXMBwQQqEfixVOdv/YWHtS6Cg5gge/4EE202nJs1sAvr5JQCS6Bi/1f+4iGXbRAbyjKgW+0tBG9TQJdoE= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:42 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:42 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 23/44] drm/amdkfd: SVM API call to restore page tables Date: Mon, 22 Mar 2021 06:58:39 -0400 Message-Id: <20210322105900.14068-24-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:42 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 52deecea-80f1-4c41-69fd-08d8ed22b69f X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:569; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 4k6A1gTuUjEcg+taAZYLSFjrC3oxd0QxFMwT27OQMREvINxriBtg5pAVdb6kkAP4OlchmJpnlmmGnnkiisWc4fNdIcuczQMyq/lyTJi2SUa56MeNXUSljZk/1drjrKUuGC/QMptOQRDHlg4NSINhCdFzZQLupThzgi86yKd9f1GWdCTERwVt2QtMymlJLL0w1Vui3Kws5iSAcfjtDYZ+vWdpDDJ0cFiiCknfwmnD9IPW5fyItb5Dukkq+yrL+N7pAJolyRle3wp+4MtRp94KAURnR8XmJROAz2xeBLKjkflYeiybYbjgGF8sUOEjta8lzn3vkQX1Mq6/ojQE/6aLUi6ktIAbUHvKlQvCh7xU8K0LZBRNaTSNnjGV9pM3ySsauLTPvvOgNl4pKX71mUuguEtGX+6M69cCWuXNtqGsO+hxLhpvFp7bB9olDRCJZm4C81TbhPU6gGnAwpeLtuHoaZKe/vG16QaDgOixbX+bW1KP+G+U6jQlFj8D7EgmffhVvVZJCH/TLJFd2p5YzuxSQxTHQyDm2378xe0aI7ZnmX7wLPNhCaJPjua/l8P78QNs45va5/NyLvTsvVn92VmOCb9KcY+TSPxKppp4jFjsbVnFecact1XVmLLkaLVKWCC3AWIJUqO6dh+HsxR+a48ec/RzcE1U93VQ6vMJIhoH/Bs= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(316002)(2906002)(7696005)(1076003)(52116002)(38100700001)(6666004); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 09k1TyptuQatTBQE2RTJo4qr/65MckT3/MghfjimDVKilgYhbCBtZ+bxxkG4avdv1l8cY39FZ+932tqQYUDpsi7EABLRqfB88TxSldX2dv2XwzkCUL1ZJ67GevH8UQC5pMV7C0Mbo9Hu20avoPIV8D9/PrGr958W38x5UyFaqXC4ApfTYVcE2kI74E6SFIl4ixyEE7qKgeSFFYUQjFHoeXA+ARJ0TPKL4OW6Sjl7qB6aHcB7vTK6rYLBT8DAogQwwgFBmTJX5ZtFTqMKn+GaelNQxBX/T6eWJbupRXgfV8J3r+P8DgXQPr8AF5Q4FOP7YRFLN9wlEGFQ0dh80ulGbn2dxtQrePwB1PZRUtHIoaA56mADeinRj0BA8U3rO7N45OvPDlcCL4MqJzSHYnS2IsYInyegEWW2EgCvfphXHjnlSufrcuVbsa+87D7TflqneXUZwgNVPzOrcdVX4KUSshe45NZbrUrobgbkTDIJIZ4CQa7aSLx3olHq4IbSzytALCSgsJj7ItVv3CM3nR/yggWN8FXDNufuT6mEXLavHrYPUBU88J5PjNeq/9T1Q9Oluj+zvDv7mfUGlFLbD9HMz0LyJlypnmsvtp10AZRNMiaub6mYZbNjySbfwjonYMZJMdWH+IOswyhN3IMuLXMY5iAyKoC6UCC0JEJSvJ1+i83DMmRmis7Fh7R6AFGp6+Iy58eckcp/hhrC0OXKmsMiugzxsQle46gi0s0zTJoRzz++Z+arrmmphyrSbiLzI+AOoSvnt1JRczxfjQiHInba6UYGugxHgKJ2uIhqcjp8gw+SbpqXw5Ldkk0aT3vatfW68vrxrgYvwHQvdC55qP60KIlJ4OCwnnOEt6n6o+g4wZUrus9tRIvE2oz+n8YFY/J67Jn94I1F/hj2OGLhsW6KgoR0thH931oqRmzFWuhRijWcgwgSjAeoR1eui/kKyDgDss+OBGVBPzj/HPF9Rg/0Pj2yC3nG593ok1WZISHsZcG2UcWaBvEZji81EAEh/uVWlX7KqJN/XoC8DNUIcRPBzufykq4j69JXXCzYzsfZ4Pqyz2ncQFXx8C4nMPNiEgpkV2/xbwnIouXMAG0FiHMI3j98GNP9jIEFUytsB71qnbA9xNVOrvYPIDA3GLN0rBMMHz8J9RWjTFRlTl/K9IbunVME6NUlOuhTajK5tO/rwtp78DJLt9rACrQ4l8sLG2t+cEyubUoz56cJg+qvLw9GfYPK0jVD+OLTfSvBK2msK+UIUoOu6b7CnRvzHr3sx1WBbdWIKxaLVmTOFgXBWIKMqZwnRhumh2xcdIvnKba+S0XSJQlj6WwIwVNNdY1aPTNI X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 52deecea-80f1-4c41-69fd-08d8ed22b69f X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:42.3249 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: JBbtCG8cpYVvuDlFAAPdBG7Zs1EKowhOflE0SpdCw0p7/dQv/kyga29fXSKYOVIOtDE0X0FXNb8uoYiPkilo3g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra Use SVM API to restore page tables when retry fault and compute context are enabled. Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 20 +++++++++++++++----- 1 file changed, 15 insertions(+), 5 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index 0e9ae5f91c7c..a61df234f012 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -37,6 +37,7 @@ #include "amdgpu_gmc.h" #include "amdgpu_xgmi.h" #include "amdgpu_dma_buf.h" +#include "kfd_svm.h" /** * DOC: GPUVM @@ -3302,18 +3303,29 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid, uint64_t value, flags; struct amdgpu_vm *vm; long r; + bool is_compute_context = false; spin_lock(&adev->vm_manager.pasid_lock); vm = idr_find(&adev->vm_manager.pasid_idr, pasid); - if (vm) + if (vm) { root = amdgpu_bo_ref(vm->root.base.bo); - else + is_compute_context = vm->is_compute_context; + } else { root = NULL; + } spin_unlock(&adev->vm_manager.pasid_lock); if (!root) return false; + addr /= AMDGPU_GPU_PAGE_SIZE; + + if (!amdgpu_noretry && is_compute_context && + !svm_range_restore_pages(adev, pasid, addr)) { + amdgpu_bo_unref(&root); + return true; + } + r = amdgpu_bo_reserve(root, true); if (r) goto error_unref; @@ -3327,18 +3339,16 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid, if (!vm) goto error_unlock; - addr /= AMDGPU_GPU_PAGE_SIZE; flags = AMDGPU_PTE_VALID | AMDGPU_PTE_SNOOPED | AMDGPU_PTE_SYSTEM; - if (vm->is_compute_context) { + if (is_compute_context) { /* Intentionally setting invalid PTE flag * combination to force a no-retry-fault */ flags = AMDGPU_PTE_EXECUTABLE | AMDGPU_PDE_PTE | AMDGPU_PTE_TF; value = 0; - } else if (amdgpu_vm_fault_stop == AMDGPU_VM_FAULT_STOP_NEVER) { /* Redirect the access to the dummy page */ value = adev->dummy_page_addr; From patchwork Mon Mar 22 10:58:40 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154515 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 77E71C433C1 for ; Mon, 22 Mar 2021 11:08:38 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 466BE61879 for ; Mon, 22 Mar 2021 11:08:38 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 466BE61879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id BBBB16E0CF; Mon, 22 Mar 2021 11:07:55 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2067.outbound.protection.outlook.com [40.107.243.67]) by gabe.freedesktop.org (Postfix) with ESMTPS id 201FD6E0BA; Mon, 22 Mar 2021 11:07:51 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YSDZVWCX3cvuVQS/nT4zwG0wp7AIL2Yynn8YoSXQg+UoXIWTH2vmSgEfAeC3IKO1t90uNAePEaSxo6dWsBKb8VLClX5JZGWgcqfApWSqhB+2GF55frW8O2q8TLNcua3LECPoiUI1foUTZaQQjGCZdHvJ5Rrcd+VWubkAY78TwuG0zrVFM/X3ykRtJ6mfwOeWRegDAr9PiR1tA2mLUHtlDBEv+jlfxiOOQtn4tACfgQqKeVTqjCM/FHj4MLYfGJrIYOiIMyzqWCh+LVxaGqSgxAmFy2wd/aCNR1uiZ8KlK7xri1fqUB2QfnTW8dyy5KhMB+f00ZpQG4vroxttsdwYqw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yS1W1O47IQlwMsVh/EWDH+eDzzFPWhv1qN5EoiJsLDE=; b=DKwp3jUeHWm3HO680aBKs/bC6Ktsm0h/K7mWml0CGjdbbeVZhFsMHG06GU/Vb94xiXltVBleaeGoXPYV9S3vy7XXSOdlAyCUKrhkedUwFpVmiyoxQXZt+vR88rQ9LfDJlwDlh7MTEp6bviKR9z6wCm9nc5lE5hRE9gsBYQDNqpZLXz+ZTmsKX1564om6b/FzZ1nT8BhFaeys52ynEYbWbfTcC0djWWnpbSf4iMjIo/zxwe+gTuZ2TjYMwdb2WJMbcT22ro38+L4FAypdBJt7SmUQQQFTRVvInnfl8gT6unsUUjjfu0jnZuAtvr4CiIoRACtjiFVUj7HQOIGBOODR3g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yS1W1O47IQlwMsVh/EWDH+eDzzFPWhv1qN5EoiJsLDE=; b=QP8hPZUSOQcgaOr/12TNqm/fu8FN2cU7f/7AeLaeelOvUTpu+8Q4rbVp04RRB4dipTls8k1UvqeiFANgECPuwp/sdvkM6kSx2v+YcPC+j1mGF4dg6uPdFHiR0eCCtQOxQ/taoUdoXMWxwKORx6e0avul19Yw2PgS7M7CbLtXcYs= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:43 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:42 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 24/44] drm/amdkfd: add svm_bo reference for eviction fence Date: Mon, 22 Mar 2021 06:58:40 -0400 Message-Id: <20210322105900.14068-25-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:42 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 170885c4-671c-4b7a-7015-08d8ed22b6ec X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3826; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: HzWRQ5heJ1/ntiYxB/biA15sidMxHp5QiU/RPcqbvNx+jdoCWemCIpT64q6OgbpxQ1PcGuMxcHNSzTyUiRU63TX58sNPVvIk59hgOdJIG6whl5T33EGnF3ZO5V8/oIo6eqWRWsLDdsqd20km/qQVM0pEQkU9/7jDZBMOfDb12yl0aQNgP2y/VXWu6XIP5P6McL3G82+3izzBSjj4e3QZTMAFLrB8dhbqIVrBZxexyPsCLP6RDLwmydxv3ZywOJnAWXzGMPU54b/lMs6HUFIX2d3i7e3ufD5BrKBRhAsCgKlmZeiQFX/drOKcnAHJCk/3MALobWMos7/ZjzXuV0pnEwIc89KzxkaMNQcXosbJgvpZ3DzJJ6AanIwAMz3cNOeInXtPS5tjP1Z9WgBhvjmyC3VMos7pPPWU67PmuEgKaIc7wOM6e2jKcG+ZCMo3TImsCXXZOSfmmmS0h0N2EliDwjW9o60w9UA29m9EgInfH7/qQik8IE9hztiwMLJt4AxLF2Tzle0WJHNoO9haYVClJoBEfQ601rJVVmWze8f908eXiR+Paos3oyxVdNaygf0Q+ubTzQpd4+fJwgCKjJ0mLFN7D1yChaJG7qXLA299XvRYC5No3I+J99hKexc9fMgwEoj8YAKwOTXWSWy2kZP/9Ee1q+vLsLckx22cu0XLfuc= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(316002)(2906002)(7696005)(1076003)(52116002)(38100700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: x/YjuFx74Kun70zKLs/s+WxN5wYU3Ogme5aXe2itNssIYIEgmqt4Fiz64D73KtikYLBnW5iiCmGgTZSD6lOQQzTX/V4dokwbtf5NKk2Ttkl46q0zTj4V2vFEtFOnceE/uHppl8esEu/F+xTQ/p6RYewHW453IcbnVId5ezQjOVhUxNgqVznGU41426rL/hK1JpF3a2PyOSJnvAIhHt1oXx6CXlZOcoY7IDsF3tl3Gc+dJ+VVSrBTuWxlxkS5SgBcoDXCvQK8059lLxmt5sdkyzHQBaD2v4LoTElI4PXc0Rf6joh+hyQNt2EdqFazstqUnsrLedgHaQdaaO4805tB+IvdkJDt1eN5I3eCsb05eRrNA3P2Y1Pvqy9+xMgxhKsI0WFi1VWNu0gglMwcTAr6t75hkqy/hdMPQij4ZhgGU/wQey+q7iwyzdpLyuT+wmJNqtKuRe2vQ9NAcU2V/Rf/3dlpQyHcgFGlLpr3mQ4tDIjG/17atVb/9RIbKAKPppypvx4bu+gdJAu71F6jGEuaGI6ZPALW10Mn7ZYlf9t0kfVS5hx3kp6jy0WohsHDKuVPvLULFMm3iTzlNk5bDduCxqiVEKE70c4r3dnMTbj7n/VZ+hV99XhUAdzBuP2vMYRCgEGI5RLdFTIkG8NMAuRmWrqkKQLvEQ5Zozxh4Ilfw4RydLL6HZuD1MR86Ts4zMVDG8PskZj9OFbrq7gyKUj3dHy1g4C08rShqCv8h4SJgBW6cst+heTJBHzuKT3ce5R3q7DsldWNw8J2vywB0BcdjmW4RI6LponQfd1oueibeWIXQW01AB6A+iyoR3surHFFhLxbyL3T1ijRqva0ZzcwZNcTKjSzA6D5xEGqFbg6n01ayIWsA41J4CP1ss4F/EL0rbsjEtpfzBTX35UCXlc/AoFmH+mQCGIsdCmWD7CEm4WWK4hS/GCOqmuWCcDpw9TTaW/zuzAuHixlnd1A+wTkg/zum6ZzqpCR+ZqYGlRkfgp9Lp+tEJOx7pR5Dsmk4uwalTS4tRjb76JIuO/O3GuixxMDFkvcbCOFnBrfh/IWhb3N7N+7a6gREqIhjAIcbkC7aqG2nYOqBS88cEQaOc9UV36FgqRRRlWt1wUdIGrwysW2+RheZAQnGjr43Rt2NliCFAdC5C/dOGSPG+ruD8BJdsR801z/h4oYaOQ+TJfXe0IPkh0M4uBVH1Icl+Z0Pdg3hGh+qD/QQyaUcScUg6xGrSVn/Q3UjJ81mzGEeZYe+mohxtvNO9LgucPzxMzouSvDu+P2fFVV+MzXyhWKasGAKSAyuA7Ua6Lu4h6O8+Iuf0b0cAl/qccw3264a3GYpRgz X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 170885c4-671c-4b7a-7015-08d8ed22b6ec X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:42.8416 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: W6s3aMDqTKmv2HpqysgxEntyEJyLK+zBp87xPlNyiPaY+wiC98PncFrBJhd83/SdCEsFx5wGJ0HrZk45RvRG4g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra [why] As part of the SVM functionality, the eviction mechanism used for SVM_BOs is different. This mechanism uses one eviction fence per prange, instead of one fence per kfd_process. [how] A svm_bo reference to amdgpu_amdkfd_fence to allow differentiate between SVM_BO or regular BO evictions. This also include modifications to set the reference at the fence creation call. Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h | 4 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c | 5 +++-- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 6 ++++-- 3 files changed, 10 insertions(+), 5 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h index 14f68c028126..beb2ef070a0c 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h @@ -75,6 +75,7 @@ struct amdgpu_amdkfd_fence { struct mm_struct *mm; spinlock_t lock; char timeline_name[TASK_COMM_LEN]; + struct svm_range_bo *svm_bo; }; struct amdgpu_kfd_dev { @@ -148,7 +149,8 @@ int amdgpu_queue_mask_bit_to_set_resource_bit(struct amdgpu_device *adev, int queue_bit); struct amdgpu_amdkfd_fence *amdgpu_amdkfd_fence_create(u64 context, - struct mm_struct *mm); + struct mm_struct *mm, + struct svm_range_bo *svm_bo); #if IS_ENABLED(CONFIG_HSA_AMD) bool amdkfd_fence_check_mm(struct dma_fence *f, struct mm_struct *mm); struct amdgpu_amdkfd_fence *to_amdgpu_amdkfd_fence(struct dma_fence *f); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c index 5af464933976..53559643c712 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c @@ -60,7 +60,8 @@ static atomic_t fence_seq = ATOMIC_INIT(0); */ struct amdgpu_amdkfd_fence *amdgpu_amdkfd_fence_create(u64 context, - struct mm_struct *mm) + struct mm_struct *mm, + struct svm_range_bo *svm_bo) { struct amdgpu_amdkfd_fence *fence; @@ -73,7 +74,7 @@ struct amdgpu_amdkfd_fence *amdgpu_amdkfd_fence_create(u64 context, fence->mm = mm; get_task_comm(fence->timeline_name, current); spin_lock_init(&fence->lock); - + fence->svm_bo = svm_bo; dma_fence_init(&fence->base, &amdkfd_fence_ops, &fence->lock, context, atomic_inc_return(&fence_seq)); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c index e93850f2f3b1..9af644f256e9 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c @@ -970,7 +970,8 @@ static int init_kfd_vm(struct amdgpu_vm *vm, void **process_info, info->eviction_fence = amdgpu_amdkfd_fence_create(dma_fence_context_alloc(1), - current->mm); + current->mm, + NULL); if (!info->eviction_fence) { pr_err("Failed to create eviction fence\n"); ret = -ENOMEM; @@ -2188,7 +2189,8 @@ int amdgpu_amdkfd_gpuvm_restore_process_bos(void *info, struct dma_fence **ef) */ new_fence = amdgpu_amdkfd_fence_create( process_info->eviction_fence->base.context, - process_info->eviction_fence->mm); + process_info->eviction_fence->mm, + NULL); if (!new_fence) { pr_err("Failed to create eviction fence\n"); ret = -ENOMEM; From patchwork Mon Mar 22 10:58:41 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154519 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4DB21C433E3 for ; Mon, 22 Mar 2021 11:08:46 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 18FCC6188B for ; Mon, 22 Mar 2021 11:08:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 18FCC6188B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id BC8FE6E1D8; Mon, 22 Mar 2021 11:07:56 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2067.outbound.protection.outlook.com [40.107.243.67]) by gabe.freedesktop.org (Postfix) with ESMTPS id C0EBD89FF7; Mon, 22 Mar 2021 11:07:50 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=hgZEh/mIHmN+R63dcknHdWlgOkBLuSDvacOnhiRjLZjZwC65XYIl2bIcRDkV/yjcKi5TCxZ4rK1KBePfi5qxPSNOKVTv9ie8ppV1Fj5YTId6g7xR+kgGwAd/j+vCwnzMwIIfvd3uQpbhhhGpaX7Ve4YP1RY5VSSEovHIQ2kk5dJ0nhkncOsmUuoCzfuQLl6KH4oh9WgHm9uDjZDaDYPFL3vziuf0YgeDWus4hDt0pAS+558nlpBdQeln2z+xjzuVTkNCDWRanz+LNtK0azgE4A70s0YtgcDZbiO2tzg4b+EDaPLPogLk2+nstPI5Os/shjwJm+1RO47vmAHiSf3nLQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cYbK478X8wg6RjfOVrK7LwRhvQtdzhEaeqEC4YR2chQ=; b=EyzYnPhF+bS1xsOP5+wlMCeeQv+83SfHwvWzDA9Qqo43kxsL2tMwE9ZcxZT7mmpUdJvhd5CluiJHoF7bVwof7pFnXvy8V1pPZgOvB4Cs1HyXfbHhBxYhwj2F99K31QYDrqTkcusFIzi2QYdJD5e0tSPHziGT1c9hP0KZNwu7J8JNHmvtRyTT+Qgr/8iHZr42QPmmWGeR2RaL8PQBwQ3iaZdR9NQjf1KfcNQ7jwvVj0kNl0mL4NukSijZO532oVwA3QycTRqkHZ8boblzm8ka5c7k0NfWzgbOgYzfZwqj9f4J/pKqTuMgKOQJIYt5aGc8J2rmLy2EQrNWiZADjV6JkQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=cYbK478X8wg6RjfOVrK7LwRhvQtdzhEaeqEC4YR2chQ=; b=gUu1WmrjqFlRe8sMh0Gfiz2TEdSV77KZ4Jm2SkRUJxY0/U8ZqxnmBvkcdy4InerCFnf2bTpqcNyr8BhqJ3Wcf9LHBq1h4C0AC/HD8dJsxyzASm4pQp7FAbczGplwrkCcSlz1stGWZ1IXS1CP7XcYxamH4W/18lAN+MLGWXpToeE= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:43 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:43 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 25/44] drm/amdgpu: add param bit flag to create SVM BOs Date: Mon, 22 Mar 2021 06:58:41 -0400 Message-Id: <20210322105900.14068-26-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:43 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: b6decc49-d193-467b-da95-08d8ed22b738 X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5516; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LJDy+Y9QrQsRAhIdbs0L4n1w94znUzqcduArsE/QoFH4DUxwC4E+d7KdicyWC7EHcA3P2xv3yNHAr0IOigdbj3zTv8k8M9eYnJEUR/GLNoZSPez2FvhZxFFAXYniwd2TD88AsVcqL+21BRAxIXPWh2NQFx7nf6QYiBT+h1UYOg2i22cg7f7c3OqbhF7aQLrFK8USKZes75pzrRet9XewyL0Z2XSDrb4zfqd/o35BqWBCtmwyP77Dmvh7B7p109ZNyPKVSKdJclyi8L9dleG389dG9/XoFSwVBiChA5ZurAurxXKSpSCaZir9F8iLOl9toCcO3n6rL+19qwSBWor4oV3JY4DfXCISRlr+5xJQJUwme7LHKIFbZQLH3/rtBV5p73rlAX/2l7vDOT71fg0r14nQYHEPNirg1xecpARrzVfFQ2QzS+sl2EgxSCzGC88UMizSCZwj7wQT4UeY1AniUQXrnSNFdGaH3xZIePkSY/60p3yr/OoLuR6V5mw0LY9MF1PzUX3E+3+pvAnnx4PXwfyK4m6rZ8gUM8fCo+q5LLoLS4ZoQjmHxCkdZIyJAwnyv61yqOjQS97WMpqN3KZ06jiwaX3XTGtHqBmCR1iVbIFxGIkdIZIgbcGJSCRSbtXTymcSQOiU8Dzjxdary13G6dDw9qor+RvetNwhLqlio2w= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(316002)(2906002)(7696005)(1076003)(52116002)(38100700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: HZnQ1RGJvAM6dqGwGbr3UBNxVlG0+EaVj3SX6305LxUWnNJwt9Pz+kEWpFjpBaUYYlPMcq7xNDGJu/PetETdPZqq5EgYUeVtST51iS51fkQWzYbxKj96K+iw34ExauOnwro2SBsfM+EgysxMDUmFIVPxwYrV1s5b7GrLBPiWCTjkKh7383cI7S7j99R9V9myifJrgIV42c4JTg1jVSigrRLLUExuwpbSzI1Tp8Uh+3XAwXksM9c1q+LRaiMzHm8FF04gWNu7rx0jt7Wod+Inz5mxZ/nMF00kObUpLIKNOA29RI6fQ1Atl+UQlhD2H2B5aldpzbjPpJzWruHG9L+D+kea5Dj5My6Fm51cybpaGhjvwOo1xqxxmQ3ba3kMXFubKEROB9jxU7BcFNhzXTFClgahNWAGExkyqxhOBCarKBZ8ulgUKmNO8ERXpJFPOKBSQeXbt0YoRSjdB3VJAYRnwCggFCgZcXjj07DEpuFgTZiLd/m21r6JOl6K73nPAzqF7d4++TFp/9TQ5LKjQqq7ZI4d6PeF6ZC+P/au1qT8CUJ613sCM5/esPNb3nQ8EKIcLm+SYr9qDcbUNoTqrEqpYiWXg/C+GH3NREedmD6VIpdnfYJfLYTO+KR/8ZFbHOl3YLhl9yW6xPyGwoPmozWsRQtkAEhhRVd1dfIf6jqGAIT+SNdW9JHhRB/0JhOG78Dv/pjC3Jtpqvkd8a3dPeHCqeMBfvyVJccFRdrfjt3kA3E6XcSF8y1Hqdv2oURSxx5pY8zDeGIOm4APcYVceTGH1FjZuALegsFXlSzNEtlwcW2HietxN3JdBCf7Jy5qlzHeOHdb/qzptB8/qLkEoZBaBwiiGvdgOFephMYGFQAa+YaCi9uwkc1Zrj7OmnfCDpul1DT3rS/625ePMeLeuNt9jbJlXC7MkppP9KLTBi5mU+gPP1V3kXLcjs3yM7fyx1C0G0j2+R4fzRG3wMbHTbhlEw1mycuDIwfl9fQq72shJAmjSsuEi39a+OlrrGYtObrUj0hYAY+X7w/MftdzyUyxbtMkuadBU6O+sKl3oGp12UDODRWo4qccrNyU6GhhZOcdQWDYH21MPTeFg9iePbC6OTCyxioUFrku2UEUvkb4aT5wqzSFXIrF7b7wrpa3o4fh00LIG4rLjC9dkxxcl0ZEKSOzqu5YFFkoWP7U4lAHZk92ZhoXq0iIOs+YC+q0C488zStg6lqJ2z/xvkpOXgzIlLl0uqkJ5lFlKoD1s86sBfPKVKDrNDWQfBkZZpg9tyLwoK9P6DZrEZkwhPdLAc5Zzqu0FJ+1N28Bd6KCyiHnui4hEWQxXGex+NoX58QO91ln X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: b6decc49-d193-467b-da95-08d8ed22b738 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:43.3494 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: je0Rtp65j+69VJEFNfaz4XBZM/bSw3x7Of7cETmtsupJpzrBNHXWEesB78V2cIjuHqrqxsCuKa/6mSED2PIrmw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra Add CREATE_SVM_BO define bit for SVM BOs. Another define flag was moved to concentrate these KFD type flags in one include file. Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c | 7 ++----- drivers/gpu/drm/amd/amdgpu/amdgpu_object.h | 4 ++++ 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c index 9af644f256e9..bc38de8c5c38 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c @@ -33,9 +33,6 @@ #include #include "amdgpu_xgmi.h" -/* BO flag to indicate a KFD userptr BO */ -#define AMDGPU_AMDKFD_USERPTR_BO (1ULL << 63) - /* Userptr restore delay, just long enough to allow consecutive VM * changes to accumulate */ @@ -217,7 +214,7 @@ void amdgpu_amdkfd_unreserve_memory_limit(struct amdgpu_bo *bo) u32 domain = bo->preferred_domains; bool sg = (bo->preferred_domains == AMDGPU_GEM_DOMAIN_CPU); - if (bo->flags & AMDGPU_AMDKFD_USERPTR_BO) { + if (bo->flags & AMDGPU_AMDKFD_CREATE_USERPTR_BO) { domain = AMDGPU_GEM_DOMAIN_CPU; sg = false; } @@ -1278,7 +1275,7 @@ int amdgpu_amdkfd_gpuvm_alloc_memory_of_gpu( bo->kfd_bo = *mem; (*mem)->bo = bo; if (user_addr) - bo->flags |= AMDGPU_AMDKFD_USERPTR_BO; + bo->flags |= AMDGPU_AMDKFD_CREATE_USERPTR_BO; (*mem)->va = va; (*mem)->domain = domain; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h index 25411b2c4dd9..b07903d317e1 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.h @@ -37,6 +37,10 @@ #define AMDGPU_BO_INVALID_OFFSET LONG_MAX #define AMDGPU_BO_MAX_PLACEMENTS 3 +/* BO flag to indicate a KFD userptr BO */ +#define AMDGPU_AMDKFD_CREATE_USERPTR_BO (1ULL << 63) +#define AMDGPU_AMDKFD_CREATE_SVM_BO (1ULL << 62) + #define to_amdgpu_bo_user(abo) container_of((abo), struct amdgpu_bo_user, bo) struct amdgpu_bo_param { From patchwork Mon Mar 22 10:58:42 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154511 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EDC34C433E1 for ; Mon, 22 Mar 2021 11:08:28 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id BA6A561931 for ; Mon, 22 Mar 2021 11:08:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BA6A561931 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D93D56E3FE; Mon, 22 Mar 2021 11:07:57 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2067.outbound.protection.outlook.com [40.107.243.67]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4A07C6E0C5; Mon, 22 Mar 2021 11:07:51 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=BDwcBNgiIeyPjyLfwMdMbHKs2l9ZVRX/HTkUEmdNrQlOcN2aVWSVeipLt4SpBuKlRzLZOWQx0kD+xuO47JSKQQ3bKqmoMAekNDIhMiIWOa43WOUqA3tFxRtte/uXyUS9ePIudLMeXqg89GdVMuEeiS9cezbZBGcvHn38gAiDyUOKT6mExblqkCcaM5lCy/EBtOQk00rSE04e03DD/tkuQsmxTt3ryZ99W9E02skqzU+95v9y85PTnUpzcsPsG+xViPDfDGb8dNph2OgcMCBF+BhMhQqguPX7iS19W7ofQLzp9d9DO1nQdjqFfGLGoVjdvedR0WYIGlv/LQo19Y/XHw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=K+E21NBSUnbwO80lM0o7jlirRLIYUu0jUt6pQ4AuFuE=; b=ROVbTiRK/atiojmLKDY4gR0f3TTLsswr5pLCVCK8bE2V4SloW3+BUsUBc/Ctu0PIz0N9iFeq+9rFZ7EEadshR8UndLFpqsFemBSsKIYTad4SQNmlwPsRXbTgbwP48wiSSKY2bX9DFpEV0JWtPWqdNoEahkzFhLO30kWjZDjJYF1w6Rkallw0TQM5k4xk2/CLUp7bh2JSz92qyDJhNiIP7N+Z1JAuk/8wKteWlZRsnWHUFqRkjGN4NE2sWAjPj3THwMu5646CM1F2l0kaG+KYoOSBMONTMGmi/FTk99gOOLvU5dR30gZdyixuQayTU3dVHEIG2w+PStfggCnQqDtMHw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=K+E21NBSUnbwO80lM0o7jlirRLIYUu0jUt6pQ4AuFuE=; b=xpJjGDw2FGOaVGKJH7pccz1PAOR0wpqcmPobSHFd/wyk4R4v9OX3MAGQ2sNS81BRd0Q+7X8CyvF+FlLNe/Y77hhox1Mv5nTUvPqRmRPig558w6j+veEGRg6CI7kcA6SDcGDRf9VNVtk2qaXekOqZF6K2eq+xY2YN9rnKExW+Shc= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:44 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:44 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 26/44] drm/amdkfd: add svm_bo eviction mechanism support Date: Mon, 22 Mar 2021 06:58:42 -0400 Message-Id: <20210322105900.14068-27-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:43 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: b5040f11-2aad-45da-58d0-08d8ed22b78b X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: oe6VWMTVlguBtJfR9PXv2xqXDkMaPcMMNm/RvCImknYUfVfuqKLfos+daQ+05eJaIXbNgPZMxkfZDwfVEq1bgJEb2+FWj3xG+5tfzIFSGmF3gJepAYp8szutKM4C6rCJPdzvPngSRuLj2iBo4Qo53Wo5DmvHqaikNn13f8MAYETyJWra4DSocgsB3tUckk8Dwbf/bzAIdo25NabRh930VIv4q860sMZbxcEyKsUvwgKJAxwsGWYFmG6kQ3J2LFxAH5e32i+bYwGLzW9w4L/x5Ao9M0B+HxxbLWRBaA7bFgdenCcmIsorgbTPG+qbIC/yyQpeLYeYjPGvf4y/Sdb64ceIUnJt9vgzpnrbnLg8JO4OCa831zM3LCGTt3ZgMMsAJEmky67pqsFh/Swk4T7EwUh/cBS5quKH61lFuEPV9CwELw+w1i7GMlnFa2sIzr+BrqmpSLlUOdVmpAyuqkh26mEe00y5+MljF1Jud2jcp2yKlwve0qVcRCAT9UsVZoJNdYauYSDayWEc0sYjV5e61otP05HaNaiOXZ+s569suWe5MRPZLvS4DTUf0xcaEhakLxlXGJcjeC10WhCLWTfSyamIucFjq0frq9uKZuWYbIbJqAP3VzS7e5TTVRaT0hp8GJ6ay2wdAlaBdwBPqaRLSQ== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(316002)(2906002)(54906003)(7696005)(1076003)(52116002)(38100700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: QA0mKrBilHVg/wLfnssu8IqC1O6vloPd2hNmvUVWl/FqZNWlpllwmClmBZAkZxhYMNbcnq5Xm8JpQ68kJnkLGE0BGHgNQCq96GvxyT0L2grqKy4BGp1ci5H2L8ykP/aCFq4pLxcVgArvu7stkERptfTROSHL+XKayoQO732gnTHXwbPclMF+A8T613HOnxLefFXbWxRVNc2+OE8gH3cxchbM+NoV+HoVsI/cM+BbV0evCQ9lHKEee6U+mCwRc26GEyi5CW9eHGKenKULn8BIbyauote9Bue0bP5AXYI84HifW/1ZibpA9ghzWy3UGqV76JU5y+gztDkN684pD781QqxxsPJyMBJTMF8OmI5KdnwWRccwzuQW7Rp8iRfi3lGFJIqivxdZ0iji99KWU2xmASJJ3/fTWGKxaVNyPB2dhXgAuAB/HTC0KUf+gU42x0PDTDf+VSV2L1sO2feAOKi0Z0wFHiwcoaqs/jzfjLSXG0ek7v8p5GEwFRu+MpDTgYe6Vtm3ABxQg3Jw7KK11y0wp8kWKjlZdk7yPjiwcYAXWDrj2orQINCLRYk67OJaKKHZ0djbMYcuvXiE8uMCLMgUkNg914VZLWc8ltc55rBwJruOa+hdn52ddr/jVGSLrNryMw+fdiZRZeRNE9sNbyIf2imb0VttB1nQmiwIxdR1A4zeMqqVlRZQ0HqsSkz5kmd6XAOaji4FdU6kSpylX071j8dITDL833uklI9N0Nwsnri3D0CVX8t7rB9aNLVK07/T5vlL4UccTnD2LKxzPRsACrEyHKdo7qcarfUzwQw0t8LTR1xx6Jly62HxUMqVkd9UXmldFDmgHzu+BlXPirVz34uAtWblOXhlL0OFLj17MZ3+evx9LUyfbE+ZypOnw9phr7H9HdUZQiGrQTvkNOB/loR2+TdkADM6cSHceME3EVwiF7dtWmswb19lUPIRNzayaid/aIg4Bcbvo9Gu1aZtWmhfE6fMyVqR8+1VHFGxkR6eaOZKic4b1Z4DnCZu7q+5yvIDsF/ovRTLAMtNLlQYwTHIp3N2vf+tjo5/Sqs4+y1PZq+PDhfv35akMfEz2qD6n/bzU098G5GakPHx+zp1B1m7O70D6gRLlQN6e1IcrX9l0Yje3aqBls9TPHf/Gwo01+d5Nen/v+aWp4VcX9udwgl4/T00xsm8ytosi2FbRWRk+Ur+7cVs2q2gARHIkLY2b6T7TdFtyWux15ZI0FrSdQn0BWUIdoLc7oAHyeecM6J1616jZODVM6VsEpbWa3u3ekYun6Lvi0Gbu/z4vc2JL6lu+qh4P7TabN9tdFGrud/aaXvKvVjAAuxVcKkY0XG4 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: b5040f11-2aad-45da-58d0-08d8ed22b78b X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:43.8830 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: naSJKBQ+4KIuDGTwMIIJQaGc4pUh0fnnhJctifnxdlEvFaDEJ3qi8ZFH/HbyRvKBMJ/xE7c8suFnp2qRuCXXFA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" svm_bo eviction mechanism is different from regular BOs. Every SVM_BO created contains one eviction fence and one worker item for eviction process. SVM_BOs can be attached to one or more pranges. For SVM_BO eviction mechanism, TTM will start to call enable_signal callback for every SVM_BO until VRAM space is available. Here, all the ttm_evict calls are synchronous, this guarantees that each eviction has completed and the fence has signaled before it returns. Signed-off-by: Alex Sierra Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 186 +++++++++++++++++++++------ drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 13 +- 2 files changed, 153 insertions(+), 46 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index c791d91cb45d..3a7b842b362c 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -34,6 +34,7 @@ #define AMDGPU_SVM_RANGE_RESTORE_DELAY_MS 1 +static void svm_range_evict_svm_bo_worker(struct work_struct *work); static bool svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, const struct mmu_notifier_range *range, @@ -332,7 +333,15 @@ static void svm_range_bo_release(struct kref *kref) spin_lock(&svm_bo->list_lock); } spin_unlock(&svm_bo->list_lock); - + if (!dma_fence_is_signaled(&svm_bo->eviction_fence->base)) { + /* We're not in the eviction worker. + * Signal the fence and synchronize with any + * pending eviction work. + */ + dma_fence_signal(&svm_bo->eviction_fence->base); + cancel_work_sync(&svm_bo->eviction_work); + } + dma_fence_put(&svm_bo->eviction_fence->base); amdgpu_bo_unref(&svm_bo->bo); kfree(svm_bo); } @@ -345,6 +354,61 @@ static void svm_range_bo_unref(struct svm_range_bo *svm_bo) kref_put(&svm_bo->kref, svm_range_bo_release); } +static bool svm_range_validate_svm_bo(struct svm_range *prange) +{ + mutex_lock(&prange->lock); + if (!prange->svm_bo) { + mutex_unlock(&prange->lock); + return false; + } + if (prange->ttm_res) { + /* We still have a reference, all is well */ + mutex_unlock(&prange->lock); + return true; + } + if (svm_bo_ref_unless_zero(prange->svm_bo)) { + if (READ_ONCE(prange->svm_bo->evicting)) { + struct dma_fence *f; + struct svm_range_bo *svm_bo; + /* The BO is getting evicted, + * we need to get a new one + */ + mutex_unlock(&prange->lock); + svm_bo = prange->svm_bo; + f = dma_fence_get(&svm_bo->eviction_fence->base); + svm_range_bo_unref(prange->svm_bo); + /* wait for the fence to avoid long spin-loop + * at list_empty_careful + */ + dma_fence_wait(f, false); + dma_fence_put(f); + } else { + /* The BO was still around and we got + * a new reference to it + */ + mutex_unlock(&prange->lock); + pr_debug("reuse old bo svms 0x%p [0x%lx 0x%lx]\n", + prange->svms, prange->start, prange->last); + + prange->ttm_res = &prange->svm_bo->bo->tbo.mem; + return true; + } + + } else { + mutex_unlock(&prange->lock); + } + + /* We need a new svm_bo. Spin-loop to wait for concurrent + * svm_range_bo_release to finish removing this range from + * its range list. After this, it is safe to reuse the + * svm_bo pointer and svm_bo_list head. + */ + while (!list_empty_careful(&prange->svm_bo_list)) + ; + + return false; +} + static struct svm_range_bo *svm_range_bo_new(void) { struct svm_range_bo *svm_bo; @@ -364,72 +428,56 @@ int svm_range_vram_node_new(struct amdgpu_device *adev, struct svm_range *prange, bool clear) { - struct amdkfd_process_info *process_info; struct amdgpu_bo_param bp; struct svm_range_bo *svm_bo; struct amdgpu_bo_user *ubo; struct amdgpu_bo *bo; struct kfd_process *p; + struct mm_struct *mm; int r; - pr_debug("[0x%lx 0x%lx]\n", prange->start, prange->last); - mutex_lock(&prange->lock); - if (prange->svm_bo) { - if (prange->ttm_res) { - /* We still have a reference, all is well */ - mutex_unlock(&prange->lock); - return 0; - } - if (svm_bo_ref_unless_zero(prange->svm_bo)) { - /* The BO was still around and we got - * a new reference to it - */ - mutex_unlock(&prange->lock); - pr_debug("reuse old bo [0x%lx 0x%lx]\n", - prange->start, prange->last); - - prange->ttm_res = &prange->svm_bo->bo->tbo.mem; - return 0; - } - - mutex_unlock(&prange->lock); - - /* We need a new svm_bo. Spin-loop to wait for concurrent - * svm_range_bo_release to finish removing this range from - * its range list. After this, it is safe to reuse the - * svm_bo pointer and svm_bo_list head. - */ - while (!list_empty_careful(&prange->svm_bo_list)) - ; + p = container_of(prange->svms, struct kfd_process, svms); + pr_debug("pasid: %x svms 0x%p [0x%lx 0x%lx]\n", p->pasid, prange->svms, + prange->start, prange->last); - } else { - mutex_unlock(&prange->lock); - } + if (svm_range_validate_svm_bo(prange)) + return 0; svm_bo = svm_range_bo_new(); if (!svm_bo) { pr_debug("failed to alloc svm bo\n"); return -ENOMEM; } - + mm = get_task_mm(p->lead_thread); + if (!mm) { + pr_debug("failed to get mm\n"); + kfree(svm_bo); + return -ESRCH; + } + svm_bo->svms = prange->svms; + svm_bo->eviction_fence = + amdgpu_amdkfd_fence_create(dma_fence_context_alloc(1), + mm, + svm_bo); + mmput(mm); + INIT_WORK(&svm_bo->eviction_work, svm_range_evict_svm_bo_worker); + svm_bo->evicting = 0; memset(&bp, 0, sizeof(bp)); bp.size = prange->npages * PAGE_SIZE; bp.byte_align = PAGE_SIZE; bp.domain = AMDGPU_GEM_DOMAIN_VRAM; bp.flags = AMDGPU_GEM_CREATE_NO_CPU_ACCESS; bp.flags |= clear ? AMDGPU_GEM_CREATE_VRAM_CLEARED : 0; + bp.flags |= AMDGPU_AMDKFD_CREATE_SVM_BO; bp.type = ttm_bo_type_device; bp.resv = NULL; r = amdgpu_bo_create_user(adev, &bp, &ubo); if (r) { pr_debug("failed %d to create bo\n", r); - kfree(svm_bo); - return r; + goto create_bo_failed; } bo = &ubo->bo; - - p = container_of(prange->svms, struct kfd_process, svms); r = amdgpu_bo_reserve(bo, true); if (r) { pr_debug("failed %d to reserve bo\n", r); @@ -442,8 +490,7 @@ svm_range_vram_node_new(struct amdgpu_device *adev, struct svm_range *prange, amdgpu_bo_unreserve(bo); goto reserve_bo_failed; } - process_info = p->kgd_process_info; - amdgpu_bo_fence(bo, &process_info->eviction_fence->base, true); + amdgpu_bo_fence(bo, &svm_bo->eviction_fence->base, true); amdgpu_bo_unreserve(bo); @@ -459,8 +506,10 @@ svm_range_vram_node_new(struct amdgpu_device *adev, struct svm_range *prange, return 0; reserve_bo_failed: - kfree(svm_bo); amdgpu_bo_unref(&bo); +create_bo_failed: + dma_fence_put(&svm_bo->eviction_fence->base); + kfree(svm_bo); prange->ttm_res = NULL; return r; @@ -2267,6 +2316,59 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange, return r; } +int svm_range_schedule_evict_svm_bo(struct amdgpu_amdkfd_fence *fence) +{ + if (!fence) + return -EINVAL; + + if (dma_fence_is_signaled(&fence->base)) + return 0; + + if (fence->svm_bo) { + WRITE_ONCE(fence->svm_bo->evicting, 1); + schedule_work(&fence->svm_bo->eviction_work); + } + + return 0; +} + +static void svm_range_evict_svm_bo_worker(struct work_struct *work) +{ + struct svm_range_bo *svm_bo; + struct svm_range *prange; + struct kfd_process *p; + struct mm_struct *mm; + + svm_bo = container_of(work, struct svm_range_bo, eviction_work); + if (!svm_bo_ref_unless_zero(svm_bo)) + return; /* svm_bo was freed while eviction was pending */ + + /* svm_range_bo_release destroys this worker thread. So during + * the lifetime of this thread, kfd_process and mm will be valid. + */ + p = container_of(svm_bo->svms, struct kfd_process, svms); + mm = p->mm; + if (!mm) + return; + + mmap_read_lock(mm); + list_for_each_entry(prange, &svm_bo->range_list, svm_bo_list) { + pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, + prange->start, prange->last); + mutex_lock(&prange->migrate_mutex); + svm_migrate_vram_to_ram(prange, svm_bo->eviction_fence->mm); + mutex_unlock(&prange->migrate_mutex); + } + mmap_read_unlock(mm); + + dma_fence_signal(&svm_bo->eviction_fence->base); + /* This is the last reference to svm_bo, after svm_range_vram_node_free + * has been called in svm_migrate_vram_to_ram + */ + WARN_ONCE(kref_read(&svm_bo->kref) != 1, "This was not the last reference\n"); + svm_range_bo_unref(svm_bo); +} + static int svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, uint32_t nattr, struct kfd_ioctl_svm_attribute *attrs) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 3aa6f6b97481..f8e282ec9d8a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -33,10 +33,14 @@ #include "kfd_priv.h" struct svm_range_bo { - struct amdgpu_bo *bo; - struct kref kref; - struct list_head range_list; /* all svm ranges shared this bo */ - spinlock_t list_lock; + struct amdgpu_bo *bo; + struct kref kref; + struct list_head range_list; /* all svm ranges shared this bo */ + spinlock_t list_lock; + struct amdgpu_amdkfd_fence *eviction_fence; + struct work_struct eviction_work; + struct svm_range_list *svms; + uint32_t evicting; }; enum svm_work_list_ops { @@ -161,6 +165,7 @@ int svm_range_split_by_granularity(struct kfd_process *p, struct mm_struct *mm, struct list_head *deferred_update_list); int svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, uint64_t addr); +int svm_range_schedule_evict_svm_bo(struct amdgpu_amdkfd_fence *fence); void svm_range_add_list_work(struct svm_range_list *svms, struct svm_range *prange, struct mm_struct *mm, enum svm_work_list_ops op); From patchwork Mon Mar 22 10:58:43 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154513 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 36DD0C433E3 for ; Mon, 22 Mar 2021 11:08:32 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id EBA0B61879 for ; Mon, 22 Mar 2021 11:08:31 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EBA0B61879 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 985D089B33; Mon, 22 Mar 2021 11:07:53 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2067.outbound.protection.outlook.com [40.107.243.67]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9A06989F49; Mon, 22 Mar 2021 11:07:50 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Vge3C1g22Z98xhoUwq0BZ6hpFQsVbfZJtHr8PiFIW64mrcN97ag8x0t4eJ7sepQJwjTCf/eXCaxzGRqhqRvORlZY2S44iUhqur0ZwKy9A6FSg8mJJzEKKKUJOyJsc8SGD6b11S8qV0PIf49NuxccO4az4ZSVrfxrHnncMwAWkn/C+O4b1qcnr3ryXb99Vo3Y3fKErdAiXmfYO2jU2s1ocmI13694T4UTflDQ0NXFxvT3oPkmx8Liv/PwtbgPe6OxUfZsb10YepUTNFam9UnW7iw3kZlWet3OoUwsXoRyimmfQPJiWpNQf7NK3xUppuIUw/Xtl0TY8JlcZhpa5vjgiQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qgNyPmXptTB7W68tG8a63FoS6cZBhpgPCx0Z4L92z+c=; b=l0qVqvnxaX5ujDrW5DGEffo6EALVgiLIbGXbAKlNcQ4N5YeRVNeuUp6eynLDl4CU/4NHC4M+wN8OVrFDFWQake0IpISqMfyKdK3EHjwwEF8JW497yf7VHKTSPKMIWYowBIzm8biGCaOxt31Hz/t4b6TXnLiSB7qcxN8oXLXtmyE9+md0zyySrOPMMpH6sjapQev+jgVuqLlz8z/2x3Hu40DlzFt00lfcpdcvnsHSb6zCllEvtNJF6wDhZdJXU/Ym+uNTFfACu7OBz4VLK/z+kWRGVhcQtG1zuAcYMYy/REGnlXMQkZ31Hf7/FiKRX0Tx1mr/cpE7UrAPda+dApSX3A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=qgNyPmXptTB7W68tG8a63FoS6cZBhpgPCx0Z4L92z+c=; b=245xWdtf1cfSuBMNDKVfVochByorBZMKKSG/hUiTaafXvr2eDNCaaW9mB5CZZZss64KWQ4WxbqaDQLW/oeYqgwWGfK22mlPxtiuP8E0pFflhf0CWeLSqWuvml6W+sioTXbs7ZcAK25I0N5FEc82WvLRTidTDfKePmarjRjO9Ors= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:44 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:44 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 27/44] drm/amdgpu: svm bo enable_signal call condition Date: Mon, 22 Mar 2021 06:58:43 -0400 Message-Id: <20210322105900.14068-28-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:44 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: bf9d77c4-c61f-4cac-1af6-08d8ed22b7db X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:8273; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: FUIojhwpDFO5Of5Yd+6ndV+W+ZnUJOIrTqgTGZfzU6QW+fqKxlhqT34oarus8SCXicTZFKfmFQU/CbI9fIT4uueyinoxA1ukPvtvuJ1PM2rF4X5q0Q9SvYg+go+xlxShIeZAp16IwZEKScD737VgKHNlDNzNpYHLHz8eYT4+bm3AI8LSB324bpWxbLec33etvLDFfw89nU11vB9zkl1MFlM65C31Y6zg16KiHJgIZ88XpivB8uGYHjDVNxFY2TwfVOjlRFPdG+29S49hmei4wgLT/I9GJB49BXz44VsTbFIw/bYezGXvznok132B5rdmq6q6KvKr/hIu2+V90Whwv7Y50arpHETawJ1IinO6yTfGpEqbiiKBA/Y+CAprrMR8Kvdv/6tr4LnE/SFycHMCK4z//PEI1WOASc2QVFXRD8DHPu1EgX5QGCOnsw2V6rcD17BQHxBv3kFYV+EfqtsmepthgVeE/DcIyaKQUSGZ2cObqOa4z5JXdKKEEROYRQBD5Q7dv9OBqWYna5zNhrItmow8SOP5TB3jlpnafBHbkA3f/3eZikn43EiFANMmjxyg6aPrlK6nTNMvBIyw5ymcy72tdH5X2X9D1x5ym5vVFhDOV4Boh/deHYm0RwAUCkFWDkiKL/8oBE3WtZnDm5O0P4sRsYqizQ5+6Lwdv63lwNI= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(316002)(2906002)(7696005)(1076003)(52116002)(38100700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: TnKYtHHCg06VqzGrBVPtcCGGH1R/bnEYdhhUzOzMk8jU+OqFUdoRxda262dgmeVFmrauCYTL+TUz6b8QagVM083z9WjfDujIincOSIkNv7IF/9I726F5jUy/Cc+nru3T/iStqX9s4bZaMJEr36ZNMKVFhCkOkkycINksmuXJuEoyVElSXtsvZG9RFJWDxoUUYEhXmnegjldI+T4EGK7HI2B4tqrGNKRjInFTk8fACMQRFblfETiVCTI5KU/v4FM0f0JPahMKJ4fEoPCft6mXJ1iK6RCgV9yr+fEv1BpEVXp3bhPzagnl5EedJr1dgetXKC4+XbV0L1GO3XpDRpo60DOGEVZpV7YC4rxIQ2ZDkh3kVkUDTtB9Isd5wQu0EXu2AbPj+lBBQ2T0+JTLiJ4Z4J5f9LeOlJKgf3iEbiWR5VNdMjeKiKYqQxUGbqQvRdpYRlznrhWswsjQZk6Yxvrxbthaego06bBXGBOFB+F5/QzmUelOW+rU0j604pBlz0gUa50undUBbXmYQPzXYqNn8/69kbhwhGTJfu9S7KIotBZrUXOyJgsVBzoF/XkgPsxnGkFXX/JbE3v+IiNdTYtN0UmkAYw66WA6/W4RN27D3YHiREYrgVG2Fj4jdvQM0o8nHAfbP2jN5E8bqaL4Ubj+RJjOwdVvyGkyvdfjAO4CAHAk2t+cYCeGTwsouRIP7t2pdwOEjW4Q0vtG2TvIvFx7Y68ksJUZ1Tcx6Jqedo1IE/TTE+KXgxl5nm3f1FNuPT+RU5TyIUxZiIaOLEa6JlHEN90xppuZBX30RWfREcnXnEnQA/IIb4fqMRuyjcuS8H7FkU1QSdJGnt7YcZiqa2uFv5HB5HvDpMz0y9GnwRZp+fLpuBXZ9pYvH/wdAbu/lX7JQygp26cR0VeJ1B9DdPY+5TCBqMYGNrGlTHcrJt4zEbdPe4nRKB9XgWCWumhgC7HyWWikmvpMmL2QGf97TrafyBQokIvEOcsUsEvvokxbwEDWLDhdpLiNTW/4E6HZAR8bRgfnt5Bax2IbnKme9828Lpd2STWE24qaVGFmsc+ajwrkwmXI211m37XHk07oBuvECjoYKt+X/8nuTRLc2XjOvRPbBECzrVp9PxO8nCRogSl3pM8XWIAZvF92mxilY0JYNCrS5hOzUENyaBx8PqqyRmzRatWFo6OiDnMDiKOy7IS/FZDEc5xbn1ub1uHP/6oBB7XDPZuSaUb23UHyD2poEdl92UmkFkUa7rojfvCDZd1V7M7LqPBOofbQHpARskqdZ/7fOY7FuRFbOuGvIGy1OUvBsv+clkYxmsVgCdYmlSSoIoT2W1Ov0/gwg/qeaE8R X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: bf9d77c4-c61f-4cac-1af6-08d8ed22b7db X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:44.3699 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 1n7+w2chOhDQdEQVwEJMdEBX5yGkugF+tXlRra8+BpToAyIxvJqeFu+c1R11+KkW05TvvpPbnIf9DITCJL0FoA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra [why] To support svm bo eviction mechanism. [how] If the BO crated has AMDGPU_AMDKFD_CREATE_SVM_BO flag set, enable_signal callback will be called inside amdgpu_evict_flags. This also causes gutting of the BO by removing all placements, so that TTM won't actually do an eviction. Instead it will discard the memory held by the BO. This is needed for HMM migration to user mode system memory pages. Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c index aca5a29f6d2a..2d80eb3fa571 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ttm.c @@ -111,6 +111,20 @@ static void amdgpu_evict_flags(struct ttm_buffer_object *bo, } abo = ttm_to_amdgpu_bo(bo); + if (abo->flags & AMDGPU_AMDKFD_CREATE_SVM_BO) { + struct dma_fence *fence; + struct dma_resv *resv = &bo->base._resv; + + rcu_read_lock(); + fence = rcu_dereference(resv->fence_excl); + if (fence && !fence->ops->signaled) + dma_fence_enable_sw_signaling(fence); + + placement->num_placement = 0; + placement->num_busy_placement = 0; + rcu_read_unlock(); + return; + } switch (bo->mem.mem_type) { case AMDGPU_PL_GDS: case AMDGPU_PL_GWS: From patchwork Mon Mar 22 10:58:44 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154523 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D5ADAC433E3 for ; Mon, 22 Mar 2021 11:08:49 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9F3936188B for ; Mon, 22 Mar 2021 11:08:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F3936188B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A76736E1D2; Mon, 22 Mar 2021 11:07:56 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2067.outbound.protection.outlook.com [40.107.243.67]) by gabe.freedesktop.org (Postfix) with ESMTPS id 658EC89F35; Mon, 22 Mar 2021 11:07:50 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Qs2vOBobAE9wcvsfzAEBuTxKUgJTY4Bsap0CBxFLPP53gg1aour4h7ki42M6kMpdvLis55LgiJVV4hc9ffInSpZVb7d9Wfe61Xm5lhTzkVD3qFOErW/jkauri6fltoBykO0GBp3SgTok6kBzeQCoLJN7lsynoJ32uxYtEbE/4hitbJdbrlnViEl2sezIIyRyjcYW6cWkfBrhpMK7fBx4fej3s4GtnmeQdoF7xU12a/cO53JQhDT+/QZv2Hf/qw9HQdBAf6KxFXJo+/h1m3Nv3G2OQvdyL4MyhyRwY6AjUKLMhV8TDZhsvjel8YHO41PDsg3uhOHefJHRtHyDNF2mxQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AAEHUI5OL68JjCeuRyb1wAw5q/46pYRqvaZUcxEN10I=; b=fbN64gyJkIEl1M5OdQ4tuqTMhMTgpaA9Q4AaTOKHim8Y6ioZlOPvccvVo36aG/oXAhnTW5/SLR5UR0iCc/WEHtQ98sBG3bh6wK10G//bvAGNF6CrAtVS1yHfAu4AmRVhwEHu7TC7PYypcpsLnSty5Gf9E0d+476QV7Z97Yhi4dcRTqn155RAaRsPkjf/u6iJHwlWEgmACAr4T+5PLTX+bZOYj4qtvGQ/8Wvwr+LiPvtv4+dy3GaZvj4dCXRq87SFX+tcbliwJpeZX5jzZQ2ZRL9HIzW1oFnPB+sEZXxZbhos9uGY1pD9WpjXEv8nWevSQ4F7GQ2cPq3M2OmJAAHqjg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AAEHUI5OL68JjCeuRyb1wAw5q/46pYRqvaZUcxEN10I=; b=dwlVVJuzaCD8SPpljIiGiCyNv2YoF83vyWuVzL/zkigCMokQinxF5Pkt6Rg7uhvu8eKDZw2h87GqAe7qcro4BFLTasD63DKDRJXoeaaTQ3Rsas+UfQIyyNk1KXUcisJY3QXV7joOxckS70Woven0i6plpHUsPu2WxzPWMYRlPXg= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:45 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:45 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 28/44] drm/amdgpu: add svm_bo eviction to enable_signal cb Date: Mon, 22 Mar 2021 06:58:44 -0400 Message-Id: <20210322105900.14068-29-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:44 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 063cb8b6-dcf4-4632-e684-08d8ed22b821 X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: Ks2v4JIGAIMJav2HQjMBEtIDnJT3ADbXZeRYLQVQtZMPY1VHJrIs/GyfmDbImIaWHb+nvSpRIJMllRsuZG41fKqdOpfR+48CtXF3tlXFnudMyExHRZPfPkUcA6q+to9fxqdk4Z0YFt1ofUNngVwEsQkflKOz4cvr04afEwVGq38wW4fsCJI74LGc6wzscx4099sD4/gfl591NUo313Ox0uF/CsP+K2JrHGlpZgZPkibuL5ml1xxlQRoPnVHe6IeRy3FfgsVmAXfjBQdx66MqxHF2GLLBZUzpfs/NEawNTIYpiXP55XsEw6t9RwKFz+S7gAIIyW0v5YPf9AnRJSpr2ICK29AevhRbGy58XBoLBMFewYQVrSorH7cm4ig9Nvfhn+26m7giEkhc3GcoyK1HNU5psJ9u0kOviaSnnBheZGaPzWdYg+17d664AcVgRUCz4G+Zs2dIdC5NJmYLF3VzFHAdN8lxGcVi7UHi1/SXhq13GSuy6nU6/h21AIr83hKLPgvHIcYqKc/aHXRta1D7GY7nUHcxTu0KfFhc0DQGEmnMwM9kJitVO8LMyPRn8GwfPedG3Q2V2Wisng18gTBhoMctaultBIQrntdYhOVJcO6YZAbXTF2/unzk9F4T6teV/mVc71DnEElF/XD7tvNMEC8aX+2neLDpScgLgWJKsgo= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(316002)(2906002)(7696005)(1076003)(52116002)(38100700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: fmMOyC3XWtAcv3dnyvEv1fW87R1winQOJXIvKIrwlMzNuXy2uUT10e6EGAHBO4HbZo6+2pNb4gJ6EAcXgBj+hc/0NqfJ2F/tuB+mG33ReKgFwN5jHhzvd/xzA6TMBbFJbNxetXtKchbN+uUEtRXM0TKtuHv/id5Zq2vQMEkXO5hcrsYoyHnpSVco59yoakaMKpcsU8EvKfD/i9Ij9R6DBYUhs1iTyrw4YShlxJO0vOtzlat3I+7KW0I/zEk4K/nQKBwyLN9JHqy+X4WSNBEXEYsgCI/Q8t/0LUjqpQYZXuECWwU649n1arqyzdoxn/dMRoFlaPEWa7xTk8ViaFV7gI7kgdmC88ln2+ogKHH0a4T2p87yuYaQWvGpMMfynvPNIVwYck0nVp1EmfeZtOmi58RhOEEf+5Nfl0TK26e5k87YwSkddxRDd6TL4XrEyx5gSZmzJtZgW73/B6+N8zjSmKIVl+9LWPHSf/xM8tnqDRMfWBC3IsICSJ49ZewRtut82Va5CchIMVeWxHf1246peFIqoChKeoOr5vWRWKsFxJ3nxc7rpFCWqhhPQ5knd0+HiDkFcKGCOnT7ch+KnMrdg24S9o36eG86AKZ6YfU5gHMYWdClrvtbOj/9lLNmAN5lY9MevvC60C6iSJuX/zls7LGzxC4f5pd8y+E2/3YcNq2BpJDKriS2oELQWt8HDfidmf2kUpTHiDaMPP4fgZm69DNYILdWz1G6nexHvqHYxoe1wISczotlmfNPDvci05C+AxuW653bPtXgRtdBm6/naA1Ja9Ajwgp8DTv2Xd6ZsrYgbdQWWoEZAzp4Yo8qAyqniwzUkoZ+tMRv3wcg5z0OGYSBdAN9voxHmI1Tx2AeVpxQZwkam52RgNlealoow7+aozOrzVkH0faHSbsMqd4LL2/+X817rPKQgpc0NOQhrdS54ZGqkdMrD6pM1wu/2e5bH1Qin21Qh7hWE7j6M6DX2GjNlzThYc1yNbxPMOG648FxLn/2E8hH8TQrdBJzwail+bNE1I/q39e1pvMB/JQm0kvX8lZCrv7ysuPxa06CGBjKrDNeMdIAagHOW/baPxet/pdlpBRFo8qlu1bByoekgeXPoeXgT3K7vMAwwND6/lMMi9iNFSpVaMJHi74eWwGKjmOwCG/4IqOF7Gmo677uXMt25yEoJNm27iDocOQtCs9psXjYix6inWtYzVWxZeQt+t6ZZuXFGhLlhij6OQhk/VbhynF1Hp1qF0djfJYfTGOsd+DB3UO16bfNLuKe2Qz+i7A6Rb+S+9e7gcIvqiQH5qiPKAGoP8vZxxAlavTloQ+IdXrKyJ8Rr54ikUBBKwCE X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 063cb8b6-dcf4-4632-e684-08d8ed22b821 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:44.8677 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: ZpQvr4so2M+iDg+VozzUcLHPeTpGY539ymDe3j9GzQamC5O4hVO552tZU9JBzFeXmqOq5iLDcG3+kDLC8g3nAg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Alex Sierra Add to amdgpu_amdkfd_fence.enable_signal callback, support for svm_bo fence eviction. Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c index 53559643c712..1fe233cddb20 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_fence.c @@ -28,6 +28,7 @@ #include #include #include "amdgpu_amdkfd.h" +#include "kfd_svm.h" static const struct dma_fence_ops amdkfd_fence_ops; static atomic_t fence_seq = ATOMIC_INIT(0); @@ -123,9 +124,13 @@ static bool amdkfd_fence_enable_signaling(struct dma_fence *f) if (dma_fence_is_signaled(f)) return true; - if (!kgd2kfd_schedule_evict_and_restore_process(fence->mm, f)) - return true; - + if (!fence->svm_bo) { + if (!kgd2kfd_schedule_evict_and_restore_process(fence->mm, f)) + return true; + } else { + if (!svm_range_schedule_evict_svm_bo(fence)) + return true; + } return false; } From patchwork Mon Mar 22 10:58:45 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154521 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 98707C433DB for ; Mon, 22 Mar 2021 11:08:47 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 6AEF26188B for ; Mon, 22 Mar 2021 11:08:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6AEF26188B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7656D6E175; Mon, 22 Mar 2021 11:07:56 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2067.outbound.protection.outlook.com [40.107.243.67]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2FCDD89F43; Mon, 22 Mar 2021 11:07:50 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=P9d4tzKEOEJ2hKkAWyTdELXeIzUDbrMU/Umyh2jbRrX+0fU2vv5v6jGSD2q6S4ZqKm5d/Sv6BKWaoeBX9xrNuVvcdJeP/FeV99ELvEiNljXii08oTSS9DYLqNvfK3qAWruEu45hXPzYn/UJxKz5AnFheHQVeuoy/GOgCiCeHDaC9LMSMo1HgKcNlQkJBbKoncsrm4AgTEaRW71PYP5DroIclicW/rvPHaYy8aiHtHwr4Z2j0+7R1py/AlkIzs5+QkJSIMsDpHSiNE8tv5shWzS8SAnjh4RpSTgda1bstNodO5SK4zDSiUYt8KgLvI4u0PMcyEIRyoewe4qM7Q1XFjQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8WsocFKaaMfkqXNLfv8YmfOY+VmBdAjI3ft21MomVWU=; b=N7yx2Ve5wKzBEbtfoHeXJff4GxODriWNa8S0v1miECtyqnv6Qr2UmoyS8nkNmyHKnzX/I9PFaUj+XU1/0ICxOGjeAcejrWrbbuPzEGUv4kjj7zCiZrlkPfym3WW+w9DZPSxMBmQ9U5QmGyYrcsA0ctAPgBdISLgElBSR1K6CmIDxgLos7LJnV79X+7ZPFynT68hQSFHYFtmwZvyhgOGC3brgGgXVPY3dKWnLRmZEt40tx+O29nKM/JT2oXsn0QLUz/J6+OKotNFBdBMQTlt588XISgLX7/1/3r4Nn7xMhuVZHFAwEB/Sk3KPSzgI8GDQJ8ISDEHwbpKWOymeXiK1xg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8WsocFKaaMfkqXNLfv8YmfOY+VmBdAjI3ft21MomVWU=; b=M97XZQ0PgilI7TBXJcxy/thDqOGZt4MCZn39WidAFdKZQ+DfAt06QrvX4NqXGDkCCGCUxmb+jIRG8Nh8uhw/fPbC2BT8cUL1T+5bEHlodUPgJWSX5LTnNu7JtzzGO38zcyf1h01UL0lcxUI/+KW7tnKU+xcGnO3545AOxsuxtNM= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:07:45 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:07:45 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 29/44] drm/amdgpu: reserve fence slot to update page table Date: Mon, 22 Mar 2021 06:58:45 -0400 Message-Id: <20210322105900.14068-30-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:45 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: ce6e4b72-7a04-4699-26b6-08d8ed22b869 X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:126; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: vnPb+8+nKmC0v5kpva5NOcH4GIKlFKxGPO2yO6msJTMEFuny6jT0UfvghtcTPs/hRQ0py8ueyblvpclX8YLwwNbrf6FBrZgqQh6RymF8KgH6Z+g5CR6cROnix43n+2SWp440aE9yIJ61Tu/TDU2cytnnnc3dPR4RFuHT4fvrP0P9Z1GCDy+ftV94aeZ4nsjV6NrgeOS6VTkjmts4tTQ8iLgbCq2htufyTKd/zVmWPYH+0pLtaXSB/9rsdZYrmr7b2Qaak584bJwKX+vNLMFDG8hxoC7fD4VNwm34ZisxF+XYAvF1E0r/CdwcJihlo0TOoCeh2ZNgkA4c+5dpPya5Zwqo+FStZEwOs3495AwW+HcDPROYoi+IXhLG+dv5VjTjk3x5l75Z1xxiw9yVzEzuMgH6hOFKPaO8GWrUHEYmmno4fBXsPMn0+URWSSKf9l4ppN0KsWoMpPjwuPWZ5ZXHIofak95k5kqE23di4iBF8hLAMkUCWzyaKt+m8qqaPp65Q8KpLKtpgaj+4UhTYVbVKZ29a4FUq28DOaYd+HQE+Ae+YwqcnxpsbUqMzSLth6Sw8ENpF6Or5wI/a5siCJuKdI2BCMfv82l8hfEJJ7EWOigdmFSIHMVJbIQomlbQ3hBof19mbj2UGzaQEgpqJUOvM7JIBB41sTqpoSg5Ocrm30A= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(316002)(2906002)(7696005)(1076003)(52116002)(15650500001)(38100700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: +iFBReVKmzFJ6ZYVeftOEvZAbaLM+W00KKuMtP8j1z5MuMRqMPIRfOw0jrky81Y36kfdQ/Srx8aaRnqIpAlHoRtrvk4lZC8ia7WzfIEAkCyLepyGtex65M3Z4JRfHy+IO7A1LTsQZZIcRemW0YqBPv0RxMFj8kJQer7tpAzDm4kCUo7vEnMXopZRlgzMGd6kOOpUgPuBUaow9XuX/UDYcpdb6zfnMKokE9TcDg17D4C0AkDgO4wI3rm+1OZb1dNZRP9yz3HBMPHRydMT8Hw5JhBiqIcIwPD8wVPY11YwxHT1XP7t2CnhbrlmSGTqcGodvvUtj0fPLhPmj8KIMm2S17mY6tc07vKdC+N+7fA81d/wL5K3WJCcw4QLG7pe5aGrTiSrYU0JYQShYcBRSJi6mt21SbQQC2VVO6SfF43H6bQ0AIJEI4she5iW4g/37kRgf7GE9WLLC1uwhYnbR6xFcLZHGGlbwpsfiBeo/jZ/M3aT/tkGuERQGrGelc8UoCxN2tkfwKqRW1jTjs/i7LdWSCg6VNcThGHhgXmSX3tdZY6XbDFhZAFxLEufacsV6EwNIJHxxGgdbtw9i4cxguJ79bUxTs45MPZ7kQdeex9eqPrzkSqzFVqA7cRvLWf8TKBX7FMF5tPSpqzwZdTjN+eIDxd8Ihvcg1+3SQWXBvE++QuFfBt+PsqisPGnmzc/vjgAv6/D1rYG0nLjoEElGEMZ1hZBImYtLEhM69wJrTdbnD2PXKJivmcqiKqJiR3jCrce0UWG5LjGtlV5+RJmKZx0NyWppITVXTkbsAkLPYko3MHJLKS7VRDalVRJ9uyahI6sZM/cHmIrn1AzzQLQNYnzfIueaSUZp+MvhgZ4iwpcMx5T/r+6g66dzyN9oMXbTYW4BpSxFR2T/t+vl05hoW24geW0QatbuThjFLb90GdDX5+BWAJrr73zKuowdQ3TlVlbC6b3ieAiNzshWcZhuabcx9iCnAE/BPw9a6yRsn7ip2OoZ45dmIPAxEowhAxZiJOrobyS4AGlD0r+B/+AVbkyoP0bEF9DvslYqq9TTW9KHuLeSdM1tRl63gsubYtVna5HTRjnj6dsaSgeHHgue3//xGqbsU3skf8tQiecqDzYAgNtArvoMxmHfGaPLjaOTzlTT0vFxDZFP0NKJvtbmU4L5bcYao3Dsk1HXiG6fGvr+tvDGNtuWXo7mwdjnpahjNRcEP2kTEmO8IN1mLHxseJfP646VpW1IYD+c13PRQyjtVJiNT3Uy1ATlg432uEeFYw6k2wqzR8XL4a+OHTEJFFsjloEQlIOJe7kbU8Di13sUy4Y7aZdo53ngf5eS1sJn/gT X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: ce6e4b72-7a04-4699-26b6-08d8ed22b869 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:45.3526 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: TS98tCFTFiT/QdaF9hY+IbEo0DUvxGIAwIk00urOPQI1+++FlkPckDJ/83VNJFzvhpwu+NPye7RpDSKDNXZZyw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang Forgot to reserve a fence slot to use sdma to update page table, cause below kernel BUG backtrace to handle vm retry fault while application is exiting. [ 133.048143] kernel BUG at /home/yangp/git/compute_staging/kernel/drivers/dma-buf/dma-resv.c:281! [ 133.048487] Workqueue: events amdgpu_irq_handle_ih1 [amdgpu] [ 133.048506] RIP: 0010:dma_resv_add_shared_fence+0x204/0x280 [ 133.048672] amdgpu_vm_sdma_commit+0x134/0x220 [amdgpu] [ 133.048788] amdgpu_vm_bo_update_range+0x220/0x250 [amdgpu] [ 133.048905] amdgpu_vm_handle_fault+0x202/0x370 [amdgpu] [ 133.049031] gmc_v9_0_process_interrupt+0x1ab/0x310 [amdgpu] [ 133.049165] ? kgd2kfd_interrupt+0x9a/0x180 [amdgpu] [ 133.049289] ? amdgpu_irq_dispatch+0xb6/0x240 [amdgpu] [ 133.049408] amdgpu_irq_dispatch+0xb6/0x240 [amdgpu] [ 133.049534] amdgpu_ih_process+0x9b/0x1c0 [amdgpu] [ 133.049657] amdgpu_irq_handle_ih1+0x21/0x60 [amdgpu] [ 133.049669] process_one_work+0x29f/0x640 [ 133.049678] worker_thread+0x39/0x3f0 [ 133.049685] ? process_one_work+0x640/0x640 Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling Reviewed-by: Christian König --- drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index a61df234f012..3e32f76cd7bb 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -3302,7 +3302,7 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid, struct amdgpu_bo *root; uint64_t value, flags; struct amdgpu_vm *vm; - long r; + int r; bool is_compute_context = false; spin_lock(&adev->vm_manager.pasid_lock); @@ -3360,6 +3360,12 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid, value = 0; } + r = dma_resv_reserve_shared(root->tbo.base.resv, 1); + if (r) { + pr_debug("failed %d to reserve fence slot\n", r); + goto error_unlock; + } + r = amdgpu_vm_bo_update_mapping(adev, adev, vm, true, false, NULL, addr, addr, flags, value, NULL, NULL, NULL); @@ -3371,7 +3377,7 @@ bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid, error_unlock: amdgpu_bo_unreserve(root); if (r < 0) - DRM_ERROR("Can't handle page fault (%ld)\n", r); + DRM_ERROR("Can't handle page fault (%d)\n", r); error_unref: amdgpu_bo_unref(&root); From patchwork Mon Mar 22 10:58:46 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154531 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 65722C433E2 for ; Mon, 22 Mar 2021 11:08:57 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2E69D6188B for ; Mon, 22 Mar 2021 11:08:57 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2E69D6188B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 831626E131; Mon, 22 Mar 2021 11:08:18 +0000 (UTC) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2055.outbound.protection.outlook.com [40.107.244.55]) by gabe.freedesktop.org (Postfix) with ESMTPS id 543916E084; Mon, 22 Mar 2021 11:08:17 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=H2EfZHqu18yLbRXoS1D6zkGsgJGGRUys5PufWSaAUzbVumEAPKnuBn/OQv6US7mNroRCOBHU5ibw78PjG8X3yInIu8LYoihd6SFVl9XJVua79aDuVq3FSSDYVXPT0KjnRPrBZbrwLZex3xVvxSrooNOHY0Mmd3j+8Hpr4tDq45zKx5psZXovrRReJ97Gis5DrG5GitVzvuiQCERE1au+dbq2EQmmdmj51XGps2SDHWkDz1pXVLBRAzT5kI47TWu+Kqhgnq0HC3Ggm66aJYmUwb94Sy8duqa8M1bbsaDtQLu5AkPQj/Z5jJ4SRdgKXZTh0SIhJ5gaLb1bi3SXzEQ5oA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rS9NQkqydZvbsSVxX5fbDh2inCGWs7BtSqnr9rI2uGg=; b=UCF576SG10MAQ9wIAFJ3x7wHznqT0UaddpbQcaEMpS9iuKGfb1XUOvY0kHbXaYGFT363mN5MMLqXxOtZGx+yvPtdC1JfmIJLt73LYDu8MOeNrX9l+PZkV2lAzUSw3i1TrYImem/m6RtUPNIX8ppE63Ty8juvrP2yb7h9ybqKtZgOW0hx5E9xHXDXtbMpM07BVjeRBYdVvRr5J0i472Xx+fuHPr65SfaghTXjwbqHPcYyKOMs5JWZ2JV94+/YjCkvS0Lk4ZsNlXBudZabFL2BRNgdwXm5gvJYoUDEk9HlpGBNXoAerKoQkDLq4hjF00G0zbYcUYsAGaIOP8tRyMOWdA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=rS9NQkqydZvbsSVxX5fbDh2inCGWs7BtSqnr9rI2uGg=; b=025A5w0MdXHBnEpQbgLgK8CRMGYOtjdiLsQvMVxLLapaGgrbH2sJThew838IRtfS4/XiJtK9A4so33FDFqY5MPfUcydsUSsOdrQOOFRjxaGmgvrIW/epXkP0Pwbj2ORzYS739nT7ty99P3P+M0dvXgnCQw/uay3ZsnHn6c+sS6k= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:08:16 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:16 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 30/44] drm/amdkfd: refine migration policy with xnack on Date: Mon, 22 Mar 2021 06:58:46 -0400 Message-Id: <20210322105900.14068-31-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:45 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: a4b803d8-9aee-4a46-23e0-08d8ed22b8cb X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5797; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: gVzZ9MhhQ2auXzUc7ltEqwV7JmD+Dj2nlYarSR9s+fX5OXKm7sGKxuS4rMv3yFL9p77XXRl9SLmbvlUMG9YbPtCK6I4qjOf0bhUpPHpubngXQlllqN8NaCZhRJCIJ0LACZL9rMkNuTUcYEdw7tJ6wIAoaYbMXFLKRxj5z7o16fRlj0C75K1DTttT1LcBc0YxPS3iabIgjfePc2UNZdKiM1OcjMkfxyzUdqlBoV5YzmgSQ+2+4HRQNNe4HIMuTcjvBIMWGigFfJJeCf7L4GyGkGY3VlYMd94QbmDe7kDjTqsGQweD+gxh0xryudV5kBpFO0yXP7MnniwHUYBX7EvYxEt13KJjVJHCygoHQulh71Q0SN1UPF0RCfLDdMUthKYl7ejybYewhtHQuCLXMcGbGJRw9zFgRrBIhjthh46nPEEJvmvpk3f9iZxXk2XEZL58Zfl9R+iQGu0RT8dfPtRcX+pzQJx60p1KAUx7mklA5zBp3DDZ2MGDbmDm2Fz6UrE+MDg842irpZOOuy+qZ45pf3MQqlh7Pm6t4BQEBer8YaGG+uS0fxBwUI0f8bxJK1BdThpi8py03T6Dw8O8tvKXXzGGzPZo22X9j8BxSV1iTZ7WaTWT47u5ggvLkB6aXiQxseY+Kotg5aQWAO457O5Arg== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(30864003)(450100002)(316002)(2906002)(54906003)(7696005)(1076003)(52116002)(38100700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: BkRCaJjpci661i5en4NDvBuFEiBFwArTzcZbIj8hKR9CkxRyuSEcUPsaM6dnyZVrR2gbLlL8GvEWAP69mxN3z/AGI/y28QqZKR3bWr2GWHGCs3tjffCIezH4TNy8hXlOtCfSVg2WvYMb30D8G7Z+oQFdyLJrJIF2Yrlec0DIaH2Y6JKtf0yLSO73/ygJbkP/N009BvCypvqtprCQS30po7vzVTTmb0AQcKdLBeRNuyZanOKRmCBM52zMDrEncP3tzNqG9WmvqmoV4G2CnX9aXr93ZliHzEZZ3Z2cduJneoCFJmksf9UIhk53hZD5bDoDewV2F+Bvi/JN+qxuHlUVzRS0AT2Z99GuYNuueOcihBMQz49x9Dhy//JTVlt/UzAJ1JQekx9Xnr/IxEe3seyvJvI5nR2d6Ik+Hmsy2z7kRvV6NqiJhSNJVmlha8FF75d8HA22dNHC45pVLQPEg12ax0OQExVahgPca6ZEDvUJXbkndC3+TSmIqY1xj8F7+vCfK3d/5EjFuAY94S0ciF8QAmwDJ632AZ9w6vufvf2yIjfLcc5K2Kom2B+fKof/k5kxq0CosyTNCDdbcvxWXVSNFcmncP1u8UDmvOV5pwTBjstp0h3VEzaqJFO1VxQ2l24Yw7QzQumKXpvWc4pRZLK1xLM6Oy8czAi5yN+w84OuFSL5Upx+5FCEE/XrQshnFOUs7F0fVE1nut1Ea5lG5yh+WNrYks3YhUGgFBu0+0nLCrr8nZLbWIxPW72zWaUWW36U7Sod4m7IxrR1aR8P2N7fc50ogbye91qAhdnEFuRYi5RN/Lr3OjKglHgzbvuqTICmKKRxSRaoKjsZSZvSHUC4kBYyGDb3+VM3sSVjoRlB16kpJsR//Hxu915BQYD0DxjeSkD1cazNbjIS/xyjCdzdC0k9fSG/FxMJcPWRw9b0i/JDb/ulGkQgg8pBLuQxMWcr3FleCs0CO9tmA+sEUGM8AP2I+P/00cNbSnxIdzsv5L2HRye6DPYHa99M7EQkgq+QkFx+M354udrBEztPJUW7hRbpq54jCdQGpmrowpiQz6kenErLkdREVtW+8TjaBPx3nLGA8tZMUtFAyS4nLScSvbN3zsal38OGiFm0Y0esKXbjzqawRAgAzDe3mMJWhZEvQWFdSyIXZr40RyPXTNzjg++B8BZcqt/XVeyt+wQ3vT6ZI6kRbxCSRDkt/NV8pdWlt1/88qjR1Gmz+7E+uIOZHngybvTSJkkkv1pIVa3aytQKMBbEzqmR+wMByog41dS1vxI0l1cTx3awNZy4KcUnh4acybiFoQBom77I2ONv3Lc0eU9S9rPdE3js8n2SaVqc X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: a4b803d8-9aee-4a46-23e0-08d8ed22b8cb X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:45.9639 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: h/CJu6LwL7onaTin+Mb76PfTZmBaCouqI7Ydsx18kv6l5v8xd+VYlnE/umnJ80VBdjqzfN+lIBWsN9xLkleQxA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Alex Sierra , Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" With xnack on, GPU vm fault handler decide the best restore location, then migrate range to the best restore location and update GPU mapping to recover the GPU vm fault. Signed-off-by: Philip Yang Signed-off-by: Alex Sierra Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 7 +- drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 3 +- drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 3 + drivers/gpu/drm/amd/amdkfd/kfd_process.c | 16 +++ drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 150 ++++++++++++++++++++--- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 1 + 6 files changed, 157 insertions(+), 23 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 8ce3ff56a0ce..cd89b38e3d9b 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -455,18 +455,19 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange, * svm_migrate_ram_to_vram - migrate svm range from system to device * @prange: range structure * @best_loc: the device to migrate to + * @mm: the process mm structure * * Context: Process context, caller hold mmap read lock, svms lock, prange lock * * Return: * 0 - OK, otherwise error code */ -int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc) +int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, + struct mm_struct *mm) { unsigned long addr, start, end; struct vm_area_struct *vma; struct amdgpu_device *adev; - struct mm_struct *mm; int r = 0; if (prange->actual_loc == best_loc) { @@ -487,8 +488,6 @@ int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc) start = prange->start << PAGE_SHIFT; end = (prange->last + 1) << PAGE_SHIFT; - mm = current->mm; - for (addr = start; addr < end;) { unsigned long next; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h index 95fd7b21791f..9949b55d3b6a 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h @@ -37,7 +37,8 @@ enum MIGRATION_COPY_DIR { FROM_VRAM_TO_RAM }; -int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc); +int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, + struct mm_struct *mm); int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm); unsigned long svm_migrate_addr_to_pfn(struct amdgpu_device *adev, unsigned long addr); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index 755c0517867d..2ccfdb218198 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -867,6 +867,9 @@ int kfd_process_gpuid_from_gpuidx(struct kfd_process *p, int kfd_process_gpuidx_from_gpuid(struct kfd_process *p, uint32_t gpu_id); int kfd_process_device_from_gpuidx(struct kfd_process *p, uint32_t gpu_idx, struct kfd_dev **gpu); +int kfd_process_gpuid_from_kgd(struct kfd_process *p, + struct amdgpu_device *adev, uint32_t *gpuid, + uint32_t *gpuidx); void kfd_unref_process(struct kfd_process *p); int kfd_process_evict_queues(struct kfd_process *p); int kfd_process_restore_queues(struct kfd_process *p); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index c8479f6bd68c..48ea6f393353 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -1684,6 +1684,22 @@ int kfd_process_device_from_gpuidx(struct kfd_process *p, return -EINVAL; } +int +kfd_process_gpuid_from_kgd(struct kfd_process *p, struct amdgpu_device *adev, + uint32_t *gpuid, uint32_t *gpuidx) +{ + struct kgd_dev *kgd = (struct kgd_dev *)adev; + int i; + + for (i = 0; i < p->n_pdds; i++) + if (p->pdds[i] && p->pdds[i]->dev->kgd == kgd) { + *gpuid = p->pdds[i]->dev->id; + *gpuidx = i; + return 0; + } + return -EINVAL; +} + static void evict_process_worker(struct work_struct *work) { int ret; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 3a7b842b362c..98c049dc3a63 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -179,8 +179,11 @@ static void svm_range_free_dma_mappings(struct svm_range *prange) int r; p = container_of(prange->svms, struct kfd_process, svms); - bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, - MAX_GPU_INSTANCE); + if (p->xnack_enabled) + bitmap_copy(bitmap, prange->bitmap_aip, MAX_GPU_INSTANCE); + else + bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, + MAX_GPU_INSTANCE); for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { dma_addr = prange->dma_addr[gpuidx]; @@ -1262,7 +1265,7 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, return r; } -static int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) +int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) { DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE); struct kfd_process_device *pdd; @@ -1279,9 +1282,29 @@ static int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) else bo_adev = NULL; - bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, - MAX_GPU_INSTANCE); p = container_of(prange->svms, struct kfd_process, svms); + if (p->xnack_enabled) { + bitmap_copy(bitmap, prange->bitmap_aip, MAX_GPU_INSTANCE); + + /* If prefetch range to GPU, or GPU retry fault migrate range to + * GPU, which has ACCESS attribute to the range, create mapping + * on that GPU. + */ + if (prange->actual_loc) { + gpuidx = kfd_process_gpuidx_from_gpuid(p, + prange->actual_loc); + if (gpuidx < 0) { + WARN_ONCE(1, "failed get device by id 0x%x\n", + prange->actual_loc); + return -EINVAL; + } + if (test_bit(gpuidx, prange->bitmap_access)) + bitmap_set(bitmap, gpuidx, 1); + } + } else { + bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, + MAX_GPU_INSTANCE); + } for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { r = kfd_process_device_from_gpuidx(p, gpuidx, &dev); @@ -1995,15 +2018,76 @@ svm_range_from_addr(struct svm_range_list *svms, unsigned long addr, return NULL; } +/* svm_range_best_restore_location - decide the best fault restore location + * @prange: svm range structure + * @adev: the GPU on which vm fault happened + * + * This is only called when xnack is on, to decide the best location to restore + * the range mapping after GPU vm fault. Caller uses the best location to do + * migration if actual loc is not best location, then update GPU page table + * mapping to the best location. + * + * If vm fault gpu is range preferred loc, the best_loc is preferred loc. + * If vm fault gpu idx is on range ACCESSIBLE bitmap, best_loc is vm fault gpu + * If vm fault gpu idx is on range ACCESSIBLE_IN_PLACE bitmap, then + * if range actual loc is cpu, best_loc is cpu + * if vm fault gpu is on xgmi same hive of range actual loc gpu, best_loc is + * range actual loc. + * Otherwise, GPU no access, best_loc is -1. + * + * Return: + * -1 means vm fault GPU no access + * 0 for CPU or GPU id + */ +static int32_t +svm_range_best_restore_location(struct svm_range *prange, + struct amdgpu_device *adev) +{ + struct amdgpu_device *bo_adev; + struct kfd_process *p; + int32_t gpuidx; + uint32_t gpuid; + int r; + + p = container_of(prange->svms, struct kfd_process, svms); + + r = kfd_process_gpuid_from_kgd(p, adev, &gpuid, &gpuidx); + if (r < 0) { + pr_debug("failed to get gpuid from kgd\n"); + return -1; + } + + if (prange->preferred_loc == gpuid) + return prange->preferred_loc; + + if (test_bit(gpuidx, prange->bitmap_access)) + return gpuid; + + if (test_bit(gpuidx, prange->bitmap_aip)) { + if (!prange->actual_loc) + return 0; + + bo_adev = svm_range_get_adev_by_id(prange, prange->actual_loc); + if (amdgpu_xgmi_same_hive(adev, bo_adev)) + return prange->actual_loc; + else + return 0; + } + + return -1; +} + int svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, uint64_t addr) { - int r = 0; + struct amdgpu_device *bo_adev; struct mm_struct *mm = NULL; - struct svm_range *prange; struct svm_range_list *svms; + struct svm_range *prange; struct kfd_process *p; + int32_t best_loc; + int r = 0; p = kfd_lookup_process_by_pasid(pasid); if (!p) { @@ -2036,21 +2120,50 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, mutex_lock(&prange->migrate_mutex); + best_loc = svm_range_best_restore_location(prange, adev); + if (best_loc == -1) { + pr_debug("svms %p failed get best restore loc [0x%lx 0x%lx]\n", + svms, prange->start, prange->last); + r = -EACCES; + goto out_unlock_range; + } + + pr_debug("svms %p [0x%lx 0x%lx] best restore 0x%x, actual loc 0x%x\n", + svms, prange->start, prange->last, best_loc, + prange->actual_loc); + + if (prange->actual_loc != best_loc) { + if (best_loc) + r = svm_migrate_ram_to_vram(prange, best_loc, mm); + else + r = svm_migrate_vram_to_ram(prange, mm); + if (r) { + pr_debug("failed %d to migrate svms %p [0x%lx 0x%lx]\n", + r, svms, prange->start, prange->last); + goto out_unlock_range; + } + } + r = svm_range_validate(mm, prange); if (r) { - pr_debug("failed %d to validate svms 0x%p [0x%lx 0x%lx]\n", r, + pr_debug("failed %d to validate svms %p [0x%lx 0x%lx]\n", r, svms, prange->start, prange->last); - goto out_unlock_range; } - pr_debug("restoring svms 0x%p [0x%lx %lx] mapping\n", - svms, prange->start, prange->last); + if (prange->svm_bo && prange->ttm_res) + bo_adev = amdgpu_ttm_adev(prange->svm_bo->bo->tbo.bdev); + else + bo_adev = NULL; + + pr_debug("restoring svms 0x%p [0x%lx %lx] mapping, bo_adev is %s\n", + svms, prange->start, prange->last, + bo_adev ? "not NULL" : "NULL"); r = svm_range_map_to_gpus(prange, true); if (r) - pr_debug("failed %d to map svms 0x%p [0x%lx 0x%lx] to gpu\n", r, - svms, prange->start, prange->last); + pr_debug("failed %d to map svms 0x%p [0x%lx 0x%lx] to gpus\n", + r, svms, prange->start, prange->last); out_unlock_range: mutex_unlock(&prange->migrate_mutex); @@ -2184,7 +2297,7 @@ svm_range_add(struct kfd_process *p, uint64_t start, uint64_t size, return 0; } -/* svm_range_best_location - decide the best actual location +/* svm_range_best_prefetch_location - decide the best prefetch location * @prange: svm range structure * * For xnack off: @@ -2206,7 +2319,8 @@ svm_range_add(struct kfd_process *p, uint64_t start, uint64_t size, * Return: * 0 for CPU or GPU id */ -static uint32_t svm_range_best_location(struct svm_range *prange) +static uint32_t +svm_range_best_prefetch_location(struct svm_range *prange) { DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE); uint32_t best_loc = prange->prefetch_loc; @@ -2283,7 +2397,7 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange, int r = 0; *migrated = false; - best_loc = svm_range_best_location(prange); + best_loc = svm_range_best_prefetch_location(prange); if (best_loc == KFD_IOCTL_SVM_LOCATION_UNDEFINED || best_loc == prange->actual_loc) @@ -2304,10 +2418,10 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange, } pr_debug("migrate from ram to vram\n"); - r = svm_migrate_ram_to_vram(prange, best_loc); + r = svm_migrate_ram_to_vram(prange, best_loc, mm); } else { pr_debug("migrate from vram to ram\n"); - r = svm_migrate_vram_to_ram(prange, current->mm); + r = svm_migrate_vram_to_ram(prange, mm); } if (!r) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index f8e282ec9d8a..fed28e487878 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -166,6 +166,7 @@ int svm_range_split_by_granularity(struct kfd_process *p, struct mm_struct *mm, int svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, uint64_t addr); int svm_range_schedule_evict_svm_bo(struct amdgpu_amdkfd_fence *fence); +int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm); void svm_range_add_list_work(struct svm_range_list *svms, struct svm_range *prange, struct mm_struct *mm, enum svm_work_list_ops op); From patchwork Mon Mar 22 10:58:47 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60977C433C1 for ; Mon, 22 Mar 2021 11:08:59 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 28AB96188B for ; Mon, 22 Mar 2021 11:08:59 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 28AB96188B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id ADEA96E199; Mon, 22 Mar 2021 11:08:19 +0000 (UTC) Received: from NAM12-MW2-obe.outbound.protection.outlook.com (mail-mw2nam12on2055.outbound.protection.outlook.com [40.107.244.55]) by gabe.freedesktop.org (Postfix) with ESMTPS id 87F8D6E11F; Mon, 22 Mar 2021 11:08:17 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZwcDDpDZGiSqOyajA0IcEjWsmdAoTnzntSrF1RVrK6E9jMPec24AE8Xn/TO3d23rRbvz4idatzs3UlGpVQH+dUHtHRaaTjZYrwVoZ/Ufn3R7Rf+HI1Jr4LSCjEtbDNdGjcK9e0uILJ7HxXfUtrxWFxhs0mnJKDOo6aEU4/XMYckyPoPhZQOTbDZkGC5yZAgO2TuZDHNpkb8kkltmRJtKQMKBHX7ipjd9Wd2vDiPqTFYeyYZzHnEJ8vcPS2VxQleRXkecwysBDP7srcW2dycDHiTh+cUJo4DGD3K/5v5c6rgvaI94ZQr2GaD8EQGZnXkGUj3HVR3PRy20m2COdNrSng== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BlEYzgsQYI2T3/t9V8hO0Z/JKWzWyGyu+FmeHzpja8k=; b=LI+HdTBKeVB/AhM8NJAfxNsRDGzgkrRYQrm4aRsfZJ1EGpFgz/KUQQkqf/XRUIV5lmxbpNGvTBn4vHVSmawOprWqiq/Yp6ryJIqIcxew9DFUEN0uGJYev3BDfCKc7WlIb+evYj01xv0CCwf/GVJweJyXux7JieXzWM3KVnuMMkLie6NxkDu9LSWH/uDGNFpS8FnSo+bQ9F8Fr+vEfcenlaoWX4/gXZrhrLf00+OikJKLNYIVHdMUhbu6czwlm210240uJSwUowZrFqV1yiEgt9/k+/Vv/4CopGc+V2L4+NssmBqlGGCud9CxSz0l/XMRtGTelAOUrYs+FkJBoUEqag== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=BlEYzgsQYI2T3/t9V8hO0Z/JKWzWyGyu+FmeHzpja8k=; b=WVMiZ0SICsFyF3YoYqBMb4C1P4DeS2RDMoohBhUuBRJvvcNHNF8xmyVnCqJusEXaWC4YHlt9mxfA2MVUPQGUzi2+1T6bgesMvRcdBle6NUDH7TJcsGepCdcuOzvpXEO31pyB5bSeEASR4ixjf9QSVeAWrwpyG8jj74+xW6VREnc= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by BL0PR12MB4899.namprd12.prod.outlook.com (2603:10b6:208:1cf::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.18; Mon, 22 Mar 2021 11:08:16 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:16 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 31/44] drm/amdkfd: add svm range validate timestamp Date: Mon, 22 Mar 2021 06:58:47 -0400 Message-Id: <20210322105900.14068-32-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:46 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 1815aaa5-d16d-4f52-40a0-08d8ed22b917 X-MS-TrafficTypeDiagnostic: BL0PR12MB4899: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: O7sCbuQ3lHkDWewFzIneioVfNifHBNVCUnGI02tig37tEuEIYQofNVjcPsWIOk5FNesTO8DW1b4wJkgX2P0JkVb0rLwuD16wcUx0urdhF+asN5ve3LlIy6Bre5pKsKLIivigBVU25Sif75Ov9C0u6DuQlH9gfoBfw7hks33Tru5s6wlmYQGQpBmDLKGuXIzDHZK5jNbczA9g+YrLJwd4zKhdy9sd3zZgNCKrUT89qh2yV6LyU49NMOpgpNucenraa1wfNWkdGUZxZ6RNOOrNFkSu8gGx7IrP5K3XNtR6NkxBbGYwNbuGiRdrL2jOn7DoIU5rvb+qdeXqLR5R5OiVX0YihRGxv4wcmUzwL9//QWx6S5DpOLfCTHpQIo3dN92/LjnLC8/exwAUbA1q74rGN8q8Iw8GVtKjogyL92mSwiIoLNKM78pqnONC5DMu7VzqjXgPpqat4fPsW2MCdREJJndukXDCWF5269AKr82nQCWY6ISHEP++UjqMA/nNd3uMWDLRix0WIifqXuSLHVatuJmfUd6BHgkjXQv5vxgfB34hjqIly5L6ZFH3JMkdvqlrnaRhoky1Xu9yIfzCysFkY7NpoTOmYGbCxvHjiB6zFqNiRBJILi5XDgeoJxX0Qd7rVcOQS7cRD39hjn8Y+5oAP6PyhnTfA7L100TEXbYhXt8= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(396003)(39860400002)(346002)(376002)(136003)(36756003)(83380400001)(66556008)(66476007)(186003)(66946007)(2616005)(8676002)(956004)(16526019)(86362001)(8936002)(26005)(478600001)(6486002)(5660300002)(4326008)(450100002)(316002)(2906002)(7696005)(1076003)(52116002)(15650500001)(38100700001); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: crkFh1ai1qbAe1YOP+T3Ck6ysmkEZMSs6aR3ju4o+k8+0Rv1iwO+dDHYr9ZVCgJgSDci2RoGI4XRrS1OtOch8u4BFwtI49dJuoQkjsidvM+/0dpvGFWkmZLR1wSggqnQ+5XLKPd8NBOK9JrXV4wxMGvXs77cpOu8mTZeju5d49+tojyaPZ7vlsAnvJADMyVFoxWoEXXwtUOLFBonqJmj4gzUQm0RhU1B+pg4UatSsf7CLccHtLDwFJB6OFxchN6DdUN0e4vT6Zpl366HdmgANU68ZpixdKtqIzhKOOoqiXvEMfzcKYiYkAGh2qfvZRkgdXocTf9mTDfKegiC1dIvNf4Nx0EXRROKO5DMgdjNZjcqWWXix3HvFIWIE6IvKxZoPzkCTupznDzpy11zkULWjG+hULoOw7fMo9NwG6SrZFCACtnOIfbbpzSQQogtQ0YeYMS0UjrJtFNJzRMVtPMswqiB0QvNFDFg9008q6l8XHPAJfSTzx0XvIuu9xyOf8H1HkBT0icEkegilk+Y8Hy2+gHzQjJgP9yCG6kFVpMT1rkoiZdNyjbkQGKd0Gtp7WFsIUHN9Mswg6rMz1DILiyW90uFUu++HdfkIzBuQboFOPQ7bJjivUPKoxagwPNG1eGFC5VJnXtH99CHc71OYbZccyVPH67uv6Z7dhlUpHfKtp+6PTw+RLPi+BtHxJq3Lbgkasj6ceArDPlasiETdRqIsZJwHzhJ920NVuD6F3idT9RXYfhd80F7ZbV7HFqkQnd42AOVebTIxAfvIiHHSYwNj/1vQwHkEo2C8z7y3/PKYAawmyB84+gPSEOlWSDXyKn5kS+2VGqOai9uFj1YeziwKglPl+JcSVwOO2LSMntmPQ6x6vluvGWBvJGZCxngitl+hw73x1z57LEK9kAklkArZ6mO79nUtc+jhc9SL+QuTHfRoVSAtWQCfydvf3N7hsCbnYfEWK/mBbIdomjgZqXo49NTeYBQshH/eBSbZxQKK8xPwe9dT3Bra9v4uaootbDvda3RWwgLtUXq5C291hXu31uHq0k3wURjZNKXX4kqvEj1E2ji55RNc8uRuD4yiQr3LtbX+SP4IBEI5yYhwLDbmePHokVbAeaUEP/7+I1V2N3j3pnKaKNYq/i2xyieXGc/jG73ZS7DT7vuXGSNUJc5Uo7LuUHqwiOLtlavGzjdApzVwsuKRl/yhnPb1xLYnjmTgWynX/1Vu1gRjGa86+4k3Lr1tdC+FMrv2IDvI0D+Fvl+ghZNDyNjiaoAhakUlS5Q1jYgvq0PsvwSFnrTqJAOL64C1wTCYIn2jpOftkrFaysvZnsbNy4DOsp0JTNxTjPL X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 1815aaa5-d16d-4f52-40a0-08d8ed22b917 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:46.4806 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: fZTwiOj3uu3kREoTOItAB+WKVI1GQRHgSXAj9usQ7Eqr6bREu/AxQOdyTCiKA2W4RfzDihVk4KQ/huiges50Cg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL0PR12MB4899 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" With xnack on, add validate timestamp in order to handle GPU vm fault from multiple GPUs. If GPU retry fault need migrate the range to the best restore location, use range validate timestamp to record system timestamp after range is restored to update GPU page table. Because multiple pages of same range have multiple retry fault, define AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING to the long time period that pending retry fault may still comes after page table update, to skip duplicate retry fault of same range. If difference between system timestamp and range last validate timestamp is bigger than AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING, that means the retry fault is from another GPU, then continue to handle retry fault recover. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 24 ++++++++++++++++++++++-- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 2 ++ 2 files changed, 24 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 98c049dc3a63..f4b4fea06ac9 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -34,6 +34,11 @@ #define AMDGPU_SVM_RANGE_RESTORE_DELAY_MS 1 +/* Long enough to ensure no retry fault comes after svm range is restored and + * page table is updated. + */ +#define AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING 2000 + static void svm_range_evict_svm_bo_worker(struct work_struct *work); static bool svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, @@ -246,6 +251,7 @@ svm_range *svm_range_new(struct svm_range_list *svms, uint64_t start, INIT_LIST_HEAD(&prange->deferred_list); INIT_LIST_HEAD(&prange->child_list); atomic_set(&prange->invalid, 0); + prange->validate_timestamp = ktime_to_us(ktime_get()); mutex_init(&prange->migrate_mutex); mutex_init(&prange->lock); svm_range_set_default_attributes(&prange->preferred_loc, @@ -578,19 +584,25 @@ static int svm_range_validate_vram(struct svm_range *prange) static int svm_range_validate(struct mm_struct *mm, struct svm_range *prange) { + struct kfd_process *p; int r; pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx] actual loc 0x%x\n", prange->svms, prange, prange->start, prange->last, prange->actual_loc); + p = container_of(prange->svms, struct kfd_process, svms); + if (!prange->actual_loc) r = svm_range_validate_ram(mm, prange); else r = svm_range_validate_vram(prange); - pr_debug("svms 0x%p [0x%lx 0x%lx] ret %d invalid %d\n", prange->svms, - prange->start, prange->last, r, atomic_read(&prange->invalid)); + if (!r) + prange->validate_timestamp = ktime_to_us(ktime_get()); + + pr_debug("svms 0x%p [0x%lx 0x%lx] ret %d\n", prange->svms, + prange->start, prange->last, r); return r; } @@ -2086,6 +2098,7 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, struct svm_range_list *svms; struct svm_range *prange; struct kfd_process *p; + uint64_t timestamp; int32_t best_loc; int r = 0; @@ -2119,6 +2132,13 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, } mutex_lock(&prange->migrate_mutex); + timestamp = ktime_to_us(ktime_get()) - prange->validate_timestamp; + /* skip duplicate vm fault on different pages of same range */ + if (timestamp < AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING) { + pr_debug("svms 0x%p [0x%lx %lx] already restored\n", + svms, prange->start, prange->last); + goto out_unlock_range; + } best_loc = svm_range_best_restore_location(prange, adev); if (best_loc == -1) { diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index fed28e487878..fea9c63b5f95 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -87,6 +87,7 @@ struct svm_work_list_item { * @actual_loc: the actual location, 0 for CPU, or GPU id * @granularity:migration granularity, log2 num pages * @invalid: not 0 means cpu page table is invalidated + * @validate_timestamp: system timestamp when range is validated * @notifier: register mmu interval notifier * @work_item: deferred work item information * @deferred_list: list header used to add range to deferred list @@ -125,6 +126,7 @@ struct svm_range { uint32_t actual_loc; uint8_t granularity; atomic_t invalid; + uint64_t validate_timestamp; struct mmu_interval_notifier notifier; struct svm_work_list_item work_item; struct list_head deferred_list; From patchwork Mon Mar 22 10:58:48 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154549 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 31BF2C433E2 for ; Mon, 22 Mar 2021 11:09:19 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F040661919 for ; Mon, 22 Mar 2021 11:09:18 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F040661919 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D1D626E40B; Mon, 22 Mar 2021 11:08:31 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2045.outbound.protection.outlook.com [40.107.243.45]) by gabe.freedesktop.org (Postfix) with ESMTPS id 853716E3F5; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=S/cRShrkke9tXjc6RjvXlyUfWT/BCqEb00HVzItFXU/jztFlr0Bu62mHz3kINMxPJNlnizYadrBIf+zo3lBXVxfHMq6q/PFkFhDuFSFcmmUyew0pWqrSjU4/8xf9lNXhb+XhsEl/Me2MHZA2UDTmzVYyiUMl0HWnF1JtNzwV12XQ+VLfs+nZEntPQs2wUhJXw4wcfLg+86g7XWZX2bD5ZX1p33C0vWPKJX22j5Tx+w2bDNjH0nbhfTb4yB7KROMW6dJXNFtBr5DQeJupufAafWsmrQ2TYLByITa+QCi21FO0BF7k+kxCwbFrN4E2N9T6Ru4+9l7Sk6al4mXFZFIjdw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IZGaiprQoPgtAliB5UZrAeVNkhMD+rhDpCVNxwt3t1w=; b=gCfQS1XQOCBfFJuhCzg+1HM+Og+FW5QfwFtGexsf6OS6tYWgbAcohJ3bsHons0Wz28ezYDnyIbiJBX0iIZYly0B+6uyI42r8jPi9iwEbjObL+KKgipBJGkCs5HNtvvtmhrjEqaYAZv4yq9AJ5H2DjYiEo31itQFsCcQLvx3HdA4U4vOwm2Wr/E/p8luwr00MG2oRylgctbMYXhQIVcMeJ6Bbmx/8/IeZEUC92IUCC6ano8V7M3mjHCtwvlvBVaZWK5b0tgep98yUZWCVHEHvgjXBBUJpXfSZf8b2OzBXMMA7nlMrZoxcK/PICNko8q7ohyoB05OCY6l+YhjQfBJ+wA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=IZGaiprQoPgtAliB5UZrAeVNkhMD+rhDpCVNxwt3t1w=; b=1DzDBL/44CbDKfIL3Cy/bvDRVnGHtmxkfwplRfVAqM1OKVAqry89kS1OH8EfcrWcJvYv66vG6xKDGBiRW0wtPUFexgmMpK3aYe/hidTKin3JsXtKLyESx7jazXojPBJwh71Wiv+gkMXgv+AQjZaQQzj9CtdhJfAOH1uPbbDcEGw= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:16 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:16 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 32/44] drm/amdkfd: multiple gpu migrate vram to vram Date: Mon, 22 Mar 2021 06:58:48 -0400 Message-Id: <20210322105900.14068-33-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:46 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 3d451ac3-369d-4c24-538b-08d8ed22b960 X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: IAGyQnuQuic7e/GPdND2lwGL1olPk7eWqmQPbQKuIjQbBYdqo5G1H6p1oClQhKyh1uen6Hp2+sbqGiU8fT1FTsZehWadGjiH4qiXZq8FaSVVaDBMDYwmC86zwH0fBPmbSYiYlduXJFH9vAS5Vy6Z7wTHLIGNYx5s29cw8LBbUhV278IXACHYmQEAB+KFdMzossRtkM8knjZGIPcLtwCpX0ieJ4nGebt5B2BVqeN1EhA+W7DoJJhumOcfkY6y/8fheHhm5rJ23myffERPstaPK50S5hNUZGKGvUEbqpT1liBhcTWn4Zu69KuFuTLo8lkv3Go+buJPUHCU1XBWdg5R6ZKJZygzu5UfzqVVDFZ3M6JPJ8GxbA2Y+XVzEV0SpQwfSpZvoO1fX53kCbKjm3K3ac7cicOR+GDvFIZto4281wAgk+FXfEpM+LdsKNKKy9NSa+3h/anKanpClIjurk0895NjS31jtRDFUfvmY5vKygerIOIhoJxMthrBaiW5570yqYr9dRu8fl7dB0qVv15rUw/Vre94n4xm2e29FRN73tI1jAWDLDCQQ7HDk8TTJLlrFPTR9HSde+V4kGbnDF38HS5vQ/A48WuVGBgJbTz2iqscEZmIwykCj34UPFYRCzY31lZJlLGEMqKyHFD7F0fFuXJ5yPHYMqkwRuWDLYCalVM= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(4326008)(450100002)(1076003)(36756003)(38100700001)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: nGYv+mQQxrgy0RYrBUP1GXfYrrKCqj75cLQ2mrimUwPzpAdp8XRqfawpId/2eXWJ/4fglUxwiUmmWy3ggn0TOiCyv+SVFh4rIuxsgOmiX4tUhNdW4CZHgziWsSvrZXcZuJvrPk6qSMTv7cvCjc1r7SWBvviFRr8zkfOj3KTJcXTEiiPRFVDCURAimcDul4q1EKCtmQzDggNppCjWsAo6WdaOXXJZe0OnuBF2jVAythtJldNll+bHprAVlRen0D1yWbYZk/q1zRD6/6h6tBgEfQv6dtILsru+8bGgjrYQ6WsvFIXihKW41Nw8Y9NK1AxnKMAE1g7UTguHqyhwqXsMkH9pgrCfUf4+TLzvIjYEQJHB0ot4zNjPUYgyZ9ixjerNUDMcx8OUyScX06HziVeBwBLJpSpv7uKaYc1cUeWdxud+ltnIphcxd6rusFgh4UjDnkwFRLCTzOC22+ts9W+PdmEl5SCOCFX3h7gdDYgxze/SlQyOoxGUHlzunZwD4YHUaSZPQrgONzKIOoDV02gEvX/tT0zrmSiR+d+/W9e4MrVSyKxHnJWUzgFbSPzVohbQqlzrgltXJ+Wb/oBbizGQj2oEWoAUeAVHhMBaRzq982sYlL5VHv1pPstHD5koWkfqdeWTuh1+M4QrttTaWWrGTs9DUHPmvQPyI+rktPJ8+Q8lTbtLvCPLdnzDnq3L8B8WjyMdruy+HkIY2SnrXG/KrMm+5eIITQ72pRKMb7z++wBZWTUqCx2O70sQ7P9PJ/NRcPkyGIgzRsU2xajjwtJb0T6bw2rPY/JCwo6F2ulbMumyoiYzpOd1EQky3G7IFy0m3q2iX5Dbb61Y1PP3NVRD9J7xVX7TJicC/Sh9853YMYit7Uvi5/01h3laWTXvGMNoZRGErAQHtgcmbH5VOSjB24syCNtrkTDWn5yGlU7xIJTl1tNs4eCSRJK0XZ1KI2SVAGvoHsXBfP09Z/RzkjZaeXaO7JFEgBWqdRdd4TVQskKCQoXhUdSLiMRNej+xp9ebhTQvsDn17HUgkppawekRMGUl7UkXRK6HxPJFhZY9caK0dSqfW0yhpCYjGER+QZCmimCueBS0k6zDi4i3vEp/4ga+J707rGAZmmN9aQSiNJkDsnFdn9EFhj3C5zrw7dc7pofNNcnMkV6764+irMkU4/UE8+NWCRIydR1qBrwwBNPDYwfwVgBJWdVl1Zp1dD/+8KdcR0uHMXBAQPYdeNs/MUzv3zhJCItMozPCFQqVaslnj3hoiOD2As0cjqGd0U7ZtHkegJKN8e3x+kWgs32uJdNRuSlryaJzb70EBfCnfHc0rN4KluyBbGA4XYf3vfsP X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 3d451ac3-369d-4c24-538b-08d8ed22b960 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:46.9107 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: S8G0R1tsIuJcb2zvY8Zw/NPk/zo1hiB0GLVpATVn6zFBLLxrIXLsKmhN57pfoQjopv2D6Q9W1LqK1OPtcM6UeA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" If prefetch range to gpu with acutal location is another gpu, or GPU retry fault restore pages to migrate the range with acutal location is gpu, then migrate from one gpu to another gpu. Use system memory as bridge because sdma engine may not able to access another gpu vram, use sdma of source gpu to migrate to system memory, then use sdma of destination gpu to migrate from system memory to gpu. Print out gpuid or gpuidx in debug messages. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 47 +++++++++++++++- drivers/gpu/drm/amd/amdkfd/kfd_migrate.h | 4 +- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 69 +++++++++++++++++------- 3 files changed, 96 insertions(+), 24 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index cd89b38e3d9b..cf71b98303c2 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -462,8 +462,9 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange, * Return: * 0 - OK, otherwise error code */ -int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, - struct mm_struct *mm) +static int +svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, + struct mm_struct *mm) { unsigned long addr, start, end; struct vm_area_struct *vma; @@ -723,6 +724,48 @@ int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm) return r; } +/** + * svm_migrate_vram_to_vram - migrate svm range from device to device + * @prange: range structure + * @best_loc: the device to migrate to + * @mm: process mm, use current->mm if NULL + * + * Context: Process context, caller hold mmap read lock, svms lock, prange lock + * + * Return: + * 0 - OK, otherwise error code + */ +static int +svm_migrate_vram_to_vram(struct svm_range *prange, uint32_t best_loc, + struct mm_struct *mm) +{ + int r; + + /* + * TODO: for both devices with PCIe large bar or on same xgmi hive, skip + * system memory as migration bridge + */ + + pr_debug("from gpu 0x%x to gpu 0x%x\n", prange->actual_loc, best_loc); + + r = svm_migrate_vram_to_ram(prange, mm); + if (r) + return r; + + return svm_migrate_ram_to_vram(prange, best_loc, mm); +} + +int +svm_migrate_to_vram(struct svm_range *prange, uint32_t best_loc, + struct mm_struct *mm) +{ + if (!prange->actual_loc) + return svm_migrate_ram_to_vram(prange, best_loc, mm); + else + return svm_migrate_vram_to_vram(prange, best_loc, mm); + +} + /** * svm_migrate_to_ram - CPU page fault handler * @vmf: CPU vm fault vma, address diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h index 9949b55d3b6a..bc680619d135 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.h @@ -37,8 +37,8 @@ enum MIGRATION_COPY_DIR { FROM_VRAM_TO_RAM }; -int svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, - struct mm_struct *mm); +int svm_migrate_to_vram(struct svm_range *prange, uint32_t best_loc, + struct mm_struct *mm); int svm_migrate_vram_to_ram(struct svm_range *prange, struct mm_struct *mm); unsigned long svm_migrate_addr_to_pfn(struct amdgpu_device *adev, unsigned long addr); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index f4b4fea06ac9..6c46d43bf613 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -363,8 +363,11 @@ static void svm_range_bo_unref(struct svm_range_bo *svm_bo) kref_put(&svm_bo->kref, svm_range_bo_release); } -static bool svm_range_validate_svm_bo(struct svm_range *prange) +static bool +svm_range_validate_svm_bo(struct amdgpu_device *adev, struct svm_range *prange) { + struct amdgpu_device *bo_adev; + mutex_lock(&prange->lock); if (!prange->svm_bo) { mutex_unlock(&prange->lock); @@ -376,6 +379,22 @@ static bool svm_range_validate_svm_bo(struct svm_range *prange) return true; } if (svm_bo_ref_unless_zero(prange->svm_bo)) { + /* + * Migrate from GPU to GPU, remove range from source bo_adev + * svm_bo range list, and return false to allocate svm_bo from + * destination adev. + */ + bo_adev = amdgpu_ttm_adev(prange->svm_bo->bo->tbo.bdev); + if (bo_adev != adev) { + mutex_unlock(&prange->lock); + + spin_lock(&prange->svm_bo->list_lock); + list_del_init(&prange->svm_bo_list); + spin_unlock(&prange->svm_bo->list_lock); + + svm_range_bo_unref(prange->svm_bo); + return false; + } if (READ_ONCE(prange->svm_bo->evicting)) { struct dma_fence *f; struct svm_range_bo *svm_bo; @@ -449,7 +468,7 @@ svm_range_vram_node_new(struct amdgpu_device *adev, struct svm_range *prange, pr_debug("pasid: %x svms 0x%p [0x%lx 0x%lx]\n", p->pasid, prange->svms, prange->start, prange->last); - if (svm_range_validate_svm_bo(prange)) + if (svm_range_validate_svm_bo(adev, prange)) return 0; svm_bo = svm_range_bo_new(); @@ -1319,6 +1338,7 @@ int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) } for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { + pr_debug("mapping to gpu idx 0x%x\n", gpuidx); r = kfd_process_device_from_gpuidx(p, gpuidx, &dev); if (r) { pr_debug("failed to find device idx %d\n", gpuidx); @@ -2154,7 +2174,7 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, if (prange->actual_loc != best_loc) { if (best_loc) - r = svm_migrate_ram_to_vram(prange, best_loc, mm); + r = svm_migrate_to_vram(prange, best_loc, mm); else r = svm_migrate_vram_to_ram(prange, mm); if (r) { @@ -2361,6 +2381,11 @@ svm_range_best_prefetch_location(struct svm_range *prange) goto out; bo_adev = svm_range_get_adev_by_id(prange, best_loc); + if (!bo_adev) { + WARN_ONCE(1, "failed to get device by id 0x%x\n", best_loc); + best_loc = 0; + goto out; + } bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, MAX_GPU_INSTANCE); @@ -2423,29 +2448,33 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange, best_loc == prange->actual_loc) return 0; + /* + * Prefetch to GPU without host access flag, set actual_loc to gpu, then + * validate on gpu and map to gpus will be handled afterwards. + */ if (best_loc && !prange->actual_loc && - !(prange->flags & KFD_IOCTL_SVM_FLAG_HOST_ACCESS)) + !(prange->flags & KFD_IOCTL_SVM_FLAG_HOST_ACCESS)) { + prange->actual_loc = best_loc; return 0; + } - if (best_loc) { - if (!prange->actual_loc && !prange->pages_addr) { - pr_debug("host access and prefetch to gpu\n"); - r = svm_range_validate_ram(mm, prange); - if (r) { - pr_debug("failed %d to validate on ram\n", r); - return r; - } - } - - pr_debug("migrate from ram to vram\n"); - r = svm_migrate_ram_to_vram(prange, best_loc, mm); - } else { - pr_debug("migrate from vram to ram\n"); + if (!best_loc) { r = svm_migrate_vram_to_ram(prange, mm); + *migrated = !r; + return r; } - if (!r) - *migrated = true; + if (!prange->actual_loc && !prange->pages_addr) { + pr_debug("host access and prefetch to gpu\n"); + r = svm_range_validate_ram(mm, prange); + if (r) { + pr_debug("failed %d to validate on ram\n", r); + return r; + } + } + + r = svm_migrate_to_vram(prange, best_loc, mm); + *migrated = !r; return r; } From patchwork Mon Mar 22 10:58:49 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154553 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 353D7C433C1 for ; Mon, 22 Mar 2021 11:09:25 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id F32C66188B for ; Mon, 22 Mar 2021 11:09:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org F32C66188B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7372B6E3FC; Mon, 22 Mar 2021 11:08:37 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2058.outbound.protection.outlook.com [40.107.243.58]) by gabe.freedesktop.org (Postfix) with ESMTPS id 34BF06E342; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=j4bkjw6HIj+aFgkLjbD3T0cJYvrdCWTz6Jnzqm6Ptr/BNTzrTLnpQmSH2X1D6o6L3DorGJ/S4BXvQX0kNzd32S7Y3acbSy/fmdtKi0m3CThQQo65T9KjjxZwX3pTyAzR4MR1pv5BgB5CMd0wGaSlNE8MCkcCW8fUj+oLi+QrG5bFdT8r5gETC7410LMANIaBm0n0GA8iWj43KP96UOhkNBPFbtRmyXpPYDXrIKkbN6hbarE4dSQdonRc59keDhr8K9jBMYrWll0Vyd1zytVGuLUKegQtlQNiFdVCcbBx+FcXbTTjcHeWHfKmoADHq09PedM8iE+WXDk9lF0BuVjXZQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n2W3r3ZCMp2kGUfBRMwtw+bbMRnIEZgpjqnr05kcOA4=; b=oSyMzRE7DMY0c1H8rAKaAB48jf6RM1f8iqg9lVnLL0CyjxMCV1ShWMkkB+d1RsJB4fcedTu3fj9u3CYbWsrXM4IuUXI27zpa5UY9sabGQYfLcauFRVeskfEybOwLM4wDKSmMb1LinGMlGesAvuDGrhwqtiuWE0x5Zd7sI0CMgEMX+qUtzybg1gPNJ3nQa3bvzBCITT9YzK1cX8lQ3zLh/4F0z7q/Z0XF8YCXtWf5//9p1EGXOqAU/irLQwrOekx7fTp3mbvgcDEyEGVIACf+3A3KDJmxnEG+o00y0PT9bXTY9Rwq8b3XZ2UOaso4yr8Mkvnd37cWvstuHn51aPQ+DA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=n2W3r3ZCMp2kGUfBRMwtw+bbMRnIEZgpjqnr05kcOA4=; b=pvsrwhpKfX0V6SG9SuF/CZL9ekp856kbtGpmq4xZbFvNjDIbnlcu6+v9OJUMYZOT02l451AjGj2tUDpRS5e4NIgknS+6AqpzuzSstBKm5UsYqLq4QeSNvkGF7GqU8Zwqg1NdFVSDfzjN7GhHrcgLf7olWJLekQj28CoKgpjmBcA= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:17 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:17 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 33/44] drm/amdkfd: Add SVM API support capability bits Date: Mon, 22 Mar 2021 06:58:49 -0400 Message-Id: <20210322105900.14068-34-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:47 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 7a077ba7-91fa-40ed-76af-08d8ed22b9aa X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5516; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3oL+bN/vO/+xjA8bE5dkfel5dmqi8ELit+GYckavhM8hiX3GO8CKF3CrOJE5Okx1Dkj0U0VBKIPdX7MdoIsle6bkuzvwgXTixcXFkELBUPyKRxM2wI+zkVTlTZEyaEUXCMVCh3Wa/A3NCWDPozLx/mU6FEzyXSN12lIfL5PAMMHFyBwAToPqEP+At3q/CjASJB2CwcO11JVF82vll6iJWknaQurvESpqkhdcC9Qoj5kvTilTWVfwbVC8meLMzQ9anS/YvvfLNrRpwRQEWYFucdjBk6n/nw8cDFD7ry+FHMW5peT0+VCmFdfvConGSbLRFFDXSapz3i7UyjI/4eOUutlGDvYO0T0t5vyquBRuohsKP9zHOSv1BukmgqhMeKarz4oewcCc9wFNX7oQKVxgHK9DHjShUvwDWoBrU0WTUTrfGfwJ6U8Nf2QPCFHEXc2CzttcXGX3ZwUF7fakY6LPssbo2cL0ZyeHN5ewx/rTLHz3AQHcpT/BsPQSEVs26iHfKy5YNmHxR5I4rUiCWI8qaB/7KNarz+n+0ygM2oFjb1eznz6uKBT6AmwiT4tbibfH7oD9DA+SO07/NInMz2Sojm5fE/9hHZERcvU5P0lbyNxjOic6LYDSNO0zO5smhXG3gedytoFLeh+Lpn0sRSO3h+Cfej1I2qdhMk5Wmekcre4= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(4326008)(450100002)(1076003)(36756003)(38100700001)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: Z+Efho5K04dp3hrqUC87Dm+NUUU2WbnNaWkQWVsNTOaRdAM00Uz+mOR0KWpdoP60GKFafle9vucMMgr51rKFiL/sTbezTQMmKrnOsr3oS3Rp/mfqm6NoIkVeB1VWInQr562qno4y0V39wC+QumDHSuww257gJKVFq5wAtuUsAUsjXIY24hwe/GJMG59RqE42X/Urs10UzEVZzL2dsyJBLYgNud/cLCa4BiolOlnoro5VwahaiV4pr+0Vgeo1ch2xUsrL7kdpIfj2OnOIJKdtHQ6hRMlYOGxB1a4MLG1tz0DkhpVcBpNy7mn4uKE5DYx6AEXi0xaNs1Ju27EMui/Nf5nvV5QtYuGLkrsrnj+EI0U/jvvpxXTdBNFAR/FCMbImaa4xtm7MvLTNBpsfFShdLwi1rQnlQS8EivkUml9pjDnjnNeYfaCtMQ0pTMkXL6xr5x8P6khiRvs16PS38BFRArY2HrtYO4GcVDR2t17T2S7FrdW1tREhOdfgn7n4abU8LiQdEdnZqYedPt2O9lx1QdKyrIden6eJzIUwJJTJcBpGjuCdjpEU8utE+mF+7AwzcLe+nbBubD6p7uEHEXhrFH4YBOwwFjAdIBBC07rGAfnOfZFM+Oho09w9kQJuEnRvLymF5+e5wA5lfL5D+wvBVHNh+j8iKLbdmKJT+kGs0Gau7XoiPYcphuDQa7EE1ow5VjIP8cb+vqUUPbX8c/TF2f3PdQJMxtvF4b2qeFn8f+j72Y/uj2w4t7NNR5iPIjYALn73wYOfYXWN+PVCEp2xY6O12qvYjENnx6srk30bBXixUdXrrA9VcGOkhBF2wfWUFVb782+v/GgjbfU7+uNDRU7iD1vvkL+Dk+SPq/v3nCHUP8k4oiDMvgU+lOYQ0QhEncb1g1xz+MBnrf0INFiihAM3spQ6WGJHKluC0czxohf1ijjFot5TGXdd+4UzInRjDZXqdDchQsIDWEoByFXuLiX3ljEfv/Sd2enrmuIVoWMxB4/gAxbJtneZw5pfUDAaJtnx5o8ipR31F6QCH56gcMZc934sQS9aZImCKPvz4i8V8mwFxlp1Km0kMGp4DWHNEqJTpy7+APFrXsYiOOXEp7hAOZoHr9bO2AJuyCyjFvdvUsJvnZjcRXKKEQkuTJnZ19crkAgBqCITeDcqfF/+KT83hUv1Q3dkl2Fi/I21EauDqAAUPeExk6L4jq3lG6QpC5FkVqlEZI8MsbudjI3i/oUhdDSZOV7lJryRNtC7ZaKzNtLrsEW+Mum9MhJpkwcBxGEw85SpjegJDyZx9FDo9JlbIMUUCirjGWXZ3QpAp5/mqCUiOSh4xZfwjx/hFlEN X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 7a077ba7-91fa-40ed-76af-08d8ed22b9aa X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:47.4026 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: /fU334iWklLV5jfE91/HvGolSG2OETo9NW3VVQN0aAf2N2blBozHgR66GMwPV9/PjJUpkHsJTYb0gb166WI8vw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Philip Yang Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Philip Yang SVMAPISupported property added to HSA_CAPABILITY, the value match HSA_CAPABILITY defined in Thunk spec: SVMAPISupported: it will not be supported on older kernels that don't have HMM or on systems with GFXv8 or older GPUs without support for 48-bit virtual addresses. CoherentHostAccess property added to HSA_MEMORYPROPERTY, the value match HSA_MEMORYPROPERTY defined in Thunk spec: CoherentHostAccess: whether or not device memory can be coherently accessed by the host CPU. Signed-off-by: Philip Yang Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_topology.c | 6 ++++++ drivers/gpu/drm/amd/amdkfd/kfd_topology.h | 10 ++++++---- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c index cdef608db4f4..ab9fe854b4d8 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.c @@ -1419,6 +1419,12 @@ int kfd_topology_add_device(struct kfd_dev *gpu) dev->node_props.capability |= (adev->ras_features != 0) ? HSA_CAP_RASEVENTNOTIFY : 0; + /* SVM API and HMM page migration work together, device memory type + * is initalized to not 0 when page migration register device memory. + */ + if (adev->kfd.dev->pgmap.type != 0) + dev->node_props.capability |= HSA_CAP_SVMAPI_SUPPORTED; + kfd_debug_print_topology(); if (!res) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_topology.h b/drivers/gpu/drm/amd/amdkfd/kfd_topology.h index b8b68087bd7a..6bd6380b0ee0 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_topology.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_topology.h @@ -53,8 +53,9 @@ #define HSA_CAP_ASIC_REVISION_MASK 0x03c00000 #define HSA_CAP_ASIC_REVISION_SHIFT 22 #define HSA_CAP_SRAM_EDCSUPPORTED 0x04000000 +#define HSA_CAP_SVMAPI_SUPPORTED 0x08000000 -#define HSA_CAP_RESERVED 0xf80f8000 +#define HSA_CAP_RESERVED 0xf00f8000 struct kfd_node_properties { uint64_t hive_id; @@ -98,9 +99,10 @@ struct kfd_node_properties { #define HSA_MEM_HEAP_TYPE_GPU_LDS 4 #define HSA_MEM_HEAP_TYPE_GPU_SCRATCH 5 -#define HSA_MEM_FLAGS_HOT_PLUGGABLE 0x00000001 -#define HSA_MEM_FLAGS_NON_VOLATILE 0x00000002 -#define HSA_MEM_FLAGS_RESERVED 0xfffffffc +#define HSA_MEM_FLAGS_HOT_PLUGGABLE 0x00000001 +#define HSA_MEM_FLAGS_NON_VOLATILE 0x00000002 +#define HSA_MEM_FLAGS_COHERENTHOSTACCESS 0x00000004 +#define HSA_MEM_FLAGS_RESERVED 0xfffffff8 struct kfd_mem_properties { struct list_head list; From patchwork Mon Mar 22 10:58:50 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154559 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38B3EC433DB for ; Mon, 22 Mar 2021 11:09:29 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 057DE6188B for ; Mon, 22 Mar 2021 11:09:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 057DE6188B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 549116E42E; Mon, 22 Mar 2021 11:08:40 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2058.outbound.protection.outlook.com [40.107.243.58]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6055C6E3F2; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=m9x7E9jYDakQul7ljTTDjf6xsCiwSC/vBdKuKO2WeVaVmzjBfRwYefzSqJQLV7B3h5MVqx6sLn9RrAavpNKmhbCZ2eR3QjHdcq+3z3/rRMBFtxdGqxKT2ZWjwrw++E0G3d7zi7vltJ8n91uTOBhdCsWYqqk4iYXOioRXsM3MkDElxfz0UACZ+qfIzx6cPav/xUet3t5O3dJBvBA7y85HmkYA3R6im5AvuvpNcB2lFg26U/lZ1ajzCit4WZTQZ8i0tYEyknQ+gqro1KuxVTRZTBAJoRZ1pQRXa+DObiws1L3Sxo5fiXAuus38i/xdXBAS2TDHRRc+37vkFBnD9Axi/A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WJ3CIGihgakSrwmqXAIBQ6hwH2KBmZUlO61ZpPl7vOY=; b=IwOrl2Vh6f1PsNlak52s6tnqTu2m5uJGgscjWv5j40rgNQpWcC7t5fRM5rs362CEghe8b/hcdktp2DhdX/K/nM3mkrD0pkfDXIMOljvfR+4H55Kf2e2CtlvIJhH+OaOK3H4yIkDSdisiu5dOIbWv4X3HnXcBB2FMoWExePb6lQOA8Ihq1SbEdc3mpIcM2ViRP1W8+AVY73B7RYuw7tPy6M6y5Pp5Yw+a9Ty5/r+iNgbEUGLqTPOVEBItvwvB+bIA/0OjRJpYFVsPsktBdRfq7fv77oQeWapkdTG8hdj0VWcdbv4tTMqDWTIYBw6JICYGpwKAUJqwmpWx13Vge+slXQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=WJ3CIGihgakSrwmqXAIBQ6hwH2KBmZUlO61ZpPl7vOY=; b=aPjer/NuzAGFzwKj0M0mwaWIRnZ4hSwb2wknB0wSK1yKrYLH09DpkdFc/99e3ttBBE+LaSMdsW1UDYUD8US3whRhp/7LppY7eFhzWZ4u4VRLKyZ3YjyXvd4cDzj6akVakZInk6cFqdfDAU7wCJkxxbKmJdFvMxODmhp5RpEQh1s= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:17 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:17 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 34/44] drm/amdkfd: Fix dma unmapping Date: Mon, 22 Mar 2021 06:58:50 -0400 Message-Id: <20210322105900.14068-35-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:47 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 06e571af-ff6c-4bf3-bd48-08d8ed22b9ef X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:7691; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bUjqJN8Wz/0wH38tLx73FGka3Khe9iExVfSvfQmsR4RprXqM5cZjIlV3Z0+DzuCHZ2cJ1LDR9XpHMoagFJNjKM50O2V5dR2YtgN0hZMDLIcCZzZKW/cRBgbxJbed8j5hMsO/JuHUlh49CwmRtEYS5UKCGMFeILUuwhg1RNxZs0rDEQq86tVKeY9YLO32ha1/LbUnFtG6oqVo+hIIhL9cQTxRWqlOJnNBmpRW31CZXq0y9mZg19PxglVF/PgQ5Kr7cmTkWVU3lC0C0LAl/KrxQDdsb65WNtK+IN5q9t0ytEdzPUYg3D/biNPucvy63FrLC+fE0jha77oZQP2Ge3HUB8mUzlZ+p9hZHNl64KtcODXJbMDfwNrLcJrM0SRkNPzAS9Rpc65x+qLZS86dnBcOlarqn0fBXlJ5nScl77Gb1OPDjUeICRHi23T2Ej2IUOizUYXAf2zLLBQPnLg7hkusypq3EeVSTvnUThXdhHmgyIp/nUAqlyfmbzWOLKUSJVB4zUQke07i7Q7vwjcD6tXvyzvKG3mIsQJ0GQXfZBVxxtHgMu4B71XdHL52ot6AAsJ20baanVRsl6+lH1dKRIxVxYe92YRcrZQ6IAmS7X/pGbOSulownHB935tdNdaMtxMjuhFRWf43xzQgfyS4tPeElf1LARS+CJPbFCkNeZCxz+k= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(450100002)(1076003)(36756003)(38100700001)(6666004)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(8936002)(498600001)(2616005)(956004)(6486002)(2906002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: Of1Vb7faJ3YYCO0gE+K9iesDrBtRhD1kXe2zr14nnyso62iW34ITI7RG5V+IwSsX5f5HUdB9oCN3PtaT+i/OybmcTCE7NATGQfoW2yB9+m60kOWiR14C2YpjvVghO0/tCy5OPdsU/g8pNpNSiooZYVHcLp8RwZpTxgYP9XyRSZJuSq64RopyHWuD7eZqi92Q3XfqaBeLj+2LjG5T2zTLIyw+AhDIqX7e4vb/iCBFg3wvz2DL2hrJx8WRwhTlwwBzlVghdfADl1nBxwLKt72caimEPj7y2fn411UR4R6Bn41oLa0J8G+Kc6RdbI3Vp1nKZNyLxdckQzjROrEZpnUpJI4BEuQX6vA1nzGczBQCST3ZmmZZ7t3yL5bJmjrIHVDpg+4L6bjK4SlxHYgwZ2vq1hZz74H+3fkMyv0Zi+CkA2bEIyIUkC2d66AdcceFhYJYWgCRazTiXVhfJjeX6QiBjRuuKbj2r65R1Ip5ymucMc1T6gZOoyWOC+gLy2b5VUW0ktDSEumk+gJlsygigeVMaS3qt3lRy5sZ33IKvTzJvnZvM1TnP7E/VMlQxl58meSLVlKC3/E4VTbgc7A+AA6Zgyr1TMIUPpLj2TQ4gMvhmeagAom63Onf5F+WzHRS5va0ymrBIArMCrxnQ1ctk5Qx0y/sSpC8Pr/Eo9ZErfTUBpGUGfKGiajgsOH1EP6WwOi/IMuciPidkR9YmRZJsQ6Gtz3TYNUhQoC+vPlr1m+asuG+lzWSJXsCctHp7VDLKbFhHLbTOAM6tgV7suKlrOmhsLp60jWT07iuxD89ePLfwabFZpDiTKaj7UhDZHs1Xyp7zBcqUAiu5zdSlLDXrHYeku1/izBOm0lNQVd3eayNIKpjfmbOx3ko3o8DWyAisrBjhvXoLIMiALob38C9KFRlZ4Nl4jP0YS6vRjyidn3RoZTT7ZqZuOzCg4UQ523OmYdyHjkIi9xbP2VDnClkLFtx0gyFZtLojAEM/RIMGbVqgfucOyow7vyDK7KhMaefO16B1gZMPOllBzab+w7swA2GfHxGl5hLLYjDnlsxryWmI0k1XKhmYdGDlR/FrLFmlcxH2nCTLPhfAHubVTfRQXQCYWZPfKDdIAfvgYbbtgHefdb882AGR/jFZ3g5AX6VsfbXL27mM6CKjUgZ7+ODXEYZVj74S27TSXtp4vUv6JhzSNK+KmU0+P9apiXmRXKot7CkFmOXUGkUwyH9Y6ATMuSf37XMhXn/9A+hh6YU5337oBtV5z0fuJWpfKs8/Mu9kmJ0RJB1XCLy5cPNfK7L1IZCf5uGlnYo8ampp/9oT+j5ivz7djbJ7i1QhCbuQglZ9xH2 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 06e571af-ff6c-4bf3-bd48-08d8ed22b9ef X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:47.8396 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nXxXMczbhs0q76qkgjNKgE9idadyivIZBzPTJ9T9PVzNV0+ZsRARTijsXDN9FgLEMaEJk/p+paWifWXrnciEgg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Don't dma_unmap in unmap_from_gpu. The dma_addr arrays are protected by the migrate_mutex, which we cannot hold when unmapping in MMU notifiers. Instead dma_unmap and free dma_addr arrays whenever the pages_array is invalidated: when migrating to VRAM and when re-validating RAM. Freeing dma_addr arrays in svm_migrate_vma_to_vram fixes a bug where the stale system memory pages were mapped instead of VRAM after a migration. When freeing dma_addr arrays, ignore the access bitmasks. Those masks may have changed since the dma_addr arrays were allocated and mapped. Change-Id: I01a6121c4c9908c1da4f303e87dcafd509fabc86 Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 1 + drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 16 +++------------- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 1 + 3 files changed, 5 insertions(+), 13 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index cf71b98303c2..da2ff655812e 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -444,6 +444,7 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange, prange->pages_addr = NULL; svm_range_dma_unmap(adev->dev, scratch, 0, npages); + svm_range_free_dma_mappings(prange); out_free: kvfree(buf); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 6c46d43bf613..3eea8f87724d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -173,9 +173,8 @@ void svm_range_dma_unmap(struct device *dev, dma_addr_t *dma_addr, } } -static void svm_range_free_dma_mappings(struct svm_range *prange) +void svm_range_free_dma_mappings(struct svm_range *prange) { - DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE); struct kfd_dev *kfd_dev; dma_addr_t *dma_addr; struct device *dev; @@ -184,13 +183,8 @@ static void svm_range_free_dma_mappings(struct svm_range *prange) int r; p = container_of(prange->svms, struct kfd_process, svms); - if (p->xnack_enabled) - bitmap_copy(bitmap, prange->bitmap_aip, MAX_GPU_INSTANCE); - else - bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, - MAX_GPU_INSTANCE); - for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { + for (gpuidx = 0; gpuidx < MAX_GPU_INSTANCE; gpuidx++) { dma_addr = prange->dma_addr[gpuidx]; if (!dma_addr) continue; @@ -292,6 +286,7 @@ svm_range_validate_ram(struct mm_struct *mm, struct svm_range *prange) } kvfree(prange->pages_addr); + svm_range_free_dma_mappings(prange); prange->pages_addr = prange->hmm_range->hmm_pfns; prange->hmm_range->hmm_pfns = NULL; @@ -1192,11 +1187,6 @@ svm_range_unmap_from_gpus(struct svm_range *prange, unsigned long start, if (r) break; } - - svm_range_dma_unmap(adev->dev, prange->dma_addr[gpuidx], - start - prange->start, - last - start + 1); - amdgpu_amdkfd_flush_gpu_tlb_pasid((struct kgd_dev *)adev, p->pasid); } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index fea9c63b5f95..b2ab920ab884 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -175,5 +175,6 @@ void svm_range_add_list_work(struct svm_range_list *svms, void schedule_deferred_list_work(struct svm_range_list *svms); void svm_range_dma_unmap(struct device *dev, dma_addr_t *dma_addr, unsigned long offset, unsigned long npages); +void svm_range_free_dma_mappings(struct svm_range *prange); #endif /* KFD_SVM_H_ */ From patchwork Mon Mar 22 10:58:51 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154551 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B75BEC433C1 for ; Mon, 22 Mar 2021 11:09:20 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 861556191F for ; Mon, 22 Mar 2021 11:09:20 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 861556191F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D9F766E406; Mon, 22 Mar 2021 11:08:34 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2045.outbound.protection.outlook.com [40.107.243.45]) by gabe.freedesktop.org (Postfix) with ESMTPS id 288CC6E2F2; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Y5Zzk1tR7G+OAtAdw+amsqQvy8jxsoNT8mKtO+x7z+1mvZzx5k+5G2bQMHT4YdcPDBDlCQhW5ALNUNuUdjzGqVwzoFA7Zg+kgtZ5ZLLrCwAn/fOlUmNa8rKPMFsdpGtddaN9WXYq2iL24bKFmlqiRWPg9IxnSQZvbelizqtu0nmKjaOX92/X83BHLW80pICD4VeE/nwHfgK1YmX6pTtcSfbXSg0oVQ9DBef7fFPYWeoGsxm+lXrNSOB5mA51EeVf4eICnpiKCsklv7ZmjD8b47XDKvs3z4VpKJFKFUaNTwwhvr8+j8wpkdtWdcM4q3lF3sFuWUhikhv3L8SSL6xbww== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ezA62SBigG5ssBduvpwkfGnOrOzYvInAKXNP97VjJdQ=; b=DpWOMnYeVRJFceSCZq3ohuSxc3OpZILTn9RZ4TRguzwb159ry89NZRNO0zRuGNVQIJa5xkPju6icFQdlM1De7v6BLL+swZ13D6QGvLl8FsjiJNhEURXKXSqTDDWjj8JIsHrT9LOb7zzbk2A6CAD0RRkVKxPuKbWViHSYrJRMz+G4ApGENU7Uvusr0jWcgv+UET2Vswmgs/gQ13pVd5jZckHjRZ65QhADnyGq9o3uta9QhtLlpa/2zfZPz1FXaN/t+lrS//IvTkEutl25mW+kFMwwY+3i9m5w3uwWa+HzrSAR2w+Ef4KtpERpqhqBMPvE8VL5M4lhtvk6y2MwQ14TSg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ezA62SBigG5ssBduvpwkfGnOrOzYvInAKXNP97VjJdQ=; b=X69AdaKzpYfLNczMQrZn8Y2UJ6EedGVIq2UTGkbHtN+l6nCWNMV/36mXaSu+uRGqM/pFsffw2+/eSNEmA6uTYdD167w1ACvpYc/fUCxSq1AJaIMWwV64aSnRQ6pLKyG1KyOZNay/rE2/ILyA8uSo3DEsmLA1KiYApSELmV64wAg= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:19 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:18 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 35/44] drm/amdkfd: Call mutex_destroy Date: Mon, 22 Mar 2021 06:58:51 -0400 Message-Id: <20210322105900.14068-36-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:48 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: bf84f3f4-5bb5-4f23-2dd1-08d8ed22ba2e X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:66; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 0nIRQMEp0zxgdF9TnZ8Af/erlBCeDmaZf8CO62ChY9LZGl5opMJFAqHkTXkHP8kpI5urlB3TVD/wSt7jLBn+Yo3ziqPx6C5VGGiyR8K6IRcRGlwG5BR0ajm+yiBKXt+UP1I9Z+0V2yjegD1Yv6uOAvpfdRw5lswOtLcdhtIEqu8fDX72qGXAZCRBdLDTHOO2F/YII1uPkP1m7yYEy91J+pZ5S8xvX7JCX+HI17S6AeOR/xGMXmsXIvDqJspCoOsnhQzOFfTLsUq5L/dUve46xl/r5ZppC5UndLgZw6CchP36A8VsFpgB9ERi0bsbqEyVx4rG+JAZ0Anf2UIUxmJBN6h0OZONdSqsKo6GGIz4rhBJ+nkc5V8gkWx7fOUaXeZcKyo0afMcuekvpBzJzk1PS88HqqGRU/IYF9VTXFE8rqKdZSHlsBHQJ3tIwz5ZFZ13Ds5wUR1qkssci8V2BxmX6cNyRvUCbbF8+KJ+dr7sYcL7cvByn78b1wXeSbZIz7DZHmstLhUsJEwExI5TRKSYUxtsFeENovaugotBbb/XkQ6rjq9sfpUjycUEDETqIsqyg5V3RKWH3/aMVgD1b8YL1EwQgWr9DR14s1GVjR53aGWuH8eTFUkMl+q1vQgnMWZqoIu4lHs1tWODqYbP9WuVJiWBlV+P6blG3qylE4L6eJs= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(450100002)(1076003)(36756003)(4744005)(38100700001)(6666004)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: CGuswZyE6deIsW3cOnWbkDA7QdvSfF0valyt/S1QJ4g2Xc5GPwPGsYdzSD8gkWjqBRfqBJo111h8nNMQ8rDyq0ed6rtUiFf0V43kTUBmljBB6B1A1A0B0MFJNBa08HSDiAE+qT+beIcv5XC0NGSozailMC26UGJdxqaTVKRl2OM7oxF1AU78Ds658OjIay6V2aF0xfSB1PnDT5RljW51G8iuLo68Eq58bCDwVjfgD1mdTLOAlnDSRrDxYx29t39IaTTzi5FWBP0cd5MF7370S6IxBeJTXoqOtPzgH1IU0u03fW5HrIv3l8AT4WvYKRVtx8xRioatSzxTMuaPzMIsGz44ipBCMFpemG+wOvKUy4QC+cjAR1Xfqxbo16rqHDe/EyBkYLr+N+xz2zH8IaPhpD25RufztvgUFi1xNjXZruB6PEWlDjT1UMh3Rmq/QW5IWlKjaOzLOOQR10TXalUdzYqJVxn+V6YGBx2eHkECQJ8VIU46d/X6Z0kEsCQXnL26AkcYXXwllQH+wptbpdhaT0ae1m57xLtkMMWTLdcEZ3R7TIUdgNYh9d6JtUw+UWH178Z7/2ThyItUNGZFafSUWM+zZkebQ05yHMi36SWnIHgteTuvFrsSZVfUTh2nSRCUYp/JZLTxyRbzRVPLTGimIZd150WgBcMQK8x6IYPk1DNFHABdD9+XKPnEMG7o7DOad3XJHD4w72POqbCKz+CbxQn1ef165wV8v3nSBAPXpefOzlSVCFxoCi94hR3irKR+tssnBtt1uXtCPFSfDvbN7RptkrDAEQcqNemUZEFj48/u8z/jHGyoATS5gJi/6qVDCeRO73ezQzU0UvFNh5bb0nLg0Q6x5xdzBJCManIVmvUG2Uyd07ZBKvpNQoTAqmNX5P1Ob3nkF15qAhQtGti3hU3HtziMrovewFN8GGPYmpx85DoYH/eJiJxn2nv8YE7bzKa8/3DFKHKiDvoyni+vUWODgGg7OKstGTfUEdRlNkfnKfGwsgafFPOlIJSVatyePJ/kQnoRru+7QD5OvrJ8EWA1K9+WU8mRKSrzme9woi4S/V7fa7go8Q/Hj54Xumss2l8nX+wH1UzGjnWY9OTP+5olndRYb18P+SrZAIWIxBnP+xfv+0ZOHRv9Jf+1WHQBZ9BF+lbff8N8FYsRDi8MHfhHVvlacwyy06OVpUhTczLzSFZkKMPYM3NGFiDoCvew9LSgvJmGW+Ngj0X59ouZ9C6bHynGhf1aZ7wCiDYafcu7B7/OhF9twyVpoclPIeLQpHJk7DWUT226UE7ISwyZ1uqyYKni6FGoWlWgd/K1GQTm7gewohuNHVYn+U5gvo4E X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: bf84f3f4-5bb5-4f23-2dd1-08d8ed22ba2e X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:48.3066 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: hw6puH3du/iQ+S7YIhwPGnTpFv9E5/QMyAGgQeTI8EG83NjRg6quQqsJvUv9mxk9mAB20LgmiyMwpnNFMSwLPA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Destroy SVM-related mutexes correctly. Change-Id: I85da30b1b0dce72433e6d3b507cb0b55b83b433c Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 3eea8f87724d..0fbc037b06e3 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -209,6 +209,8 @@ static void svm_range_free(struct svm_range *prange) svm_range_vram_node_free(prange); svm_range_free_dma_mappings(prange); kvfree(prange->pages_addr); + mutex_destroy(&prange->lock); + mutex_destroy(&prange->migrate_mutex); kfree(prange); } @@ -2220,6 +2222,8 @@ void svm_range_list_fini(struct kfd_process *p) list_for_each_entry_safe(prange, next, &p->svms.list, list) svm_range_free(prange); + mutex_destroy(&p->svms.lock); + pr_debug("pasid 0x%x svms 0x%p done\n", p->pasid, &p->svms); } From patchwork Mon Mar 22 10:58:52 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154537 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 085A6C433C1 for ; Mon, 22 Mar 2021 11:09:06 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C4B6161919 for ; Mon, 22 Mar 2021 11:09:05 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C4B6161919 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id ECCBB6E413; Mon, 22 Mar 2021 11:08:31 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2045.outbound.protection.outlook.com [40.107.243.45]) by gabe.freedesktop.org (Postfix) with ESMTPS id 75C6E6E11F; Mon, 22 Mar 2021 11:08:26 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=iU3WA0U8ehH+3kkU2kEwFkUsQxuztpkXTgWVb4L3wCCU949EoK0ztaD/aSMiDteX1w64mHxVg/Qz9dYe4ueTDp4FgfQMxXjrObusp9FOQe44j0bIX4LEUDf9326cctxn23xn5sYFRcTfnIBqAuHOx+1Owhb7KKk66TnCOAV3sRBkadgzfhbrUBmkB069NpsyPjS+s4kJ+9VEhLuWOZUdLoicU+g8qQqo/wsQmkRx6LechIzTUpESRVuzBYrp4G7BSJ7l8gJ2zETiT1bTNy5Z8Al/PHZM2v8AJYAl28WaFvoPbZNJ4t7bTH438zGwwiDm2PUvM5p54z3lkrB/NoXcaA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gMGDertcs7Fvzjb8Z2jqvjc2mhUdSVMOse2sccDSUFQ=; b=JU6qfg/iNbOf4bj9Y84Xz2Jvdmw7kUaZH/MDFpr0CyTN/i00OdEyGYF4VOD3cEt2fDNwo3YvbALDYAW3KGlGsavrGbVCqGaicUPeFGkD3Rflm5tGEUnjkcWie3rRHafzvvlzg7EbAFioZA3ldnU2Oiq4zhXKUmZXi4PBQ9MZNnNOdvP7BHtt+04ihX39eYGOwPUt40ln55UegiddsVdGyHIFhTdFSR8YtltfSS9OcDA2Z4MBcmkMV3g95WeMb5ySTzaidGB/xPQnEdUiLQo3hvFzSjGPiswWPLajaaYbixPfzgNVz4dN68F6Dbg8q4XBRRZWfWALRtrVPRc9Ktv2AQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=gMGDertcs7Fvzjb8Z2jqvjc2mhUdSVMOse2sccDSUFQ=; b=Ou+XCPPbILxZCBRm0W/t6y/hTzlHVAl0bm0JBggfv5vmBYZXFNIJwT8CYiTsS76RF4Id7w2X1vjPYTpJH1Ub33jaKrmH0e3CACq+8AGv49zNI0x9i6bKpcMwk9XKFEKO/amhcoEzdI5mgVtL6lNdVt39CZrXV8SFoyC0/xXw4i4= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:19 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:19 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 36/44] drm/amdkfd: Fix spurious restore failures Date: Mon, 22 Mar 2021 06:58:52 -0400 Message-Id: <20210322105900.14068-37-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:48 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: b8bb80e8-a8dd-4f46-b867-08d8ed22ba70 X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5797; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: WbEJ3QipaMRdVZpVkd3PYkaZ8DqGgLGujnZwzzMT4uSY5wc9r2eCIk6hj0Hp9nvlARHwqtekTfGkKAVkpbUxdDe41zVPfuXlTUgMoO4295D1PYZSfPJEhRrxiO/vl/paBsckZ6u6wCDnrcmN28kpJj3F4ozuX0jRHHJJAe4dULqjFXEmbjPrvb6iolEK1uu/6bsNRyiLewzW1rKxMyMbb1POzRdrLrt3snkDsHlFgvZn6rTmbS7RHDGZokrhWUfMSjfuX9FKxW45C84okcAEp5iuXJ7+0mOJV4qROiODKptLX+EOSaX3JtTYGoHGOWug8cAQB/fuTKy/t6tUyXTIcL/d6dhBouYYHOXPGO2eHMcJqiIVrTjMgqQpSVII1M9PLTHI433H/J5nTTyugG84mGYd3nvmdIMWP2x3/AJI0JL4adp29UpR3scD2sXchnj8+PRdBKLR5Ss1Tbc9h9u+vy65kwBlLb9DmQYsAJyI7AufXTz7iQcZm4FL1lbw+ibMQBGU0dbVcEof7K64+7NYR5FEsohZD1EQFJ41Y8JqvUTgOBHmENQIC/5E/o8TGqk7BR9S764zkC4oOSXx361EIox5Lr0K8lLq3VVt2VqOEXIhF3jBNHOgxLSvu+z3trGCLqkmoupsSPCbumalzPIyetOPrYSxkab3Yqs2tbMvoOE= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(450100002)(1076003)(36756003)(4744005)(38100700001)(6666004)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: C7N1o/jsodyAbNqBRbVejvu/cQVXCsF4uMugY2XSEObqufIa5Jg1M2alEsLQhFqWuRyXovskWSLCaffXI7B6uhUKQDCHmqa2TkRMT38x/GBYYewAUR4hOpYJoC0pGRshLXcl/msSd7vzdKX6SbBF0mbHbWcghFydzR4/qYa1gYip8H6rVsJePk3I/DeSqywb4LckeYtKzOMG36ZTsfgbJMGG5inbN87nPpXroMmIoIXuzfQoNC1aqMDWJkYJleCu7g0Yk4ma4YlUOuiBCpWe9i7uRmvWA1nTcvivQfZuW8NYpJ9LlX9vReG1kiajGXVfNADycp47haJ0nq3+XrtR2u0cHtScn11V7l5nTs8+jnySi6XPdUrbTe5c1lKGkZZUl6N9x+vmWXUzaY2Ijc/89AY8roanIa8UEykf5oqWwp/Skua0ZML5ppvHIVsrpd4fQ3EFg7QCo6k6DI4En1BISWHh947a2H57b3Vb9vOOAOk/PZ0JoJWEu17lIOqODC8C9DOFyr8YBdtxorB9jOPdCRPijRCr17Lqb5wKYsL2SzDLZM9xyFGmKLOTQ6ep0CuWvKzB5DtJzVl+tpJdLqayGsLJ5UW9W1Fg+XKabADBUvD9cNBj8Vrudy9M8Gp6r5CHJLesUMo4NdU5j2eW5f8K/nfCJRCyn9iYzjktaRbWEXPS/3N9bLWi5UZa6mD0nwxBkYBpTDeWKJpDlnDcMBOtQMfJW+jP14wT2FoD9KHuDRZ96Fo+6r9TXH6JZ8+PrYyjhXaAuTIB3YR5ROVhypL/KUFqg6RnnTEh5tZxVjixOGFAEPMqQBnrBiJFBtmpCx7zXyVD/F52WzxrITtc/ldIQGJ7iXiotUwzTtb/dRs3yKlBEEtDCg6g8hW1JNb2qT8Q+FqInDwCHj2F8XzXI7BCuGT8SGpLaIiV9i1YdCmzhTLbZwxBlRXX+4sVWjRGXDajdr9XCrmz5lp7sbv3EMLQ/ygIfBw4JTB+BmZP5XuRXHfI2lP8GAJe+uD7Sdrrf4vF5xh6DtNNrUa2cTaqTpM+m0D40seBU5rNQXg37CU/40ZNffUnaKlCaApqinWr+I0euYF+SBqKTCEirqv9supqKwUYu5YqDteXtz4BvD8WPV/77QAjlODpxgq+wrhrVCgOgYCs6YFGO4t86Lu8HJ6jpwyO8/gNHa2pN4SHUI/5iZOjczTzD4ZVGLY7/UDtNNqynI5PAUFURQVrrl1UovbVdAhmqZa1/2Jz8rjFQw2sUHm1vEYmCiiVplnAHoC/32hbkLnnp1cXYx4H5LsrL/FDt9WPTcqLZnxackoPYJsjLWkW7uGQN7gzQBLxjPAiQJnJ X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: b8bb80e8-a8dd-4f46-b867-08d8ed22ba70 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:48.8313 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: lxWpzpLqdiCYOPB41wIDdY9v2Ei/w4iomLSG0pM64ZN7kYkuhQi7tA47zylM5Rf4Eaqx3W3s4Bf2FemcJrcqPg== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Restore can appear to fail if the svms->evicted counter changes before the function can acquire the necessary locks. Re-read the counter after acquiring the lock to minimize the chances of having to reschedule the worker. Change-Id: I236b912bddf106583be264abde2f6bd1a5d5a083 Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 0fbc037b06e3..49aca4664411 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1429,6 +1429,8 @@ static void svm_range_restore_work(struct work_struct *work) svm_range_list_lock_and_flush_work(svms, mm); mutex_lock(&svms->lock); + evicted_ranges = atomic_read(&svms->evicted_ranges); + list_for_each_entry(prange, &svms->list, list) { invalid = atomic_read(&prange->invalid); if (!invalid) From patchwork Mon Mar 22 10:58:53 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154535 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 95192C433DB for ; Mon, 22 Mar 2021 11:09:04 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4B89161919 for ; Mon, 22 Mar 2021 11:09:04 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4B89161919 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id ECCDD6E416; Mon, 22 Mar 2021 11:08:31 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2045.outbound.protection.outlook.com [40.107.243.45]) by gabe.freedesktop.org (Postfix) with ESMTPS id EF9C86E239; Mon, 22 Mar 2021 11:08:26 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ncSTnM87+PbktwphdecS1YO1rh3Yxnp9WDAVeef9utxq3hVFq6yyj83hKuLeuvtlZxJXymSiGHIFUe6iXA0OlaBQSMy4eyH0VVTPK9iNU5nirpdVQfiCF+3BcahMIRv9pUExY/L0g+F+L+Wp7NkmJFbpT9tfXAGh4iHQ4ouwsSdYT4FmmrdwbpFVALNfCJlI0EsZWD4XYghPAvskww23YaIk9cfpO+GSqzIMDjW4IpFn9TrZVz45R2+xZHCOip2aAItN8z6PZ7YjKvdHU2JrkSj7J9qjlq01zuzPR5eNmc5GXmvjGLxNxr6NrOCpHZgdLityTxYvCY4+ORu1LQNdjg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=73WwPGFnb6tZhN5DiEHDt8iZDiauT7Uyyo0XfI2bg8U=; b=OW4yuAT3SOiA88LTN9TRwLqXStYhzuWvuPwYHpolhHDBkuSa4R9rnx0IWdRYfRSVwRGrMLpw3OVdSXcdm5OayCSFwsH/Ow75bQwPsSms1AW9n5E9363RNLnOE/P5jtjxNSD4bom+dHrKF7J6aqccQ6DNaR44YUXB9o7mDZae0fDIHJxXTaQybtRaomknasa/EYtr5hBmChpaw2gHV2xoEgsnsKLHYRu4maFOt06eq26EOo/iWK7Edy9XhV4YCMTZdx0xGjCu+Yq5mwj+u43nkvNi4XGJ9QahxnM/kQtDSbeVk1hOq0/JeW1y5iIHGqhnGH1pRImzT5i6t0bOsl2dMw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=73WwPGFnb6tZhN5DiEHDt8iZDiauT7Uyyo0XfI2bg8U=; b=U8LJfkw/4UCGfufi9Zbxg6tXRsfAea6nVf2/1GDiViWnOyZB9VlL2H9kIzIX11flUAqMAxBb/C0KkreOWKHfdxxTFoFNs/ckwGMLCn1jfMNry447gaKtpYQBLQqWThwXadvmglHhuJ+Ebt7xe28y7N6rzL3Zk1SxA8MzEptdva4= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:20 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:19 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 37/44] drm/amdkfd: Fix svm_bo_list locking in eviction worker Date: Mon, 22 Mar 2021 06:58:53 -0400 Message-Id: <20210322105900.14068-38-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:49 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: bcf6a082-16ce-4a33-8e2b-08d8ed22bac6 X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4125; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 3bfZl8k6eHWmKUktqLbWqAdLj9rKpDkBj7Jn+nLxA4bXR2YYF8h6XnRuEDkFSOa4pmqGf0qWWEJ/Kj0teK4Uoc9AgKxkysI+62xl3AmRF6L1bBe7msk8TuWeS4vAGrMqTUD/0Ipzt42b/5Ox5rQzjttc4QIraJpClFNBLcc71jK7NMPElqkWB0vUJKd5fwW2tPJuhbVngDwjEjd6YX0tz9obI6tWwD1v5/quikTrvFvyCRJHXoVFmpI7jH49RIR+Gw+2IVPUCOLazciiXk50j/ffIa4c+Szkz/tUGTGKboE3HNJry+Gwf3Nu6JNtTBx8kaaY2KAg/mJdrlUGBzUBgE/i7ak8gF3qYPTrVFp4MV0UWrbBEfVoWO23vMq95Z1p+PLLanIlUmnRZM5WYazNiXBGDsWeuhuX8iiVqS5zrBRG1cW1tZOJxstzbHxlxB3fwL5Dd474JfRztGJQT5sHnyUnY5phCm18aH4xfkWHZhZ3b1xWODmRTjWu5KgjPXTi0IavAe3IXBLIPbQ+Yaq36LrHma/gC2iqPd0L2asiaZVxs4l3MUZr1h3L5i1vsXezW/rX+2DBCoCrC0XDm32OLdZvfBcmt+/YOOm3JybuCoWk0Usa1hmooUsk4mfpxKruV/mv8sgfcQNzSHl7mJ1YJQs81uqHrqZwYP1TZMaso0I= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(450100002)(1076003)(36756003)(38100700001)(6666004)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: nNO8T2Vb3mwrqcRDhyGsQFC4cuy2axJuz9Xn4AzgtGV+AfBb2sBdgNEwbf1T5Q0ioTr6U/gSD8z8WTQSewxWm2kFcLOhjI+bQTJ9ySTnVhqt1nc3sJynYu3ASYXii676UVoITEz7771O+FuOf+JYktlzEt5tHko+meGx2Oi6Fv+of7y3FIaHkRVQvYFyV7/NM/FBydg+j4AGf9MgOhy0VD0nbVRSJWpgV3yn28OfnS3OJazAvc2HalMxGpQ1xfZ2HmlJ+oKM56+DJIW4KW3Q6cQdEwZ+6AhvgSq4GJivPSuH9jds9L6M8zmfDfirJOqv6rgIlrBqSZ0UMj8BqLPJGQYhBfC3cN71zjO7aHt6CfvV2rbF9Zt5AsIYB0u7cAH/2xke9jlkx97n8LOLvKnoptDYjKqQUJJVk6aZ2xeFRrXT6HJ/VqbHAYdcXQmD2v3uRT6F2f221+ekUTYNqPqtGsSX+wSc3pqBZVF5GaaOSzgUCtnaKRDfAvDZdpAyMbJyFJRCFW1eg2kZ2kcLeoaAZnkm8waEdB422GDPmvtEvbt9KkM+V6m6x9dtZzlhJxt+p/ln/Fi1S9JKx3toeUU+PnVMy0ArNPOgDP+U6BhcLVIOuZTwuCD+QPahhDN+RwqqdKSw0sCwzcHrKc5L6EnuQiandhqfGbRDy2FaMjvPPTuK6k72khS5aadUcejX74M3LG5QjjeGP4VN7ydh5rXk/SMF9OqbixeqxUX8d+SMmni6VHaoMsNNYBblaQf1kLLsA+AxuchtMy5byoLy47mVvO6Wd7qoROcke3Y/7pLvY7bHaqw7g0IZuYREG8GjTNOBg0DeSM2Wy/B1zpMOgA3ajHIJKTTWL9oUXilJIvjSF1qEuAV3AXqxVi96t6eM0hSaG8RIBhBMqyzE5BbXi7DlMrMogkvOck3yYP2/k/dIQF2rJZsZbTk+CPDt3yTWwRp2jfxXbU+yUVHUTABewzZF3FXbuCORJUIkllKY07l2mJuUYFbxDFT8x7UJsffdxUHl97CKYPyU9ld1xlHtXy8s48LOeRN4q2SfG7MYBmxXkhOkG6BbWlNmJBvqqh8MCQeqUDXqAFOtozo6FK5elJEKFTC0+Izs1+zuFwmUUJ7uvYmQuA1TkCknSlKzc/kbxQRL5h/XUXz7ueVk8Tdsl+j4ow0zA39IByn9iaWhdXaJHisBm1I4d8iXBoTDseDDyoJNBcdbRKs2ISAQNyKYroUDWNeEd5XQ0XRW4K9v4/pA6D7w6zAPDSG03FuoVuYQptxvp9K6PpIMnswoNGpRsPkm/iO5X2iIFeYibZoly8R9tJUXWDQoaLmldO3NmSBHxdAw X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: bcf6a082-16ce-4a33-8e2b-08d8ed22bac6 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:49.2714 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: DjpoN8B9qS8NqYcFS3gGlGZQ0EOX2gmyxliEY4Eo3R0a86sl//zNPSghVIPRTkO72hQcwIwpRT7dBywjkQHZMQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Take the svm_bo_list spin lock when iterating of the range list during eviction. Change-Id: I979d959e06c32e114cea8d151933b8ee7455627e Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 49aca4664411..3a7030d9f331 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -2494,7 +2494,6 @@ int svm_range_schedule_evict_svm_bo(struct amdgpu_amdkfd_fence *fence) static void svm_range_evict_svm_bo_worker(struct work_struct *work) { struct svm_range_bo *svm_bo; - struct svm_range *prange; struct kfd_process *p; struct mm_struct *mm; @@ -2511,13 +2510,29 @@ static void svm_range_evict_svm_bo_worker(struct work_struct *work) return; mmap_read_lock(mm); - list_for_each_entry(prange, &svm_bo->range_list, svm_bo_list) { + spin_lock(&svm_bo->list_lock); + while (!list_empty(&svm_bo->range_list)) { + struct svm_range *prange = + list_first_entry(&svm_bo->range_list, + struct svm_range, svm_bo_list); + list_del_init(&prange->svm_bo_list); + spin_unlock(&svm_bo->list_lock); + pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, prange->last); + mutex_lock(&prange->migrate_mutex); svm_migrate_vram_to_ram(prange, svm_bo->eviction_fence->mm); + + mutex_lock(&prange->lock); + prange->svm_bo = NULL; + mutex_unlock(&prange->lock); + mutex_unlock(&prange->migrate_mutex); + + spin_lock(&svm_bo->list_lock); } + spin_unlock(&svm_bo->list_lock); mmap_read_unlock(mm); dma_fence_signal(&svm_bo->eviction_fence->base); From patchwork Mon Mar 22 10:58:54 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154543 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17057C433E0 for ; Mon, 22 Mar 2021 11:09:12 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D8B816188B for ; Mon, 22 Mar 2021 11:09:11 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D8B816188B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0D0356E41D; Mon, 22 Mar 2021 11:08:33 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2058.outbound.protection.outlook.com [40.107.243.58]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8FF506E3FC; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=AzavCaa7lVsFPeJv4NhQN+vBbUHV9hnTUe1OLq2VrCK99rZkXEg97Jf+IPcirR9ZhVZlTr5pcRsVKaN5tnMnksHPGwcOZkehJSMv/2mS9YDjYEyxCRUPcVtO/ixqa+KrSrTCoudLNJN5JyQaXn5AhY0ebEZcZleMTBmpb/t2aJMjZl5j+Uf60f7+XbDUSAcJVUSce13mkilgZSpS1bdZeNeybVIfxv7ywd59hN75DJEqTbeAGFYVdrKwe0HpgTD3WJonRxdl7ZfigrYgva0fQcCSBNDXLHTbLgHdB9S7EcjYlcVqwGaUQojsG8Oc31BGd1PZnXv1tH2NESVIc4ht3Q== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DZCt1lqVtIYxz+pdsCdtNIA6HPeG7QA6q+VZqw/gBFc=; b=jY9f4VQ6po48Zp8PPH8b2fO6gnGajiGHINzAIjC3neUOMwHdqNWDtlWIF/c//gCCgE7H3S95sH2QIw6zFrEvCTj9qJUHxa0YH9t0KLCdqEYrhJYU4gVF40TFHNUdGo4B3+eEPsDnNh1EWjmLhWchLtJyVVhUxT1H3C40a21ZtVlA5V5V6opqHyOHO9+XFmZFKH7dYaQ6LlC0WxLXE+xC94Vdg/+6EPEPNe3YUgJaHCSqWNh0qNz5/8+y0BEGmUNucFOc1Eq3c1NPphmWdaGhIejvrMcxP49z+ruh2u0+xKMd93OrniU6WXVPD6/F2eHWgvpzFrBMvLncTBdmzaGV+g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=DZCt1lqVtIYxz+pdsCdtNIA6HPeG7QA6q+VZqw/gBFc=; b=cgxrZxtBNALkOsUtp4kP9/8RnjuPtK6+n31QeSiLtQkH1p1uIcHMcwKaTsSZAYvSfz7X5lhlZu1uaq38/gF9tf7SeB+J4H9BzlWhzmICJVzkBM3vZXX2GtP/3JvoPhHxG0vaF1pG1yX+H690mqGghpAaFwiYOlrldbl04+H11HQ= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:20 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:20 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 38/44] drm/amdkfd: Simplify split_by_granularity Date: Mon, 22 Mar 2021 06:58:54 -0400 Message-Id: <20210322105900.14068-39-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:49 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: d5fd3120-15f2-475f-86ff-08d8ed22bb0b X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:660; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: tpHmFvlIMmfw7B6TAJdEj/ijK290viHNEk28YmrTHHQt0/FvgG8WfSgs1m78pKFp5924z7VI94ifba1p0zVW0VKvPl7VJyjBl1i715y2h7azsNoamhrbkODIvEhql0QKNHCeOTwSfHv3E5dcs/QIOl8Utz1xGh9Fwut/FGcDHcV4bF/zJJfaXNt2BpxRojN0nzK6MPQhiJtVtOu1HJyoEXKZzFEKJdZo3x2pGBwhUkDoHFdfdyVcjniPF7IuGYwsbIxtLUXKDzbXcOfBk/E2l2Va3poyF94J+IUs1SUfwX7+/3hF34UgBIjjUqdDnydKD7h67QNdcE+Wd4lEsaZsWe1C5aO09CHquhV1OFawJG2wIaJyyVeWTCHz2BSf8MVTxxwz9kCK70ngatTSe1qifkxORmulLUfH/fJYa8mXitLSz/GAvhan6E0sF1T8lGK+JPhX9KiIovCiCBMJYOuvtJMyXpBbisHWlK42OZPGfaBAxplzXBwEB1hHDbdTHFkZRw2kNsEJdY/ODrrF60nu7iE3vVOfxLmDOT4mGshwkLi4KTw4FQcvxIiAEIX4R4evl5cQXMpZQ3vvLgXmzKL2IgP2GF0FE7mxfA5qbEifKtnZNkFGq7lOuMq0rMPJ5GxA8NtrI/PKZlfv3deSSyK9MDpGqrbeFb7Sd7/1yK0FWWs= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(450100002)(1076003)(36756003)(38100700001)(6666004)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 3ruPRgm2t+2trutAnowqxUk9TfNhv1cVDk7KCWrXQivENxAkQuzxQgv+CZ0FRN9oADCmsY2ewgfqsV2XVI+IowIx/QDn0YTepkbRihXk7z5xLIh+eEjkDG2v6bngVE/cBIaDJad2UwkCPgFXISbr9NiJbohvW9E8dROi5Gb0s0C2CXzpWwFUNI7uLZWIg0LSTw2GI1mhs3jC+Wc7qCirAz/6jGNKuQ8hQea5jAUXGIjOzxdfro6VWUbUxqwTtdD99jh3BqoClhlRm75ompIg0SUauuFC4GGXMx5xoIRyV647M+EV9lLdzEysMcUVazHuAvYVRdk8rjXIdxgPR8yLQ/xyojuf6YwIWkWUZRUqv447Q48PejvUy03QGlRiG8UDxA2G0kp002Qx4XuT8b72IHEDYq10TBqlFcfMQkpGl0P23Cd6oZ4Wn+X/opK9Cm0N754ILq0rgY0dU/rfiaziE9EEvLEa2Lwov7PVhdVtbg59svEoYNYhlkl5aPYqDhbKEtPNVkTDiLx80YDmsyq/bjv6vAf3eYdMSCo6dCmN8rxGkMW3sNjE2Ol/nRKyJ6QQJfAR2BoMUmF/1n6RNpsT/bL0AF7eLEzxdeXheLJXnF6qI8RrUTY3qEkuxjtygBYQVBnxks4azeByoU18mYR874sSS8lAkVYdTo+4ZZZXPeRnClR3dEiW3s80OveNFq+5VIKFI0i9cp7rXC9mSak1tWjY2wdmDE8ZjS3wS/DCztMSycfbDqG2XnqXiJ4NZqFSBJJ5cnUDsAlHtaeW9JYcn83QwW7ak3wDbnqp3vEVAy/9OAf2Qkr1c9w7wHPvf5Sa4LO9OtN/yMuonwDmOJL6sgoWzXv8oUzKg1GMVwrWp1S40LD+R1MsX3Gw+33OJQknDas2NyBAxCCK9teNxePvcQpWVDm+HaemR8dmNuwT5aIcJ1TKQbz80AmIzzLcm4hY0DT6k034ZgbxI/HrWm/tpGTusagMaWfzvbbZ7Yi1QX27VtKNH/8hNu64sjqUPOS6+zHtXXdd8a+RCxGpES6jvMOXIVtYexvwTsgMkiceYTxIsXc9UW6jndhaDMy5aAZlI5ZjZsUqv1WAkzCAeaByvp1Cmc2u3DOHwE9RGedj7PtrC8dG1fJqO88VnGFl73IZT6JhOMMsOOozF8NOf94fqO9/2nYPeBHRlodBt5qDCDqa6Kh+rlk2s3SBmNPyj+KGZS16y8V6+qAay3kui4FkNF0H0lw/14KYYw2OKUo9d4TgB6U5TlGGK6ta4Do/LyHHXLyKdO7F90tnDyKuHLKjwV2T8w7QGqOMQNvu+vnU/OSMJmjmCjA6wPM3ihqfrIyX X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: d5fd3120-15f2-475f-86ff-08d8ed22bb0b X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:49.7094 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: gVcwO2ZilYPjbR4ZJiD4xGWEOCGDv1uEDJDO8JVSRFOsIjFnP2jOp4rmPyrkF6mzZfW7IUK8xudAx565Hu+Kpw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" svm_range_split_by_granularity always added the parent range and only the parent range to the update list for the caller to add it to the deferred work list. So just do that in the caller unconditionally and eliminate the update_list parameter. Split the range so that the original prange is always the one that will be migrated. That way we can eliminate the pmigrate parameter and simplify the code further. Update the outdated documentation. Change-Id: Ifdc8d29b2abda67478e0d41daf5b46b861802ae7 Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 29 ++++------ drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 73 ++++++------------------ drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 4 +- 3 files changed, 30 insertions(+), 76 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index da2ff655812e..5c8b32873086 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -780,12 +780,10 @@ svm_migrate_to_vram(struct svm_range *prange, uint32_t best_loc, static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) { unsigned long addr = vmf->address; - struct list_head update_list; - struct svm_range *pmigrate; struct vm_area_struct *vma; + enum svm_work_list_ops op; struct svm_range *parent; struct svm_range *prange; - struct svm_range *next; struct kfd_process *p; struct mm_struct *mm; int r = 0; @@ -816,31 +814,24 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) goto out_unlock_prange; svm_range_lock(parent); - r = svm_range_split_by_granularity(p, mm, addr, parent, prange, - &pmigrate, &update_list); + r = svm_range_split_by_granularity(p, mm, addr, parent, prange); svm_range_unlock(parent); if (r) { pr_debug("failed %d to split range by granularity\n", r); goto out_unlock_prange; } - r = svm_migrate_vram_to_ram(pmigrate, mm); + r = svm_migrate_vram_to_ram(prange, mm); if (r) pr_debug("failed %d migrate 0x%p [0x%lx 0x%lx] to ram\n", r, - pmigrate, pmigrate->start, pmigrate->last); - - list_for_each_entry_safe(prange, next, &update_list, update_list) { - enum svm_work_list_ops op; - - /* xnack on, update mapping on GPUs with ACCESS_IN_PLACE */ - if (p->xnack_enabled && prange == pmigrate) - op = SVM_OP_UPDATE_RANGE_NOTIFIER_AND_MAP; - else - op = SVM_OP_UPDATE_RANGE_NOTIFIER; + prange, prange->start, prange->last); - svm_range_add_list_work(&p->svms, prange, mm, op); - list_del_init(&prange->update_list); - } + /* xnack on, update mapping on GPUs with ACCESS_IN_PLACE */ + if (p->xnack_enabled && parent == prange) + op = SVM_OP_UPDATE_RANGE_NOTIFIER_AND_MAP; + else + op = SVM_OP_UPDATE_RANGE_NOTIFIER; + svm_range_add_list_work(&p->svms, parent, mm, op); schedule_deferred_list_work(&p->svms); out_unlock_prange: diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 3a7030d9f331..fbcb1491e987 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1005,16 +1005,14 @@ void svm_range_add_child(struct svm_range *prange, struct mm_struct *mm, * * @p: the process with svms list * @mm: mm structure + * @addr: the vm fault address in pages, to split the prange * @parent: parent range if prange is from child list * @prange: prange to split - * @addr: the vm fault address in pages, to split the prange - * @pmigrate: output, the range to be migrated to ram - * @update_list: output, the ranges to update notifier * - * Collects small ranges that make up one migration granule and splits the first - * and the last range at the granularity boundary + * Trims @prange to be a single aligned block of prange->granularity if + * possible. The head and tail are added to the child_list in @parent. * - * Context: caller hold svms lock + * Context: caller must hold mmap_read_lock and prange->lock * * Return: * 0 - OK, otherwise error code @@ -1022,75 +1020,42 @@ void svm_range_add_child(struct svm_range *prange, struct mm_struct *mm, int svm_range_split_by_granularity(struct kfd_process *p, struct mm_struct *mm, unsigned long addr, struct svm_range *parent, - struct svm_range *prange, - struct svm_range **pmigrate, - struct list_head *update_list) + struct svm_range *prange) { - struct svm_range *tail; - struct svm_range *new; - unsigned long start; - unsigned long last; - unsigned long size; - int r = 0; + struct svm_range *head, *tail; + unsigned long start, last, size; + int r; /* Align splited range start and size to granularity size, then a single * PTE will be used for whole range, this reduces the number of PTE * updated and the L1 TLB space used for translation. */ - size = 1ULL << prange->granularity; + size = 1UL << prange->granularity; start = ALIGN_DOWN(addr, size); last = ALIGN(addr + 1, size) - 1; - INIT_LIST_HEAD(update_list); - INIT_LIST_HEAD(&parent->update_list); pr_debug("svms 0x%p split [0x%lx 0x%lx] to [0x%lx 0x%lx] size 0x%lx\n", prange->svms, prange->start, prange->last, start, last, size); if (start > prange->start) { - r = svm_range_split(prange, prange->start, start - 1, &new); + r = svm_range_split(prange, start, prange->last, &head); if (r) return r; - - svm_range_add_child(parent, mm, new, SVM_OP_ADD_RANGE); - - if (parent == prange) { - pr_debug("add to update list prange 0x%p [0x%lx 0x%lx]\n", - parent, parent->start, parent->last); - list_add(&parent->update_list, update_list); - } - } else { - new = prange; - } - - if (last >= new->last) { - pr_debug("entire prange 0x%p [0x%lx 0x%lx] on prange %s list\n", - new, new->start, new->last, - (parent == prange) ? "" : "child"); - goto out_update; + svm_range_add_child(parent, mm, head, SVM_OP_ADD_RANGE); } - pr_debug("split remaining last 0x%lx [0x%lx 0x%lx] from prange %s\n", - last, new->start, new->last, (parent == new) ? "" : "child"); - r = svm_range_split(new, new->start, last, &tail); - if (r) - return r; - svm_range_add_child(parent, mm, tail, SVM_OP_ADD_RANGE); - -out_update: - /* If parent is not on update list, add it to put into deferred work */ - if (list_empty(&parent->update_list)) { - pr_debug("add to update list parange 0x%p [0x%lx 0x%lx]\n", - prange, parent->start, parent->last); - list_add(&parent->update_list, update_list); + if (last < prange->last) { + r = svm_range_split(prange, prange->start, last, &tail); + if (r) + return r; + svm_range_add_child(parent, mm, tail, SVM_OP_ADD_RANGE); } - *pmigrate = new; - /* xnack on, update mapping on GPUs with ACCESS_IN_PLACE */ - if (p->xnack_enabled && (*pmigrate)->work_item.op == SVM_OP_ADD_RANGE) { - (*pmigrate)->work_item.op = SVM_OP_ADD_RANGE_AND_MAP; + if (p->xnack_enabled && prange->work_item.op == SVM_OP_ADD_RANGE) { + prange->work_item.op = SVM_OP_ADD_RANGE_AND_MAP; pr_debug("change prange 0x%p [0x%lx 0x%lx] op %d\n", - *pmigrate, (*pmigrate)->start, (*pmigrate)->last, + prange, prange->start, prange->last, SVM_OP_ADD_RANGE_AND_MAP); } return 0; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index b2ab920ab884..7fce3fccfe58 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -162,9 +162,7 @@ int svm_range_vram_node_new(struct amdgpu_device *adev, void svm_range_vram_node_free(struct svm_range *prange); int svm_range_split_by_granularity(struct kfd_process *p, struct mm_struct *mm, unsigned long addr, struct svm_range *parent, - struct svm_range *prange, - struct svm_range **pmigrate, - struct list_head *deferred_update_list); + struct svm_range *prange); int svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, uint64_t addr); int svm_range_schedule_evict_svm_bo(struct amdgpu_amdkfd_fence *fence); From patchwork Mon Mar 22 10:58:55 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154547 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A2225C433E0 for ; Mon, 22 Mar 2021 11:09:15 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 627D661919 for ; Mon, 22 Mar 2021 11:09:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 627D661919 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 8F7766E11F; Mon, 22 Mar 2021 11:08:31 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2045.outbound.protection.outlook.com [40.107.243.45]) by gabe.freedesktop.org (Postfix) with ESMTPS id A2AA36E185; Mon, 22 Mar 2021 11:08:26 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KQV4PpCK55auiYme0lHr2SvaKnBs9Tw/MP3VXMVy3Q2b9VMQQgc4AhErXj2sElz077+AE6mlq7nj7qfXz7Td6erf4ynQRb3GRuJEkMoqbDAvC/y2gVgHIJEUiBRUdtTjDeEqBNl2KIi/YBY9SJxTZVy1LITHj7DS02mtCZTJrroeJy3fuxQU8pYUcEJE4OgE7lAh0RwoTZLOnrliY30Ote5tgD+gLrSYnHPJkuPX9VXlNEjMGeRWkvC0j8D4d44PrmI4DihdHx3CqeVRABqdjlYj6LrEVN3XX3FHrDehfxfN/rnzL2ffu3JqwOkHB3UfDd9Kb/3fayTW+21mCB1ICw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bN4tz8cNX6WVe/mLrmsJdfxYKxUfO/5nj+k1b953QSI=; b=nlxPGEuKTF6S15wTRZSinrwgrY6wMWFIcbAvc+fZvh6J+aeohdOX28fbOnmmOy85UyP5cxvoD3Pr+fCVsNFMhePNH4IbDwKJRNK61xRuHcNnQkKELTD3shEjB7rwfBYygEd73us+Y1Arr4ti+Vi5iNA3+rvlTwLgJ9eaE1soR3+Jzfh3TqflVu+q+rmlVbLdMlchRwFSKxAg5MsMrLriMVQ6bcoP2DXjxqWci6oznJBmai576goktkYENOCS9rU6oht3hGtgbwUgK7AgJNluvgUIPGB0MKiSauqO5iopdiqzH0SpQdPOH1YjsnDsMEruiae1hlYqX4NNDenVfE6/6g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=bN4tz8cNX6WVe/mLrmsJdfxYKxUfO/5nj+k1b953QSI=; b=QjKbHZTo+lIFdQnk5DRZRsKutmunGR+/wDUMcPQsQpvdjWyaf1Wm7p4dlJqDeomeIma8P6gmMV5ShfHtGzWSARMoOabZFSeGsIulNzSvn28gMKHsTDzioo2OAfwiGzVpAVSO1Hl0R/CKrJWOd6Yg7qU+pLNr8xFTppyVJ2DSUlw= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:20 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:20 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 39/44] drm/amdkfd: Point out several race conditions Date: Mon, 22 Mar 2021 06:58:55 -0400 Message-Id: <20210322105900.14068-40-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:49 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 4159afca-a365-4632-8646-08d8ed22bb4e X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: F66uq32gxrIznDpLIhxIr/+A7VXZzo4OLlhCnr4VLmmXL4zTVA9CWkG6YnRBMI46c8gX+TAXsjOHdZoiIMdJ5tMPmXFGrpEzGaoHU3YaseeSGYkeZgdVCyIRzCGpnLoIzxSz0qwC3UnOH1azLFaivnNAr+I9elc/D+APo2exy47AhW5G/leuBpa/5c2J9NMU/OlSphLNaTNQWOSDIMfvOm6lddvtLyiUVzVY9i6wA2rqTFe13cCl/lGAGPoZc+xJ9vdY4whEP0SPO3qMyAh7UkSwrvvWq5ImRCATMG0oYrn3rv505Xe8GFuuAo4kY+pg/Jpk9C+P5CmmHbsCGySuXS8+vRuBXR8CKlxZSuEYzaYFFSrv6iEzdSzUhw4uymDBAm37Zy1Ij2hvG6iG/h4Bsv4IkOUmr/OMgBF7+wMOiU0Mq0MfPQakPQhO2qZjUtoFWwRN7MvtfwA9x1HxTGcLE6NP8lw/g7veusgSLMfXxhd2Xl7tt+f3tB2fyvNn4om78LgNYg0IAyumhvXArqTWwYo8pASE7kKr/SgQyxf4lmq64Hll7ZLfU/3cruz6L2bKwctVoornym6vmZy1Poa68Niht2XKrBGGDzucEekF0Mo99GeeQmK1k7xn3GnQhLGnlYHFeQWqS3nZmMVQANtpdLoZVWqORm5C+/nKJBe5X7w= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(450100002)(1076003)(36756003)(38100700001)(6666004)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 2fUaomc/1LeDdQo3dwCOTq1cXQ9OCJ5qv3XVLD5vcPrFSjRfURweagMdgcTkProOhg+M+dAgwaXY6IyMCOZaaCjlp6QwuGVbXAAOD8GNyHeq8fvt6PrTg43a2163GeSrw2SKTRSMMcvi9yVeJGxtDJsl+fnCfZGDqROTJRxcinWFpdj72rYfWdoLRVMv9Ut68BTvcQ+/d8U+eon5e9T6hplH3hM0g76GCg7nAeqb00BvdjJzjov7FHliM7Sk49fiWSNzKOIuOgtM6S6FwmGKzOM7S/xpDpgefsMVGenwMyaw+JJW6wEMI3DYo91FTUlArW9nL0EZNiSfvarBwDrecIAmCd5qFbz/LTWxcccrpPELKkXExyEfkhMoztpLrukN1FJFcIVO2Lu+FRSdR0vtL8DZqJEbq6pnJGWfvsu84yTeJAmwvDbLqs9gsagO89rmJZlUyaUdyg7tzAJrrE4PfI4vQxW79Q1l8mg+i7cTrCS/Yg8HCTCEbOepWepmyGZIpeuEHMqM3BImvvkJYBU2DQy1ixrs+xqj8GLMNLkBRlFmnKro4kH/4UxwxZt9Rrya3Id4P2yPXz599B647qJwBuICMhY3GoNO5F7K/2LTURCoFxaplSRGL8LYjXeJt79lwz+xL7VH9EkWMW+80H/rJ+HUETB3wK5i/YjxsjQaGzebUQeeqvWpS9mT6nrjizdAwFKX4azV+RtGexWN07M0M9c2OXjN/Zb7MfFire7ZsR+xL1cbI44trCKsh7mIV4/JjW02KhRVdn3Zcw6Fe05v/AUX1Zy/32dOmzqYsMMJ7LMFLlgU0MOTkn/CQfbuScIXoA7nCVQXloTHswIfGZpEHtuJhCJy7q2C2VFR+7dBhRIIOwBp9bT+sEzsCZCGLRu6XAjn4ro9eS9aJV+UZFxxM1uwsOmjd8rHNv0aklHkTenH+/eeBUZ7etZS7477pbpqdx9fouDT7g5QxaRmxAs+Q4r6hL9JlSr/ND5B2DLZj1J+xdZYv21nhU74agiovP8vDo4YqUltRz8x1qjtHF1qB1szD2ohP1W4IUpGasQ70VczgF3QlWgquSsqL5jDzLvkgwr4eEFZvz2jdUwFZ5WKf9pdlIGGLkmok/EU3s0uagLPY9wqmSLoUOWjeaEcqZgPSbeqi9K2gsXWR3wkwmiFs63mbMnauslmUHfdoLC51h17KpX3/XeGV6WVHs19TdYWaKmMI0a8tTnRaZ7n1HWNNRpRwDIefgEmZo1pDrx9kl7s3PEZVUA1A63DPKdiNjaWKTzoKPD0YlHclz+SUwUOoRenC8XNA2f18vXkNwtR/LrgmDWaCnZRO66pPwp71+x8 X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 4159afca-a365-4632-8646-08d8ed22bb4e X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:50.1595 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: GCX5lr1QlcgCzWzBmxWw+2P8J81n1UWmA5+1M8g2vZ5pN0obng4fShfWMZSE8WKYEOHGmYGnpVaNcmfQjlimHQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" There are several race conditions with XNACK enabled. For now just some FIXME comments with ideas how to fix it. Change-Id: If0abab6dcb8f4e95c9d8820f6c569263eda29a89 Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 5 +++++ drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 21 ++++++++++++++++++++- 2 files changed, 25 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 5c8b32873086..101d1f71db84 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -539,6 +539,11 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, src = (uint64_t *)(scratch + npages); dst = scratch; + /* FIXME: Is it legal to hold on to this page array? We don't have + * proper references to the pages and we may not have an MMU notifier + * set up for the range at this point that could invalidate it (if + * it's a child range). + */ prange->pages_addr = kvmalloc_array(npages, sizeof(*prange->pages_addr), GFP_KERNEL | __GFP_ZERO); if (!prange->pages_addr) { diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index fbcb1491e987..c48fe2f276b9 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1727,7 +1727,7 @@ svm_range_handle_list_op(struct svm_range_list *svms, struct svm_range *prange) pr_debug("update and map 0x%p prange 0x%p [0x%lx 0x%lx]\n", svms, prange, prange->start, prange->last); svm_range_update_notifier_and_interval_tree(mm, prange); - + /* FIXME: need to validate somewhere */ r = svm_range_map_to_gpus(prange, true); if (r) pr_debug("failed %d map 0x%p [0x%lx 0x%lx]\n", @@ -1744,6 +1744,7 @@ svm_range_handle_list_op(struct svm_range_list *svms, struct svm_range *prange) prange, prange->start, prange->last); svm_range_add_to_svms(prange); svm_range_add_notifier_locked(mm, prange); + /* FIXME: need to validate somewhere */ r = svm_range_map_to_gpus(prange, true); if (r) pr_debug("failed %d map 0x%p [0x%lx 0x%lx]\n", @@ -2068,6 +2069,14 @@ svm_range_best_restore_location(struct svm_range *prange, return -1; } +/* FIXME: This function can race with MMU notifiers. MMU notifiers can + * invalidate the page addresses concurrently, so we may end up mapping + * invalid addresses here. We cannot hold the prange->lock (held in MMU + * notifier) while updating page tables because of lock dependencies, + * as SDMA page table updates need reservation locks. Only unmapping + * works without reservations. May need to hold the mmap_write_lock to + * prevent concurrent MMU notifiers. + */ int svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, uint64_t addr) @@ -2592,6 +2601,16 @@ svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, continue; } + /* FIXME: With xnack on, this can race with MMU notifiers. + * They may invalidate page addresses before we map them. + * Then we end up mapping invalid addresses in the GPU page + * table. May need to find a way to still hold the mmap write + * for map_to_gpus but drop it for validate to allow + * concurrent evictions. This will lead to some retry logic + * and the need to protect the update list differently. + * Maybe factor migration and validation into a common helper + * function shared with the GPU page fault handler. + */ r = svm_range_validate(mm, prange); if (r) { pr_debug("failed %d to validate svm range\n", r); From patchwork Mon Mar 22 10:58:56 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154539 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1967C433DB for ; Mon, 22 Mar 2021 11:09:07 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id AB68F61919 for ; Mon, 22 Mar 2021 11:09:07 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AB68F61919 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 63B2F6E417; Mon, 22 Mar 2021 11:08:32 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2045.outbound.protection.outlook.com [40.107.243.45]) by gabe.freedesktop.org (Postfix) with ESMTPS id B41FF6E0C5; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mu7ukDqIuGb794u8Lub0ij7Ft1rHYCVHEGMRI5EVgrliKgBD600FsF1k8mU1QPVRJfi5GsF/zoPwclhyZh11ZGFJ8vq9mKV6cOJbLRSLVoKAe8rFAgShRr6Om50eLrKg2CyUryhGtvHHEaG7RmtWqrZ6VQE4BnCwL2A2j+7GZ8oGSRq7OQ9pwZHPFi+f9tyH5UKnhxuA2jUYVNrErm1YA/dR57NTYGF7uRh7E+x37JmNH9dbGsjAUv+pAJycLk2rJ2WDTS6WWMZ8sX6Czu/1/qI/ENAHFI/ePLqIZH4pEyUjaVYCESMhK/q+DrVZlmatGy0YnitFixGklRJj0It/Hg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6RIXm+RcZ99c0tX2BZ7PVXMjpQmls35a4G/yPgBRk1I=; b=UA+fPFa3l0pn+ts+y3sW+9AipHwESOtOQVF9/PB67edbsTjqNmE96CsCugpqY2lngCOmZMHtn7ZO67bNR3UZ8zlcn5S1gZ1GKbL0iOhW9L1KLUBmmnRvf8Hjbpx8Nx7WjoCJhkoPbNfofQW/GLLa5ChUMQC65/+sznNfm2QaJ4rDeVWfrilGPwtXn3D7Un+3Qaxk8mn+GPH0lpAgm2KfXeoYSwL6cAh/YuuO7hUfoAQfKsQXIMbPY5tW2oiuPZM2SJg2p0ZpwEmltjWFpJzsXTHqe9tcncc8J1pe08iZ5Kb3eSOHtzbmzlT9eLDwv5tUmZmzJXu/HkVH4uSnq1hA/Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=6RIXm+RcZ99c0tX2BZ7PVXMjpQmls35a4G/yPgBRk1I=; b=aNkd6q2/RB5kV9Gb7Gj3ywGuMl1UiQeW2AzNZ3hEiZOvdRUeDLeEeUgz9EXsmQ5+g1SeFulcB5arTA4Cu2gJFOykTKoaSZC76C8h4Qu7GFlYWtDGgVtkknXM0zYceHdcNMSoEc8U81CpibbDELBhy9kWNqWJGEl0ksSU+8YzAgY= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:21 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:21 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 40/44] drm/amdkfd: Return pdd from kfd_process_device_from_gduid Date: Mon, 22 Mar 2021 06:58:56 -0400 Message-Id: <20210322105900.14068-41-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:50 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: e570be07-e5ad-4a40-6211-08d8ed22bb96 X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:3383; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: mPzVK1Do7DHcMoEyneqSeFtizynbkLyDt7lZfjQ+c7WHkgxbp56QEFCF0iOxlhSxTMCt26Lkizty84K3G9L6JuKCvRBNON8EK8B9UKnEwcP2dWhjbC8uZMj28FD4Z8J9Xjdia9a5m71YXwJZWiZampEHieVRzdkh0YsCPkaUFeF9l0lgcz4854WpH/c20IrAsJBGDB98HrhbH9eSjphtnYrFgLNhFOdx31ViZgcwbmo3zyc1B8n3ugFeCbfxXzpj60QvMj8vReqVoR859JU2SbA+r8yyY7aPz60AbRiz6n2WtQY08422F9QlwpN7WNVVIhy/1D/CfOz0uvVYKQdXgXeGTqJaAnYRhDJhL85XSNsVblkEej0RPsWasqze64ttjD+nOC9j/Q4SmnOYNBdM/qdkwd1gc1/n9KYOClHxRiGJsVFjt3uFRCJcyhtqa6Y7eKs+R53vKafr62wkBlbj1O21DpWTzwVqf9qs2wzHqn+OVp+6aAHX0tj8eo88WF5YqLqjryI3KDhrcAbHLeojbwBtfzpweCeK3kXxxYWJE5FlAkpmMS7suDH+ApiXpiHerY3ECqMB29K7fmjH2xdk+9NRpLJcHhmYw/6xmmPRADtoYG068sLvpiyMVlELxV4E+XvLyTn8aOorIzXArAaG5wjyZqxA7lxbqhUzFs0YPMw= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(450100002)(1076003)(36756003)(38100700001)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: qt0kbarHi0vItPtMJo0b3HkntJ+HFBjEUCllWK/C1edyTQFADq2vR2o9QNmXcihq/3EcnZh6sdk5JudT90Xwo9eg+3JcqDmhpb4ePFaJSZrqJMmUvU1bDmm8o5/D/RcbSR2Q75u8ESMd18APuL5ITZujjdi1th1a4C9ZJP/WAsNosxhX8FtpY6b/BnZEAfU5HMO4Krr4mU/JdgojineYd40rZyRuJQ1S1n8+yQwZ7CUWwrVnnyipCpIxXYagpZ6E4OA+rFenq4agZieTneU4dULhML4t+OsYa/MFDOEILd+8FFpsd77MPJsyhGsYAWAfJ8cNUtwlSPh71boieRMi8vWnDVvQOpkm0Hqo89ipJMnNRAYE8MyjgRYZC0/mYBeC+mQe7ktWV5k6GczymDUJ/Tx57JZPKUbje7qAyBC34eZCeP9tW584ML3fYuNg7NmQp8id+vTNps13nZ375WjYDD15q3p6HD1/rxy5Dge0f3vZexUVLFfL8AWsVTBf6Ui53VOpPmSOJl3s4WAh4jhepZBBkDTXE15roPpTLbfygkwfMD0tw8ltRjoiJqzTpdwY4Xqh5p5e/eL8IKVAe+nixJ8XTAlPBsUcddFU9aPL5URAW8hmbevrZa/T3b6Eururidkmrld8tlrQ68i7HaY4qru9BA1LdWimjY84/U5+tuTEEulr8serwI8pDnMWu+2A1XOayxg12ToC47Sr1SexgRN6CxBQAvau5/ETQzXe5rwt0WfGXquQ9VHz5xJkBky9QzK0ODdzAMqONvzPxOknMlitPqy/99+No8iWJHx205t29bolJKLipIeKonq0mYCoAbs5j+/Qr810fBp5zZuCF+b2zSgn/PdEToS3vu2V6bmOOA0qZs04ZIMEwpOKYnbQ7mQTPOdaeNKfeftst3dJnrIobGOsrTyh58lEZudZTlQK4eJHEeVksHBE71beFFSWoyAzzgnFurgFazSc0wKkMJdej2FPAoZuesqcanK4MrPxNAI/c2oKALylPV1L83AxMCYxr9htoybbimv5xJB9hDKNUegFp87O4zRyV8Nzzl7xBGriXSheveSSJ8xQT8GAR1TkPvdabIF7Fn2HRlR29vvs3RH8vE+spm8aLdX9hOp9hJZCKhQj9kk6LOYR7O+hR7AeMQ23MvRv4dZT2XFxTQxCCgLWHjHbk88OXr295zYdNhMkDhi6yKu4+Uo5rQVOJCbjYoY/bwN+p5g1SKVnPrZ+hp/hWRl1+AzIO+AJ0MX2eVt0ZjFeYRyJDPdv+xNMbhE60m85kxjWxvwHIjD/tqR+uWv+hCHUsQvhIDD+GA0LaJPvGQvSbsV+bjhQqKiG X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: e570be07-e5ad-4a40-6211-08d8ed22bb96 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:50.6463 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Fap9uQqL86cWeeYavP67JR/8n/VUev9R3IOUC/8Ph/UMV7Jtlio8Ab8rVhTtax+wmr0qh96vWO5oPqlzEcGuzA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This saves callers from looking up the pdd with a linear search later. Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_priv.h | 8 +++- drivers/gpu/drm/amd/amdkfd/kfd_process.c | 10 ----- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 51 +++++++++++------------- 3 files changed, 29 insertions(+), 40 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index 2ccfdb218198..ca44547c46a0 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -865,11 +865,15 @@ struct kfd_process *kfd_lookup_process_by_mm(const struct mm_struct *mm); int kfd_process_gpuid_from_gpuidx(struct kfd_process *p, uint32_t gpu_idx, uint32_t *gpuid); int kfd_process_gpuidx_from_gpuid(struct kfd_process *p, uint32_t gpu_id); -int kfd_process_device_from_gpuidx(struct kfd_process *p, - uint32_t gpu_idx, struct kfd_dev **gpu); int kfd_process_gpuid_from_kgd(struct kfd_process *p, struct amdgpu_device *adev, uint32_t *gpuid, uint32_t *gpuidx); + +static inline struct kfd_process_device *kfd_process_device_from_gpuidx( + struct kfd_process *p, uint32_t gpuidx) { + return gpuidx < p->n_pdds ? p->pdds[gpuidx] : NULL; +} + void kfd_unref_process(struct kfd_process *p); int kfd_process_evict_queues(struct kfd_process *p); int kfd_process_restore_queues(struct kfd_process *p); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process.c b/drivers/gpu/drm/amd/amdkfd/kfd_process.c index 48ea6f393353..cc988bf6057d 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process.c @@ -1674,16 +1674,6 @@ int kfd_process_gpuidx_from_gpuid(struct kfd_process *p, uint32_t gpu_id) return -EINVAL; } -int kfd_process_device_from_gpuidx(struct kfd_process *p, - uint32_t gpu_idx, struct kfd_dev **gpu) -{ - if (gpu_idx < p->n_pdds) { - *gpu = p->pdds[gpu_idx]->dev; - return 0; - } - return -EINVAL; -} - int kfd_process_gpuid_from_kgd(struct kfd_process *p, struct amdgpu_device *adev, uint32_t *gpuid, uint32_t *gpuidx) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index c48fe2f276b9..081d6bb75b09 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -175,12 +175,11 @@ void svm_range_dma_unmap(struct device *dev, dma_addr_t *dma_addr, void svm_range_free_dma_mappings(struct svm_range *prange) { - struct kfd_dev *kfd_dev; + struct kfd_process_device *pdd; dma_addr_t *dma_addr; struct device *dev; struct kfd_process *p; uint32_t gpuidx; - int r; p = container_of(prange->svms, struct kfd_process, svms); @@ -189,12 +188,12 @@ void svm_range_free_dma_mappings(struct svm_range *prange) if (!dma_addr) continue; - r = kfd_process_device_from_gpuidx(p, gpuidx, &kfd_dev); - if (r) { + pdd = kfd_process_device_from_gpuidx(p, gpuidx); + if (!pdd) { pr_debug("failed to find device idx %d\n", gpuidx); - return; + continue; } - dev = &kfd_dev->pdev->dev; + dev = &pdd->dev->pdev->dev; svm_range_dma_unmap(dev, dma_addr, 0, prange->npages); kvfree(dma_addr); prange->dma_addr[gpuidx] = NULL; @@ -549,10 +548,9 @@ void svm_range_vram_node_free(struct svm_range *prange) struct amdgpu_device * svm_range_get_adev_by_id(struct svm_range *prange, uint32_t gpu_id) { + struct kfd_process_device *pdd; struct kfd_process *p; - struct kfd_dev *dev; int32_t gpu_idx; - int r; p = container_of(prange->svms, struct kfd_process, svms); @@ -561,13 +559,13 @@ svm_range_get_adev_by_id(struct svm_range *prange, uint32_t gpu_id) pr_debug("failed to get device by id 0x%x\n", gpu_id); return NULL; } - r = kfd_process_device_from_gpuidx(p, gpu_idx, &dev); - if (r < 0) { + pdd = kfd_process_device_from_gpuidx(p, gpu_idx); + if (!pdd) { pr_debug("failed to get device by idx 0x%x\n", gpu_idx); return NULL; } - return (struct amdgpu_device *)dev->kgd; + return (struct amdgpu_device *)pdd->dev->kgd; } static int svm_range_validate_vram(struct svm_range *prange) @@ -1120,7 +1118,6 @@ svm_range_unmap_from_gpus(struct svm_range *prange, unsigned long start, struct dma_fence *fence = NULL; struct amdgpu_device *adev; struct kfd_process *p; - struct kfd_dev *dev; uint32_t gpuidx; int r = 0; @@ -1130,17 +1127,12 @@ svm_range_unmap_from_gpus(struct svm_range *prange, unsigned long start, for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { pr_debug("unmap from gpu idx 0x%x\n", gpuidx); - r = kfd_process_device_from_gpuidx(p, gpuidx, &dev); - if (r) { + pdd = kfd_process_device_from_gpuidx(p, gpuidx); + if (!pdd) { pr_debug("failed to find device idx %d\n", gpuidx); return -EINVAL; } - - pdd = kfd_get_process_device_data(dev, p); - if (!pdd) - return -EINVAL; - - adev = (struct amdgpu_device *)dev->kgd; + adev = (struct amdgpu_device *)pdd->dev->kgd; r = svm_range_unmap_from_gpu(adev, pdd->vm, start, last, &fence); @@ -1260,7 +1252,6 @@ int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) struct amdgpu_device *bo_adev; struct amdgpu_device *adev; struct kfd_process *p; - struct kfd_dev *dev; struct dma_fence *fence = NULL; uint32_t gpuidx; int r = 0; @@ -1296,16 +1287,16 @@ int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { pr_debug("mapping to gpu idx 0x%x\n", gpuidx); - r = kfd_process_device_from_gpuidx(p, gpuidx, &dev); - if (r) { + pdd = kfd_process_device_from_gpuidx(p, gpuidx); + if (!pdd) { pr_debug("failed to find device idx %d\n", gpuidx); return -EINVAL; } + adev = (struct amdgpu_device *)pdd->dev->kgd; - pdd = kfd_bind_process_to_device(dev, p); + pdd = kfd_bind_process_to_device(pdd->dev, p); if (IS_ERR(pdd)) return -EINVAL; - adev = (struct amdgpu_device *)dev->kgd; if (bo_adev && adev != bo_adev && !amdgpu_xgmi_same_hive(adev, bo_adev)) { @@ -2334,9 +2325,9 @@ svm_range_best_prefetch_location(struct svm_range *prange) { DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE); uint32_t best_loc = prange->prefetch_loc; + struct kfd_process_device *pdd; struct amdgpu_device *bo_adev; struct amdgpu_device *adev; - struct kfd_dev *kfd_dev; struct kfd_process *p; uint32_t gpuidx; @@ -2360,8 +2351,12 @@ svm_range_best_prefetch_location(struct svm_range *prange) MAX_GPU_INSTANCE); for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { - kfd_process_device_from_gpuidx(p, gpuidx, &kfd_dev); - adev = (struct amdgpu_device *)kfd_dev->kgd; + pdd = kfd_process_device_from_gpuidx(p, gpuidx); + if (!pdd) { + pr_debug("failed to get device by idx 0x%x\n", gpuidx); + continue; + } + adev = (struct amdgpu_device *)pdd->dev->kgd; if (adev == bo_adev) continue; From patchwork Mon Mar 22 10:58:57 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154541 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5EC3CC433E1 for ; Mon, 22 Mar 2021 11:09:10 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 270DE61919 for ; Mon, 22 Mar 2021 11:09:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 270DE61919 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B33BE6E2EF; Mon, 22 Mar 2021 11:08:31 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2058.outbound.protection.outlook.com [40.107.243.58]) by gabe.freedesktop.org (Postfix) with ESMTPS id 07E116E2EF; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=GfBHFqcQ/M8n4y0xlL6RfbmD8mhEXAkiAxcNoIBUPVNgmfIiuTqiQ/8pNVokaFCVqBbME/2RDXMH4hORjwWOctDH7npaUy5hYwCfrLPidEB6OGJAYvJKRhHQBA0e5++PmzFs4mLK2IWhokJilaXgPsg6rAcb3DQHP9ishbFmEnsPclX7ZYI2uccphalxLbTd3N0bx9ITlouD0VRQ8fd6NIaOqfnXV3PAwCdCQecFz51ir6qXYifp0rt0foC7cAN9Lfo1ISRUXQvA4LnKN1NVjj1hz15Jy2HJ+jUcMF14UUFopSAC7aYXPlYzvrPqfA19otW7PNDeV7QyKuQC/ch7kw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/h0RV2QwtgDvA+ZVdz/fvOzdSK0Ty+NCHkPWdLlzqYE=; b=hM8IYSKldsYaoxPv27tNI75hDbDKFsmBQdDHgwHnV8LErdJA65w22vcLqy+yvGan9LWpF//xCm7YMboAUOi+9yveQKwTaDX80A1LaQCKVZRbU1UNrjyxXXWNoBlYP7AhFtR+CFSmdDCo79EjjC/3S9QPgLC671BLsx7mS0EVjU1IvIccaHfcbBkmzf2jGs7PmXVbpL9oIUMrLtKN+vcTJGSZv0UP6n05QJ5SrtGP/hmss6UD7LArIw9VLaKycWyyo1e7+rkMsfxA2IzWAokKHwbBnL0TYspk2r1XTI4UKcij4/8D9i/ysjSBuZ8LCdR2I9MKI02b3Tvt5rRDloiU2w== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/h0RV2QwtgDvA+ZVdz/fvOzdSK0Ty+NCHkPWdLlzqYE=; b=Ou+jeLHb8Oql7vM9FCeZ0TQOQACLMzYaaVIS8KB6hixYq7sXGNGDQo+xityf53SXQ9GEeA2QKqFEbhdPwRqw9n0gZ7G70g5NYxhZz2voQPfFR60sMkbicjl3hd6Ces3ayRn/LHfeVqBtAKWhk3xh9F8B6q4r3YXIvxRy75j4jHY= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:22 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:21 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 41/44] drm/amdkfd: Remove broken deferred mapping Date: Mon, 22 Mar 2021 06:58:57 -0400 Message-Id: <20210322105900.14068-42-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:50 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 8fcba44f-2e9b-4e8b-a244-08d8ed22bbdd X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:5797; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: LOzFvRpWPZJwvxS/iuE80Aq+L0zZm1VnTvuvXbEjw6gHh8FJlPNI+XrdUjiPvjVbfRAe79uWAXubdmNHy71YDyPntmLpGIJMmSGOtIw7KlaK3C5LQ35/YEB4dcWx5LAM9lYweu0XVoi3NyUZZAHzu86XI8i7HwX7Zo4qR3P3u7P+qxUdLkCJL9TtIyOLa1bXrjuHmQ3/7/J84hUHYYuPSNWtjlzpvgjP1s9CG+Hfyo3S4C7W78SmoXy6iuqRwD9BBeKz2vj3SJd6jKUf6EIzbQu/8pAvQjSbXrFFEmSeNXAFtdWDrS1BbkE4H+VN95jIigW/g0wbbmdIM55g4Hu/XsviN6kIwxbdas9b1OzlKXaCZGqBgaNJlsMwr5wQk1VeFjvm70Z0pqEChglHSX3Ei/dtaL68ugnX72uIIX6Ulfwrvwnuyr80qd5NhjR0LP9VUMpUm3ftzYcZed5dRIj+iwIBt/RQTssGkKfaAF35Blb8LOenn43y/z0QlqHmDvQFiUPGRXK3lL559XFIuZBGE4LyCgE0C5eBpCy1fxD0BfwHlX3ZTkKoQJtaJlsGrHESYE8qrkWsFp9JBgvECJJ7G8jADbKDLE+VdrhzE25agYRE0Lu34/bWp0KwRFj2Q2OtqJ13/KbW2dPFcuUQLYDlFaXYSobnvtmMGKeNzjP3gGw= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(450100002)(1076003)(36756003)(38100700001)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: 7lszXtnwJRWesomBNlJEIanHaDMp0wY72kmMju+0eHDPjcSadun97f74pUrBYxfqub2CrmN9YfhbIiv1CV0DPoJTuKp+BzE9S4ZVqn5Ge2FiYun9BPAuZi1rC6CJ8el+pTpHGs5j0J17jpqMBvVkocLuYsd5lHIRdzXLguz6z7A+h11B3G+/MNQ8rXAYRCaZ+m1h3scEg2PYX2OEXCK/AJp5Ovl+WBirouM5clkuD1JabN3WD1CBbpSoGQTAhIWyy9kntuJ9g0aPFZk74SXx6DZMOTgHCFPzoW1UmRzlbIaPpodK7fszamwRykb6cC9VXPeATb9qc/PDCUD2CBEywGFAmiYDqRwgROXSRGA21PgOoHWel8gnxmtaRwws62nznDd75I6fi0qCMtXhJmt1T766MCj+/v1GLcyWep/MhfC/ZNAzgSsBPmrL8na9KSNatKFJ+W9krylFnfR28UJlMa75nY6AeinDyBBjE8L7WRcJ+8wqHx3BTCo6F/+YvmZm0uYmdxkGcXpymUklvFAEtAH9EpWi8GSXCSHYP2AgwJcM/hIgrLu+JP7Eov+aQ7y+zL9EVC+rajJTUlOiL0L1NFVAojqIvotwnafapxtLwLAxdKfpuClZJPQZdxx2W20N+VCLoEmICBs8ZXUaY62/ERtan3cuEYPkOf+1Vta8duD/IjAOJigIYyMb5AIuaF5TumTQX1hNQIqYDhzioe+whqhg+7xJBE7HU6jHrX+K/e5WajZDkHBZiJEOPdEAaHjHj7trM3hKrbxJ3lhCcutDQFio+TQk+oWIefRJlTSRI+h6/CsZLjcDfGc+EWKatYP9R3h91ichC5AWvwPjnzDIZM6gqIQyKB8E+VpMdY96EMEUDwvc76O+LwDLQ28hfLxlbiIKvWEePCcnqHhg7kWxUqImftKt6iwDCj+N416UaNzxriQuGi8OcTiU9H4GO2yZJGZ2RhOmSg0EcYdM1Wy9WLJmw3yyI7mSPXzQ7av+WUqHAl7w5KoTLAlpjPR27YeRi/1aRhJOK1KWWsp6/0z/nsEXxc/KJAr1iJctWlLqij2iNwWkuiv+Vae5PIbi2vIBJKoDXZXaxpcz1fHkfFOIXkg0BTDm3qgddpz1N07OIOGCZFbZeHbfjAI/z/Au4/suzbPgffnAKCit+rVfs+bw+0XMXvZjAap1bNK6Ukxm00cmwD1H9ZZm9UpLYShil/AI4HJE63xQpd6d1rYpDM+6H3j0efEMzS+0ft1yIblCFU4gezF6NxSW6shIhJzFpAsh1p5zyypp3dPy2ydDSdOiEVRYMri3POKo4Ii0zGrPuQhTE1oNZY7pV7ojOthVLWf/ X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8fcba44f-2e9b-4e8b-a244-08d8ed22bbdd X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:51.0565 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: Og55hli4izhHQLY5zHZsETb4hPbwusJS5ys3cROf3xoVMeMhP6n6u+7/G/uUHUWcSsPtmp93XSAIXp1rNfUT4g== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Mapping without validation is broken. Also removed saving the pages from the last migration. They may be invalidated without an MMU notifier to catch it, so let the next proper validation take care of it. Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 14 -------------- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 13 ++----------- 2 files changed, 2 insertions(+), 25 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 101d1f71db84..6748c5db64f5 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -539,18 +539,6 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, src = (uint64_t *)(scratch + npages); dst = scratch; - /* FIXME: Is it legal to hold on to this page array? We don't have - * proper references to the pages and we may not have an MMU notifier - * set up for the range at this point that could invalidate it (if - * it's a child range). - */ - prange->pages_addr = kvmalloc_array(npages, sizeof(*prange->pages_addr), - GFP_KERNEL | __GFP_ZERO); - if (!prange->pages_addr) { - r = -ENOMEM; - goto out_oom; - } - for (i = 0, j = 0; i < npages; i++, j++, addr += PAGE_SIZE) { struct page *spage; @@ -590,8 +578,6 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, pr_debug("dma mapping dst to 0x%llx, page_to_pfn 0x%lx\n", dst[i] >> PAGE_SHIFT, page_to_pfn(dpage)); - prange->pages_addr[i] = page_to_pfn(dpage); - migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)); migrate->dst[i] |= MIGRATE_PFN_LOCKED; } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 081d6bb75b09..aedb2c84131e 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1695,7 +1695,6 @@ static void svm_range_handle_list_op(struct svm_range_list *svms, struct svm_range *prange) { struct mm_struct *mm = prange->work_item.mm; - int r; switch (prange->work_item.op) { case SVM_OP_NULL: @@ -1718,11 +1717,7 @@ svm_range_handle_list_op(struct svm_range_list *svms, struct svm_range *prange) pr_debug("update and map 0x%p prange 0x%p [0x%lx 0x%lx]\n", svms, prange, prange->start, prange->last); svm_range_update_notifier_and_interval_tree(mm, prange); - /* FIXME: need to validate somewhere */ - r = svm_range_map_to_gpus(prange, true); - if (r) - pr_debug("failed %d map 0x%p [0x%lx 0x%lx]\n", - r, svms, prange->start, prange->last); + /* TODO: implement deferred validation and mapping */ break; case SVM_OP_ADD_RANGE: pr_debug("add 0x%p prange 0x%p [0x%lx 0x%lx]\n", svms, prange, @@ -1735,11 +1730,7 @@ svm_range_handle_list_op(struct svm_range_list *svms, struct svm_range *prange) prange, prange->start, prange->last); svm_range_add_to_svms(prange); svm_range_add_notifier_locked(mm, prange); - /* FIXME: need to validate somewhere */ - r = svm_range_map_to_gpus(prange, true); - if (r) - pr_debug("failed %d map 0x%p [0x%lx 0x%lx]\n", - r, svms, prange->start, prange->last); + /* TODO: implement deferred validation and mapping */ break; default: WARN_ONCE(1, "Unknown prange 0x%p work op %d\n", prange, From patchwork Mon Mar 22 10:58:58 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154545 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D79EFC433C1 for ; Mon, 22 Mar 2021 11:09:13 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A2D0B6191A for ; Mon, 22 Mar 2021 11:09:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A2D0B6191A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 090966E0C5; Mon, 22 Mar 2021 11:08:34 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2045.outbound.protection.outlook.com [40.107.243.45]) by gabe.freedesktop.org (Postfix) with ESMTPS id 567B06E11F; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=oW8IEqFehzF6tnete/0TAgXDSg1qU68biCHhpcF1KnRGgppTBHP1/yYyjoNrali4anBrEjNzZ0yJz/wv5Q15s+q0g/XDxoaiEhPIKHEW8nIzugGe8wU5SWpxe/x4+jgqmuCY93XrFI9Ly6W2rlfpzbpmXdLhqOLb+pFR88lNqepWLVrVz247vUQKoCpgl8Dh1S2RuXW2nWaZICXzyTH/d9FWERga08YupQEqlL9zooiIdMvr7bGERJe3ABgfGWFmqLU1k5CLUSae94KkKB2dXzLOX642n0U2YtA4tnr8smSc4ZZP3SJOBefdTR1QUEQK+ZOgplELEgW8mDLLDV0org== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E4TUZyjnEs4AAf4tGf2dhJLnXpsdr0sRrv2Mc1wI4A8=; b=V0CE/UGTkRCc93fMGfwSmW+op9e2NvVEZ6PJjKxBoMTe+CfO2RMSRUH3LMHtAp96fPvnW9UP7R4aZDTPM4sBu08rL45qpKPocYfpinPMHWjagffm5xGyNF32kTBHcF9QGrzJTdwOkikigiovugK4168pTl9iPprktpwFfMGW68Zzqo+NAojxw87tpWOPWTQj0ioPfMSZtLuL9Q2ywUsiOnNN9ke93t09xiS3HC9w9vojcMwm1CRCXu+KFGMuGya+A3E624SpjtPePnfKwcwei4V11tPjpS5A/VhpxJbLgDwQsb3rjOW7u88DWfV0nbgzNlYncvuZ96zo7kkXKQO20Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=E4TUZyjnEs4AAf4tGf2dhJLnXpsdr0sRrv2Mc1wI4A8=; b=z9kdmCTka7V243Uchy/qvcbY3Kl1f0W5+0KbjvaF5Tu6NHnc0nk9E6ibJVr6q7MLkBjhkb8oXDnTzojHgdJwPpSMd2blRNb8lhUbjKIsDG35UZVnwRMknShC0goTa5AkkBR4F7V8TJHl7mDYpQ/9sMDa6b0507ueWEf1R+logX4= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:23 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:23 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 42/44] drm/amdkfd: Allow invalid pages in migration.src Date: Mon, 22 Mar 2021 06:58:58 -0400 Message-Id: <20210322105900.14068-43-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:51 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 840e5017-3ff8-4726-ae53-08d8ed22bc1c X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:6790; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: W8xNFpO93iHYqkdHFcG2Ka9gxVqBcAQqoWE/ymtKL/H6Ia0HqoyZNBVSYWxQZHhIOhngShhvh6BfVe3FEnD6drYKo014UmGqaPrytC7tjR4JM5mgs6J6btVg+IRl5hRxlcj44IsV5DhI9uRjX0CaJgmWck8PFg3wuzckbOcW9um1zKDYRGFqq1Gwk51Yfa259ThT7zzSEOIBSJ8rFvNI1H1fVTcrKxMG11y3LAfGQo/73DE7wKPODXX5L5h4FngOXfUbjHeKTWYOoi2HRSLjCNkHS8HVYA+9XSg+h3HgjjsEV/IQETzpDJ8joaZOUJ/ag2d9iXYj0+2YpDSewosGkKSnODiKMvz8NjCxQEeo+kscjueRNaO6vMquDFSPHwSHhxR7QYODH8TXaZUJqSK8VMNjQ2kQHywoYPOWeuq/ErXWkxFm+TF6CzXjSHR1ex6+IN2Kvvovm8ory16iXk8dgQp4hBdfcT/XifBfuhJidJj/giFEKcQVLd05HkODmB5ZyMYAFI9IRnHY7DKe+W20gvwVj2impnY09Z5I+qFEMt2u3uQZ5HVHgqIhCf9aFFGvJjoAnhXiNx0BNf07c0UyoEL4TEmGFXckr0X04Cf43OGR+pvLGhicML4qmafEMmBG3ewHxo1I3xRetqV1CNc38zmmyiiTLq/VAluZJm2uzIY= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(366004)(450100002)(1076003)(36756003)(38100700001)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(8936002)(498600001)(2616005)(956004)(6486002)(2906002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: MSV3yHoLos7uEGP2WbWg2KHnyLrBr2C5Zu9h6+UQXegyqUaa+2w9QaWlxljXpKfmd+wTG1G0MvqP++YrkbznlkNs8RBhSrRHgMXbKrikHGhKjXN7X2bmZCP1SZ+1rDWs2ge4kkT/6sbAXAzgylyIdOeZ6NeuDeSFlAtbviiRf2hGi+foscjM/9Z0o7eD+TiCn0AMzIQOqDVJyeOZ9wgN1D/v0YNDzS7Rfgl+tjGcTRymL1Y3qpoBoKi1Ne8NE3OIOiLgUwEIpaZQBWc7UA8EgFmhsYQZrPvxjP/9OxUDaFr7Z33UdD1NdDbIKq3gKEHAa2TQzG7zmY6i72Ee+0Pepy/DqEdZjQyWdHirXxYMb6I+Vif3f2EV1ZXsTXg74IqZc8ldwvI8s71oDnlJBQ1wd/N7uUsIIY5/fXYqYdaZufKkTdX0d/VIDpdHEHH2Q0E6J3yUwHMWiE9bHRJ/GKMo0yJDXiFBALShMrtGQv/gjIKQC5l9+CTMYL/I9KxbYnHdce454T70ePNYhIQKOGwuj1tFanojxLKsYguNKKVyQaB6tQHyJSHoFp8b/Obzf4pk+ZJuBfUpqDwA3K3D9xVwiOimaU8794IGw0lTjh4oWyyLFrdLOOsoh18sAnshiA/fzdYpoetkCKQACoAQDHML/WMgPY30lgd3TZqlRzWTsyFB8ZL9qL6jLlZ6PtCPOApevqLsWzARfu/GT/Z6Hp70/Pks37B8IIs3Xi4yoLGtJgLX8QVPQNb3vjdF9OPZfRXCSGRN+6z8MupgIcuz7mPKgxsFzoy38ejBKTyJRwSmGJP3zF7MoAH4P+CtPjNRhu+b+5R5uiw6DHuCnKVzD85D9ejMwX3SCTFcy9WJVheMFzV+4tx7UngJT9gBo3pbqAzJ1kre/OAT77brFBJmhXtww/asHTspL+XrJcRFprUn4LyZf7RXDSexXbk6xntbxAEnIyQEVirDDiYA6/k1HYo+4PvmGfWjhZmSHEpIL8oMF5nFL9kCAGdRR8SlihCC7nLyf2B8gZ9COd/g68VU9ZYrGoLeLJeqxP9Lh0dl0JowlRRKkX5S0mUigzW+vVgGBdnt0HgyTp3DTgFtqcesQINKKMlbGRHJOewnXn/0wF1Lm2JXga7Gc+jAOMKDmpFMkRHfCinPtiC1G0mcTTn/043G9RFXVCOB4tNzeVAWBX7QNgKf2i+kWLxLwEtlM1cV5wAK6wMSoGThsitd6OIKsUJdifY8Qg91MCzcH/XfCC0KAdZUSRw18gMsgjkdBy9BcFbluyxyXpZwYh0hp3YnIXXFKbEnAPTTKnvU9pfDaWwImvrUtPBLD6GC5p7pgPREvBSi X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 840e5017-3ff8-4726-ae53-08d8ed22bc1c X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:51.5653 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: IgJ1QnFB3iyCWwUqVJdGH2AM033kEvmkpDURnWqXdcTg8aLth3KktTNnxRRwRGpO5idqF2MD479kBX/aQuTqDA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This can happen when syste memory page were never allocated. Skip them during the migration. 0-initialize the BO. Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 50 ++++++++++++++++++------ 1 file changed, 38 insertions(+), 12 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 6748c5db64f5..87561b907543 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -310,7 +310,7 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange, src = scratch; dst = (uint64_t *)(scratch + npages); - r = svm_range_vram_node_new(adev, prange, false); + r = svm_range_vram_node_new(adev, prange, true); if (r) { pr_debug("failed %d get 0x%llx pages from vram\n", r, npages); goto out; @@ -328,17 +328,6 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange, for (i = j = 0; i < npages; i++) { struct page *spage; - spage = migrate_pfn_to_page(migrate->src[i]); - src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE, DMA_TO_DEVICE); - r = dma_mapping_error(dev, src[i]); - if (r) { - pr_debug("failed %d dma_map_page\n", r); - goto out_free_vram_pages; - } - - pr_debug("dma mapping src to 0x%llx, page_to_pfn 0x%lx\n", - src[i] >> PAGE_SHIFT, page_to_pfn(spage)); - dst[i] = vram_addr + (j << PAGE_SHIFT); migrate->dst[i] = svm_migrate_addr_to_pfn(adev, dst[i]); svm_migrate_get_vram_page(prange, migrate->dst[i]); @@ -346,6 +335,43 @@ svm_migrate_copy_to_vram(struct amdgpu_device *adev, struct svm_range *prange, migrate->dst[i] = migrate_pfn(migrate->dst[i]); migrate->dst[i] |= MIGRATE_PFN_LOCKED; + if (migrate->src[i] & MIGRATE_PFN_VALID) { + spage = migrate_pfn_to_page(migrate->src[i]); + src[i] = dma_map_page(dev, spage, 0, PAGE_SIZE, + DMA_TO_DEVICE); + r = dma_mapping_error(dev, src[i]); + if (r) { + pr_debug("failed %d dma_map_page\n", r); + goto out_free_vram_pages; + } + } else { + if (j) { + j--; + r = svm_migrate_copy_memory_gart( + adev, src + i - j, + dst + i - j, j + 1, + FROM_RAM_TO_VRAM, + mfence); + if (r) + goto out_free_vram_pages; + offset = j; + vram_addr = (node->start + offset) << PAGE_SHIFT; + j = 0; + } + offset++; + vram_addr += PAGE_SIZE; + if (offset >= node->size) { + node++; + pr_debug("next node size 0x%llx\n", node->size); + vram_addr = node->start << PAGE_SHIFT; + offset = 0; + } + continue; + } + + pr_debug("dma mapping src to 0x%llx, page_to_pfn 0x%lx\n", + src[i] >> PAGE_SHIFT, page_to_pfn(spage)); + if (j + offset >= node->size - 1 && i < npages - 1) { r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j + 1, From patchwork Mon Mar 22 10:58:59 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C40F6C433E6 for ; Mon, 22 Mar 2021 11:09:23 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 67EA561919 for ; Mon, 22 Mar 2021 11:09:23 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 67EA561919 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 76BAB6E425; Mon, 22 Mar 2021 11:08:37 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2045.outbound.protection.outlook.com [40.107.243.45]) by gabe.freedesktop.org (Postfix) with ESMTPS id E46E96E226; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XtR0l+aKFOuWgtGScWtkAFl7ijxu8V1ldxWxA1VzRME9Y9gCrTOUCBSSdXg3XC3reu7O6XPbP5Jdqg55vK0w7nGf6SWxBXHbtrSZ4ZlGtpvOj0cCOWzbCEeTTzMynmhLoBsdCyg/chJAfOCliSxXo/anDZT9z0KKtVaDBEucjj50dqb2MIxIiKKWTOoiHbcLfyP4yZ55oJJUNpMiXXrVODcyMZnIHzvOP/udQcanYbFNDBWy8fhP92QeFaL0k44lDiyDIuK24nrVXudWl9EBwOs3D3Wt+piq+/Nw0Em+oILoZCLPhqLG62xaX02suhpGDMCstEIRxBrxZ4onppA8oQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ScsvdE99eeiugznlhS42lTQYOKbMlIoPsISEjrkiK6s=; b=YFod+NSGNbzbJDqB8KO2GSUkCK4Ih4sJkPoxjiVfkaBDyPTiT4p0TyKoCn7B58/qpbdKbh8XDMdFjqA9f2TJfHb/6SvhMXShni8jti3b1mCq3K3+WhGRoyJH7v1UCmcH9ywAcK/36PvAVKi3NyBLAnCCUbjA/RqnirNyDTif7VEy2jGQBEkMyEWuVchIvMChziAuPYH6sPzX3go7nWUtBE2hwljtNcVJXielaeLFBQAg5qsMyg0ABydQIzMjf5fJdR0//rN3QHjQrkCW5uAsPhdnjMNANXOFIPOrJPZ1j9JuOqYfHnfMu3iuVw6Bxz0zW1xPEnXbaPDAe0aosJ76LQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=ScsvdE99eeiugznlhS42lTQYOKbMlIoPsISEjrkiK6s=; b=i7KXgGSg+T4Qj/zjbJbwfJ/aN0bBdtOc+Aw2elfI4GWtMD8MxSCypJplWRtarIaAJxmwUd2PLfaHabFIDxx86g6FVkYxVbvYjXHduJ/ppo/ewW8IJzpvJx821sHfPVl1f7KrlqIslvALhhJPhQ1YF7KuXHBHfnaNEBnOkwoizE0= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:23 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:23 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 43/44] drm/amdkfd: Correct locking during migration and mapping Date: Mon, 22 Mar 2021 06:58:59 -0400 Message-Id: <20210322105900.14068-44-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:51 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: a9c7d274-b24c-4c96-93f1-08d8ed22bc67 X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: bURxYpGoAswTPyYQwMLO8hoo2j/8SDRWkdmjkIHkqdn6sG6EtmEezplRN/katlmH8rcZbEgkIdbvuUz+1y9N3o92KMaGS5qXf3qGot8v8LZPGOkyqtwhk6C0tfk9usVPLoL+6ogKegbezvTXB1E/f0F+drjfi2/cGgjy7jzVRY+nq4PSC9/mrO6mCtGXF/6+mJvsp/ZXYM2i6RWOK9fCQxwipGLx7BQE490D1jRkUau+OVwfDsFCID2wnTUJlFJ/6KAvL1c8Z4EGY6iAF1igMkkS8wMRthi6BzgWiLPOnPj5Gzxc/IJFt2+YZUMQ7co1vHHApAcXpR6udNZeE8o3QgLFFZoGO5qNgk19jl9gJGfGoraA7H9VumFATbTVeAST/ifa2+7R5fmTqzdmkM4Cm0jDgzSjYNTfkaSHumZ5PRTzo6KfHmQJBB3Dq78DTeDc8e9WMUGTMsSqOxBjR98di3wkrm+hVrNTBZtnTh8gZoEYi00IBI2xN2OkKO4Hh9Us2hENLw8Apzd4bVvn2NwEKXbsM2qlbMG4G3fJ0Ga775t+m0PHfjMGEXWghRGEN3bHrX1vSch1ng41E6jkequ+za9uGxpjkWU/ed+CQ6R4+kgVIc0pcJUf2Jf/SB9aXlOKdCuKsZqIikqElmbClzTu4w== X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(450100002)(1076003)(36756003)(38100700001)(30864003)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: uO2THRP66z7Ws1mXtWbb+bVhRe03dovp2VjQKOrjnpT4sUdcn4eSrSTk+3xL2zJ2a06mHP6JUrury/nmkmIlwc+2gkJuZgYkSzxvxvEFLDKUVrKsP43Sa9hF9gYHnYPYiL+V/fvbL00qVNLVfT5M+zSISEGK035Y5qDJL1mC7CnxiG3uIvsfNSEj9qFrAhMLA9tWGSG2i6ZJekLME7J9Sn5nNIJUZs+plQZJUuaMxbeMy5IHpKDUJF9loq/k7ma/4XKLXsYNOjqx/6+FtGol3mWh41JGPHI9S2Pc93efXIOkTRX1TwCBQDtoG4etB1JRlXpjrKsZjM+5/YM1S8AdVfVqKxKqBDMWFBCb6k5vUYFOq2SpgQ8oAhhZODbCaanwgHtGNcTXAoA2btR2qAR6vIJ1tZjLqHSHAF/KDIAg8euE9bG76/ZuCAa2Vt1UW7Xb1tiz1TAs/8n1jvsgLkjyj1QhNp8pSzZv6vAv/Gqzr8HNvU3MS01HDRkMHoB4t0hCDGIyokj9D4L1x6MScgWmAIxgoqEx2LWy52ztITQa21kxSmptpbHJz1HuBr8BXJBdJSmpjz47tiHtI4EHtWirvdVaI/3BQxz7a8JyQIL1sgF19NDGa103gi3lJBm6qyIXHfbDIYT1zZxGFDbzi7Y9edltIb4z5uc26mlwt9Jcgm3WENcFGWjJMJcCoyaIibLEsDOvzZI7PmweYj5YlMBXl7y/ctz38mh6zm9L0ZsrBJHNjCbPxHSRtTytHtQKPzQjUhpWquPYKdi6tK3p5oLEdsdi1UHTu77ythHGO+rU6DVVY2UlcS3oUUMJ8Y0bIaJ5ARYYRfSDTH/3Cjj/sxJLG8RgpIyLue5I5DIBntp14qLx4XuQXkA9BkybS8GkXh+nlilFzjeX5/HjYj0JDTdcM7mRaHSVacjWffeIH+QW14qYnHwzAzUy1e146ML0ULvLnqcuEZoahDD5VBa4LehsfSKx0Ab1iv3Yz0cfJJBxkG51bZ2PIq0gpdmYRvKiyNO8dqwCG1kp7Pq4/fWM6ONQ379lbXB7MVvg8IoNFSKMYaB4yYL77YsTRks1GbebP177smpt40JROF6ofrc4ThAOmIk/5mrcPQ4Bjh/36x84PZc+CjIqZ+obh9igE9MXgA9tGyOEWRmtiLOKCGo/G+XUtGIWXf/PldjPJ3DXxwfj/krmvJ1nT5PqgNP/ikSGF3fXWIu6rD8POOlexgiRT5iCmhHDejRgaXdQbG1qiox16YD+CgLPnNQLadTNtDF7NBhHZoOAomBCx5TTV634WEGEAb5dT85lrSwgcniKkSndxE859/JXbPX9YnRWLS6dtzZc X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: a9c7d274-b24c-4c96-93f1-08d8ed22bc67 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:52.0452 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: 6irYgKIYOMd5KNvbf0XRuL61Rl7ME37MTHBkdJc/mPcWJ7bmXcMXU5H/w5QdLG7BpWN5DZ1ktJzbnyLjKvfRlA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This fixes potential race conditions between any code that validates and maps SVM ranges and MMU notifiers. The whole sequence is encapsulated in svm_range_validate_and_map. The page_addr and hmm_range structures are not useful outside that function, so they were removed from struct svm_range. Validation of system memory pages before migration to VRAM is maintained as an explicit workaround. It should not be needed, but without it there are still some page locking deadlocks to be investigated. Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 6 +- drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 573 ++++++++++++----------- drivers/gpu/drm/amd/amdkfd/kfd_svm.h | 9 +- 3 files changed, 310 insertions(+), 278 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 87561b907543..4d79d69d8aac 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -466,9 +466,6 @@ svm_migrate_vma_to_vram(struct amdgpu_device *adev, struct svm_range *prange, migrate_vma_finalize(&migrate); } - kvfree(prange->pages_addr); - prange->pages_addr = NULL; - svm_range_dma_unmap(adev->dev, scratch, 0, npages); svm_range_free_dma_mappings(prange); @@ -513,6 +510,9 @@ svm_migrate_ram_to_vram(struct svm_range *prange, uint32_t best_loc, pr_debug("svms 0x%p [0x%lx 0x%lx] to gpu 0x%x\n", prange->svms, prange->start, prange->last, best_loc); + /* FIXME: workaround for page locking bug with invalid pages */ + svm_range_prefault(prange, mm); + start = prange->start << PAGE_SHIFT; end = (prange->last + 1) << PAGE_SHIFT; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index aedb2c84131e..0a6e28f0dcaf 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -39,6 +39,16 @@ */ #define AMDGPU_SVM_RANGE_RETRY_FAULT_PENDING 2000 +struct svm_validate_context { + struct kfd_process *process; + struct svm_range *prange; + bool intr; + unsigned long bitmap[MAX_GPU_INSTANCE]; + struct ttm_validate_buffer tv[MAX_GPU_INSTANCE+1]; + struct list_head validate_list; + struct ww_acquire_ctx ticket; +}; + static void svm_range_evict_svm_bo_worker(struct work_struct *work); static bool svm_range_cpu_invalidate_pagetables(struct mmu_interval_notifier *mni, @@ -118,17 +128,14 @@ static void svm_range_remove_notifier(struct svm_range *prange) } static int -svm_range_dma_map(struct device *dev, dma_addr_t **dma_addr, - unsigned long *pages_addr, uint64_t npages) +svm_range_dma_map_dev(struct device *dev, dma_addr_t **dma_addr, + unsigned long *hmm_pfns, uint64_t npages) { enum dma_data_direction dir = DMA_BIDIRECTIONAL; dma_addr_t *addr = *dma_addr; struct page *page; int i, r; - if (!pages_addr) - return 0; - if (!addr) { addr = kvmalloc_array(npages, sizeof(*addr), GFP_KERNEL | __GFP_ZERO); @@ -142,7 +149,7 @@ svm_range_dma_map(struct device *dev, dma_addr_t **dma_addr, "leaking dma mapping\n")) dma_unmap_page(dev, addr[i], PAGE_SIZE, dir); - page = hmm_pfn_to_page(pages_addr[i]); + page = hmm_pfn_to_page(hmm_pfns[i]); addr[i] = dma_map_page(dev, page, 0, PAGE_SIZE, dir); r = dma_mapping_error(dev, addr[i]); if (r) { @@ -155,6 +162,37 @@ svm_range_dma_map(struct device *dev, dma_addr_t **dma_addr, return 0; } +static int +svm_range_dma_map(struct svm_range *prange, unsigned long *bitmap, + unsigned long *hmm_pfns) +{ + struct kfd_process *p; + uint32_t gpuidx; + int r; + + p = container_of(prange->svms, struct kfd_process, svms); + + for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { + struct kfd_process_device *pdd; + struct amdgpu_device *adev; + + pr_debug("mapping to gpu idx 0x%x\n", gpuidx); + pdd = kfd_process_device_from_gpuidx(p, gpuidx); + if (!pdd) { + pr_debug("failed to find device idx %d\n", gpuidx); + return -EINVAL; + } + adev = (struct amdgpu_device *)pdd->dev->kgd; + + r = svm_range_dma_map_dev(adev->dev, &prange->dma_addr[gpuidx], + hmm_pfns, prange->npages); + if (r) + break; + } + + return r; +} + void svm_range_dma_unmap(struct device *dev, dma_addr_t *dma_addr, unsigned long offset, unsigned long npages) { @@ -207,7 +245,6 @@ static void svm_range_free(struct svm_range *prange) svm_range_vram_node_free(prange); svm_range_free_dma_mappings(prange); - kvfree(prange->pages_addr); mutex_destroy(&prange->lock); mutex_destroy(&prange->migrate_mutex); kfree(prange); @@ -258,45 +295,6 @@ svm_range *svm_range_new(struct svm_range_list *svms, uint64_t start, return prange; } -/** - * svm_range_validate_ram - get system memory pages of svm range - * - * @mm: the mm_struct of process - * @prange: the range struct - * - * After mapping system memory to GPU, system memory maybe invalidated anytime - * during application running, we use HMM callback to sync GPU with CPU page - * table update, so we don't need use lock to prevent CPU invalidation and check - * hmm_range_get_pages_done return value. - * - * Return: - * 0 - OK, otherwise error code - */ -static int -svm_range_validate_ram(struct mm_struct *mm, struct svm_range *prange) -{ - int r; - - r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL, - prange->start << PAGE_SHIFT, - prange->npages, &prange->hmm_range, - false, true); - if (r) { - pr_debug("failed %d to get svm range pages\n", r); - return r; - } - - kvfree(prange->pages_addr); - svm_range_free_dma_mappings(prange); - prange->pages_addr = prange->hmm_range->hmm_pfns; - prange->hmm_range->hmm_pfns = NULL; - - amdgpu_hmm_range_get_pages_done(prange->hmm_range); - prange->hmm_range = NULL; - - return 0; -} - static bool svm_bo_ref_unless_zero(struct svm_range_bo *svm_bo) { if (!svm_bo || !kref_get_unless_zero(&svm_bo->kref)) @@ -568,57 +566,13 @@ svm_range_get_adev_by_id(struct svm_range *prange, uint32_t gpu_id) return (struct amdgpu_device *)pdd->dev->kgd; } -static int svm_range_validate_vram(struct svm_range *prange) -{ - struct amdgpu_device *adev; - int r; - - pr_debug("svms 0x%p [0x%lx 0x%lx] actual_loc 0x%x\n", prange->svms, - prange->start, prange->last, prange->actual_loc); - - if (prange->ttm_res) { - pr_debug("validation skipped after migration\n"); - return 0; - } - - adev = svm_range_get_adev_by_id(prange, prange->actual_loc); - if (!adev) { - pr_debug("failed to get device by id 0x%x\n", - prange->actual_loc); - return -EINVAL; - } - - r = svm_range_vram_node_new(adev, prange, true); - if (r) - pr_debug("failed %d to alloc vram\n", r); - - return r; -} - -static int -svm_range_validate(struct mm_struct *mm, struct svm_range *prange) +static int svm_range_bo_validate(void *param, struct amdgpu_bo *bo) { - struct kfd_process *p; - int r; - - pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx] actual loc 0x%x\n", - prange->svms, prange, prange->start, prange->last, - prange->actual_loc); - - p = container_of(prange->svms, struct kfd_process, svms); - - if (!prange->actual_loc) - r = svm_range_validate_ram(mm, prange); - else - r = svm_range_validate_vram(prange); - - if (!r) - prange->validate_timestamp = ktime_to_us(ktime_get()); + struct ttm_operation_ctx ctx = { false, false }; - pr_debug("svms 0x%p [0x%lx 0x%lx] ret %d\n", prange->svms, - prange->start, prange->last, r); + amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_VRAM); - return r; + return ttm_bo_validate(&bo->tbo, &bo->placement, &ctx); } static int @@ -722,7 +676,7 @@ svm_range_apply_attrs(struct kfd_process *p, struct svm_range *prange, * svm_range_debug_dump - print all range information from svms * @svms: svm range list header * - * debug output svm range start, end, pages_addr, prefetch location from svms + * debug output svm range start, end, prefetch location from svms * interval tree and link list * * Context: The caller must hold svms->lock @@ -733,25 +687,23 @@ static void svm_range_debug_dump(struct svm_range_list *svms) struct svm_range *prange; pr_debug("dump svms 0x%p list\n", svms); - pr_debug("range\tstart\tpage\tend\t\tpages_addr\tlocation\n"); + pr_debug("range\tstart\tpage\tend\t\tlocation\n"); list_for_each_entry(prange, &svms->list, list) { - pr_debug("0x%p 0x%lx\t0x%llx\t0x%llx\t0x%lx\t0x%x\n", + pr_debug("0x%p 0x%lx\t0x%llx\t0x%llx\t0x%x\n", prange, prange->start, prange->npages, prange->start + prange->npages - 1, - prange->pages_addr ? *prange->pages_addr : 0, prange->actual_loc); } pr_debug("dump svms 0x%p interval tree\n", svms); - pr_debug("range\tstart\tpage\tend\t\tpages_addr\tlocation\n"); + pr_debug("range\tstart\tpage\tend\t\tlocation\n"); node = interval_tree_iter_first(&svms->objects, 0, ~0ULL); while (node) { prange = container_of(node, struct svm_range, it_node); - pr_debug("0x%p 0x%lx\t0x%llx\t0x%llx\t0x%lx\t0x%x\n", + pr_debug("0x%p 0x%lx\t0x%llx\t0x%llx\t0x%x\n", prange, prange->start, prange->npages, prange->start + prange->npages - 1, - prange->pages_addr ? *prange->pages_addr : 0, prange->actual_loc); node = interval_tree_iter_next(node, 0, ~0ULL); } @@ -817,12 +769,6 @@ svm_range_split_pages(struct svm_range *new, struct svm_range *old, return r; } - r = svm_range_split_array(&new->pages_addr, &old->pages_addr, - sizeof(*old->pages_addr), old->start, npages, - new->start, new->npages); - if (r) - return r; - return 0; } @@ -860,7 +806,7 @@ svm_range_split_nodes(struct svm_range *new, struct svm_range *old, * @start: the old range adjust to start address in pages * @last: the old range adjust to last address in pages * - * Copy system memory pages, pages_addr or vram ttm_res in old range to new + * Copy system memory dma_addr or vram ttm_res in old range to new * range from new_start up to size new->npages, the remaining old range is from * start to last * @@ -882,15 +828,16 @@ svm_range_split_adjust(struct svm_range *new, struct svm_range *old, return -EINVAL; } - if (old->pages_addr) - r = svm_range_split_pages(new, old, start, last); - else if (old->actual_loc && old->ttm_res) - r = svm_range_split_nodes(new, old, start, last); - else - WARN_ONCE(1, "split adjust invalid pages_addr and nodes\n"); + r = svm_range_split_pages(new, old, start, last); if (r) return r; + if (old->actual_loc && old->ttm_res) { + r = svm_range_split_nodes(new, old, start, last); + if (r) + return r; + } + old->npages = last - start + 1; old->start = start; old->last = last; @@ -1088,9 +1035,9 @@ svm_range_get_pte_flags(struct amdgpu_device *adev, struct svm_range *prange) /* Apply ASIC specific mapping flags */ amdgpu_gmc_get_vm_pte(adev, &prange->mapping, &pte_flags); - pr_debug("svms 0x%p [0x%lx 0x%lx] vram %d system %d PTE flags 0x%llx\n", + pr_debug("svms 0x%p [0x%lx 0x%lx] vram %d PTE flags 0x%llx\n", prange->svms, prange->start, prange->last, - prange->ttm_res ? 1:0, prange->pages_addr ? 1:0, pte_flags); + prange->ttm_res ? 1:0, pte_flags); return pte_flags; } @@ -1153,56 +1100,18 @@ svm_range_unmap_from_gpus(struct svm_range *prange, unsigned long start, return r; } -static int svm_range_bo_validate(void *param, struct amdgpu_bo *bo) -{ - struct ttm_operation_ctx ctx = { false, false }; - - amdgpu_bo_placement_from_domain(bo, AMDGPU_GEM_DOMAIN_VRAM); - - return ttm_bo_validate(&bo->tbo, &bo->placement, &ctx); -} - static int svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, - struct svm_range *prange, dma_addr_t *pages_addr, - bool reserve_vm, struct amdgpu_device *bo_adev, - struct dma_fence **fence) + struct svm_range *prange, dma_addr_t *dma_addr, + struct amdgpu_device *bo_adev, struct dma_fence **fence) { - struct ttm_validate_buffer tv[2]; - struct ww_acquire_ctx ticket; struct amdgpu_bo_va bo_va; - struct list_head list; uint64_t pte_flags; int r = 0; pr_debug("svms 0x%p [0x%lx 0x%lx]\n", prange->svms, prange->start, prange->last); - if (reserve_vm) { - INIT_LIST_HEAD(&list); - - tv[0].bo = &vm->root.base.bo->tbo; - tv[0].num_shared = 4; - list_add(&tv[0].head, &list); - if (prange->svm_bo && prange->ttm_res) { - tv[1].bo = &prange->svm_bo->bo->tbo; - tv[1].num_shared = 1; - list_add(&tv[1].head, &list); - } - r = ttm_eu_reserve_buffers(&ticket, &list, true, NULL); - if (r) { - pr_debug("failed %d to reserve bo\n", r); - goto out; - } - - r = amdgpu_vm_validate_pt_bos(adev, vm, svm_range_bo_validate, - NULL); - if (r) { - pr_debug("failed %d validate pt bos\n", r); - goto unreserve_out; - } - } - if (prange->svm_bo && prange->ttm_res) { bo_va.is_xgmi = amdgpu_xgmi_same_hive(adev, bo_adev); prange->mapping.bo_va = &bo_va; @@ -1220,10 +1129,10 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, prange->mapping.offset, prange->ttm_res ? prange->ttm_res->mm_node : NULL, - pages_addr, &vm->last_update); + dma_addr, &vm->last_update); if (r) { pr_debug("failed %d to map to gpu 0x%lx\n", r, prange->start); - goto unreserve_out; + goto out; } @@ -1231,23 +1140,21 @@ svm_range_map_to_gpu(struct amdgpu_device *adev, struct amdgpu_vm *vm, if (r) { pr_debug("failed %d to update directories 0x%lx\n", r, prange->start); - goto unreserve_out; + goto out; } if (fence) *fence = dma_fence_get(vm->last_update); -unreserve_out: - prange->mapping.bo_va = NULL; - if (reserve_vm) - ttm_eu_backoff_reservation(&ticket, &list); out: + prange->mapping.bo_va = NULL; + return r; } -int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) +static int svm_range_map_to_gpus(struct svm_range *prange, + unsigned long *bitmap, bool wait) { - DECLARE_BITMAP(bitmap, MAX_GPU_INSTANCE); struct kfd_process_device *pdd; struct amdgpu_device *bo_adev; struct amdgpu_device *adev; @@ -1262,29 +1169,6 @@ int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) bo_adev = NULL; p = container_of(prange->svms, struct kfd_process, svms); - if (p->xnack_enabled) { - bitmap_copy(bitmap, prange->bitmap_aip, MAX_GPU_INSTANCE); - - /* If prefetch range to GPU, or GPU retry fault migrate range to - * GPU, which has ACCESS attribute to the range, create mapping - * on that GPU. - */ - if (prange->actual_loc) { - gpuidx = kfd_process_gpuidx_from_gpuid(p, - prange->actual_loc); - if (gpuidx < 0) { - WARN_ONCE(1, "failed get device by id 0x%x\n", - prange->actual_loc); - return -EINVAL; - } - if (test_bit(gpuidx, prange->bitmap_access)) - bitmap_set(bitmap, gpuidx, 1); - } - } else { - bitmap_or(bitmap, prange->bitmap_access, prange->bitmap_aip, - MAX_GPU_INSTANCE); - } - for_each_set_bit(gpuidx, bitmap, MAX_GPU_INSTANCE) { pr_debug("mapping to gpu idx 0x%x\n", gpuidx); pdd = kfd_process_device_from_gpuidx(p, gpuidx); @@ -1304,14 +1188,9 @@ int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) continue; } - r = svm_range_dma_map(adev->dev, &prange->dma_addr[gpuidx], - prange->pages_addr, prange->npages); - if (r) - break; - r = svm_range_map_to_gpu(adev, pdd->vm, prange, - prange->dma_addr[gpuidx], reserve_vm, - bo_adev, &fence); + prange->dma_addr[gpuidx], bo_adev, + wait ? &fence : NULL); if (r) break; @@ -1329,6 +1208,207 @@ int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm) return r; } +int svm_range_reserve_bos(struct svm_validate_context *ctx) +{ + struct kfd_process_device *pdd; + struct amdgpu_device *adev; + struct amdgpu_vm *vm; + uint32_t gpuidx; + int r; + + INIT_LIST_HEAD(&ctx->validate_list); + for_each_set_bit(gpuidx, ctx->bitmap, MAX_GPU_INSTANCE) { + pdd = kfd_process_device_from_gpuidx(ctx->process, gpuidx); + if (!pdd) { + pr_debug("failed to find device idx %d\n", gpuidx); + return -EINVAL; + } + adev = (struct amdgpu_device *)pdd->dev->kgd; + vm = pdd->vm; + + ctx->tv[gpuidx].bo = &vm->root.base.bo->tbo; + ctx->tv[gpuidx].num_shared = 4; + list_add(&ctx->tv[gpuidx].head, &ctx->validate_list); + } + if (ctx->prange->svm_bo && ctx->prange->ttm_res) { + ctx->tv[MAX_GPU_INSTANCE].bo = &ctx->prange->svm_bo->bo->tbo; + ctx->tv[MAX_GPU_INSTANCE].num_shared = 1; + list_add(&ctx->tv[MAX_GPU_INSTANCE].head, &ctx->validate_list); + } + + r = ttm_eu_reserve_buffers(&ctx->ticket, &ctx->validate_list, + ctx->intr, NULL); + if (r) { + pr_debug("failed %d to reserve bo\n", r); + return r; + } + + for_each_set_bit(gpuidx, ctx->bitmap, MAX_GPU_INSTANCE) { + pdd = kfd_process_device_from_gpuidx(ctx->process, gpuidx); + if (!pdd) { + pr_debug("failed to find device idx %d\n", gpuidx); + r = -EINVAL; + goto unreserve_out; + } + adev = (struct amdgpu_device *)pdd->dev->kgd; + + r = amdgpu_vm_validate_pt_bos(adev, pdd->vm, + svm_range_bo_validate, NULL); + if (r) { + pr_debug("failed %d validate pt bos\n", r); + goto unreserve_out; + } + } + + return 0; + +unreserve_out: + ttm_eu_backoff_reservation(&ctx->ticket, &ctx->validate_list); + return r; +} + +void svm_range_unreserve_bos(struct svm_validate_context *ctx) +{ + ttm_eu_backoff_reservation(&ctx->ticket, &ctx->validate_list); +} + +/* + * Validation+GPU mapping with concurrent invalidation (MMU notifiers) + * + * To prevent concurrent destruction or change of range attributes, the + * svm_read_lock must be held. The caller must not hold the svm_write_lock + * because that would block concurrent evictions and lead to deadlocks. To + * serialize concurrent migrations or validations of the same range, the + * prange->migrate_mutex must be held. + * + * The range must be in the inverval tree and have an MMU notifier to catch + * concurrent invalidations of the virtual address range. This means it cannot + * be a child range. + * + * For VRAM ranges, the SVM BO must be allocated and valid (protected by its + * eviction fence. + * + * The following sequence ensures race-free validation and GPU mapping: + * + * 1. Reserve page table (and SVM BO if range is in VRAM) + * 2. hmm_range_fault to get page addresses (if system memory) + * 3. DMA-map pages (if system memory) + * 4-a. Take notifier lock + * 4-b. Check that pages still valid (mmu_interval_read_retry) + * 4-c. Check that the range was not split or otherwise invalidated + * 4-d. Update GPU page table + * 4.e. Release notifier lock + * 5. Release page table (and SVM BO) reservation + */ +static int svm_range_validate_and_map(struct mm_struct *mm, + struct svm_range *prange, + uint32_t gpuidx, bool intr, bool wait) +{ + struct svm_validate_context ctx; + struct hmm_range *hmm_range; + int r = 0; + + ctx.process = container_of(prange->svms, struct kfd_process, svms); + ctx.prange = prange; + ctx.intr = intr; + + if (gpuidx < MAX_GPU_INSTANCE) { + bitmap_zero(ctx.bitmap, MAX_GPU_INSTANCE); + bitmap_set(ctx.bitmap, gpuidx, 1); + } else if (ctx.process->xnack_enabled) { + bitmap_copy(ctx.bitmap, prange->bitmap_aip, MAX_GPU_INSTANCE); + + /* If prefetch range to GPU, or GPU retry fault migrate range to + * GPU, which has ACCESS attribute to the range, create mapping + * on that GPU. + */ + if (prange->actual_loc) { + gpuidx = kfd_process_gpuidx_from_gpuid(ctx.process, + prange->actual_loc); + if (gpuidx < 0) { + WARN_ONCE(1, "failed get device by id 0x%x\n", + prange->actual_loc); + return -EINVAL; + } + if (test_bit(gpuidx, prange->bitmap_access)) + bitmap_set(ctx.bitmap, gpuidx, 1); + } + } else { + bitmap_or(ctx.bitmap, prange->bitmap_access, + prange->bitmap_aip, MAX_GPU_INSTANCE); + } + + if (bitmap_empty(ctx.bitmap, MAX_GPU_INSTANCE)) + return 0; + + if (prange->actual_loc && !prange->ttm_res) { + struct amdgpu_device *adev; + + /* This should never happen. actual_loc gets set by + * svm_migrate_ram_to_vram after allocating a BO. + */ + WARN(1, "creating new VRAM BO during validation\n"); + adev = svm_range_get_adev_by_id(prange, prange->actual_loc); + if (!adev) { + pr_debug("failed to get device by id 0x%x\n", + prange->actual_loc); + return -EINVAL; + } + + r = svm_range_vram_node_new(adev, prange, true); + if (r) { + pr_debug("failed %d to allocate vram\n", r); + return r; + } + } + + svm_range_reserve_bos(&ctx); + + if (!prange->actual_loc) { + r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL, + prange->start << PAGE_SHIFT, + prange->npages, &hmm_range, + false, true); + if (r) { + pr_debug("failed %d to get svm range pages\n", r); + goto unreserve_out; + } + + r = svm_range_dma_map(prange, ctx.bitmap, + hmm_range->hmm_pfns); + if (r) { + pr_debug("failed %d to dma map range\n", r); + goto unreserve_out; + } + + prange->validated_once = true; + } + + svm_range_lock(prange); + if (!prange->actual_loc) { + if (amdgpu_hmm_range_get_pages_done(hmm_range)) { + r = -EAGAIN; + goto unlock_out; + } + } + if (!list_empty(&prange->child_list)) { + r = -EAGAIN; + goto unlock_out; + } + + r = svm_range_map_to_gpus(prange, ctx.bitmap, wait); + +unlock_out: + svm_range_unlock(prange); +unreserve_out: + svm_range_unreserve_bos(&ctx); + + if (!r) + prange->validate_timestamp = ktime_to_us(ktime_get()); + + return r; +} + /** * svm_range_list_lock_and_flush_work - flush pending deferred work * @@ -1401,20 +1481,12 @@ static void svm_range_restore_work(struct work_struct *work) */ mutex_lock(&prange->migrate_mutex); - r = svm_range_validate(mm, prange); - if (r) { - pr_debug("failed %d to validate [0x%lx 0x%lx]\n", r, - prange->start, prange->last); - - goto out_unlock; - } - - r = svm_range_map_to_gpus(prange, true); + r = svm_range_validate_and_map(mm, prange, MAX_GPU_INSTANCE, + false, true); if (r) - pr_debug("failed %d to map 0x%lx to gpu\n", r, + pr_debug("failed %d to map 0x%lx to gpus\n", r, prange->start); -out_unlock: mutex_unlock(&prange->migrate_mutex); if (r) goto out_reschedule; @@ -1498,7 +1570,9 @@ svm_range_evict(struct svm_range *prange, struct mm_struct *mm, } else { pr_debug("invalidate unmap svms 0x%p [0x%lx 0x%lx] from GPUs\n", prange->svms, start, last); + svm_range_lock(prange); svm_range_unmap_from_gpus(prange, start, last); + svm_range_unlock(prange); } return r; @@ -1512,17 +1586,6 @@ struct svm_range *svm_range_clone(struct svm_range *old) if (!new) return NULL; - if (old->pages_addr) { - new->pages_addr = kvmalloc_array(new->npages, - sizeof(*new->pages_addr), - GFP_KERNEL); - if (!new->pages_addr) { - kfree(new); - return NULL; - } - memcpy(new->pages_addr, old->pages_addr, - old->npages * sizeof(*old->pages_addr)); - } if (old->svm_bo) { new->ttm_res = old->ttm_res; new->offset = old->offset; @@ -2051,19 +2114,10 @@ svm_range_best_restore_location(struct svm_range *prange, return -1; } -/* FIXME: This function can race with MMU notifiers. MMU notifiers can - * invalidate the page addresses concurrently, so we may end up mapping - * invalid addresses here. We cannot hold the prange->lock (held in MMU - * notifier) while updating page tables because of lock dependencies, - * as SDMA page table updates need reservation locks. Only unmapping - * works without reservations. May need to hold the mmap_write_lock to - * prevent concurrent MMU notifiers. - */ int svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, uint64_t addr) { - struct amdgpu_device *bo_adev; struct mm_struct *mm = NULL; struct svm_range_list *svms; struct svm_range *prange; @@ -2134,23 +2188,7 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, } } - r = svm_range_validate(mm, prange); - if (r) { - pr_debug("failed %d to validate svms %p [0x%lx 0x%lx]\n", r, - svms, prange->start, prange->last); - goto out_unlock_range; - } - - if (prange->svm_bo && prange->ttm_res) - bo_adev = amdgpu_ttm_adev(prange->svm_bo->bo->tbo.bdev); - else - bo_adev = NULL; - - pr_debug("restoring svms 0x%p [0x%lx %lx] mapping, bo_adev is %s\n", - svms, prange->start, prange->last, - bo_adev ? "not NULL" : "NULL"); - - r = svm_range_map_to_gpus(prange, true); + r = svm_range_validate_and_map(mm, prange, MAX_GPU_INSTANCE, false, true); if (r) pr_debug("failed %d to map svms 0x%p [0x%lx 0x%lx] to gpus\n", r, svms, prange->start, prange->last); @@ -2366,6 +2404,27 @@ svm_range_best_prefetch_location(struct svm_range *prange) return best_loc; } +/* FIXME: This is a workaround for page locking bug when some pages are + * invalid during migration to VRAM + */ +void svm_range_prefault(struct svm_range *prange, struct mm_struct *mm) +{ + struct hmm_range *hmm_range; + int r; + + if (prange->validated_once) + return; + + r = amdgpu_hmm_range_get_pages(&prange->notifier, mm, NULL, + prange->start << PAGE_SHIFT, + prange->npages, &hmm_range, + false, true); + if (!r) { + amdgpu_hmm_range_get_pages_done(hmm_range); + prange->validated_once = true; + } +} + /* svm_range_trigger_migration - start page migration if prefetch loc changed * @mm: current process mm_struct * @prange: svm range structure @@ -2420,15 +2479,6 @@ svm_range_trigger_migration(struct mm_struct *mm, struct svm_range *prange, return r; } - if (!prange->actual_loc && !prange->pages_addr) { - pr_debug("host access and prefetch to gpu\n"); - r = svm_range_validate_ram(mm, prange); - if (r) { - pr_debug("failed %d to validate on ram\n", r); - return r; - } - } - r = svm_migrate_to_vram(prange, best_loc, mm); *migrated = !r; @@ -2587,23 +2637,8 @@ svm_range_set_attr(struct kfd_process *p, uint64_t start, uint64_t size, continue; } - /* FIXME: With xnack on, this can race with MMU notifiers. - * They may invalidate page addresses before we map them. - * Then we end up mapping invalid addresses in the GPU page - * table. May need to find a way to still hold the mmap write - * for map_to_gpus but drop it for validate to allow - * concurrent evictions. This will lead to some retry logic - * and the need to protect the update list differently. - * Maybe factor migration and validation into a common helper - * function shared with the GPU page fault handler. - */ - r = svm_range_validate(mm, prange); - if (r) { - pr_debug("failed %d to validate svm range\n", r); - goto out_unlock_range; - } - - r = svm_range_map_to_gpus(prange, true); + r = svm_range_validate_and_map(mm, prange, MAX_GPU_INSTANCE, + true, true); if (r) pr_debug("failed %d to map svm range\n", r); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h index 7fce3fccfe58..af853726b861 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.h @@ -70,16 +70,14 @@ struct svm_work_list_item { * @update_list:link list node used to add to update_list * @remove_list:link list node used to add to remove list * @insert_list:link list node used to add to insert list - * @hmm_range: hmm range structure used by hmm_range_fault to get system pages * @mapping: bo_va mapping structure to create and update GPU page table * @npages: number of pages - * @pages_addr: list of system memory physical page address * @dma_addr: dma mapping address on each GPU for system memory physical page * @ttm_res: vram ttm resource map * @offset: range start offset within mm_nodes * @svm_bo: struct to manage splited amdgpu_bo * @svm_bo_list:link list node, to scan all ranges which share same svm_bo - * @lock: protect prange start, last, child_list, svm_bo_list, pages_addr + * @lock: protect prange start, last, child_list, svm_bo_list * @saved_flags:save/restore current PF_MEMALLOC flags * @flags: flags defined as KFD_IOCTL_SVM_FLAG_* * @perferred_loc: perferred location, 0 for CPU, or GPU id @@ -109,10 +107,8 @@ struct svm_range { struct list_head update_list; struct list_head remove_list; struct list_head insert_list; - struct hmm_range *hmm_range; struct amdgpu_bo_va_mapping mapping; uint64_t npages; - unsigned long *pages_addr; dma_addr_t *dma_addr[MAX_GPU_INSTANCE]; struct ttm_resource *ttm_res; uint64_t offset; @@ -133,6 +129,7 @@ struct svm_range { struct list_head child_list; DECLARE_BITMAP(bitmap_access, MAX_GPU_INSTANCE); DECLARE_BITMAP(bitmap_aip, MAX_GPU_INSTANCE); + bool validated_once; }; static inline void svm_range_lock(struct svm_range *prange) @@ -166,7 +163,6 @@ int svm_range_split_by_granularity(struct kfd_process *p, struct mm_struct *mm, int svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, uint64_t addr); int svm_range_schedule_evict_svm_bo(struct amdgpu_amdkfd_fence *fence); -int svm_range_map_to_gpus(struct svm_range *prange, bool reserve_vm); void svm_range_add_list_work(struct svm_range_list *svms, struct svm_range *prange, struct mm_struct *mm, enum svm_work_list_ops op); @@ -174,5 +170,6 @@ void schedule_deferred_list_work(struct svm_range_list *svms); void svm_range_dma_unmap(struct device *dev, dma_addr_t *dma_addr, unsigned long offset, unsigned long npages); void svm_range_free_dma_mappings(struct svm_range *prange); +void svm_range_prefault(struct svm_range *prange, struct mm_struct *mm); #endif /* KFD_SVM_H_ */ From patchwork Mon Mar 22 10:59:00 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Felix Kuehling X-Patchwork-Id: 12154557 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id AC797C433C1 for ; Mon, 22 Mar 2021 11:09:27 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 76F5D6188B for ; Mon, 22 Mar 2021 11:09:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 76F5D6188B Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C33776E436; Mon, 22 Mar 2021 11:08:39 +0000 (UTC) Received: from NAM12-DM6-obe.outbound.protection.outlook.com (mail-dm6nam12on2058.outbound.protection.outlook.com [40.107.243.58]) by gabe.freedesktop.org (Postfix) with ESMTPS id BBF5E6E406; Mon, 22 Mar 2021 11:08:27 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=PaGN+iRTZ5U6/b79ik0i2URYpQ6IjyH426ofgz44NMuzUpJGJEjxbBcXV6fOWBwH5QYBl45+Cq0Zr1hDHKvhkorZ8A/ImW5qynkproP58Eru676xp/E6qWPUDzuj8EBdmx7BwzoI/Yi8x5H2Yiur0RQokVxk1AevP9xwMOZ8V1XfZCHVm1+kIgsmr+JAVzj/Xuw9ojDzGYaQogHCFgYlDhkmRA/jH+kj6MmrnARjId8/dT6eOtQXAOcEc38qzj5OGzLvV6EYaGFqbbQ9IBb0XylP3MmiKkofVp0DwPEkaE321Yr/DWw/v8CTzxfcfojWbIV+zXs+SmKUhH1HLUEr2A== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8RCoLsH7sTr9z/n56GvfDtsxb9pzYOKmQXozkxbljZI=; b=UuNq5PWPV3OAxhqDFIdiQFgoF1+oaH0V+GvjGCbBCUDLymrLQNf9Y8daehExa5WJIHZzcTeBwR6x4BGC/t/rykJnFgDal6h+HqtGUQH/Bppq5baiVtFWnvd2Cda4eFk2seq/aUZHWyhtEUPLJCUPrk4ClIbrSNRwQQfls8E/F/H57swYyy6H++gApESv1gwHI9hdaj4D1+xuEUKux5+5GQC4DGQl3dmZnjdvF4VmBgtxsghDXbymLa8IspRG0IAt6ZpxVq/lOoW3pH3w0Kfxjs1SKxZ03BfCRy+bNhRDhpeasYkrUkmrNzcIh+8HHnSoJOWx+LougjeCJNLTvVfJMQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=amd.com; dmarc=pass action=none header.from=amd.com; dkim=pass header.d=amd.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amd.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=8RCoLsH7sTr9z/n56GvfDtsxb9pzYOKmQXozkxbljZI=; b=ytPnHo5zGYuwBe5xxirGJ3YBkWk1HDgrdexy23FSmmn60jT1AqG9biFXiqcfVcM9yVYDlhxNymv8B6W/a+f/dRJXWNdXBOd+ttfhpDeGhiFIW2Ogpo447YGEwwSlYFx9Ih3OhYuT7LXa3nhpJhxV7sYABfHnozsaLpvmDtuC2nw= Authentication-Results: lists.freedesktop.org; dkim=none (message not signed) header.d=none; lists.freedesktop.org; dmarc=none action=none header.from=amd.com; Received: from BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) by MN2PR12MB4255.namprd12.prod.outlook.com (2603:10b6:208:198::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24; Mon, 22 Mar 2021 11:08:23 +0000 Received: from BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033]) by BL0PR12MB4948.namprd12.prod.outlook.com ([fe80::70f5:99ed:65a1:c033%5]) with mapi id 15.20.3933.036; Mon, 22 Mar 2021 11:08:23 +0000 From: Felix Kuehling To: dri-devel@lists.freedesktop.org, amd-gfx@lists.freedesktop.org Subject: [PATCH 44/44] drm/amdkfd: Nested locking and invalidation of child ranges Date: Mon, 22 Mar 2021 06:59:00 -0400 Message-Id: <20210322105900.14068-45-Felix.Kuehling@amd.com> X-Mailer: git-send-email 2.31.0 In-Reply-To: <20210322105900.14068-1-Felix.Kuehling@amd.com> References: <20210322105900.14068-1-Felix.Kuehling@amd.com> X-Originating-IP: [165.204.55.251] X-ClientProxiedBy: YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) To BL0PR12MB4948.namprd12.prod.outlook.com (2603:10b6:208:1cc::20) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from Harpoon.amd.com (165.204.55.251) by YTOPR0101CA0008.CANPRD01.PROD.OUTLOOK.COM (2603:10b6:b00:15::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3955.24 via Frontend Transport; Mon, 22 Mar 2021 11:07:52 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-HT: Tenant X-MS-Office365-Filtering-Correlation-Id: 32e7e30e-316d-4bb0-426b-08d8ed22bcb1 X-MS-TrafficTypeDiagnostic: MN2PR12MB4255: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:466; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: SD/CFpV5+eHsbBbCrd3wDPevIdQgsgnyk9s3/kcm3tF4j5zcTLGiuCmRjG4rpzl2d6r75ftHUJHDIB4DH26xledPNbgs2/wHrZFb8iGn5inMfTFJen3akIZhqJXrCKI+KoPOfimbPNS3vmpVNpVUinB7RHbe257KMtv0ic+WWhVXnssW1XUYO3JOWe5mQojnRzyi2ZTe0adN+aQIkiC9BJ9lWX8s3nAUVe/WglQugDncwvnj2PLI8GwQmCbR2L8ipVeN4nEtJklZeQtQC2R3LxUY0MQzImX8QFq6UqArJ4W1PECwENCehF4yevZv1Aq5Nf8GIKRMjmN+yAJ29W/tHijxVNFSSQfkWPq/5kHYVoXYbXxQXqpK+Rm0My86sSaO4ohwo0XU8D8JDttsCOZY5w1AU2k7ZeeM0FEp+OW7dzPmY3qUekbFPaHlY26fVJBGsM+qqAx3Z3onAfajmUrYuHHJnO9XYyWQWHgLT8RwvQoQXB1hcIhVc84pSQETiwIMU/1MeoOFHQOwoABF+cQ479K+C3kA7989UlD/E104V6F9qmmSI8GZbrhn1bQLvZQ9duA0p/IFard9VSROzl8+kcIP+VvAeasTxyinglBA8/fdlgFNr822Tv3Znm1Qy4/fTabjJNgxT9dXuaKZKLS25lGgSDs5x7ThZrsu7LfuNEY= X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BL0PR12MB4948.namprd12.prod.outlook.com; PTR:; CAT:NONE; SFS:(4636009)(396003)(136003)(366004)(39860400002)(376002)(346002)(450100002)(1076003)(36756003)(38100700001)(6666004)(66946007)(83380400001)(66556008)(66476007)(5660300002)(16526019)(186003)(8676002)(26005)(7696005)(52116002)(86362001)(478600001)(8936002)(2616005)(956004)(6486002)(2906002)(316002); DIR:OUT; SFP:1101; X-MS-Exchange-AntiSpam-MessageData: /RM9hJQsXr+yLRe1Mp/qPqcIow3BpQ4rVNeMpejan7qlXh0hPLMc0hM7T/5eVzxsmYI1uP/N0/K87j4pNmSrPkHOK4r/yUvDHuLzFr48nuGG7SaN22pmsTMWeof3EB4N3lKobVBTkuf0TD/Q5Nk9JbORIlICoPzneQpWcVcvPTFflfW+ChBrsBfhlO1wrdipCzt2RJZFc6QE15bfgIRouIg2yKI2MtuIEbChno4FfKLJrkTpX2eAOHW4hluqChCLkQP91nMd9qns2dhZo3YFuSHeRrzeT4s/y3f8L9qRgFToNFEIrdWu7A2RVaj5BARi2LU5dw17xWXGi2TMal2Ikw7a1/54HP3gjQhnDpUBsxelhs4julWdQhpMJsC3Os0ANrZkG7gExakvPVoEFSI5+suuD343yIxKKePrxzQbKxMy4Hf9m/7WXjqhpd3krSIASYsM9tI/0voo59Y2GVPRcxy4k5q/cQhAJP9M7NdePey83SYXXsePdNHCvo9NFMvXFNkht+TjOnOJPz9BGwACNfmhRX9qOjYBya6O281/jPV5a0S0XMJlGOeHZK5qSfjiVFFwfaYToTki2kaahzYbm/Da8b3ODPudOAKCLWMaUZDvsU7LrNSY5rYyLXM7VFnvk4B4yV20AVOwIANqPyYyf1WKb7n3lw9clo9lMYf6XHNHaKByBpXnVZzF+iKkzgFQHGrR/1UVQCzREh9bzceqMPzfLxsVB12mRTS5JV7/MrVmypHL6Q4OgygHxji/hk6jHuYS860XKXBDAOeRLnHO29TojdUzWYkVwOr7r45MIoNqfeyV5w6NS2sqH/dTZw9gpyfEAHL2kRGW3vTVEHULBDzjKK4OxaNkfcr1y5xNSfjHSh8B6mtJPV+hiUXtR5QQzCKLuwC+cJLNMyl11oIj5OuKCgpQpNzab1sI1urnPqw77B2wQh6CWDTKWR0NWcMSJT0/+pUsUfZFtplnuQRbe+OoeXo23ubphJ3DxmzgtEgtkKzpg6fJ6qStcHdFQOZy0JVfWCVCh98+P0p4LfIvgHMWRGkguc9a9jYWKYiFItUvj3hrpAhIT8Aa4mRqH5Q8lTZNRp/0EH5ZtArXqrrIB7QWsLPhNDFg3IDSlj2uBFEzadqN8HwzUm+SBVf5qvvF5s0t9L8XD7+0vioSizQvYAFyCDJau8a1ZvEauWPG4DAsgn7B5XaKrptOxeUXSy2rEyBEPMEQoGBoIRLrrI0pu6j8fLNnInezVWD2WqBGEua2CWO4qmf4HpdkybVL4fnVJVWihhwthqBxaEhjGYOcZO4cpdHPsUVrOOn6hfYReQxbxy6vxLFvvsL54saclFiM X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-Network-Message-Id: 32e7e30e-316d-4bb0-426b-08d8ed22bcb1 X-MS-Exchange-CrossTenant-AuthSource: BL0PR12MB4948.namprd12.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 22 Mar 2021 11:07:52.4534 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: nIj4hc8Bxj6JhnOdb0QUf3c3sQ6XYiuKPKTTyMJipVPE5I97y0NSAnN9TB9ytTPqamLz8t3VgfUVUIVsQIPsGQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: MN2PR12MB4255 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This allows validation of child ranges, so the GPU page fault handler can be more light-weight. Signed-off-by: Felix Kuehling --- drivers/gpu/drm/amd/amdkfd/kfd_migrate.c | 8 +++++ drivers/gpu/drm/amd/amdkfd/kfd_svm.c | 40 +++++++++++++++++------- 2 files changed, 37 insertions(+), 11 deletions(-) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 4d79d69d8aac..cc8bf6438383 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -826,12 +826,18 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) } mutex_lock(&parent->migrate_mutex); + if (prange != parent) + mutex_lock_nested(&prange->migrate_mutex, 1); if (!prange->actual_loc) goto out_unlock_prange; svm_range_lock(parent); + if (prange != parent) + mutex_lock_nested(&prange->lock, 1); r = svm_range_split_by_granularity(p, mm, addr, parent, prange); + if (prange != parent) + mutex_unlock(&prange->lock); svm_range_unlock(parent); if (r) { pr_debug("failed %d to split range by granularity\n", r); @@ -852,6 +858,8 @@ static vm_fault_t svm_migrate_to_ram(struct vm_fault *vmf) schedule_deferred_list_work(&p->svms); out_unlock_prange: + if (prange != parent) + mutex_unlock(&prange->migrate_mutex); mutex_unlock(&parent->migrate_mutex); out: mutex_unlock(&p->svms.lock); diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c index 0a6e28f0dcaf..bc1a9e9ba722 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_svm.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_svm.c @@ -1281,10 +1281,6 @@ void svm_range_unreserve_bos(struct svm_validate_context *ctx) * serialize concurrent migrations or validations of the same range, the * prange->migrate_mutex must be held. * - * The range must be in the inverval tree and have an MMU notifier to catch - * concurrent invalidations of the virtual address range. This means it cannot - * be a child range. - * * For VRAM ranges, the SVM BO must be allocated and valid (protected by its * eviction fence. * @@ -1568,10 +1564,24 @@ svm_range_evict(struct svm_range *prange, struct mm_struct *mm, schedule_delayed_work(&svms->restore_work, msecs_to_jiffies(AMDGPU_SVM_RANGE_RESTORE_DELAY_MS)); } else { + struct svm_range *pchild; + unsigned long s, l; + pr_debug("invalidate unmap svms 0x%p [0x%lx 0x%lx] from GPUs\n", prange->svms, start, last); svm_range_lock(prange); - svm_range_unmap_from_gpus(prange, start, last); + list_for_each_entry(pchild, &prange->child_list, child_list) { + mutex_lock_nested(&pchild->lock, 1); + s = max(start, pchild->start); + l = min(last, pchild->last); + if (l >= s) + svm_range_unmap_from_gpus(pchild, s, l); + mutex_unlock(&pchild->lock); + } + s = max(start, prange->start); + l = min(last, prange->last); + if (l >= s) + svm_range_unmap_from_gpus(prange, s, l); svm_range_unlock(prange); } @@ -1927,6 +1937,7 @@ svm_range_unmap_from_cpu(struct mm_struct *mm, struct svm_range *prange, struct svm_range_list *svms; struct svm_range *pchild; struct kfd_process *p; + unsigned long s, l; bool unmap_parent; p = kfd_lookup_process_by_mm(mm); @@ -1937,14 +1948,23 @@ svm_range_unmap_from_cpu(struct mm_struct *mm, struct svm_range *prange, pr_debug("svms 0x%p prange 0x%p [0x%lx 0x%lx] [0x%lx 0x%lx]\n", svms, prange, prange->start, prange->last, start, last); - svm_range_unmap_from_gpus(prange, start, last); - svm_range_lock(prange); unmap_parent = start <= prange->start && last >= prange->last; - list_for_each_entry(pchild, &prange->child_list, child_list) + list_for_each_entry(pchild, &prange->child_list, child_list) { + mutex_lock_nested(&pchild->lock, 1); + s = max(start, pchild->start); + l = min(last, pchild->last); + if (l >= s) + svm_range_unmap_from_gpus(pchild, s, l); svm_range_unmap_split(mm, prange, pchild, start, last); + mutex_unlock(&pchild->lock); + } + s = max(start, prange->start); + l = min(last, prange->last); + if (l >= s) + svm_range_unmap_from_gpus(prange, s, l); svm_range_unmap_split(mm, prange, prange, start, last); svm_range_unlock(prange); @@ -2142,12 +2162,10 @@ svm_range_restore_pages(struct amdgpu_device *adev, unsigned int pasid, goto out; } - svm_range_list_lock_and_flush_work(svms, mm); + mmap_read_lock(mm); mutex_lock(&svms->lock); prange = svm_range_from_addr(svms, addr, NULL); - mmap_write_downgrade(mm); - if (!prange) { pr_debug("failed to find prange svms 0x%p address [0x%llx]\n", svms, addr);