From patchwork Sun Jun 21 06:03:00 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrey Grodzovsky X-Patchwork-Id: 11616445 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 45A511731 for ; Sun, 21 Jun 2020 06:05:57 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 2370320B80 for ; Sun, 21 Jun 2020 06:05:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=amdcloud.onmicrosoft.com header.i=@amdcloud.onmicrosoft.com header.b="MboF1QqJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 2370320B80 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 167EF6E436; Sun, 21 Jun 2020 06:05:49 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2085.outbound.protection.outlook.com [40.107.94.85]) by gabe.freedesktop.org (Postfix) with ESMTPS id 388816E429; Sun, 21 Jun 2020 06:05:44 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=U4WS2HIIfWRKIOn303JLdI/iPL4cQ1CkFnVxlMHU0/xaZaxUqcdnAr2Lm3F8xwcwUDTXz63vALfe80cCaGy4oDbxOIUV2wdRdAt1z4//FCgLnF5C/pEAJHzhsiuwA6OSErAqXrtZ3ZGwW2yEtiwknfwKj3Rh+K1r+xQ0iFuUSigUS40yS3eX4AkD2CjELUNRZT91V9VsQLn0EjVGqRKVFnqQ2l831+e0sbjTHy5A9aIFTZIY6RoUtYyUXGCeGUK/8WsQI78DELP5WOH0Q73x+SXzhjcC75WTI1Ar+XIbDzDdonM7hJvdCYSXlDPY32lge4g1mTkrsuZTw02vfaY58w== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+GBdDvH0tDwuZP6mHThM03IsBFn8f8bv7nIxscPgQ+s=; b=FfVj2fY7+X3uAwfPkGRoUmE1aaiatoo2Gh5MhV4sSekADeUA1jchwCwCFcjq0F/inw1JfxVvBnmOzpSOA6wJAcjrph3CJyPgScc8l/VPyGGP9JCbyo2dM5a9H+BMvNYVQT4E5Xng3rh8p8EW+nU+8dFskNtZc3ypcoiMU+qCDOqqwZFNYlR1Cikk4kEp6t13PrA8dQ2TU7jczFjcJiuRtos8D0BVOTB4bbgnLG/1jBMYQ/yIPWs1Xvw7pG+LGaC5E/c1Ikheqx+52aYblBCwtrNOY3/y+oYckGKMYhmpR/OYVk9c882HhjVfxmG0Tz4MGb9HA9AWXJi/C3e6O8VoUQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.freedesktop.org smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=+GBdDvH0tDwuZP6mHThM03IsBFn8f8bv7nIxscPgQ+s=; b=MboF1QqJr741evwhIJoFX5VXDX4zhJesSQNPscSIalcUviXRXPsQjCSewYqbCX11mv3Q8YpuASTHyU5RT14lDCqHzbKGUFIPP9dASVVhd7+TaoXjAViqJJuTMudD6M2zg0hXpNhzGXGGDsnQ+6MXDxAn3Z7jv6iZTav6Dy9aOPo= Received: from BN6PR21CA0011.namprd21.prod.outlook.com (2603:10b6:404:8e::21) by DM5PR1201MB0089.namprd12.prod.outlook.com (2603:10b6:4:50::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3109.23; Sun, 21 Jun 2020 06:05:37 +0000 Received: from BN8NAM11FT065.eop-nam11.prod.protection.outlook.com (2603:10b6:404:8e:cafe::57) by BN6PR21CA0011.outlook.office365.com (2603:10b6:404:8e::21) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3153.3 via Frontend Transport; Sun, 21 Jun 2020 06:05:37 +0000 X-MS-Exchange-Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; lists.freedesktop.org; dkim=none (message not signed) header.d=none;lists.freedesktop.org; dmarc=permerror action=none header.from=amd.com; Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB02.amd.com (165.204.84.17) by BN8NAM11FT065.mail.protection.outlook.com (10.13.177.63) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.3109.22 via Frontend Transport; Sun, 21 Jun 2020 06:05:37 +0000 Received: from SATLEXMB01.amd.com (10.181.40.142) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Sun, 21 Jun 2020 01:05:37 -0500 Received: from agrodzovsky-All-Series.amd.com (10.180.168.240) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Sun, 21 Jun 2020 01:05:36 -0500 From: Andrey Grodzovsky To: , Subject: [PATCH v2 0/8] RFC Support hot device unplug in amdgpu Date: Sun, 21 Jun 2020 02:03:00 -0400 Message-ID: <1592719388-13819-1-git-send-email-andrey.grodzovsky@amd.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:SATLEXMB02.amd.com; PTR:InfoDomainNonexistent; CAT:NONE; SFTY:; SFS:(4636009)(396003)(39860400002)(136003)(376002)(346002)(46966005)(5660300002)(2906002)(316002)(54906003)(110136005)(8676002)(70586007)(966005)(70206006)(36756003)(26005)(478600001)(8936002)(83380400001)(86362001)(4326008)(82740400003)(47076004)(356005)(82310400002)(81166007)(336012)(2616005)(44832011)(186003)(426003)(7696005)(6666004); DIR:OUT; SFP:1101; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 913f2b42-4684-4a30-c967-08d815a91e47 X-MS-TrafficTypeDiagnostic: DM5PR1201MB0089: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-Forefront-PRVS: 04410E544A X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 6o1wnwmYvt7hgRhYse5ixcN2ABsuNoe2MK/LdIyiyG1oKGGDeOYlqU50+HjKSjI0/c1MiW3V3ML6RgBGUuwdGLR1+Ir8cVm4ToeAa4bzAS95wYWhlJBQ/NO7Z29cUkUqCUhLxcpFE2kP8qGwBLLQzFsJ46g/cvELIk305wBi8iPfdawp7NPMrVkeynMJJof0C2SjC/L5ebEGvshCifim0+5lse/knT4lfNAXtdIqsiPsfwkB2/9Ie/gQ4az7MZbZ3//IxG9JcTCLIMxYeqypTh6I0FzB+rFmddNdTIBjstwt6YtDsJxc6O0SG7q5xEpPmjrcldSWQxKDi9Xinvi9lYzL6bygRvyTC+pCE2e12JJFXvQddBqiyKYO2CKzv13/T8wKt7uLuRem6OXQetglYAGU4HskOFkJGZrvNOlk7uSR2+Zj1ZZsr23MVwSvIs+syLqXw7g/g9qETok6sd1WNR80mMC7nnoCWkhvuP+l+4U= X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 21 Jun 2020 06:05:37.4172 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 913f2b42-4684-4a30-c967-08d815a91e47 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB02.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR1201MB0089 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: daniel.vetter@ffwll.ch, michel@daenzer.net, ckoenig.leichtzumerken@gmail.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" This RFC is more of a proof of concept then a fully working solution as there are a few unresolved issues we are hoping to get advise on from people on the mailing list. Until now extracting a card either by physical extraction (e.g. eGPU with thunderbolt connection or by emulation through syfs -> /sys/bus/pci/devices/device_id/remove) would cause random crashes in user apps. The random crashes in apps were mostly due to the app having mapped a device backed BO into its address space was still trying to access the BO while the backing device was gone. To answer this first problem Christian suggested to fix the handling of mapped memory in the clients when the device goes away by forcibly unmap all buffers the user processes has by clearing their respective VMAs mapping the device BOs. Then when the VMAs try to fill in the page tables again we check in the fault handler if the device is removed and if so, return an error. This will generate a SIGBUS to the application which can then cleanly terminate. This indeed was done but this in turn created a problem of kernel OOPs were the OOPSes were due to the fact that while the app was terminating because of the SIGBUS it would trigger use after free in the driver by calling to accesses device structures that were already released from the pci remove sequence. This was handled by introducing a 'flush' sequence during device removal were we wait for drm file reference to drop to 0 meaning all user clients directly using this device terminated. With this I was able to cleanly emulate device unplug with X and glxgears running and later emulate device plug back and restart of X and glxgears. v2: Based on discussions in the mailing list with Daniel and Pekka [1] and based on the document produced by Pekka from those discussions [2] the whole approach with returning SIGBUS and waiting for all user clients having CPU mapping of device BOs to die was dropped. Instead as per the document suggestion the device structures are kept alive until the last reference to the device is dropped by user client and in the meanwhile all existing and new CPU mappings of the BOs belonging to the device directly or by dma-buf import are rerouted to per user process dummy rw page. Also, I skipped the 'Requirements for KMS UAPI' section of [2] since i am trying to get the minimal set of requiremnts that still give useful solution to work and this is the 'Requirements for Render and Cross-Device UAPI' section and so my test case is removing a secondary device, which is render only and is not involved in KMS. This iteration is still more of a draft as I am still facing a few unsolved issues such as a crash in user client when trying to CPU map imported BO if the map happens after device was removed and HW failure to plug back a removed device. Also since i don't have real life setup with external GPU connected through TB I am using sysfs to emulate pci remove and i expect to encounter more issues once i try this on real life case. I am also expecting some help on this from a user who volunteered to test in the related gitlab ticket. So basically this is more of a way to get feedback if I am moving in the right direction. [1] - Discussions during v1 of the patchset https://lists.freedesktop.org/archives/dri-devel/2020-May/265386.html [2] - drm/doc: device hot-unplug for userspace https://www.spinics.net/lists/dri-devel/msg259755.html [3] - Related gitlab ticket https://gitlab.freedesktop.org/drm/amd/-/issues/1081   Andrey Grodzovsky (8): drm: Add dummy page per device or GEM object drm/ttm: Remap all page faults to per process dummy page. drm/ttm: Add unampping of the entire device address space drm/amdgpu: Split amdgpu_device_fini into early and late drm/amdgpu: Refactor sysfs removal drm/amdgpu: Unmap entire device address space on device remove. drm/amdgpu: Fix sdma code crash post device unplug drm/amdgpu: Prevent any job recoveries after device is unplugged. drivers/gpu/drm/amd/amdgpu/amdgpu.h | 19 +++++++- drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c | 7 ++- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 50 +++++++++++++++++---- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 23 ++++++++-- drivers/gpu/drm/amd/amdgpu/amdgpu_gtt_mgr.c | 12 +++-- drivers/gpu/drm/amd/amdgpu/amdgpu_irq.c | 24 ++++++---- drivers/gpu/drm/amd/amdgpu/amdgpu_irq.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_job.c | 8 ++++ drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c | 23 +++++++--- drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c | 8 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 3 ++ drivers/gpu/drm/amd/amdgpu/amdgpu_vm_sdma.c | 21 ++++++--- drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c | 17 +++++++- drivers/gpu/drm/amd/amdgpu/amdgpu_xgmi.c | 13 +++++- drivers/gpu/drm/amd/amdgpu/df_v3_6.c | 10 +++-- drivers/gpu/drm/drm_file.c | 8 ++++ drivers/gpu/drm/drm_prime.c | 10 +++++ drivers/gpu/drm/ttm/ttm_bo.c | 8 +++- drivers/gpu/drm/ttm/ttm_bo_vm.c | 65 ++++++++++++++++++++++++---- include/drm/drm_file.h | 2 + include/drm/drm_gem.h | 2 + include/drm/ttm/ttm_bo_driver.h | 7 +++ 22 files changed, 286 insertions(+), 55 deletions(-)