From patchwork Mon Nov 25 20:51:29 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Andrey Grodzovsky X-Patchwork-Id: 11260831 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6992E14C0 for ; Mon, 25 Nov 2019 20:51:42 +0000 (UTC) Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 51A1920659 for ; Mon, 25 Nov 2019 20:51:42 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 51A1920659 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=amd.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=dri-devel-bounces@lists.freedesktop.org Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 04D9D89C2C; Mon, 25 Nov 2019 20:51:40 +0000 (UTC) X-Original-To: dri-devel@lists.freedesktop.org Delivered-To: dri-devel@lists.freedesktop.org Received: from NAM02-CY1-obe.outbound.protection.outlook.com (mail-eopbgr760074.outbound.protection.outlook.com [40.107.76.74]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8E88F89C2C; Mon, 25 Nov 2019 20:51:38 +0000 (UTC) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=mlxBkXNCNj8ZPEoGgYxWGmIeXGu+Jm2IatLafQGSTLbTov/l+aOpZGJMmazyTcaSH6fnw4Rf+BN+7diUwrPWs3I9ppX/hrLiJ0IFGzAAJ/lxPQ+QyfYdbQWE6JB/ybLUXSCzdjXjD3vO1j5DcuKv18nZzg8k0CEzTWrcTf7HlXkGG64NQhzaouNjKiNeBrw0CuVV4iWV5pwKccPCtnR0H4GK0Y4cpua8vaWUYz0Gg8/7Nd9YErr2vldpmrUIuCBPBj+bxzNMrhLaDbzcioes+tH/Gwy+6Y7Qx+mm+8b66/5+MH6/9CjOrltsPjYudexEvMdb4tUN7Fx55/YCG30xrg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Eu+JJt5L4aDFZyDUnfoj/2453/8J/y18mkGy07h1jF0=; b=lkjK3de4V+MtlhWUeAsRfu2frkPVTkJOoCTwj7ukZGuOhfHcTsaQKjL9ilBULGwVWZnYgkUb8P24jl9EfTq/bCLdrnYSGjAsz86NHw/WVTzoj4rGkLC0ChfIso5fKwBgOwWev8OCFYWZDKVtVgAw2LaZFYr+Ofvov8keUgVbyqni+GFtinWKNkuCV32c7RjbecwgB5CQQo+D3yjhTP61GGnpBajVJmESCYQBW6c3WkZgpEBmyF3vPOQsQvqRnh7KpkuCVPpj2StEtwodRgcsHZ/xnBByQq2gBbQutm+LfZChGiSCbFdzXB2unwLutKtOJvD05EIkiH57cUsDw4jxpw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none (sender ip is 165.204.84.17) smtp.rcpttodomain=lists.freedesktop.org smtp.mailfrom=amd.com; dmarc=permerror action=none header.from=amd.com; dkim=none (message not signed); arc=none Received: from MWHPR12CA0033.namprd12.prod.outlook.com (2603:10b6:301:2::19) by MWHPR1201MB0061.namprd12.prod.outlook.com (2603:10b6:301:52::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2474.16; Mon, 25 Nov 2019 20:51:36 +0000 Received: from CO1NAM11FT025.eop-nam11.prod.protection.outlook.com (2a01:111:f400:7eab::200) by MWHPR12CA0033.outlook.office365.com (2603:10b6:301:2::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2474.18 via Frontend Transport; Mon, 25 Nov 2019 20:51:36 +0000 Received-SPF: None (protection.outlook.com: amd.com does not designate permitted sender hosts) Received: from SATLEXMB02.amd.com (165.204.84.17) by CO1NAM11FT025.mail.protection.outlook.com (10.13.175.232) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.20.2451.23 via Frontend Transport; Mon, 25 Nov 2019 20:51:36 +0000 Received: from SATLEXMB04.amd.com (10.181.40.145) by SATLEXMB02.amd.com (10.181.40.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 25 Nov 2019 14:51:35 -0600 Received: from SATLEXMB01.amd.com (10.181.40.142) by SATLEXMB04.amd.com (10.181.40.145) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1713.5; Mon, 25 Nov 2019 14:51:34 -0600 Received: from agrodzovsky-All-Series.amd.com (10.180.168.240) by SATLEXMB01.amd.com (10.181.40.142) with Microsoft SMTP Server id 15.1.1713.5 via Frontend Transport; Mon, 25 Nov 2019 14:51:34 -0600 From: Andrey Grodzovsky To: Subject: [PATCH v4] drm/scheduler: Avoid accessing freed bad job. Date: Mon, 25 Nov 2019 15:51:29 -0500 Message-ID: <1574715089-14875-1-git-send-email-andrey.grodzovsky@amd.com> X-Mailer: git-send-email 2.7.4 MIME-Version: 1.0 X-EOPAttributedMessage: 0 X-MS-Office365-Filtering-HT: Tenant X-Forefront-Antispam-Report: CIP:165.204.84.17; IPV:NLI; CTRY:US; EFV:NLI; SFV:NSPM; SFS:(10009020)(4636009)(376002)(396003)(39860400002)(136003)(346002)(428003)(189003)(199004)(54906003)(8676002)(109986005)(26005)(478600001)(23676004)(7696005)(186003)(44832011)(336012)(1671002)(305945005)(5820100001)(4326008)(6666004)(356004)(53416004)(50466002)(2870700001)(2906002)(5660300002)(50226002)(2616005)(86362001)(8936002)(70206006)(81166006)(81156014)(70586007)(36756003)(426003)(14444005)(66574012)(316002)(47776003)(266003); DIR:OUT; SFP:1101; SCL:1; SRVR:MWHPR1201MB0061; H:SATLEXMB02.amd.com; FPR:; SPF:None; LANG:en; PTR:InfoDomainNonexistent; MX:1; A:1; X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: b6d86045-a807-4bf7-fdfe-08d771e94356 X-MS-TrafficTypeDiagnostic: MWHPR1201MB0061: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:9508; X-Forefront-PRVS: 0232B30BBC X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: PPgbqAdyM/XsXMum7YUM7z95l4ZXpYz9hOEJm7SM3QBYS49y2CCbTDpwGl8EO7Bf0GxHKcQ38Xm4xls5GCjahBvJMtICC5MXRnNGoxtyXvc4gnciq4mOw+FjClqEmpseBzidb4cRFgqfSPXG6P/3jj/Zzyk+Rww77CS9IY3ShBnRsyeCtMtTvnYHGg9JAh2uFvK0uYfSmtfVVWF6AEa414pSl6Jr909v41v2/7uaJfTsg8QHrE2ffJZUUdtf8NxeuuyayHkEZpkC1R2JSLGckbIfUSIbgoKaY3Uon8/5mQ6bznfglQzZ9pC5EZErJRb2rWsbWYcyF7K0XqCdWFkMhZSx/1pWSjo2keyERMc7AVGo/MJQ1MMHzZtZy0dyTFQuH00sIb3caIyepTAMbJhlZZHAvSrOcdXQbygCnET0Ha5eUD2g2t5bdEN2GKEdj6UX X-OriginatorOrg: amd.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Nov 2019 20:51:36.5716 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: b6d86045-a807-4bf7-fdfe-08d771e94356 X-MS-Exchange-CrossTenant-Id: 3dd8961f-e488-4e60-8e11-a82d994e183d X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=3dd8961f-e488-4e60-8e11-a82d994e183d; Ip=[165.204.84.17]; Helo=[SATLEXMB02.amd.com] X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR1201MB0061 X-Mailman-Original-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amdcloud.onmicrosoft.com; s=selector2-amdcloud-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=Eu+JJt5L4aDFZyDUnfoj/2453/8J/y18mkGy07h1jF0=; b=PK7g+1G8iFUyub3ubvOCKdVfxIgO96m2M6fvf59uFsny5rc2eYEDUotEcQ8vWtOODz1+HVrpy8AkJpK2biAu3RnW8b+TNy0ylBLhEgz14K1QAA4NRceIEWE/gfoxkwBAg/zbImFRVVFlA9qxz5220O6nZ6uv/es0E54whgt41Q4= X-Mailman-Original-Authentication-Results: spf=none (sender IP is 165.204.84.17) smtp.mailfrom=amd.com; lists.freedesktop.org; dkim=none (message not signed) header.d=none;lists.freedesktop.org; dmarc=permerror action=none header.from=amd.com; X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amd-gfx@lists.freedesktop.org, steven.price@arm.com, Emily.Deng@amd.com, dri-devel@lists.freedesktop.org, Christian.Koenig@amd.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Problem: Due to a race between drm_sched_cleanup_jobs in sched thread and drm_sched_job_timedout in timeout work there is a possiblity that bad job was already freed while still being accessed from the timeout thread. Fix: Instead of just peeking at the bad job in the mirror list remove it from the list under lock and then put it back later when we are garanteed no race with main sched thread is possible which is after the thread is parked. v2: Lock around processing ring_mirror_list in drm_sched_cleanup_jobs. v3: Rebase on top of drm-misc-next. v2 is not needed anymore as drm_sched_get_cleanup_job already has a lock there. v4: Fix comments to relfect latest code in drm-misc. Signed-off-by: Andrey Grodzovsky Reviewed-by: Christian König Tested-by: Emily Deng Reviewed-by: Emily Deng --- drivers/gpu/drm/scheduler/sched_main.c | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 6774955..1bf9c40 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -284,10 +284,21 @@ static void drm_sched_job_timedout(struct work_struct *work) unsigned long flags; sched = container_of(work, struct drm_gpu_scheduler, work_tdr.work); + + /* Protects against concurrent deletion in drm_sched_get_cleanup_job */ + spin_lock_irqsave(&sched->job_list_lock, flags); job = list_first_entry_or_null(&sched->ring_mirror_list, struct drm_sched_job, node); if (job) { + /* + * Remove the bad job so it cannot be freed by concurrent + * drm_sched_cleanup_jobs. It will be reinserted back after sched->thread + * is parked at which point it's safe. + */ + list_del_init(&job->node); + spin_unlock_irqrestore(&sched->job_list_lock, flags); + job->sched->ops->timedout_job(job); /* @@ -298,6 +309,8 @@ static void drm_sched_job_timedout(struct work_struct *work) job->sched->ops->free_job(job); sched->free_guilty = false; } + } else { + spin_unlock_irqrestore(&sched->job_list_lock, flags); } spin_lock_irqsave(&sched->job_list_lock, flags); @@ -370,6 +383,20 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad) kthread_park(sched->thread); /* + * Reinsert back the bad job here - now it's safe as + * drm_sched_get_cleanup_job cannot race against us and release the + * bad job at this point - we parked (waited for) any in progress + * (earlier) cleanups and drm_sched_get_cleanup_job will not be called + * now until the scheduler thread is unparked. + */ + if (bad && bad->sched == sched) + /* + * Add at the head of the queue to reflect it was the earliest + * job extracted. + */ + list_add(&bad->node, &sched->ring_mirror_list); + + /* * Iterate the job list from later to earlier one and either deactive * their HW callbacks or remove them from mirror list if they already * signaled.