From patchwork Tue Oct 17 15:09:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13425503 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 36E64CDB482 for ; Tue, 17 Oct 2023 15:09:46 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D828710E303; Tue, 17 Oct 2023 15:09:36 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id 508C910E043; Tue, 17 Oct 2023 15:09:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697555375; x=1729091375; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=CrT1x5/BtckFOYhrOtsw0sETMqiXrD4lRSZ3PNh3neA=; b=gubXkJIFtTFMUXtuJRjbmGQm8fldqfHaE8BZazV+siONjtmlFSrYzBsO gRX60QWbxmnNmlyo0HbFsHtzAb5WR9fbPJF760wch3dK7LbLy9TbzMvOc XoPyUSKReDzWOhEHmdNWVcsyAiDcmXM+J6toU4W9s6ZfHD5Z2CitKfs4J lG3Wc9TUfNoYRjpFDyhf50Illizwt/PfhadUBKzp8j85kRvANNe6h0hdI 8yRDdvDQD39kUZ3TCl68rWkqkP6Hp53Z5HNdwoksMjQW4uhlxk53Lydah uh6CEGCPcC//YPufVf7Pz4H2S39s0ftLi4ZW1C+u0q6jbv1LXUYzf3FP0 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="388665075" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="388665075" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="791257560" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="791257560" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:33 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v6 1/7] drm/sched: Add drm_sched_wqueue_* helpers Date: Tue, 17 Oct 2023 08:09:52 -0700 Message-Id: <20231017150958.838613-2-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017150958.838613-1-matthew.brost@intel.com> References: <20231017150958.838613-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , lina@asahilina.net, sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, mcanal@igalia.com, luben.tuikov@amd.com, dakr@redhat.com, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add scheduler wqueue ready, stop, and start helpers to hide the implementation details of the scheduler from the drivers. v2: - s/sched_wqueue/sched_wqueue (Luben) - Remove the extra white line after the return-statement (Luben) - update drm_sched_wqueue_ready comment (Luben) Cc: Luben Tuikov Signed-off-by: Matthew Brost Reviewed-by: Luben Tuikov --- .../drm/amd/amdgpu/amdgpu_amdkfd_arcturus.c | 2 +- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 15 +++---- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 12 +++--- drivers/gpu/drm/msm/adreno/adreno_device.c | 6 ++- drivers/gpu/drm/scheduler/sched_main.c | 39 ++++++++++++++++++- include/drm/gpu_scheduler.h | 3 ++ 6 files changed, 59 insertions(+), 18 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_arcturus.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_arcturus.c index 625db444df1c..10d56979fe3b 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_arcturus.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_arcturus.c @@ -290,7 +290,7 @@ static int suspend_resume_compute_scheduler(struct amdgpu_device *adev, bool sus for (i = 0; i < adev->gfx.num_compute_rings; i++) { struct amdgpu_ring *ring = &adev->gfx.compute_ring[i]; - if (!(ring && ring->sched.thread)) + if (!(ring && drm_sched_wqueue_ready(&ring->sched))) continue; /* stop secheduler and drain ring. */ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c index a4faea4fa0b5..a4c0bb358db7 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c @@ -1659,9 +1659,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused) for (i = 0; i < AMDGPU_MAX_RINGS; i++) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !drm_sched_wqueue_ready(&ring->sched)) continue; - kthread_park(ring->sched.thread); + drm_sched_wqueue_stop(&ring->sched); } seq_puts(m, "run ib test:\n"); @@ -1675,9 +1675,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused) for (i = 0; i < AMDGPU_MAX_RINGS; i++) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !drm_sched_wqueue_ready(&ring->sched)) continue; - kthread_unpark(ring->sched.thread); + drm_sched_wqueue_start(&ring->sched); } up_write(&adev->reset_domain->sem); @@ -1897,7 +1897,8 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val) ring = adev->rings[val]; - if (!ring || !ring->funcs->preempt_ib || !ring->sched.thread) + if (!ring || !ring->funcs->preempt_ib || + !drm_sched_wqueue_ready(&ring->sched)) return -EINVAL; /* the last preemption failed */ @@ -1915,7 +1916,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val) goto pro_end; /* stop the scheduler */ - kthread_park(ring->sched.thread); + drm_sched_wqueue_stop(&ring->sched); /* preempt the IB */ r = amdgpu_ring_preempt_ib(ring); @@ -1949,7 +1950,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val) failure: /* restart the scheduler */ - kthread_unpark(ring->sched.thread); + drm_sched_wqueue_start(&ring->sched); up_read(&adev->reset_domain->sem); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 2b8356699f23..b1aafe815f28 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -4588,7 +4588,7 @@ bool amdgpu_device_has_job_running(struct amdgpu_device *adev) for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !drm_sched_wqueue_ready(&ring->sched)) continue; spin_lock(&ring->sched.job_list_lock); @@ -4727,7 +4727,7 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !drm_sched_wqueue_ready(&ring->sched)) continue; /* Clear job fence from fence drv to avoid force_completion @@ -5266,7 +5266,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = tmp_adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !drm_sched_wqueue_ready(&ring->sched)) continue; drm_sched_stop(&ring->sched, job ? &job->base : NULL); @@ -5341,7 +5341,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = tmp_adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !drm_sched_wqueue_ready(&ring->sched)) continue; drm_sched_start(&ring->sched, true); @@ -5667,7 +5667,7 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !drm_sched_wqueue_ready(&ring->sched)) continue; drm_sched_stop(&ring->sched, NULL); @@ -5795,7 +5795,7 @@ void amdgpu_pci_resume(struct pci_dev *pdev) for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !drm_sched_wqueue_ready(&ring->sched)) continue; drm_sched_start(&ring->sched, true); diff --git a/drivers/gpu/drm/msm/adreno/adreno_device.c b/drivers/gpu/drm/msm/adreno/adreno_device.c index fa527935ffd4..8fa9ce3746b6 100644 --- a/drivers/gpu/drm/msm/adreno/adreno_device.c +++ b/drivers/gpu/drm/msm/adreno/adreno_device.c @@ -809,7 +809,8 @@ static void suspend_scheduler(struct msm_gpu *gpu) */ for (i = 0; i < gpu->nr_rings; i++) { struct drm_gpu_scheduler *sched = &gpu->rb[i]->sched; - kthread_park(sched->thread); + + drm_sched_wqueue_stop(sched); } } @@ -819,7 +820,8 @@ static void resume_scheduler(struct msm_gpu *gpu) for (i = 0; i < gpu->nr_rings; i++) { struct drm_gpu_scheduler *sched = &gpu->rb[i]->sched; - kthread_unpark(sched->thread); + + drm_sched_wqueue_start(sched); } } diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 5a3a622fc672..6f2f7dd4ba0a 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -439,7 +439,7 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad) { struct drm_sched_job *s_job, *tmp; - kthread_park(sched->thread); + drm_sched_wqueue_stop(sched); /* * Reinsert back the bad job here - now it's safe as @@ -552,7 +552,7 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery) spin_unlock(&sched->job_list_lock); } - kthread_unpark(sched->thread); + drm_sched_wqueue_start(sched); } EXPORT_SYMBOL(drm_sched_start); @@ -1206,3 +1206,38 @@ void drm_sched_increase_karma(struct drm_sched_job *bad) } } EXPORT_SYMBOL(drm_sched_increase_karma); + +/** + * drm_sched_wqueue_ready - Is the scheduler ready for submission + * + * @sched: scheduler instance + * + * Returns true if submission is ready + */ +bool drm_sched_wqueue_ready(struct drm_gpu_scheduler *sched) +{ + return !!sched->thread; +} +EXPORT_SYMBOL(drm_sched_wqueue_ready); + +/** + * drm_sched_wqueue_stop - stop scheduler submission + * + * @sched: scheduler instance + */ +void drm_sched_wqueue_stop(struct drm_gpu_scheduler *sched) +{ + kthread_park(sched->thread); +} +EXPORT_SYMBOL(drm_sched_wqueue_stop); + +/** + * drm_sched_wqueue_start - start scheduler submission + * + * @sched: scheduler instance + */ +void drm_sched_wqueue_start(struct drm_gpu_scheduler *sched) +{ + kthread_unpark(sched->thread); +} +EXPORT_SYMBOL(drm_sched_wqueue_start); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index f9544d9b670d..38578fe74573 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -550,6 +550,9 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, void drm_sched_job_cleanup(struct drm_sched_job *job); void drm_sched_wakeup_if_can_queue(struct drm_gpu_scheduler *sched); +bool drm_sched_wqueue_ready(struct drm_gpu_scheduler *sched); +void drm_sched_wqueue_stop(struct drm_gpu_scheduler *sched); +void drm_sched_wqueue_start(struct drm_gpu_scheduler *sched); void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad); void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery); void drm_sched_resubmit_jobs(struct drm_gpu_scheduler *sched); From patchwork Tue Oct 17 15:09:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13425501 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 88A7BCDB483 for ; Tue, 17 Oct 2023 15:09:37 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 81FDE10E128; Tue, 17 Oct 2023 15:09:36 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id C919410E043; Tue, 17 Oct 2023 15:09:34 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697555374; x=1729091374; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=BAjPJbceZOFhjSKTldp5d08JKSJVJkWjCF2CbFSj9t0=; b=X+JlJQNvd/4AM7BYcRVk/wyZZcG7eiRnZexWQVCqlXFjCiMoEtpefw/f OHAiQA24Lu3KtfhCCBauZBBhQOQ2Pk7M73aZyuIlf1sFT4TnUMJwSYuKS pKrRJHgLIJgFNruAaQzuAT8BLuBVTeSFxkQhDQdM5820AmHp5RkArtYQO pD4iV9l+CB0hUWK7zec5NfSoYWGp3/x6WZTefIIX2wX0bdkTDfNOdJAiG fDoazdA3LiXLb1VB7bUvvUl6X9AEJNrbSKBFsNsBg/u1+e+scAXNfPFrG rJ7g5KLqQuqKgV5bs+zyGEk7aK+HH/EiEUtnUVYHE/IY+3/42iSNI8SMv w==; X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="388665088" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="388665088" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:33 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="791257563" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="791257563" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:33 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v6 2/7] drm/sched: Convert drm scheduler to use a work queue rather than kthread Date: Tue, 17 Oct 2023 08:09:53 -0700 Message-Id: <20231017150958.838613-3-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017150958.838613-1-matthew.brost@intel.com> References: <20231017150958.838613-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , lina@asahilina.net, sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, mcanal@igalia.com, luben.tuikov@amd.com, dakr@redhat.com, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" In Xe, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below. 1. In Xe the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in Xe we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem. 2. In Xe submissions are done via programming a ring buffer (circular buffer), a drm_gpu_scheduler provides a limit on number of jobs, if the limit of number jobs is set to RING_SIZE / MAX_SIZE_PER_JOB we get flow control on the ring for free. A problem with this design is currently a drm_gpu_scheduler uses a kthread for submission / job cleanup. This doesn't scale if a large number of drm_gpu_scheduler are used. To work around the scaling issue, use a worker rather than kthread for submission / job cleanup. v2: - (Rob Clark) Fix msm build - Pass in run work queue v3: - (Boris) don't have loop in worker v4: - (Tvrtko) break out submit ready, stop, start helpers into own patch v5: - (Boris) default to ordered work queue v6: - (Luben / checkpatch) fix alignment in msm_ringbuffer.c - (Luben) s/drm_sched_submit_queue/drm_sched_wqueue_enqueue - (Luben) Update comment for drm_sched_wqueue_enqueue - (Luben) Positive check for submit_wq in drm_sched_init - (Luben) s/alloc_submit_wq/own_submit_wq Signed-off-by: Matthew Brost Reviewed-by: Luben Tuikov --- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/etnaviv/etnaviv_sched.c | 2 +- drivers/gpu/drm/lima/lima_sched.c | 2 +- drivers/gpu/drm/msm/msm_ringbuffer.c | 7 +- drivers/gpu/drm/nouveau/nouveau_sched.c | 2 +- drivers/gpu/drm/panfrost/panfrost_job.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 118 ++++++++++----------- drivers/gpu/drm/v3d/v3d_sched.c | 10 +- include/drm/gpu_scheduler.h | 14 ++- 9 files changed, 82 insertions(+), 77 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index b1aafe815f28..b54c4d771104 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -2279,7 +2279,7 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev) break; } - r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, + r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, NULL, ring->num_hw_submission, 0, timeout, adev->reset_domain->wq, ring->sched_score, ring->name, diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c index 345fec6cb1a4..618a804ddc34 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c @@ -134,7 +134,7 @@ int etnaviv_sched_init(struct etnaviv_gpu *gpu) { int ret; - ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, + ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, NULL, etnaviv_hw_jobs_limit, etnaviv_job_hang_limit, msecs_to_jiffies(500), NULL, NULL, dev_name(gpu->dev), gpu->dev); diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c index ffd91a5ee299..8d858aed0e56 100644 --- a/drivers/gpu/drm/lima/lima_sched.c +++ b/drivers/gpu/drm/lima/lima_sched.c @@ -488,7 +488,7 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name) INIT_WORK(&pipe->recover_work, lima_sched_recover_work); - return drm_sched_init(&pipe->base, &lima_sched_ops, 1, + return drm_sched_init(&pipe->base, &lima_sched_ops, NULL, 1, lima_job_hang_limit, msecs_to_jiffies(timeout), NULL, NULL, name, pipe->ldev->dev); diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.c b/drivers/gpu/drm/msm/msm_ringbuffer.c index 40c0bc35a44c..1097f8e93d6b 100644 --- a/drivers/gpu/drm/msm/msm_ringbuffer.c +++ b/drivers/gpu/drm/msm/msm_ringbuffer.c @@ -94,9 +94,10 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, /* currently managing hangcheck ourselves: */ sched_timeout = MAX_SCHEDULE_TIMEOUT; - ret = drm_sched_init(&ring->sched, &msm_sched_ops, - num_hw_submissions, 0, sched_timeout, - NULL, NULL, to_msm_bo(ring->bo)->name, gpu->dev->dev); + ret = drm_sched_init(&ring->sched, &msm_sched_ops, NULL, + num_hw_submissions, 0, sched_timeout, + NULL, NULL, to_msm_bo(ring->bo)->name, + gpu->dev->dev); if (ret) { goto fail; } diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c b/drivers/gpu/drm/nouveau/nouveau_sched.c index 3b7ea5221226..4c959dec42b3 100644 --- a/drivers/gpu/drm/nouveau/nouveau_sched.c +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c @@ -435,7 +435,7 @@ int nouveau_sched_init(struct nouveau_drm *drm) if (!drm->sched_wq) return -ENOMEM; - return drm_sched_init(sched, &nouveau_sched_ops, + return drm_sched_init(sched, &nouveau_sched_ops, NULL, NOUVEAU_SCHED_HW_SUBMISSIONS, 0, job_hang_limit, NULL, NULL, "nouveau_sched", drm->dev->dev); } diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index fb16de2d0420..934b7b930c76 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -852,7 +852,7 @@ int panfrost_job_init(struct panfrost_device *pfdev) js->queue[j].fence_context = dma_fence_context_alloc(1); ret = drm_sched_init(&js->queue[j].sched, - &panfrost_sched_ops, + &panfrost_sched_ops, NULL, nentries, 0, msecs_to_jiffies(JOB_TIMEOUT_MS), pfdev->reset.wq, diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 6f2f7dd4ba0a..8b1d52cff1e9 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -48,7 +48,6 @@ * through the jobs entity pointer. */ -#include #include #include #include @@ -256,6 +255,16 @@ drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq) return rb ? rb_entry(rb, struct drm_sched_entity, rb_tree_node) : NULL; } +/** + * drm_sched_wqueue_enqueue - enqueue scheduler submission + * @sched: scheduler instance + */ +static void drm_sched_wqueue_enqueue(struct drm_gpu_scheduler *sched) +{ + if (!READ_ONCE(sched->pause_submit)) + queue_work(sched->submit_wq, &sched->work_submit); +} + /** * drm_sched_job_done - complete a job * @s_job: pointer to the job which is done @@ -275,7 +284,7 @@ static void drm_sched_job_done(struct drm_sched_job *s_job, int result) dma_fence_get(&s_fence->finished); drm_sched_fence_finished(s_fence, result); dma_fence_put(&s_fence->finished); - wake_up_interruptible(&sched->wake_up_worker); + drm_sched_wqueue_enqueue(sched); } /** @@ -868,7 +877,7 @@ static bool drm_sched_can_queue(struct drm_gpu_scheduler *sched) void drm_sched_wakeup_if_can_queue(struct drm_gpu_scheduler *sched) { if (drm_sched_can_queue(sched)) - wake_up_interruptible(&sched->wake_up_worker); + drm_sched_wqueue_enqueue(sched); } /** @@ -978,61 +987,42 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list, } EXPORT_SYMBOL(drm_sched_pick_best); -/** - * drm_sched_blocked - check if the scheduler is blocked - * - * @sched: scheduler instance - * - * Returns true if blocked, otherwise false. - */ -static bool drm_sched_blocked(struct drm_gpu_scheduler *sched) -{ - if (kthread_should_park()) { - kthread_parkme(); - return true; - } - - return false; -} - /** * drm_sched_main - main scheduler thread * * @param: scheduler instance - * - * Returns 0. */ -static int drm_sched_main(void *param) +static void drm_sched_main(struct work_struct *w) { - struct drm_gpu_scheduler *sched = (struct drm_gpu_scheduler *)param; + struct drm_gpu_scheduler *sched = + container_of(w, struct drm_gpu_scheduler, work_submit); + struct drm_sched_entity *entity; + struct drm_sched_job *cleanup_job; int r; - sched_set_fifo_low(current); + if (READ_ONCE(sched->pause_submit)) + return; - while (!kthread_should_stop()) { - struct drm_sched_entity *entity = NULL; - struct drm_sched_fence *s_fence; - struct drm_sched_job *sched_job; - struct dma_fence *fence; - struct drm_sched_job *cleanup_job = NULL; + cleanup_job = drm_sched_get_cleanup_job(sched); + entity = drm_sched_select_entity(sched); - wait_event_interruptible(sched->wake_up_worker, - (cleanup_job = drm_sched_get_cleanup_job(sched)) || - (!drm_sched_blocked(sched) && - (entity = drm_sched_select_entity(sched))) || - kthread_should_stop()); + if (!entity && !cleanup_job) + return; /* No more work */ - if (cleanup_job) - sched->ops->free_job(cleanup_job); + if (cleanup_job) + sched->ops->free_job(cleanup_job); - if (!entity) - continue; + if (entity) { + struct dma_fence *fence; + struct drm_sched_fence *s_fence; + struct drm_sched_job *sched_job; sched_job = drm_sched_entity_pop_job(entity); - if (!sched_job) { complete_all(&entity->entity_idle); - continue; + if (!cleanup_job) + return; /* No more work */ + goto again; } s_fence = sched_job->s_fence; @@ -1063,7 +1053,9 @@ static int drm_sched_main(void *param) wake_up(&sched->job_scheduled); } - return 0; + +again: + drm_sched_wqueue_enqueue(sched); } /** @@ -1071,6 +1063,8 @@ static int drm_sched_main(void *param) * * @sched: scheduler instance * @ops: backend operations for this scheduler + * @submit_wq: workqueue to use for submission. If NULL, an ordered wq is + * allocated and used * @hw_submission: number of hw submissions that can be in flight * @hang_limit: number of times to allow a job to hang before dropping it * @timeout: timeout value in jiffies for the scheduler @@ -1084,14 +1078,25 @@ static int drm_sched_main(void *param) */ int drm_sched_init(struct drm_gpu_scheduler *sched, const struct drm_sched_backend_ops *ops, + struct workqueue_struct *submit_wq, unsigned hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, atomic_t *score, const char *name, struct device *dev) { - int i, ret; + int i; sched->ops = ops; sched->hw_submission_limit = hw_submission; sched->name = name; + if (submit_wq) { + sched->submit_wq = submit_wq; + sched->own_submit_wq = false; + } else { + sched->submit_wq = alloc_ordered_workqueue(name, 0); + if (!sched->submit_wq) + return -ENOMEM; + + sched->own_submit_wq = true; + } sched->timeout = timeout; sched->timeout_wq = timeout_wq ? : system_wq; sched->hang_limit = hang_limit; @@ -1100,23 +1105,15 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) drm_sched_rq_init(sched, &sched->sched_rq[i]); - init_waitqueue_head(&sched->wake_up_worker); init_waitqueue_head(&sched->job_scheduled); INIT_LIST_HEAD(&sched->pending_list); spin_lock_init(&sched->job_list_lock); atomic_set(&sched->hw_rq_count, 0); INIT_DELAYED_WORK(&sched->work_tdr, drm_sched_job_timedout); + INIT_WORK(&sched->work_submit, drm_sched_main); atomic_set(&sched->_score, 0); atomic64_set(&sched->job_id_count, 0); - - /* Each scheduler will run on a seperate kernel thread */ - sched->thread = kthread_run(drm_sched_main, sched, sched->name); - if (IS_ERR(sched->thread)) { - ret = PTR_ERR(sched->thread); - sched->thread = NULL; - DRM_DEV_ERROR(sched->dev, "Failed to create scheduler for %s.\n", name); - return ret; - } + sched->pause_submit = false; sched->ready = true; return 0; @@ -1135,8 +1132,7 @@ void drm_sched_fini(struct drm_gpu_scheduler *sched) struct drm_sched_entity *s_entity; int i; - if (sched->thread) - kthread_stop(sched->thread); + drm_sched_wqueue_stop(sched); for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { struct drm_sched_rq *rq = &sched->sched_rq[i]; @@ -1159,6 +1155,8 @@ void drm_sched_fini(struct drm_gpu_scheduler *sched) /* Confirm no work left behind accessing device structures */ cancel_delayed_work_sync(&sched->work_tdr); + if (sched->own_submit_wq) + destroy_workqueue(sched->submit_wq); sched->ready = false; } EXPORT_SYMBOL(drm_sched_fini); @@ -1216,7 +1214,7 @@ EXPORT_SYMBOL(drm_sched_increase_karma); */ bool drm_sched_wqueue_ready(struct drm_gpu_scheduler *sched) { - return !!sched->thread; + return sched->ready; } EXPORT_SYMBOL(drm_sched_wqueue_ready); @@ -1227,7 +1225,8 @@ EXPORT_SYMBOL(drm_sched_wqueue_ready); */ void drm_sched_wqueue_stop(struct drm_gpu_scheduler *sched) { - kthread_park(sched->thread); + WRITE_ONCE(sched->pause_submit, true); + cancel_work_sync(&sched->work_submit); } EXPORT_SYMBOL(drm_sched_wqueue_stop); @@ -1238,6 +1237,7 @@ EXPORT_SYMBOL(drm_sched_wqueue_stop); */ void drm_sched_wqueue_start(struct drm_gpu_scheduler *sched) { - kthread_unpark(sched->thread); + WRITE_ONCE(sched->pause_submit, false); + queue_work(sched->submit_wq, &sched->work_submit); } EXPORT_SYMBOL(drm_sched_wqueue_start); diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index 06238e6d7f5c..38e092ea41e6 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -388,7 +388,7 @@ v3d_sched_init(struct v3d_dev *v3d) int ret; ret = drm_sched_init(&v3d->queue[V3D_BIN].sched, - &v3d_bin_sched_ops, + &v3d_bin_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_bin", v3d->drm.dev); @@ -396,7 +396,7 @@ v3d_sched_init(struct v3d_dev *v3d) return ret; ret = drm_sched_init(&v3d->queue[V3D_RENDER].sched, - &v3d_render_sched_ops, + &v3d_render_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_render", v3d->drm.dev); @@ -404,7 +404,7 @@ v3d_sched_init(struct v3d_dev *v3d) goto fail; ret = drm_sched_init(&v3d->queue[V3D_TFU].sched, - &v3d_tfu_sched_ops, + &v3d_tfu_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_tfu", v3d->drm.dev); @@ -413,7 +413,7 @@ v3d_sched_init(struct v3d_dev *v3d) if (v3d_has_csd(v3d)) { ret = drm_sched_init(&v3d->queue[V3D_CSD].sched, - &v3d_csd_sched_ops, + &v3d_csd_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_csd", v3d->drm.dev); @@ -421,7 +421,7 @@ v3d_sched_init(struct v3d_dev *v3d) goto fail; ret = drm_sched_init(&v3d->queue[V3D_CACHE_CLEAN].sched, - &v3d_cache_clean_sched_ops, + &v3d_cache_clean_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_cache_clean", v3d->drm.dev); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 38578fe74573..211bd3cdabdc 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -473,17 +473,16 @@ struct drm_sched_backend_ops { * @timeout: the time after which a job is removed from the scheduler. * @name: name of the ring for which this scheduler is being used. * @sched_rq: priority wise array of run queues. - * @wake_up_worker: the wait queue on which the scheduler sleeps until a job - * is ready to be scheduled. * @job_scheduled: once @drm_sched_entity_do_release is called the scheduler * waits on this wait queue until all the scheduled jobs are * finished. * @hw_rq_count: the number of jobs currently in the hardware queue. * @job_id_count: used to assign unique id to the each job. + * @submit_wq: workqueue used to queue @work_submit * @timeout_wq: workqueue used to queue @work_tdr + * @work_submit: schedules jobs and cleans up entities * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the * timeout interval is over. - * @thread: the kthread on which the scheduler which run. * @pending_list: the list of jobs which are currently in the job queue. * @job_list_lock: lock to protect the pending_list. * @hang_limit: once the hangs by a job crosses this limit then it is marked @@ -492,6 +491,8 @@ struct drm_sched_backend_ops { * @_score: score used when the driver doesn't provide one * @ready: marks if the underlying HW is ready to work * @free_guilty: A hit to time out handler to free the guilty job. + * @pause_submit: pause queuing of @work_submit on @submit_wq + * @own_submit_wq: scheduler owns allocation of @submit_wq * @dev: system &struct device * * One scheduler is implemented for each hardware ring. @@ -502,13 +503,13 @@ struct drm_gpu_scheduler { long timeout; const char *name; struct drm_sched_rq sched_rq[DRM_SCHED_PRIORITY_COUNT]; - wait_queue_head_t wake_up_worker; wait_queue_head_t job_scheduled; atomic_t hw_rq_count; atomic64_t job_id_count; + struct workqueue_struct *submit_wq; struct workqueue_struct *timeout_wq; + struct work_struct work_submit; struct delayed_work work_tdr; - struct task_struct *thread; struct list_head pending_list; spinlock_t job_list_lock; int hang_limit; @@ -516,11 +517,14 @@ struct drm_gpu_scheduler { atomic_t _score; bool ready; bool free_guilty; + bool pause_submit; + bool own_submit_wq; struct device *dev; }; int drm_sched_init(struct drm_gpu_scheduler *sched, const struct drm_sched_backend_ops *ops, + struct workqueue_struct *submit_wq, uint32_t hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, atomic_t *score, const char *name, struct device *dev); From patchwork Tue Oct 17 15:09:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13425504 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 11235CDB474 for ; Tue, 17 Oct 2023 15:09:49 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4C88D10E304; Tue, 17 Oct 2023 15:09:38 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id F07B110E128; Tue, 17 Oct 2023 15:09:35 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697555376; x=1729091376; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=wkLwS++2y7AGfBPu4rf4lz4XmBGwXF+oXgNmQ3c4ZQg=; b=GXtvI1tZSm1KBjrhCYbbbAWBK3zBYvSq30sHBQMSTveqNtcu4Cu30dzW Nef6q5+v3pq0LChtKY9uXte/V9CYfFTYyJyyhiWTmq5bp7c8xBgbeqGBA 4uMGMlyY/IHhi5gpA3gf3zFJrgThNDgi/VJKRGv833aO1In1mkSy4FG16 sK1EP9N/ZmksbWf3UKRsj5Uz/1BKBE9Kv9qx8bJSKNhE+QU2NlMvrZQIu 0t0ln66J4EGH46GaEfqGqF/+N3urE3kT7RvUmipWNSBjJWEo88pNaWGjW 8/w1Gdto9ZvQt6Hsqjsz1oRlEb5uxqsVHmIQS4nD79q1F5c7Ga85IPcdS Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="388665103" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="388665103" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="791257569" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="791257569" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:33 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v6 3/7] drm/sched: Move schedule policy to scheduler Date: Tue, 17 Oct 2023 08:09:54 -0700 Message-Id: <20231017150958.838613-4-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017150958.838613-1-matthew.brost@intel.com> References: <20231017150958.838613-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , lina@asahilina.net, sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, mcanal@igalia.com, luben.tuikov@amd.com, dakr@redhat.com, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Rather than a global modparam for scheduling policy, move the scheduling policy to scheduler so user can control each scheduler policy. v2: - s/DRM_SCHED_POLICY_MAX/DRM_SCHED_POLICY_COUNT (Luben) - Only include policy in scheduler (Luben) v3: - use a ternary operator as opposed to an if-control (Luben) - s/DRM_SCHED_POLICY_DEFAULT/DRM_SCHED_POLICY_UNSET/ (Luben) - s/default_drm_sched_policy/drm_sched_policy_default/ (Luben) - Update commit message (Boris) - Fix v3d build (CI) - s/bad_policies/drm_sched_policy_mismatch/ (Luben) - Don't update modparam doc (Luben) v4: - Fix alignment in msm_ringbuffer_new (Luben / checkpatch) Signed-off-by: Matthew Brost Reviewed-by: Luben Tuikov --- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/etnaviv/etnaviv_sched.c | 3 ++- drivers/gpu/drm/lima/lima_sched.c | 3 ++- drivers/gpu/drm/msm/msm_ringbuffer.c | 2 +- drivers/gpu/drm/nouveau/nouveau_sched.c | 3 ++- drivers/gpu/drm/panfrost/panfrost_job.c | 3 ++- drivers/gpu/drm/scheduler/sched_entity.c | 24 ++++++++++++++++++---- drivers/gpu/drm/scheduler/sched_main.c | 19 ++++++++++++----- drivers/gpu/drm/v3d/v3d_sched.c | 15 +++++++++----- include/drm/gpu_scheduler.h | 20 ++++++++++++------ 10 files changed, 68 insertions(+), 25 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index b54c4d771104..e4e6f91450a4 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -2283,6 +2283,7 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev) ring->num_hw_submission, 0, timeout, adev->reset_domain->wq, ring->sched_score, ring->name, + DRM_SCHED_POLICY_UNSET, adev->dev); if (r) { DRM_ERROR("Failed to create scheduler on ring %s.\n", diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c index 618a804ddc34..15b0e2f1abe5 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c @@ -137,7 +137,8 @@ int etnaviv_sched_init(struct etnaviv_gpu *gpu) ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, NULL, etnaviv_hw_jobs_limit, etnaviv_job_hang_limit, msecs_to_jiffies(500), NULL, NULL, - dev_name(gpu->dev), gpu->dev); + dev_name(gpu->dev), DRM_SCHED_POLICY_UNSET, + gpu->dev); if (ret) return ret; diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c index 8d858aed0e56..50c2075228aa 100644 --- a/drivers/gpu/drm/lima/lima_sched.c +++ b/drivers/gpu/drm/lima/lima_sched.c @@ -491,7 +491,8 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name) return drm_sched_init(&pipe->base, &lima_sched_ops, NULL, 1, lima_job_hang_limit, msecs_to_jiffies(timeout), NULL, - NULL, name, pipe->ldev->dev); + NULL, name, DRM_SCHED_POLICY_UNSET, + pipe->ldev->dev); } void lima_sched_pipe_fini(struct lima_sched_pipe *pipe) diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.c b/drivers/gpu/drm/msm/msm_ringbuffer.c index 1097f8e93d6b..173ad2f17c50 100644 --- a/drivers/gpu/drm/msm/msm_ringbuffer.c +++ b/drivers/gpu/drm/msm/msm_ringbuffer.c @@ -97,7 +97,7 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, ret = drm_sched_init(&ring->sched, &msm_sched_ops, NULL, num_hw_submissions, 0, sched_timeout, NULL, NULL, to_msm_bo(ring->bo)->name, - gpu->dev->dev); + DRM_SCHED_POLICY_UNSET, gpu->dev->dev); if (ret) { goto fail; } diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c b/drivers/gpu/drm/nouveau/nouveau_sched.c index 4c959dec42b3..c4e09d2e77f9 100644 --- a/drivers/gpu/drm/nouveau/nouveau_sched.c +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c @@ -437,7 +437,8 @@ int nouveau_sched_init(struct nouveau_drm *drm) return drm_sched_init(sched, &nouveau_sched_ops, NULL, NOUVEAU_SCHED_HW_SUBMISSIONS, 0, job_hang_limit, - NULL, NULL, "nouveau_sched", drm->dev->dev); + NULL, NULL, "nouveau_sched", + DRM_SCHED_POLICY_UNSET, drm->dev->dev); } void nouveau_sched_fini(struct nouveau_drm *drm) diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index 934b7b930c76..95330ff402ba 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -856,7 +856,8 @@ int panfrost_job_init(struct panfrost_device *pfdev) nentries, 0, msecs_to_jiffies(JOB_TIMEOUT_MS), pfdev->reset.wq, - NULL, "pan_js", pfdev->dev); + NULL, "pan_js", DRM_SCHED_POLICY_UNSET, + pfdev->dev); if (ret) { dev_err(pfdev->dev, "Failed to create scheduler: %d.", ret); goto err_sched; diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index a42763e1429d..cf42e2265d64 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -33,6 +33,20 @@ #define to_drm_sched_job(sched_job) \ container_of((sched_job), struct drm_sched_job, queue_node) +static bool drm_sched_policy_mismatch(struct drm_gpu_scheduler **sched_list, + unsigned int num_sched_list) +{ + enum drm_sched_policy sched_policy = sched_list[0]->sched_policy; + unsigned int i; + + /* All schedule policies must match */ + for (i = 1; i < num_sched_list; ++i) + if (sched_policy != sched_list[i]->sched_policy) + return true; + + return false; +} + /** * drm_sched_entity_init - Init a context entity used by scheduler when * submit to HW ring. @@ -62,7 +76,8 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, unsigned int num_sched_list, atomic_t *guilty) { - if (!(entity && sched_list && (num_sched_list == 0 || sched_list[0]))) + if (!(entity && sched_list && (num_sched_list == 0 || sched_list[0])) || + drm_sched_policy_mismatch(sched_list, num_sched_list)) return -EINVAL; memset(entity, 0, sizeof(struct drm_sched_entity)); @@ -486,7 +501,7 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) * Update the entity's location in the min heap according to * the timestamp of the next job, if any. */ - if (drm_sched_policy == DRM_SCHED_POLICY_FIFO) { + if (entity->rq->sched->sched_policy == DRM_SCHED_POLICY_FIFO) { struct drm_sched_job *next; next = to_drm_sched_job(spsc_queue_peek(&entity->job_queue)); @@ -558,7 +573,8 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity) void drm_sched_entity_push_job(struct drm_sched_job *sched_job) { struct drm_sched_entity *entity = sched_job->entity; - bool first; + bool first, fifo = entity->rq->sched->sched_policy == + DRM_SCHED_POLICY_FIFO; ktime_t submit_ts; trace_drm_sched_job(sched_job, entity); @@ -587,7 +603,7 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) drm_sched_rq_add_entity(entity->rq, entity); spin_unlock(&entity->rq_lock); - if (drm_sched_policy == DRM_SCHED_POLICY_FIFO) + if (fifo) drm_sched_rq_update_fifo(entity, submit_ts); drm_sched_wakeup_if_can_queue(entity->rq->sched); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 8b1d52cff1e9..150e5330f0fa 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -66,14 +66,14 @@ #define to_drm_sched_job(sched_job) \ container_of((sched_job), struct drm_sched_job, queue_node) -int drm_sched_policy = DRM_SCHED_POLICY_FIFO; +int drm_sched_policy_default = DRM_SCHED_POLICY_FIFO; /** * DOC: sched_policy (int) * Used to override default entities scheduling policy in a run queue. */ MODULE_PARM_DESC(sched_policy, "Specify the scheduling policy for entities on a run-queue, " __stringify(DRM_SCHED_POLICY_RR) " = Round Robin, " __stringify(DRM_SCHED_POLICY_FIFO) " = FIFO (default)."); -module_param_named(sched_policy, drm_sched_policy, int, 0444); +module_param_named(sched_policy, drm_sched_policy_default, int, 0444); static __always_inline bool drm_sched_entity_compare_before(struct rb_node *a, const struct rb_node *b) @@ -177,7 +177,7 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq, if (rq->current_entity == entity) rq->current_entity = NULL; - if (drm_sched_policy == DRM_SCHED_POLICY_FIFO) + if (rq->sched->sched_policy == DRM_SCHED_POLICY_FIFO) drm_sched_rq_remove_fifo_locked(entity); spin_unlock(&rq->lock); @@ -898,7 +898,7 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched) /* Kernel run queue has higher priority than normal run queue*/ for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { - entity = drm_sched_policy == DRM_SCHED_POLICY_FIFO ? + entity = sched->sched_policy == DRM_SCHED_POLICY_FIFO ? drm_sched_rq_select_entity_fifo(&sched->sched_rq[i]) : drm_sched_rq_select_entity_rr(&sched->sched_rq[i]); if (entity) @@ -1072,6 +1072,7 @@ static void drm_sched_main(struct work_struct *w) * used * @score: optional score atomic shared with other schedulers * @name: name used for debugging + * @sched_policy: schedule policy * @dev: target &struct device * * Return 0 on success, otherwise error code. @@ -1081,9 +1082,15 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, struct workqueue_struct *submit_wq, unsigned hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, - atomic_t *score, const char *name, struct device *dev) + atomic_t *score, const char *name, + enum drm_sched_policy sched_policy, + struct device *dev) { int i; + + if (sched_policy >= DRM_SCHED_POLICY_COUNT) + return -EINVAL; + sched->ops = ops; sched->hw_submission_limit = hw_submission; sched->name = name; @@ -1102,6 +1109,8 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, sched->hang_limit = hang_limit; sched->score = score ? score : &sched->_score; sched->dev = dev; + sched->sched_policy = sched_policy == DRM_SCHED_POLICY_UNSET ? + drm_sched_policy_default : sched_policy; for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) drm_sched_rq_init(sched, &sched->sched_rq[i]); diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index 38e092ea41e6..dec89c5b8cb1 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -391,7 +391,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_bin_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_bin", v3d->drm.dev); + NULL, "v3d_bin", DRM_SCHED_POLICY_UNSET, + v3d->drm.dev); if (ret) return ret; @@ -399,7 +400,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_render_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_render", v3d->drm.dev); + NULL, "v3d_render", DRM_SCHED_POLICY_UNSET, + v3d->drm.dev); if (ret) goto fail; @@ -407,7 +409,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_tfu_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_tfu", v3d->drm.dev); + NULL, "v3d_tfu", DRM_SCHED_POLICY_UNSET, + v3d->drm.dev); if (ret) goto fail; @@ -416,7 +419,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_csd_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_csd", v3d->drm.dev); + NULL, "v3d_csd", DRM_SCHED_POLICY_UNSET, + v3d->drm.dev); if (ret) goto fail; @@ -424,7 +428,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_cache_clean_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_cache_clean", v3d->drm.dev); + NULL, "v3d_cache_clean", + DRM_SCHED_POLICY_UNSET, v3d->drm.dev); if (ret) goto fail; } diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 211bd3cdabdc..320f0a720486 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -72,11 +72,15 @@ enum drm_sched_priority { DRM_SCHED_PRIORITY_UNSET = -2 }; -/* Used to chose between FIFO and RR jobs scheduling */ -extern int drm_sched_policy; - -#define DRM_SCHED_POLICY_RR 0 -#define DRM_SCHED_POLICY_FIFO 1 +/* Used to chose default scheduling policy*/ +extern int default_drm_sched_policy; + +enum drm_sched_policy { + DRM_SCHED_POLICY_UNSET, + DRM_SCHED_POLICY_RR, + DRM_SCHED_POLICY_FIFO, + DRM_SCHED_POLICY_COUNT, +}; /** * struct drm_sched_entity - A wrapper around a job queue (typically @@ -489,6 +493,7 @@ struct drm_sched_backend_ops { * guilty and it will no longer be considered for scheduling. * @score: score to help loadbalancer pick a idle sched * @_score: score used when the driver doesn't provide one + * @sched_policy: Schedule policy for scheduler * @ready: marks if the underlying HW is ready to work * @free_guilty: A hit to time out handler to free the guilty job. * @pause_submit: pause queuing of @work_submit on @submit_wq @@ -515,6 +520,7 @@ struct drm_gpu_scheduler { int hang_limit; atomic_t *score; atomic_t _score; + enum drm_sched_policy sched_policy; bool ready; bool free_guilty; bool pause_submit; @@ -527,7 +533,9 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, struct workqueue_struct *submit_wq, uint32_t hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, - atomic_t *score, const char *name, struct device *dev); + atomic_t *score, const char *name, + enum drm_sched_policy sched_policy, + struct device *dev); void drm_sched_fini(struct drm_gpu_scheduler *sched); int drm_sched_job_init(struct drm_sched_job *job, From patchwork Tue Oct 17 15:09:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13425505 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8E64CCDB482 for ; Tue, 17 Oct 2023 15:09:51 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D71A910E30E; Tue, 17 Oct 2023 15:09:42 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5710710E301; Tue, 17 Oct 2023 15:09:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697555376; x=1729091376; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=i1QyKVYZyeEa+g/xWuFwx7rT61BrGCSq1TLjWhiK5Cg=; b=V4/PxWMKfACzvPHTb2G9PL/9iRJENJ70H41zQN0syEeuLO2V1JDowdSW 64NfSTOWZVTKjtY7iMyD5FiAi40sv7cuyfjLMnXK6ldAY8EBQTRcK2yCf mVjn/vi+w3FpdsjmKZxoX1+v4L5CxwZ9YJoerC+85Px4PMEm0ndgO59yX daO6L0zyYT+O+RbTVGNcQcx56uBEtawYMGGGloAZWBU5gIwvnjGlPa2RW oAa8RQAuqOoKBZ5JywFEaH134jKNHVwa7nsf6uMCE/shUDPBg55PfA0Xn Wc4+nNhLH0tGHrMTJdWS/0KJTuUWc5SoKxhlzaoDHKTJCwzxT9YDRFsU3 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="388665115" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="388665115" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="791257576" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="791257576" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:33 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v6 4/7] drm/sched: Add DRM_SCHED_POLICY_SINGLE_ENTITY scheduling policy Date: Tue, 17 Oct 2023 08:09:55 -0700 Message-Id: <20231017150958.838613-5-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017150958.838613-1-matthew.brost@intel.com> References: <20231017150958.838613-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , lina@asahilina.net, sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, mcanal@igalia.com, luben.tuikov@amd.com, dakr@redhat.com, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" DRM_SCHED_POLICY_SINGLE_ENTITY creates a 1 to 1 relationship between scheduler and entity. No priorities or run queue used in this mode. Intended for devices with firmware schedulers. v2: - Drop sched / rq union (Luben) v3: - Don't pick entity if stopped in drm_sched_select_entity (Danilo) v4: - Rework if statement in drm_sched_entity_init (Luben) - Update comment for drm_sched_entity_to_scheduler (Luben) - Reword a few things in DOC comment (Luben) - Do not check sched policy in for statement (Luben) v5: - Fix extra blank lines (Luben / Checkpatch) Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_entity.c | 69 +++++++++++++++---- drivers/gpu/drm/scheduler/sched_fence.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 87 ++++++++++++++++++------ include/drm/gpu_scheduler.h | 8 +++ 4 files changed, 130 insertions(+), 36 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index cf42e2265d64..940f63dd6965 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -83,6 +83,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, memset(entity, 0, sizeof(struct drm_sched_entity)); INIT_LIST_HEAD(&entity->list); entity->rq = NULL; + entity->single_sched = NULL; entity->guilty = guilty; entity->num_sched_list = num_sched_list; entity->priority = priority; @@ -90,8 +91,17 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, RCU_INIT_POINTER(entity->last_scheduled, NULL); RB_CLEAR_NODE(&entity->rb_tree_node); - if(num_sched_list) - entity->rq = &sched_list[0]->sched_rq[entity->priority]; + if (num_sched_list) { + if (sched_list[0]->sched_policy != + DRM_SCHED_POLICY_SINGLE_ENTITY) { + entity->rq = &sched_list[0]->sched_rq[entity->priority]; + } else if (num_sched_list == 1 && !sched_list[0]->single_entity) { + sched_list[0]->single_entity = entity; + entity->single_sched = sched_list[0]; + } else { + return -EINVAL; + } + } init_completion(&entity->entity_idle); @@ -124,7 +134,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list) { - WARN_ON(!num_sched_list || !sched_list); + WARN_ON(!num_sched_list || !sched_list || + !!entity->single_sched); entity->sched_list = sched_list; entity->num_sched_list = num_sched_list; @@ -231,13 +242,15 @@ static void drm_sched_entity_kill(struct drm_sched_entity *entity) { struct drm_sched_job *job; struct dma_fence *prev; + bool single_entity = !!entity->single_sched; - if (!entity->rq) + if (!entity->rq && !single_entity) return; spin_lock(&entity->rq_lock); entity->stopped = true; - drm_sched_rq_remove_entity(entity->rq, entity); + if (!single_entity) + drm_sched_rq_remove_entity(entity->rq, entity); spin_unlock(&entity->rq_lock); /* Make sure this entity is not used by the scheduler at the moment */ @@ -259,6 +272,20 @@ static void drm_sched_entity_kill(struct drm_sched_entity *entity) dma_fence_put(prev); } +/** + * drm_sched_entity_to_scheduler - Schedule entity to GPU scheduler + * @entity: scheduler entity + * + * Returns GPU scheduler for the entity + */ +struct drm_gpu_scheduler * +drm_sched_entity_to_scheduler(struct drm_sched_entity *entity) +{ + bool single_entity = !!entity->single_sched; + + return single_entity ? entity->single_sched : entity->rq->sched; +} + /** * drm_sched_entity_flush - Flush a context entity * @@ -276,11 +303,12 @@ long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout) struct drm_gpu_scheduler *sched; struct task_struct *last_user; long ret = timeout; + bool single_entity = !!entity->single_sched; - if (!entity->rq) + if (!entity->rq && !single_entity) return 0; - sched = entity->rq->sched; + sched = drm_sched_entity_to_scheduler(entity); /** * The client will not queue more IBs during this fini, consume existing * queued IBs or discard them on SIGKILL @@ -373,7 +401,7 @@ static void drm_sched_entity_wakeup(struct dma_fence *f, container_of(cb, struct drm_sched_entity, cb); drm_sched_entity_clear_dep(f, cb); - drm_sched_wakeup_if_can_queue(entity->rq->sched); + drm_sched_wakeup_if_can_queue(drm_sched_entity_to_scheduler(entity)); } /** @@ -387,6 +415,8 @@ static void drm_sched_entity_wakeup(struct dma_fence *f, void drm_sched_entity_set_priority(struct drm_sched_entity *entity, enum drm_sched_priority priority) { + WARN_ON(!!entity->single_sched); + spin_lock(&entity->rq_lock); entity->priority = priority; spin_unlock(&entity->rq_lock); @@ -399,7 +429,7 @@ EXPORT_SYMBOL(drm_sched_entity_set_priority); */ static bool drm_sched_entity_add_dependency_cb(struct drm_sched_entity *entity) { - struct drm_gpu_scheduler *sched = entity->rq->sched; + struct drm_gpu_scheduler *sched = drm_sched_entity_to_scheduler(entity); struct dma_fence *fence = entity->dependency; struct drm_sched_fence *s_fence; @@ -501,7 +531,8 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) * Update the entity's location in the min heap according to * the timestamp of the next job, if any. */ - if (entity->rq->sched->sched_policy == DRM_SCHED_POLICY_FIFO) { + if (drm_sched_entity_to_scheduler(entity)->sched_policy == + DRM_SCHED_POLICY_FIFO) { struct drm_sched_job *next; next = to_drm_sched_job(spsc_queue_peek(&entity->job_queue)); @@ -524,6 +555,8 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity) struct drm_gpu_scheduler *sched; struct drm_sched_rq *rq; + WARN_ON(!!entity->single_sched); + /* single possible engine and already selected */ if (!entity->sched_list) return; @@ -573,12 +606,13 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity) void drm_sched_entity_push_job(struct drm_sched_job *sched_job) { struct drm_sched_entity *entity = sched_job->entity; - bool first, fifo = entity->rq->sched->sched_policy == - DRM_SCHED_POLICY_FIFO; + bool single_entity = !!entity->single_sched; + bool first; ktime_t submit_ts; trace_drm_sched_job(sched_job, entity); - atomic_inc(entity->rq->sched->score); + if (!single_entity) + atomic_inc(entity->rq->sched->score); WRITE_ONCE(entity->last_user, current->group_leader); /* @@ -591,6 +625,10 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) /* first job wakes up scheduler */ if (first) { + struct drm_gpu_scheduler *sched = + drm_sched_entity_to_scheduler(entity); + bool fifo = sched->sched_policy == DRM_SCHED_POLICY_FIFO; + /* Add the entity to the run queue */ spin_lock(&entity->rq_lock); if (entity->stopped) { @@ -600,13 +638,14 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) return; } - drm_sched_rq_add_entity(entity->rq, entity); + if (!single_entity) + drm_sched_rq_add_entity(entity->rq, entity); spin_unlock(&entity->rq_lock); if (fifo) drm_sched_rq_update_fifo(entity, submit_ts); - drm_sched_wakeup_if_can_queue(entity->rq->sched); + drm_sched_wakeup_if_can_queue(sched); } } EXPORT_SYMBOL(drm_sched_entity_push_job); diff --git a/drivers/gpu/drm/scheduler/sched_fence.c b/drivers/gpu/drm/scheduler/sched_fence.c index 06cedfe4b486..f6b926f5e188 100644 --- a/drivers/gpu/drm/scheduler/sched_fence.c +++ b/drivers/gpu/drm/scheduler/sched_fence.c @@ -225,7 +225,7 @@ void drm_sched_fence_init(struct drm_sched_fence *fence, { unsigned seq; - fence->sched = entity->rq->sched; + fence->sched = drm_sched_entity_to_scheduler(entity); seq = atomic_inc_return(&entity->fence_seq); dma_fence_init(&fence->scheduled, &drm_sched_fence_ops_scheduled, &fence->lock, entity->fence_context, seq); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 150e5330f0fa..273e0fbc4eab 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -32,7 +32,8 @@ * backend operations to the scheduler like submitting a job to hardware run queue, * returning the dependencies of a job etc. * - * The organisation of the scheduler is the following: + * For scheduling policies DRM_SCHED_POLICY_RR and DRM_SCHED_POLICY_FIFO, the + * scheduler organization is: * * 1. Each hw run queue has one scheduler * 2. Each scheduler has multiple run queues with different priorities @@ -43,7 +44,24 @@ * * The jobs in a entity are always scheduled in the order that they were pushed. * - * Note that once a job was taken from the entities queue and pushed to the + * For DRM_SCHED_POLICY_SINGLE_ENTITY, the organization of the scheduler is: + * + * 1. One to one relationship between scheduler and entity + * 2. No priorities implemented per scheduler (single job queue) + * 3. No run queues in scheduler rather jobs are directly dequeued from entity + * 4. The entity maintains a queue of jobs that will be scheduled to the + * hardware + * + * The jobs in a entity are always scheduled in the order that they were pushed + * regardless of scheduling policy. Single-entity scheduling is essentially a + * FIFO for jobs. + * + * A policy of DRM_SCHED_POLICY_RR or DRM_SCHED_POLICY_FIFO is expected to be + * used when the kernel-mode driver is scheduling directly to the hardware while + * a scheduling policy of DRM_SCHED_POLICY_SINGLE_ENTITY is expected to be used + * when there is a firmware scheduler. + * + * Note that once a job is taken from the entities queue and pushed to the * hardware, i.e. the pending queue, the entity must not be referenced anymore * through the jobs entity pointer. */ @@ -96,6 +114,8 @@ static inline void drm_sched_rq_remove_fifo_locked(struct drm_sched_entity *enti void drm_sched_rq_update_fifo(struct drm_sched_entity *entity, ktime_t ts) { + WARN_ON(!!entity->single_sched); + /* * Both locks need to be grabbed, one to protect from entity->rq change * for entity from within concurrent drm_sched_entity_select_rq and the @@ -126,6 +146,8 @@ void drm_sched_rq_update_fifo(struct drm_sched_entity *entity, ktime_t ts) static void drm_sched_rq_init(struct drm_gpu_scheduler *sched, struct drm_sched_rq *rq) { + WARN_ON(sched->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + spin_lock_init(&rq->lock); INIT_LIST_HEAD(&rq->entities); rq->rb_tree_root = RB_ROOT_CACHED; @@ -144,6 +166,8 @@ static void drm_sched_rq_init(struct drm_gpu_scheduler *sched, void drm_sched_rq_add_entity(struct drm_sched_rq *rq, struct drm_sched_entity *entity) { + WARN_ON(!!entity->single_sched); + if (!list_empty(&entity->list)) return; @@ -166,6 +190,8 @@ void drm_sched_rq_add_entity(struct drm_sched_rq *rq, void drm_sched_rq_remove_entity(struct drm_sched_rq *rq, struct drm_sched_entity *entity) { + WARN_ON(!!entity->single_sched); + if (list_empty(&entity->list)) return; @@ -641,7 +667,7 @@ int drm_sched_job_init(struct drm_sched_job *job, struct drm_sched_entity *entity, void *owner) { - if (!entity->rq) + if (!entity->rq && !entity->single_sched) return -ENOENT; job->entity = entity; @@ -674,13 +700,16 @@ void drm_sched_job_arm(struct drm_sched_job *job) { struct drm_gpu_scheduler *sched; struct drm_sched_entity *entity = job->entity; + bool single_entity = !!entity->single_sched; BUG_ON(!entity); - drm_sched_entity_select_rq(entity); - sched = entity->rq->sched; + if (!single_entity) + drm_sched_entity_select_rq(entity); + sched = drm_sched_entity_to_scheduler(entity); job->sched = sched; - job->s_priority = entity->rq - sched->sched_rq; + if (!single_entity) + job->s_priority = entity->rq - sched->sched_rq; job->id = atomic64_inc_return(&sched->job_id_count); drm_sched_fence_init(job->s_fence, job->entity); @@ -896,6 +925,14 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched) if (!drm_sched_can_queue(sched)) return NULL; + if (sched->single_entity) { + if (!READ_ONCE(sched->single_entity->stopped) && + drm_sched_entity_is_ready(sched->single_entity)) + return sched->single_entity; + + return NULL; + } + /* Kernel run queue has higher priority than normal run queue*/ for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { entity = sched->sched_policy == DRM_SCHED_POLICY_FIFO ? @@ -1092,6 +1129,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, return -EINVAL; sched->ops = ops; + sched->single_entity = NULL; sched->hw_submission_limit = hw_submission; sched->name = name; if (submit_wq) { @@ -1111,8 +1149,10 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, sched->dev = dev; sched->sched_policy = sched_policy == DRM_SCHED_POLICY_UNSET ? drm_sched_policy_default : sched_policy; - for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) - drm_sched_rq_init(sched, &sched->sched_rq[i]); + if (sched_policy != DRM_SCHED_POLICY_SINGLE_ENTITY) { + for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) + drm_sched_rq_init(sched, &sched->sched_rq[i]); + } init_waitqueue_head(&sched->job_scheduled); INIT_LIST_HEAD(&sched->pending_list); @@ -1143,19 +1183,24 @@ void drm_sched_fini(struct drm_gpu_scheduler *sched) drm_sched_wqueue_stop(sched); - for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { - struct drm_sched_rq *rq = &sched->sched_rq[i]; - - spin_lock(&rq->lock); - list_for_each_entry(s_entity, &rq->entities, list) - /* - * Prevents reinsertion and marks job_queue as idle, - * it will removed from rq in drm_sched_entity_fini - * eventually - */ - s_entity->stopped = true; - spin_unlock(&rq->lock); + if (sched->single_entity) { + spin_lock(&sched->single_entity->rq_lock); + sched->single_entity->stopped = true; + spin_unlock(&sched->single_entity->rq_lock); + } else if (sched->sched_policy != DRM_SCHED_POLICY_SINGLE_ENTITY) { + for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { + struct drm_sched_rq *rq = &sched->sched_rq[i]; + spin_lock(&rq->lock); + list_for_each_entry(s_entity, &rq->entities, list) + /* + * Prevents reinsertion and marks job_queue as idle, + * it will removed from rq in drm_sched_entity_fini + * eventually + */ + s_entity->stopped = true; + spin_unlock(&rq->lock); + } } /* Wakeup everyone stuck in drm_sched_entity_flush for this scheduler */ @@ -1186,6 +1231,8 @@ void drm_sched_increase_karma(struct drm_sched_job *bad) struct drm_sched_entity *entity; struct drm_gpu_scheduler *sched = bad->sched; + WARN_ON(sched->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + /* don't change @bad's karma if it's from KERNEL RQ, * because sometimes GPU hang would cause kernel jobs (like VM updating jobs) * corrupt but keep in mind that kernel jobs always considered good. diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 320f0a720486..e67b53c3546b 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -79,6 +79,7 @@ enum drm_sched_policy { DRM_SCHED_POLICY_UNSET, DRM_SCHED_POLICY_RR, DRM_SCHED_POLICY_FIFO, + DRM_SCHED_POLICY_SINGLE_ENTITY, DRM_SCHED_POLICY_COUNT, }; @@ -112,6 +113,9 @@ struct drm_sched_entity { */ struct drm_sched_rq *rq; + /** @single_sched: Single scheduler */ + struct drm_gpu_scheduler *single_sched; + /** * @sched_list: * @@ -473,6 +477,7 @@ struct drm_sched_backend_ops { * struct drm_gpu_scheduler - scheduler instance-specific data * * @ops: backend operations provided by the driver. + * @single_entity: Single entity for the scheduler * @hw_submission_limit: the max size of the hardware queue. * @timeout: the time after which a job is removed from the scheduler. * @name: name of the ring for which this scheduler is being used. @@ -504,6 +509,7 @@ struct drm_sched_backend_ops { */ struct drm_gpu_scheduler { const struct drm_sched_backend_ops *ops; + struct drm_sched_entity *single_entity; uint32_t hw_submission_limit; long timeout; const char *name; @@ -587,6 +593,8 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list, atomic_t *guilty); +struct drm_gpu_scheduler * +drm_sched_entity_to_scheduler(struct drm_sched_entity *entity); long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout); void drm_sched_entity_fini(struct drm_sched_entity *entity); void drm_sched_entity_destroy(struct drm_sched_entity *entity); From patchwork Tue Oct 17 15:09:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13425508 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 30CD6CDB474 for ; Tue, 17 Oct 2023 15:09:57 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5C11210E311; Tue, 17 Oct 2023 15:09:45 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id DBDB010E305; Tue, 17 Oct 2023 15:09:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697555378; x=1729091378; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=MAtdW/AgX9+BWLux24tRMuj6GhRpFBd0bpW3xriGTIM=; b=nw24MoOYZmCCck76Nas6HnW2DUwBOAWbHogw7oQ9FY3kAu5+asHXjSiU 2VmiFaZ3TpowOid3r2oDVk16O/QfjDKbmXjKPlUGLFo4Iug4xika9UJf8 xLr4LISu3/eDOgGe/COnpW/MSaxQrSd5SeYItxcMjUuAcUmS3UUv4Z+3S CtRVPo8JLqHo6N37GklD1fiRzBXfLE0+eFQpFK78UxOkmxve6625bYiYl 8FjHPceqvFRLxWhTzCbaKf0K0myRUdP7oqe3ZTMhtf+4xqrVM6Qh5esRI gl//gb+nuxsIHm3G3MX6TZr0hAc4Ka0ElBGursmXeTHEQS155RlkRBzuH w==; X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="388665125" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="388665125" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="791257579" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="791257579" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:33 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v6 5/7] drm/sched: Split free_job into own work item Date: Tue, 17 Oct 2023 08:09:56 -0700 Message-Id: <20231017150958.838613-6-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017150958.838613-1-matthew.brost@intel.com> References: <20231017150958.838613-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , lina@asahilina.net, sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, mcanal@igalia.com, luben.tuikov@amd.com, dakr@redhat.com, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Rather than call free_job and run_job in same work item have a dedicated work item for each. This aligns with the design and intended use of work queues. v2: - Test for DMA_FENCE_FLAG_TIMESTAMP_BIT before setting timestamp in free_job() work item (Danilo) v3: - Drop forward dec of drm_sched_select_entity (Boris) - Return in drm_sched_run_job_work if entity NULL (Boris) v4: - Replace dequeue with peek and invert logic (Luben) - Wrap to 100 lines (Luben) - Update comments for *_queue / *_queue_if_ready functions (Luben) Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 287 +++++++++++++++---------- include/drm/gpu_scheduler.h | 8 +- 2 files changed, 178 insertions(+), 117 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 273e0fbc4eab..b1b8d9f96da5 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -213,11 +213,12 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq, * drm_sched_rq_select_entity_rr - Select an entity which could provide a job to run * * @rq: scheduler run queue to check. + * @peek: Just find, don't set to current. * * Try to find a ready entity, returns NULL if none found. */ static struct drm_sched_entity * -drm_sched_rq_select_entity_rr(struct drm_sched_rq *rq) +drm_sched_rq_select_entity_rr(struct drm_sched_rq *rq, bool peek) { struct drm_sched_entity *entity; @@ -227,8 +228,10 @@ drm_sched_rq_select_entity_rr(struct drm_sched_rq *rq) if (entity) { list_for_each_entry_continue(entity, &rq->entities, list) { if (drm_sched_entity_is_ready(entity)) { - rq->current_entity = entity; - reinit_completion(&entity->entity_idle); + if (!peek) { + rq->current_entity = entity; + reinit_completion(&entity->entity_idle); + } spin_unlock(&rq->lock); return entity; } @@ -238,8 +241,10 @@ drm_sched_rq_select_entity_rr(struct drm_sched_rq *rq) list_for_each_entry(entity, &rq->entities, list) { if (drm_sched_entity_is_ready(entity)) { - rq->current_entity = entity; - reinit_completion(&entity->entity_idle); + if (!peek) { + rq->current_entity = entity; + reinit_completion(&entity->entity_idle); + } spin_unlock(&rq->lock); return entity; } @@ -257,11 +262,12 @@ drm_sched_rq_select_entity_rr(struct drm_sched_rq *rq) * drm_sched_rq_select_entity_fifo - Select an entity which provides a job to run * * @rq: scheduler run queue to check. + * @peek: Just find, don't set to current. * * Find oldest waiting ready entity, returns NULL if none found. */ static struct drm_sched_entity * -drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq) +drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq, bool peek) { struct rb_node *rb; @@ -271,8 +277,10 @@ drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq) entity = rb_entry(rb, struct drm_sched_entity, rb_tree_node); if (drm_sched_entity_is_ready(entity)) { - rq->current_entity = entity; - reinit_completion(&entity->entity_idle); + if (!peek) { + rq->current_entity = entity; + reinit_completion(&entity->entity_idle); + } break; } } @@ -282,13 +290,98 @@ drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq) } /** - * drm_sched_wqueue_enqueue - enqueue scheduler submission + * drm_sched_run_job_queue - enqueue run-job work + * @sched: scheduler instance + */ +static void drm_sched_run_job_queue(struct drm_gpu_scheduler *sched) +{ + if (!READ_ONCE(sched->pause_submit)) + queue_work(sched->submit_wq, &sched->work_run_job); +} + +/** + * drm_sched_can_queue -- Can we queue more to the hardware? + * @sched: scheduler instance + * + * Return true if we can push more jobs to the hw, otherwise false. + */ +static bool drm_sched_can_queue(struct drm_gpu_scheduler *sched) +{ + return atomic_read(&sched->hw_rq_count) < + sched->hw_submission_limit; +} + +/** + * drm_sched_select_entity - Select next entity to process + * + * @sched: scheduler instance + * @peek: Just find, don't set to current. + * + * Returns the entity to process or NULL if none are found. + */ +static struct drm_sched_entity * +drm_sched_select_entity(struct drm_gpu_scheduler *sched, bool peek) +{ + struct drm_sched_entity *entity; + int i; + + if (!drm_sched_can_queue(sched)) + return NULL; + + if (sched->single_entity) { + if (!READ_ONCE(sched->single_entity->stopped) && + drm_sched_entity_is_ready(sched->single_entity)) + return sched->single_entity; + + return NULL; + } + + /* Kernel run queue has higher priority than normal run queue*/ + for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { + entity = sched->sched_policy == DRM_SCHED_POLICY_FIFO ? + drm_sched_rq_select_entity_fifo(&sched->sched_rq[i], peek) : + drm_sched_rq_select_entity_rr(&sched->sched_rq[i], peek); + if (entity) + break; + } + + return entity; +} + +/** + * drm_sched_run_job_queue_if_ready - enqueue run-job work if ready + * @sched: scheduler instance + */ +static void drm_sched_run_job_queue_if_ready(struct drm_gpu_scheduler *sched) +{ + if (drm_sched_select_entity(sched, false)) + drm_sched_run_job_queue(sched); +} + +/** + * drm_sched_free_job_queue - enqueue free-job work * @sched: scheduler instance */ -static void drm_sched_wqueue_enqueue(struct drm_gpu_scheduler *sched) +static void drm_sched_free_job_queue(struct drm_gpu_scheduler *sched) { if (!READ_ONCE(sched->pause_submit)) - queue_work(sched->submit_wq, &sched->work_submit); + queue_work(sched->submit_wq, &sched->work_free_job); +} + +/** + * drm_sched_free_job_queue_if_ready - enqueue free-job work if ready + * @sched: scheduler instance + */ +static void drm_sched_free_job_queue_if_ready(struct drm_gpu_scheduler *sched) +{ + struct drm_sched_job *job; + + spin_lock(&sched->job_list_lock); + job = list_first_entry_or_null(&sched->pending_list, + struct drm_sched_job, list); + if (job && dma_fence_is_signaled(&job->s_fence->finished)) + drm_sched_free_job_queue(sched); + spin_unlock(&sched->job_list_lock); } /** @@ -310,7 +403,7 @@ static void drm_sched_job_done(struct drm_sched_job *s_job, int result) dma_fence_get(&s_fence->finished); drm_sched_fence_finished(s_fence, result); dma_fence_put(&s_fence->finished); - drm_sched_wqueue_enqueue(sched); + drm_sched_free_job_queue(sched); } /** @@ -576,7 +669,7 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery) drm_sched_job_done(s_job, fence->error); else if (r) DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", - r); + r); } else drm_sched_job_done(s_job, -ECANCELED); } @@ -885,18 +978,6 @@ void drm_sched_job_cleanup(struct drm_sched_job *job) } EXPORT_SYMBOL(drm_sched_job_cleanup); -/** - * drm_sched_can_queue -- Can we queue more to the hardware? - * @sched: scheduler instance - * - * Return true if we can push more jobs to the hw, otherwise false. - */ -static bool drm_sched_can_queue(struct drm_gpu_scheduler *sched) -{ - return atomic_read(&sched->hw_rq_count) < - sched->hw_submission_limit; -} - /** * drm_sched_wakeup_if_can_queue - Wake up the scheduler * @sched: scheduler instance @@ -906,43 +987,7 @@ static bool drm_sched_can_queue(struct drm_gpu_scheduler *sched) void drm_sched_wakeup_if_can_queue(struct drm_gpu_scheduler *sched) { if (drm_sched_can_queue(sched)) - drm_sched_wqueue_enqueue(sched); -} - -/** - * drm_sched_select_entity - Select next entity to process - * - * @sched: scheduler instance - * - * Returns the entity to process or NULL if none are found. - */ -static struct drm_sched_entity * -drm_sched_select_entity(struct drm_gpu_scheduler *sched) -{ - struct drm_sched_entity *entity; - int i; - - if (!drm_sched_can_queue(sched)) - return NULL; - - if (sched->single_entity) { - if (!READ_ONCE(sched->single_entity->stopped) && - drm_sched_entity_is_ready(sched->single_entity)) - return sched->single_entity; - - return NULL; - } - - /* Kernel run queue has higher priority than normal run queue*/ - for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { - entity = sched->sched_policy == DRM_SCHED_POLICY_FIFO ? - drm_sched_rq_select_entity_fifo(&sched->sched_rq[i]) : - drm_sched_rq_select_entity_rr(&sched->sched_rq[i]); - if (entity) - break; - } - - return entity; + drm_sched_run_job_queue(sched); } /** @@ -974,8 +1019,10 @@ drm_sched_get_cleanup_job(struct drm_gpu_scheduler *sched) typeof(*next), list); if (next) { - next->s_fence->scheduled.timestamp = - dma_fence_timestamp(&job->s_fence->finished); + if (test_bit(DMA_FENCE_FLAG_TIMESTAMP_BIT, + &next->s_fence->scheduled.flags)) + next->s_fence->scheduled.timestamp = + dma_fence_timestamp(&job->s_fence->finished); /* start TO timer for next job */ drm_sched_start_timeout(sched); } @@ -1025,74 +1072,83 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list, EXPORT_SYMBOL(drm_sched_pick_best); /** - * drm_sched_main - main scheduler thread + * drm_sched_free_job_work - worker to call free_job * - * @param: scheduler instance + * @w: free job work */ -static void drm_sched_main(struct work_struct *w) +static void drm_sched_free_job_work(struct work_struct *w) { struct drm_gpu_scheduler *sched = - container_of(w, struct drm_gpu_scheduler, work_submit); - struct drm_sched_entity *entity; + container_of(w, struct drm_gpu_scheduler, work_free_job); struct drm_sched_job *cleanup_job; - int r; if (READ_ONCE(sched->pause_submit)) return; cleanup_job = drm_sched_get_cleanup_job(sched); - entity = drm_sched_select_entity(sched); - - if (!entity && !cleanup_job) - return; /* No more work */ - - if (cleanup_job) + if (cleanup_job) { sched->ops->free_job(cleanup_job); - if (entity) { - struct dma_fence *fence; - struct drm_sched_fence *s_fence; - struct drm_sched_job *sched_job; - - sched_job = drm_sched_entity_pop_job(entity); - if (!sched_job) { - complete_all(&entity->entity_idle); - if (!cleanup_job) - return; /* No more work */ - goto again; - } + drm_sched_free_job_queue_if_ready(sched); + drm_sched_run_job_queue_if_ready(sched); + } +} - s_fence = sched_job->s_fence; +/** + * drm_sched_run_job_work - worker to call run_job + * + * @w: run job work + */ +static void drm_sched_run_job_work(struct work_struct *w) +{ + struct drm_gpu_scheduler *sched = + container_of(w, struct drm_gpu_scheduler, work_run_job); + struct drm_sched_entity *entity; + struct dma_fence *fence; + struct drm_sched_fence *s_fence; + struct drm_sched_job *sched_job; + int r; - atomic_inc(&sched->hw_rq_count); - drm_sched_job_begin(sched_job); + if (READ_ONCE(sched->pause_submit)) + return; + + entity = drm_sched_select_entity(sched, true); + if (!entity) + return; - trace_drm_run_job(sched_job, entity); - fence = sched->ops->run_job(sched_job); + sched_job = drm_sched_entity_pop_job(entity); + if (!sched_job) { complete_all(&entity->entity_idle); - drm_sched_fence_scheduled(s_fence, fence); + return; /* No more work */ + } - if (!IS_ERR_OR_NULL(fence)) { - /* Drop for original kref_init of the fence */ - dma_fence_put(fence); + s_fence = sched_job->s_fence; - r = dma_fence_add_callback(fence, &sched_job->cb, - drm_sched_job_done_cb); - if (r == -ENOENT) - drm_sched_job_done(sched_job, fence->error); - else if (r) - DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", - r); - } else { - drm_sched_job_done(sched_job, IS_ERR(fence) ? - PTR_ERR(fence) : 0); - } + atomic_inc(&sched->hw_rq_count); + drm_sched_job_begin(sched_job); - wake_up(&sched->job_scheduled); + trace_drm_run_job(sched_job, entity); + fence = sched->ops->run_job(sched_job); + complete_all(&entity->entity_idle); + drm_sched_fence_scheduled(s_fence, fence); + + if (!IS_ERR_OR_NULL(fence)) { + /* Drop for original kref_init of the fence */ + dma_fence_put(fence); + + r = dma_fence_add_callback(fence, &sched_job->cb, + drm_sched_job_done_cb); + if (r == -ENOENT) + drm_sched_job_done(sched_job, fence->error); + else if (r) + DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", r); + } else { + drm_sched_job_done(sched_job, IS_ERR(fence) ? + PTR_ERR(fence) : 0); } -again: - drm_sched_wqueue_enqueue(sched); + wake_up(&sched->job_scheduled); + drm_sched_run_job_queue_if_ready(sched); } /** @@ -1159,7 +1215,8 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, spin_lock_init(&sched->job_list_lock); atomic_set(&sched->hw_rq_count, 0); INIT_DELAYED_WORK(&sched->work_tdr, drm_sched_job_timedout); - INIT_WORK(&sched->work_submit, drm_sched_main); + INIT_WORK(&sched->work_run_job, drm_sched_run_job_work); + INIT_WORK(&sched->work_free_job, drm_sched_free_job_work); atomic_set(&sched->_score, 0); atomic64_set(&sched->job_id_count, 0); sched->pause_submit = false; @@ -1282,7 +1339,8 @@ EXPORT_SYMBOL(drm_sched_wqueue_ready); void drm_sched_wqueue_stop(struct drm_gpu_scheduler *sched) { WRITE_ONCE(sched->pause_submit, true); - cancel_work_sync(&sched->work_submit); + cancel_work_sync(&sched->work_run_job); + cancel_work_sync(&sched->work_free_job); } EXPORT_SYMBOL(drm_sched_wqueue_stop); @@ -1294,6 +1352,7 @@ EXPORT_SYMBOL(drm_sched_wqueue_stop); void drm_sched_wqueue_start(struct drm_gpu_scheduler *sched) { WRITE_ONCE(sched->pause_submit, false); - queue_work(sched->submit_wq, &sched->work_submit); + queue_work(sched->submit_wq, &sched->work_run_job); + queue_work(sched->submit_wq, &sched->work_free_job); } EXPORT_SYMBOL(drm_sched_wqueue_start); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index e67b53c3546b..625ffe040de3 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -487,9 +487,10 @@ struct drm_sched_backend_ops { * finished. * @hw_rq_count: the number of jobs currently in the hardware queue. * @job_id_count: used to assign unique id to the each job. - * @submit_wq: workqueue used to queue @work_submit + * @submit_wq: workqueue used to queue @work_run_job and @work_free_job * @timeout_wq: workqueue used to queue @work_tdr - * @work_submit: schedules jobs and cleans up entities + * @work_run_job: schedules jobs + * @work_free_job: cleans up jobs * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the * timeout interval is over. * @pending_list: the list of jobs which are currently in the job queue. @@ -519,7 +520,8 @@ struct drm_gpu_scheduler { atomic64_t job_id_count; struct workqueue_struct *submit_wq; struct workqueue_struct *timeout_wq; - struct work_struct work_submit; + struct work_struct work_run_job; + struct work_struct work_free_job; struct delayed_work work_tdr; struct list_head pending_list; spinlock_t job_list_lock; From patchwork Tue Oct 17 15:09:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13425506 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A5E17CDB474 for ; Tue, 17 Oct 2023 15:09:53 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1061510E30C; Tue, 17 Oct 2023 15:09:44 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5439A10E305; Tue, 17 Oct 2023 15:09:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697555378; x=1729091378; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=oRblEkvFnZSBS0jyD/7sV4PXb6dVSnwnOEVPOs1zpEY=; b=YXZYq9/I+OcT11wihZGvszhqeHPMGWm/yD3/OlDa51VFsVd+tJOSoWfn GIghxJMoBM6TkBL09Klh3qhJO9cc7QAGk8V318B5WnBd2kUQf4WR6GPEa p0ic8OySQMShskArBtFrz8VFWzZcffi4lThZ/txhZ1/GWdGLCfPMnDsb2 zUavAO/Cj7gFDxxOeuG8IBReVR4/YnhSjcsJjHM2PgZM2RyE2m4dvTMpw TjW9kS5H5I/MRrKEq7XLHYuNHupJipctC4a3cxlrT1kXNiIoIhaPhSauj yOuCcd7Yf3Qen5AMCAq9toAHZ9ENMNo6VGpPKhhlr86vZ/8IaHlIcMASx w==; X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="388665148" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="388665148" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="791257581" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="791257581" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:33 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v6 6/7] drm/sched: Add drm_sched_start_timeout_unlocked helper Date: Tue, 17 Oct 2023 08:09:57 -0700 Message-Id: <20231017150958.838613-7-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017150958.838613-1-matthew.brost@intel.com> References: <20231017150958.838613-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , lina@asahilina.net, sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, mcanal@igalia.com, luben.tuikov@amd.com, dakr@redhat.com, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Also add a lockdep assert to drm_sched_start_timeout. Signed-off-by: Matthew Brost Reviewed-by: Luben Tuikov --- drivers/gpu/drm/scheduler/sched_main.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index b1b8d9f96da5..ff2cfad54342 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -427,11 +427,20 @@ static void drm_sched_job_done_cb(struct dma_fence *f, struct dma_fence_cb *cb) */ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched) { + lockdep_assert_held(&sched->job_list_lock); + if (sched->timeout != MAX_SCHEDULE_TIMEOUT && !list_empty(&sched->pending_list)) queue_delayed_work(sched->timeout_wq, &sched->work_tdr, sched->timeout); } +static void drm_sched_start_timeout_unlocked(struct drm_gpu_scheduler *sched) +{ + spin_lock(&sched->job_list_lock); + drm_sched_start_timeout(sched); + spin_unlock(&sched->job_list_lock); +} + /** * drm_sched_fault - immediately start timeout handler * @@ -544,11 +553,8 @@ static void drm_sched_job_timedout(struct work_struct *work) spin_unlock(&sched->job_list_lock); } - if (status != DRM_GPU_SCHED_STAT_ENODEV) { - spin_lock(&sched->job_list_lock); - drm_sched_start_timeout(sched); - spin_unlock(&sched->job_list_lock); - } + if (status != DRM_GPU_SCHED_STAT_ENODEV) + drm_sched_start_timeout_unlocked(sched); } /** @@ -674,11 +680,8 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery) drm_sched_job_done(s_job, -ECANCELED); } - if (full_recovery) { - spin_lock(&sched->job_list_lock); - drm_sched_start_timeout(sched); - spin_unlock(&sched->job_list_lock); - } + if (full_recovery) + drm_sched_start_timeout_unlocked(sched); drm_sched_wqueue_start(sched); } From patchwork Tue Oct 17 15:09:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13425507 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 94418CDB482 for ; Tue, 17 Oct 2023 15:09:55 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7FA7E10E30F; Tue, 17 Oct 2023 15:09:44 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.24]) by gabe.freedesktop.org (Postfix) with ESMTPS id AB21310E308; Tue, 17 Oct 2023 15:09:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1697555378; x=1729091378; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=2S8PcZ1eWRywLaL/2P6uGQDgoWGp1RlLAEFLr1zUrsg=; b=AACAKEiYlSwiK6E14a7gVFw+KGrpDLdNVrA9QeCCf8QApaVBYuFAXPqe A6tCyIvRNeGDKFZN7vMjIiVe44myTUG6JZEUovbF+QgHLmwcJOmNtT4CS X1IZt3YemmZIxtOyRGtnbirO2Ficr2kta6HRO6A/Xn4rSXxOHb0lolWFu ApAADl6RBfWrcBrYIz2Osn4gDpa9lKlbqKX53JUYUIMZau/bAx276Kz+a iwv3XtlBXA+jdZHyLhbghM+uo42WOzvmj9NNo1TM+vdG9yYEKk2243wIE XUA1c1fR2YTpwCZBVKAN70ozNBdZWrXCsQRyRGAerLV+O9xOL7GpRRPVu Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="388665152" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="388665152" Received: from orsmga001.jf.intel.com ([10.7.209.18]) by orsmga102.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:35 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10866"; a="791257584" X-IronPort-AV: E=Sophos;i="6.03,232,1694761200"; d="scan'208";a="791257584" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by orsmga001-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 17 Oct 2023 08:09:33 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v6 7/7] drm/sched: Add a helper to queue TDR immediately Date: Tue, 17 Oct 2023 08:09:58 -0700 Message-Id: <20231017150958.838613-8-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20231017150958.838613-1-matthew.brost@intel.com> References: <20231017150958.838613-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , lina@asahilina.net, sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, mcanal@igalia.com, luben.tuikov@amd.com, dakr@redhat.com, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add a helper whereby a driver can invoke TDR immediately. v2: - Drop timeout args, rename function, use mod delayed work (Luben) v3: - s/XE/Xe (Luben) - present tense in commit message (Luben) - Adjust comment for drm_sched_tdr_queue_imm (Luben) v4: - Adjust commit message (Luben) Cc: Luben Tuikov Signed-off-by: Matthew Brost Reviewed-by: Luben Tuikov --- drivers/gpu/drm/scheduler/sched_main.c | 18 +++++++++++++++++- include/drm/gpu_scheduler.h | 1 + 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index ff2cfad54342..563a4f0f6956 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -431,7 +431,7 @@ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched) if (sched->timeout != MAX_SCHEDULE_TIMEOUT && !list_empty(&sched->pending_list)) - queue_delayed_work(sched->timeout_wq, &sched->work_tdr, sched->timeout); + mod_delayed_work(sched->timeout_wq, &sched->work_tdr, sched->timeout); } static void drm_sched_start_timeout_unlocked(struct drm_gpu_scheduler *sched) @@ -441,6 +441,22 @@ static void drm_sched_start_timeout_unlocked(struct drm_gpu_scheduler *sched) spin_unlock(&sched->job_list_lock); } +/** + * drm_sched_tdr_queue_imm: - immediately start job timeout handler + * + * @sched: scheduler for which the timeout handling should be started. + * + * Start timeout handling immediately for the named scheduler. + */ +void drm_sched_tdr_queue_imm(struct drm_gpu_scheduler *sched) +{ + spin_lock(&sched->job_list_lock); + sched->timeout = 0; + drm_sched_start_timeout(sched); + spin_unlock(&sched->job_list_lock); +} +EXPORT_SYMBOL(drm_sched_tdr_queue_imm); + /** * drm_sched_fault - immediately start timeout handler * diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 625ffe040de3..998b32b8d212 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -568,6 +568,7 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list); +void drm_sched_tdr_queue_imm(struct drm_gpu_scheduler *sched); void drm_sched_job_cleanup(struct drm_sched_job *job); void drm_sched_wakeup_if_can_queue(struct drm_gpu_scheduler *sched); bool drm_sched_wqueue_ready(struct drm_gpu_scheduler *sched);