From patchwork Fri Aug 11 02:31:29 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13350007 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7771DC001E0 for ; Fri, 11 Aug 2023 02:31:42 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C291810E624; Fri, 11 Aug 2023 02:31:41 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 70EC910E623; Fri, 11 Aug 2023 02:31:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691721100; x=1723257100; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uDUIgAsDsBx8WowLbrWFGLyf+xrNmdA3RMmH7jL0wMU=; b=NGIcxvZUfG+2If3i9x14MSqY5UfE5Z3vMUq4SAfLm4Ffxm31h8RsfF1X 2RHTAn5kTH/oYkkCZofmehuWEE9OCASAqLnYe5JmAbfsho6Vg5peOq4YY XB7IQVEDih07PJtQ87AdfCid6bVb+rf+OupEcNRQGEDV3+66k2m9gx1JE 19Qe9wWYpV1Rylzh2KfU1pC83AozVmTSQ4C6FR5QeINUtLCkUY/Bn7Z8+ 8F+rZmrLcE3N4uzYmmx6vVYAOr/9Uyg6NAv6xotES41FCbeIfqiXpDtMi PJudNbpPejXXUfcD/akRBZx+D0Rp+R1dnMSoJqzdtLjNa8TTwEhZakkwj g==; X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="361714792" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="361714792" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="797838394" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="797838394" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:39 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v2 1/9] drm/sched: Convert drm scheduler to use a work queue rather than kthread Date: Thu, 10 Aug 2023 19:31:29 -0700 Message-Id: <20230811023137.659037-2-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230811023137.659037-1-matthew.brost@intel.com> References: <20230811023137.659037-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, luben.tuikov@amd.com, lina@asahilina.net, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below. 1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem. 2. In XE submissions are done via programming a ring buffer (circular buffer), a drm_gpu_scheduler provides a limit on number of jobs, if the limit of number jobs is set to RING_SIZE / MAX_SIZE_PER_JOB we get flow control on the ring for free. A problem with this design is currently a drm_gpu_scheduler uses a kthread for submission / job cleanup. This doesn't scale if a large number of drm_gpu_scheduler are used. To work around the scaling issue, use a worker rather than kthread for submission / job cleanup. v2: - (Rob Clark) Fix msm build - Pass in run work queue v3: - (Boris) don't have loop in worker v4: - (Tvrtko) break out submit ready, stop, start helpers into own patch Signed-off-by: Matthew Brost --- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 2 +- drivers/gpu/drm/etnaviv/etnaviv_sched.c | 2 +- drivers/gpu/drm/lima/lima_sched.c | 2 +- drivers/gpu/drm/msm/msm_ringbuffer.c | 2 +- drivers/gpu/drm/nouveau/nouveau_sched.c | 2 +- drivers/gpu/drm/panfrost/panfrost_job.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 106 +++++++++------------ drivers/gpu/drm/v3d/v3d_sched.c | 10 +- include/drm/gpu_scheduler.h | 12 ++- 9 files changed, 65 insertions(+), 75 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 5f5efec6c444..3ebf9882edba 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -2331,7 +2331,7 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev) break; } - r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, + r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, NULL, ring->num_hw_submission, 0, timeout, adev->reset_domain->wq, ring->sched_score, ring->name, diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c index 1ae87dfd19c4..8486a2923f1b 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c @@ -133,7 +133,7 @@ int etnaviv_sched_init(struct etnaviv_gpu *gpu) { int ret; - ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, + ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, NULL, etnaviv_hw_jobs_limit, etnaviv_job_hang_limit, msecs_to_jiffies(500), NULL, NULL, dev_name(gpu->dev), gpu->dev); diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c index ffd91a5ee299..8d858aed0e56 100644 --- a/drivers/gpu/drm/lima/lima_sched.c +++ b/drivers/gpu/drm/lima/lima_sched.c @@ -488,7 +488,7 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name) INIT_WORK(&pipe->recover_work, lima_sched_recover_work); - return drm_sched_init(&pipe->base, &lima_sched_ops, 1, + return drm_sched_init(&pipe->base, &lima_sched_ops, NULL, 1, lima_job_hang_limit, msecs_to_jiffies(timeout), NULL, NULL, name, pipe->ldev->dev); diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.c b/drivers/gpu/drm/msm/msm_ringbuffer.c index b60199184409..79aa112118da 100644 --- a/drivers/gpu/drm/msm/msm_ringbuffer.c +++ b/drivers/gpu/drm/msm/msm_ringbuffer.c @@ -93,7 +93,7 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, /* currently managing hangcheck ourselves: */ sched_timeout = MAX_SCHEDULE_TIMEOUT; - ret = drm_sched_init(&ring->sched, &msm_sched_ops, + ret = drm_sched_init(&ring->sched, &msm_sched_ops, NULL, num_hw_submissions, 0, sched_timeout, NULL, NULL, to_msm_bo(ring->bo)->name, gpu->dev->dev); if (ret) { diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c b/drivers/gpu/drm/nouveau/nouveau_sched.c index 3424a1bf6af3..2d607de5d393 100644 --- a/drivers/gpu/drm/nouveau/nouveau_sched.c +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c @@ -407,7 +407,7 @@ int nouveau_sched_init(struct nouveau_drm *drm) if (!drm->sched_wq) return -ENOMEM; - return drm_sched_init(sched, &nouveau_sched_ops, + return drm_sched_init(sched, &nouveau_sched_ops, NULL, NOUVEAU_SCHED_HW_SUBMISSIONS, 0, job_hang_limit, NULL, NULL, "nouveau_sched", drm->dev->dev); } diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index a8b4827dc425..4efbc8aa3c2f 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -831,7 +831,7 @@ int panfrost_job_init(struct panfrost_device *pfdev) js->queue[j].fence_context = dma_fence_context_alloc(1); ret = drm_sched_init(&js->queue[j].sched, - &panfrost_sched_ops, + &panfrost_sched_ops, NULL, nentries, 0, msecs_to_jiffies(JOB_TIMEOUT_MS), pfdev->reset.wq, diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index e4fa62abca41..614e8c97a622 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -48,7 +48,6 @@ * through the jobs entity pointer. */ -#include #include #include #include @@ -256,6 +255,16 @@ drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq) return rb ? rb_entry(rb, struct drm_sched_entity, rb_tree_node) : NULL; } +/** + * drm_sched_submit_queue - scheduler queue submission + * @sched: scheduler instance + */ +static void drm_sched_submit_queue(struct drm_gpu_scheduler *sched) +{ + if (!READ_ONCE(sched->pause_submit)) + queue_work(sched->submit_wq, &sched->work_submit); +} + /** * drm_sched_job_done - complete a job * @s_job: pointer to the job which is done @@ -275,7 +284,7 @@ static void drm_sched_job_done(struct drm_sched_job *s_job, int result) dma_fence_get(&s_fence->finished); drm_sched_fence_finished(s_fence, result); dma_fence_put(&s_fence->finished); - wake_up_interruptible(&sched->wake_up_worker); + drm_sched_submit_queue(sched); } /** @@ -868,7 +877,7 @@ static bool drm_sched_can_queue(struct drm_gpu_scheduler *sched) void drm_sched_wakeup_if_can_queue(struct drm_gpu_scheduler *sched) { if (drm_sched_can_queue(sched)) - wake_up_interruptible(&sched->wake_up_worker); + drm_sched_submit_queue(sched); } /** @@ -978,61 +987,42 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list, } EXPORT_SYMBOL(drm_sched_pick_best); -/** - * drm_sched_blocked - check if the scheduler is blocked - * - * @sched: scheduler instance - * - * Returns true if blocked, otherwise false. - */ -static bool drm_sched_blocked(struct drm_gpu_scheduler *sched) -{ - if (kthread_should_park()) { - kthread_parkme(); - return true; - } - - return false; -} - /** * drm_sched_main - main scheduler thread * * @param: scheduler instance - * - * Returns 0. */ -static int drm_sched_main(void *param) +static void drm_sched_main(struct work_struct *w) { - struct drm_gpu_scheduler *sched = (struct drm_gpu_scheduler *)param; + struct drm_gpu_scheduler *sched = + container_of(w, struct drm_gpu_scheduler, work_submit); + struct drm_sched_entity *entity; + struct drm_sched_job *cleanup_job; int r; - sched_set_fifo_low(current); + if (READ_ONCE(sched->pause_submit)) + return; - while (!kthread_should_stop()) { - struct drm_sched_entity *entity = NULL; - struct drm_sched_fence *s_fence; - struct drm_sched_job *sched_job; - struct dma_fence *fence; - struct drm_sched_job *cleanup_job = NULL; + cleanup_job = drm_sched_get_cleanup_job(sched); + entity = drm_sched_select_entity(sched); - wait_event_interruptible(sched->wake_up_worker, - (cleanup_job = drm_sched_get_cleanup_job(sched)) || - (!drm_sched_blocked(sched) && - (entity = drm_sched_select_entity(sched))) || - kthread_should_stop()); + if (!entity && !cleanup_job) + return; /* No more work */ - if (cleanup_job) - sched->ops->free_job(cleanup_job); + if (cleanup_job) + sched->ops->free_job(cleanup_job); - if (!entity) - continue; + if (entity) { + struct dma_fence *fence; + struct drm_sched_fence *s_fence; + struct drm_sched_job *sched_job; sched_job = drm_sched_entity_pop_job(entity); - if (!sched_job) { complete_all(&entity->entity_idle); - continue; + if (!cleanup_job) + return; /* No more work */ + goto again; } s_fence = sched_job->s_fence; @@ -1063,7 +1053,9 @@ static int drm_sched_main(void *param) wake_up(&sched->job_scheduled); } - return 0; + +again: + drm_sched_submit_queue(sched); } /** @@ -1071,6 +1063,7 @@ static int drm_sched_main(void *param) * * @sched: scheduler instance * @ops: backend operations for this scheduler + * @submit_wq: workqueue to use for submission. If NULL, the system_wq is used * @hw_submission: number of hw submissions that can be in flight * @hang_limit: number of times to allow a job to hang before dropping it * @timeout: timeout value in jiffies for the scheduler @@ -1084,14 +1077,16 @@ static int drm_sched_main(void *param) */ int drm_sched_init(struct drm_gpu_scheduler *sched, const struct drm_sched_backend_ops *ops, + struct workqueue_struct *submit_wq, unsigned hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, atomic_t *score, const char *name, struct device *dev) { - int i, ret; + int i; sched->ops = ops; sched->hw_submission_limit = hw_submission; sched->name = name; + sched->submit_wq = submit_wq ? : system_wq; sched->timeout = timeout; sched->timeout_wq = timeout_wq ? : system_wq; sched->hang_limit = hang_limit; @@ -1100,23 +1095,15 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) drm_sched_rq_init(sched, &sched->sched_rq[i]); - init_waitqueue_head(&sched->wake_up_worker); init_waitqueue_head(&sched->job_scheduled); INIT_LIST_HEAD(&sched->pending_list); spin_lock_init(&sched->job_list_lock); atomic_set(&sched->hw_rq_count, 0); INIT_DELAYED_WORK(&sched->work_tdr, drm_sched_job_timedout); + INIT_WORK(&sched->work_submit, drm_sched_main); atomic_set(&sched->_score, 0); atomic64_set(&sched->job_id_count, 0); - - /* Each scheduler will run on a seperate kernel thread */ - sched->thread = kthread_run(drm_sched_main, sched, sched->name); - if (IS_ERR(sched->thread)) { - ret = PTR_ERR(sched->thread); - sched->thread = NULL; - DRM_DEV_ERROR(sched->dev, "Failed to create scheduler for %s.\n", name); - return ret; - } + sched->pause_submit = false; sched->ready = true; return 0; @@ -1135,8 +1122,7 @@ void drm_sched_fini(struct drm_gpu_scheduler *sched) struct drm_sched_entity *s_entity; int i; - if (sched->thread) - kthread_stop(sched->thread); + drm_sched_submit_stop(sched); for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { struct drm_sched_rq *rq = &sched->sched_rq[i]; @@ -1216,7 +1202,7 @@ EXPORT_SYMBOL(drm_sched_increase_karma); */ bool drm_sched_submit_ready(struct drm_gpu_scheduler *sched) { - return !!sched->thread; + return sched->ready; } EXPORT_SYMBOL(drm_sched_submit_ready); @@ -1228,7 +1214,8 @@ EXPORT_SYMBOL(drm_sched_submit_ready); */ void drm_sched_submit_stop(struct drm_gpu_scheduler *sched) { - kthread_park(sched->thread); + WRITE_ONCE(sched->pause_submit, true); + cancel_work_sync(&sched->work_submit); } EXPORT_SYMBOL(drm_sched_submit_stop); @@ -1239,6 +1226,7 @@ EXPORT_SYMBOL(drm_sched_submit_stop); */ void drm_sched_submit_start(struct drm_gpu_scheduler *sched) { - kthread_unpark(sched->thread); + WRITE_ONCE(sched->pause_submit, false); + queue_work(sched->submit_wq, &sched->work_submit); } EXPORT_SYMBOL(drm_sched_submit_start); diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index 06238e6d7f5c..38e092ea41e6 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -388,7 +388,7 @@ v3d_sched_init(struct v3d_dev *v3d) int ret; ret = drm_sched_init(&v3d->queue[V3D_BIN].sched, - &v3d_bin_sched_ops, + &v3d_bin_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_bin", v3d->drm.dev); @@ -396,7 +396,7 @@ v3d_sched_init(struct v3d_dev *v3d) return ret; ret = drm_sched_init(&v3d->queue[V3D_RENDER].sched, - &v3d_render_sched_ops, + &v3d_render_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_render", v3d->drm.dev); @@ -404,7 +404,7 @@ v3d_sched_init(struct v3d_dev *v3d) goto fail; ret = drm_sched_init(&v3d->queue[V3D_TFU].sched, - &v3d_tfu_sched_ops, + &v3d_tfu_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_tfu", v3d->drm.dev); @@ -413,7 +413,7 @@ v3d_sched_init(struct v3d_dev *v3d) if (v3d_has_csd(v3d)) { ret = drm_sched_init(&v3d->queue[V3D_CSD].sched, - &v3d_csd_sched_ops, + &v3d_csd_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_csd", v3d->drm.dev); @@ -421,7 +421,7 @@ v3d_sched_init(struct v3d_dev *v3d) goto fail; ret = drm_sched_init(&v3d->queue[V3D_CACHE_CLEAN].sched, - &v3d_cache_clean_sched_ops, + &v3d_cache_clean_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_cache_clean", v3d->drm.dev); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index f12c5aea5294..278106e358a8 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -473,17 +473,16 @@ struct drm_sched_backend_ops { * @timeout: the time after which a job is removed from the scheduler. * @name: name of the ring for which this scheduler is being used. * @sched_rq: priority wise array of run queues. - * @wake_up_worker: the wait queue on which the scheduler sleeps until a job - * is ready to be scheduled. * @job_scheduled: once @drm_sched_entity_do_release is called the scheduler * waits on this wait queue until all the scheduled jobs are * finished. * @hw_rq_count: the number of jobs currently in the hardware queue. * @job_id_count: used to assign unique id to the each job. + * @submit_wq: workqueue used to queue @work_submit * @timeout_wq: workqueue used to queue @work_tdr + * @work_submit: schedules jobs and cleans up entities * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the * timeout interval is over. - * @thread: the kthread on which the scheduler which run. * @pending_list: the list of jobs which are currently in the job queue. * @job_list_lock: lock to protect the pending_list. * @hang_limit: once the hangs by a job crosses this limit then it is marked @@ -492,6 +491,7 @@ struct drm_sched_backend_ops { * @_score: score used when the driver doesn't provide one * @ready: marks if the underlying HW is ready to work * @free_guilty: A hit to time out handler to free the guilty job. + * @pause_submit: pause queuing of @work_submit on @submit_wq * @dev: system &struct device * * One scheduler is implemented for each hardware ring. @@ -502,13 +502,13 @@ struct drm_gpu_scheduler { long timeout; const char *name; struct drm_sched_rq sched_rq[DRM_SCHED_PRIORITY_COUNT]; - wait_queue_head_t wake_up_worker; wait_queue_head_t job_scheduled; atomic_t hw_rq_count; atomic64_t job_id_count; + struct workqueue_struct *submit_wq; struct workqueue_struct *timeout_wq; + struct work_struct work_submit; struct delayed_work work_tdr; - struct task_struct *thread; struct list_head pending_list; spinlock_t job_list_lock; int hang_limit; @@ -516,11 +516,13 @@ struct drm_gpu_scheduler { atomic_t _score; bool ready; bool free_guilty; + bool pause_submit; struct device *dev; }; int drm_sched_init(struct drm_gpu_scheduler *sched, const struct drm_sched_backend_ops *ops, + struct workqueue_struct *submit_wq, uint32_t hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, atomic_t *score, const char *name, struct device *dev); From patchwork Fri Aug 11 02:31:30 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13350013 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A0853C001E0 for ; Fri, 11 Aug 2023 02:32:01 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 810F110E633; Fri, 11 Aug 2023 02:31:48 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 93A5B10E622; Fri, 11 Aug 2023 02:31:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691721100; x=1723257100; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=3KteY21mWgh9u18A2EWLQyTeYAV8LKvHke+29qbWtPM=; b=l0sA+YzLSD0d9re4F/8TOklZIfqTtx8mxT5BAAsLFzEY2oyPDfl+HdI2 DH1CQ76Rwg1UDRfFHRwIxot1Kge8eTVChZcO3lVtH9uAfnub8bGt2aOJ8 sQaGqkFh1SZgOWwi23nlJ1qohOGZc5BjFdKZrYUxKz7sI7eq/RaLavXLm JIqBnKLgIXIC7SZ15XjoLcZg/rVCdQP0GpjcJ+EjQquWDAwiVo+jM6oR5 mcDdHQrTBc42Q9VPR3y3/QphtZQ5Fmcl1+y+gTRGu1qt6rRYYmBMZiUje jkNz3FyTEno43S1ec7wir8k6S7e/pVnu2FIaaGysYsAkSAqDwengwF/Ve A==; X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="361714802" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="361714802" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="797838397" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="797838397" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:39 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v2 2/9] drm/sched: Move schedule policy to scheduler / entity Date: Thu, 10 Aug 2023 19:31:30 -0700 Message-Id: <20230811023137.659037-3-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230811023137.659037-1-matthew.brost@intel.com> References: <20230811023137.659037-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, luben.tuikov@amd.com, lina@asahilina.net, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Rather than a global modparam for scheduling policy, move the scheduling policy to scheduler / entity so user can control each scheduler / entity policy. v2: - s/DRM_SCHED_POLICY_MAX/DRM_SCHED_POLICY_COUNT (Luben) - Only include policy in scheduler (Luben) Signed-off-by: Matthew Brost --- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/etnaviv/etnaviv_sched.c | 3 ++- drivers/gpu/drm/lima/lima_sched.c | 3 ++- drivers/gpu/drm/msm/msm_ringbuffer.c | 3 ++- drivers/gpu/drm/nouveau/nouveau_sched.c | 3 ++- drivers/gpu/drm/panfrost/panfrost_job.c | 3 ++- drivers/gpu/drm/scheduler/sched_entity.c | 24 ++++++++++++++++++---- drivers/gpu/drm/scheduler/sched_main.c | 23 +++++++++++++++------ drivers/gpu/drm/v3d/v3d_sched.c | 15 +++++++++----- include/drm/gpu_scheduler.h | 20 ++++++++++++------ 10 files changed, 72 insertions(+), 26 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 3ebf9882edba..993a637dca0a 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -2335,6 +2335,7 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev) ring->num_hw_submission, 0, timeout, adev->reset_domain->wq, ring->sched_score, ring->name, + DRM_SCHED_POLICY_DEFAULT, adev->dev); if (r) { DRM_ERROR("Failed to create scheduler on ring %s.\n", diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c index 8486a2923f1b..61204a3f8b0b 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c @@ -136,7 +136,8 @@ int etnaviv_sched_init(struct etnaviv_gpu *gpu) ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, NULL, etnaviv_hw_jobs_limit, etnaviv_job_hang_limit, msecs_to_jiffies(500), NULL, NULL, - dev_name(gpu->dev), gpu->dev); + dev_name(gpu->dev), DRM_SCHED_POLICY_DEFAULT, + gpu->dev); if (ret) return ret; diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c index 8d858aed0e56..465d4bf3882b 100644 --- a/drivers/gpu/drm/lima/lima_sched.c +++ b/drivers/gpu/drm/lima/lima_sched.c @@ -491,7 +491,8 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name) return drm_sched_init(&pipe->base, &lima_sched_ops, NULL, 1, lima_job_hang_limit, msecs_to_jiffies(timeout), NULL, - NULL, name, pipe->ldev->dev); + NULL, name, DRM_SCHED_POLICY_DEFAULT, + pipe->ldev->dev); } void lima_sched_pipe_fini(struct lima_sched_pipe *pipe) diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.c b/drivers/gpu/drm/msm/msm_ringbuffer.c index 79aa112118da..78f05be89b61 100644 --- a/drivers/gpu/drm/msm/msm_ringbuffer.c +++ b/drivers/gpu/drm/msm/msm_ringbuffer.c @@ -95,7 +95,8 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, ret = drm_sched_init(&ring->sched, &msm_sched_ops, NULL, num_hw_submissions, 0, sched_timeout, - NULL, NULL, to_msm_bo(ring->bo)->name, gpu->dev->dev); + NULL, NULL, to_msm_bo(ring->bo)->name, + DRM_SCHED_POLICY_DEFAULT, gpu->dev->dev); if (ret) { goto fail; } diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c b/drivers/gpu/drm/nouveau/nouveau_sched.c index 2d607de5d393..59c263acf347 100644 --- a/drivers/gpu/drm/nouveau/nouveau_sched.c +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c @@ -409,7 +409,8 @@ int nouveau_sched_init(struct nouveau_drm *drm) return drm_sched_init(sched, &nouveau_sched_ops, NULL, NOUVEAU_SCHED_HW_SUBMISSIONS, 0, job_hang_limit, - NULL, NULL, "nouveau_sched", drm->dev->dev); + NULL, NULL, "nouveau_sched", + DRM_SCHED_POLICY_DEFAULT, drm->dev->dev); } void nouveau_sched_fini(struct nouveau_drm *drm) diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index 4efbc8aa3c2f..c051aa01bb92 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -835,7 +835,8 @@ int panfrost_job_init(struct panfrost_device *pfdev) nentries, 0, msecs_to_jiffies(JOB_TIMEOUT_MS), pfdev->reset.wq, - NULL, "pan_js", pfdev->dev); + NULL, "pan_js", DRM_SCHED_POLICY_DEFAULT, + pfdev->dev); if (ret) { dev_err(pfdev->dev, "Failed to create scheduler: %d.", ret); goto err_sched; diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index a42763e1429d..65a972b52eda 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -33,6 +33,20 @@ #define to_drm_sched_job(sched_job) \ container_of((sched_job), struct drm_sched_job, queue_node) +static bool bad_policies(struct drm_gpu_scheduler **sched_list, + unsigned int num_sched_list) +{ + enum drm_sched_policy sched_policy = sched_list[0]->sched_policy; + unsigned int i; + + /* All schedule policies must match */ + for (i = 1; i < num_sched_list; ++i) + if (sched_policy != sched_list[i]->sched_policy) + return true; + + return false; +} + /** * drm_sched_entity_init - Init a context entity used by scheduler when * submit to HW ring. @@ -62,7 +76,8 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, unsigned int num_sched_list, atomic_t *guilty) { - if (!(entity && sched_list && (num_sched_list == 0 || sched_list[0]))) + if (!(entity && sched_list && (num_sched_list == 0 || sched_list[0])) || + bad_policies(sched_list, num_sched_list)) return -EINVAL; memset(entity, 0, sizeof(struct drm_sched_entity)); @@ -486,7 +501,7 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) * Update the entity's location in the min heap according to * the timestamp of the next job, if any. */ - if (drm_sched_policy == DRM_SCHED_POLICY_FIFO) { + if (entity->rq->sched->sched_policy == DRM_SCHED_POLICY_FIFO) { struct drm_sched_job *next; next = to_drm_sched_job(spsc_queue_peek(&entity->job_queue)); @@ -558,7 +573,8 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity) void drm_sched_entity_push_job(struct drm_sched_job *sched_job) { struct drm_sched_entity *entity = sched_job->entity; - bool first; + bool first, fifo = entity->rq->sched->sched_policy == + DRM_SCHED_POLICY_FIFO; ktime_t submit_ts; trace_drm_sched_job(sched_job, entity); @@ -587,7 +603,7 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) drm_sched_rq_add_entity(entity->rq, entity); spin_unlock(&entity->rq_lock); - if (drm_sched_policy == DRM_SCHED_POLICY_FIFO) + if (fifo) drm_sched_rq_update_fifo(entity, submit_ts); drm_sched_wakeup_if_can_queue(entity->rq->sched); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 614e8c97a622..545d5298c086 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -66,14 +66,14 @@ #define to_drm_sched_job(sched_job) \ container_of((sched_job), struct drm_sched_job, queue_node) -int drm_sched_policy = DRM_SCHED_POLICY_FIFO; +int default_drm_sched_policy = DRM_SCHED_POLICY_FIFO; /** * DOC: sched_policy (int) * Used to override default entities scheduling policy in a run queue. */ -MODULE_PARM_DESC(sched_policy, "Specify the scheduling policy for entities on a run-queue, " __stringify(DRM_SCHED_POLICY_RR) " = Round Robin, " __stringify(DRM_SCHED_POLICY_FIFO) " = FIFO (default)."); -module_param_named(sched_policy, drm_sched_policy, int, 0444); +MODULE_PARM_DESC(sched_policy, "Specify the default scheduling policy for entities on a run-queue, " __stringify(DRM_SCHED_POLICY_RR) " = Round Robin, " __stringify(DRM_SCHED_POLICY_FIFO) " = FIFO (default)."); +module_param_named(sched_policy, default_drm_sched_policy, int, 0444); static __always_inline bool drm_sched_entity_compare_before(struct rb_node *a, const struct rb_node *b) @@ -177,7 +177,7 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq, if (rq->current_entity == entity) rq->current_entity = NULL; - if (drm_sched_policy == DRM_SCHED_POLICY_FIFO) + if (rq->sched->sched_policy == DRM_SCHED_POLICY_FIFO) drm_sched_rq_remove_fifo_locked(entity); spin_unlock(&rq->lock); @@ -898,7 +898,7 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched) /* Kernel run queue has higher priority than normal run queue*/ for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { - entity = drm_sched_policy == DRM_SCHED_POLICY_FIFO ? + entity = sched->sched_policy == DRM_SCHED_POLICY_FIFO ? drm_sched_rq_select_entity_fifo(&sched->sched_rq[i]) : drm_sched_rq_select_entity_rr(&sched->sched_rq[i]); if (entity) @@ -1071,6 +1071,7 @@ static void drm_sched_main(struct work_struct *w) * used * @score: optional score atomic shared with other schedulers * @name: name used for debugging + * @sched_policy: schedule policy * @dev: target &struct device * * Return 0 on success, otherwise error code. @@ -1080,9 +1081,15 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, struct workqueue_struct *submit_wq, unsigned hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, - atomic_t *score, const char *name, struct device *dev) + atomic_t *score, const char *name, + enum drm_sched_policy sched_policy, + struct device *dev) { int i; + + if (sched_policy >= DRM_SCHED_POLICY_COUNT) + return -EINVAL; + sched->ops = ops; sched->hw_submission_limit = hw_submission; sched->name = name; @@ -1092,6 +1099,10 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, sched->hang_limit = hang_limit; sched->score = score ? score : &sched->_score; sched->dev = dev; + if (sched_policy == DRM_SCHED_POLICY_DEFAULT) + sched->sched_policy = default_drm_sched_policy; + else + sched->sched_policy = sched_policy; for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) drm_sched_rq_init(sched, &sched->sched_rq[i]); diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index 38e092ea41e6..5e3fe77fa991 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -391,7 +391,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_bin_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_bin", v3d->drm.dev); + NULL, "v3d_bin", DRM_SCHED_POLICY_DEFAULT, + v3d->drm.dev); if (ret) return ret; @@ -399,7 +400,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_render_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_render", v3d->drm.dev); + ULL, "v3d_render", DRM_SCHED_POLICY_DEFAULT, + v3d->drm.dev); if (ret) goto fail; @@ -407,7 +409,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_tfu_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_tfu", v3d->drm.dev); + NULL, "v3d_tfu", DRM_SCHED_POLICY_DEFAULT, + v3d->drm.dev); if (ret) goto fail; @@ -416,7 +419,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_csd_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_csd", v3d->drm.dev); + NULL, "v3d_csd", DRM_SCHED_POLICY_DEFAULT, + v3d->drm.dev); if (ret) goto fail; @@ -424,7 +428,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_cache_clean_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_cache_clean", v3d->drm.dev); + NULL, "v3d_cache_clean", + DRM_SCHED_POLICY_DEFAULT, v3d->drm.dev); if (ret) goto fail; } diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 278106e358a8..897d52a4ff4f 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -72,11 +72,15 @@ enum drm_sched_priority { DRM_SCHED_PRIORITY_UNSET = -2 }; -/* Used to chose between FIFO and RR jobs scheduling */ -extern int drm_sched_policy; - -#define DRM_SCHED_POLICY_RR 0 -#define DRM_SCHED_POLICY_FIFO 1 +/* Used to chose default scheduling policy*/ +extern int default_drm_sched_policy; + +enum drm_sched_policy { + DRM_SCHED_POLICY_DEFAULT, + DRM_SCHED_POLICY_RR, + DRM_SCHED_POLICY_FIFO, + DRM_SCHED_POLICY_COUNT, +}; /** * struct drm_sched_entity - A wrapper around a job queue (typically @@ -489,6 +493,7 @@ struct drm_sched_backend_ops { * guilty and it will no longer be considered for scheduling. * @score: score to help loadbalancer pick a idle sched * @_score: score used when the driver doesn't provide one + * @sched_policy: Schedule policy for scheduler * @ready: marks if the underlying HW is ready to work * @free_guilty: A hit to time out handler to free the guilty job. * @pause_submit: pause queuing of @work_submit on @submit_wq @@ -514,6 +519,7 @@ struct drm_gpu_scheduler { int hang_limit; atomic_t *score; atomic_t _score; + enum drm_sched_policy sched_policy; bool ready; bool free_guilty; bool pause_submit; @@ -525,7 +531,9 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, struct workqueue_struct *submit_wq, uint32_t hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, - atomic_t *score, const char *name, struct device *dev); + atomic_t *score, const char *name, + enum drm_sched_policy sched_policy, + struct device *dev); void drm_sched_fini(struct drm_gpu_scheduler *sched); int drm_sched_job_init(struct drm_sched_job *job, From patchwork Fri Aug 11 02:31:31 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13350014 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 56C2FC001DE for ; Fri, 11 Aug 2023 02:32:03 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 001DA10E622; Fri, 11 Aug 2023 02:31:55 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id D37A310E622; Fri, 11 Aug 2023 02:31:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691721100; x=1723257100; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nhLqGUX/txcLqiAI9htJoe7+1UJuTnE0BdAu93/CrLc=; b=XLyRX9iFD2ivrcgRnhRaG6m3VNMnPSBF75gVGlki2gQcu0YBzkfxABQ+ uX+njm1rjZbXqJT978i8YFjU+X4TBD9FjV8iC0OJzPdyKoCgRZ890pA8K xW2FPAxyFpsq8H3WiDG9GVEzpw6koAxh0ewZtgESo+uj/mKfdl8cHnP8U /lTZ0e2UbkbM8fzeTpDOGfiEuZ6oNkrOB+y2Pj2CsDB/19xQpQR93Ol/q +iVt/cXFnwbpDDjDdAdUnvjM/0iJZvDEocIfSiEpgIfnLziIF4hkB0LtJ 1vw3pC8qfT+LPjsqfNOXwUk7/V+dmh/vF9YCdQpzw7n0clUDy4qEvUFNb Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="361714811" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="361714811" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="797838400" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="797838400" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:40 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v2 3/9] drm/sched: Add DRM_SCHED_POLICY_SINGLE_ENTITY scheduling policy Date: Thu, 10 Aug 2023 19:31:31 -0700 Message-Id: <20230811023137.659037-4-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230811023137.659037-1-matthew.brost@intel.com> References: <20230811023137.659037-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, luben.tuikov@amd.com, lina@asahilina.net, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" DRM_SCHED_POLICY_SINGLE_ENTITY creates a 1 to 1 relationship between scheduler and entity. No priorities or run queue used in this mode. Intended for devices with firmware schedulers. v2: - Drop sched / rq union (Luben) Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_entity.c | 69 ++++++++++++++++++------ drivers/gpu/drm/scheduler/sched_fence.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 63 +++++++++++++++++++--- include/drm/gpu_scheduler.h | 8 +++ 4 files changed, 119 insertions(+), 23 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index 65a972b52eda..1dec97caaba3 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -83,6 +83,7 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, memset(entity, 0, sizeof(struct drm_sched_entity)); INIT_LIST_HEAD(&entity->list); entity->rq = NULL; + entity->single_sched = NULL; entity->guilty = guilty; entity->num_sched_list = num_sched_list; entity->priority = priority; @@ -90,8 +91,17 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, RCU_INIT_POINTER(entity->last_scheduled, NULL); RB_CLEAR_NODE(&entity->rb_tree_node); - if(num_sched_list) - entity->rq = &sched_list[0]->sched_rq[entity->priority]; + if (num_sched_list) { + if (sched_list[0]->sched_policy != + DRM_SCHED_POLICY_SINGLE_ENTITY) { + entity->rq = &sched_list[0]->sched_rq[entity->priority]; + } else { + if (num_sched_list != 1 || sched_list[0]->single_entity) + return -EINVAL; + sched_list[0]->single_entity = entity; + entity->single_sched = sched_list[0]; + } + } init_completion(&entity->entity_idle); @@ -124,7 +134,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list) { - WARN_ON(!num_sched_list || !sched_list); + WARN_ON(!num_sched_list || !sched_list || + !!entity->single_sched); entity->sched_list = sched_list; entity->num_sched_list = num_sched_list; @@ -231,13 +242,15 @@ static void drm_sched_entity_kill(struct drm_sched_entity *entity) { struct drm_sched_job *job; struct dma_fence *prev; + bool single_entity = !!entity->single_sched; - if (!entity->rq) + if (!entity->rq && !single_entity) return; spin_lock(&entity->rq_lock); entity->stopped = true; - drm_sched_rq_remove_entity(entity->rq, entity); + if (!single_entity) + drm_sched_rq_remove_entity(entity->rq, entity); spin_unlock(&entity->rq_lock); /* Make sure this entity is not used by the scheduler at the moment */ @@ -259,6 +272,20 @@ static void drm_sched_entity_kill(struct drm_sched_entity *entity) dma_fence_put(prev); } +/** + * drm_sched_entity_to_scheduler - Schedule entity to GPU scheduler + * @entity: scheduler entity + * + * Returns GPU scheduler for the entity + */ +struct drm_gpu_scheduler * +drm_sched_entity_to_scheduler(struct drm_sched_entity *entity) +{ + bool single_entity = !!entity->single_sched; + + return single_entity ? entity->single_sched : entity->rq->sched; +} + /** * drm_sched_entity_flush - Flush a context entity * @@ -276,11 +303,12 @@ long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout) struct drm_gpu_scheduler *sched; struct task_struct *last_user; long ret = timeout; + bool single_entity = !!entity->single_sched; - if (!entity->rq) + if (!entity->rq && !single_entity) return 0; - sched = entity->rq->sched; + sched = drm_sched_entity_to_scheduler(entity); /** * The client will not queue more IBs during this fini, consume existing * queued IBs or discard them on SIGKILL @@ -373,7 +401,7 @@ static void drm_sched_entity_wakeup(struct dma_fence *f, container_of(cb, struct drm_sched_entity, cb); drm_sched_entity_clear_dep(f, cb); - drm_sched_wakeup_if_can_queue(entity->rq->sched); + drm_sched_wakeup_if_can_queue(drm_sched_entity_to_scheduler(entity)); } /** @@ -387,6 +415,8 @@ static void drm_sched_entity_wakeup(struct dma_fence *f, void drm_sched_entity_set_priority(struct drm_sched_entity *entity, enum drm_sched_priority priority) { + WARN_ON(!!entity->single_sched); + spin_lock(&entity->rq_lock); entity->priority = priority; spin_unlock(&entity->rq_lock); @@ -399,7 +429,7 @@ EXPORT_SYMBOL(drm_sched_entity_set_priority); */ static bool drm_sched_entity_add_dependency_cb(struct drm_sched_entity *entity) { - struct drm_gpu_scheduler *sched = entity->rq->sched; + struct drm_gpu_scheduler *sched = drm_sched_entity_to_scheduler(entity); struct dma_fence *fence = entity->dependency; struct drm_sched_fence *s_fence; @@ -501,7 +531,8 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) * Update the entity's location in the min heap according to * the timestamp of the next job, if any. */ - if (entity->rq->sched->sched_policy == DRM_SCHED_POLICY_FIFO) { + if (drm_sched_entity_to_scheduler(entity)->sched_policy == + DRM_SCHED_POLICY_FIFO) { struct drm_sched_job *next; next = to_drm_sched_job(spsc_queue_peek(&entity->job_queue)); @@ -524,6 +555,8 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity) struct drm_gpu_scheduler *sched; struct drm_sched_rq *rq; + WARN_ON(!!entity->single_sched); + /* single possible engine and already selected */ if (!entity->sched_list) return; @@ -573,12 +606,13 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity) void drm_sched_entity_push_job(struct drm_sched_job *sched_job) { struct drm_sched_entity *entity = sched_job->entity; - bool first, fifo = entity->rq->sched->sched_policy == - DRM_SCHED_POLICY_FIFO; + bool single_entity = !!entity->single_sched; + bool first; ktime_t submit_ts; trace_drm_sched_job(sched_job, entity); - atomic_inc(entity->rq->sched->score); + if (!single_entity) + atomic_inc(entity->rq->sched->score); WRITE_ONCE(entity->last_user, current->group_leader); /* @@ -591,6 +625,10 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) /* first job wakes up scheduler */ if (first) { + struct drm_gpu_scheduler *sched = + drm_sched_entity_to_scheduler(entity); + bool fifo = sched->sched_policy == DRM_SCHED_POLICY_FIFO; + /* Add the entity to the run queue */ spin_lock(&entity->rq_lock); if (entity->stopped) { @@ -600,13 +638,14 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) return; } - drm_sched_rq_add_entity(entity->rq, entity); + if (!single_entity) + drm_sched_rq_add_entity(entity->rq, entity); spin_unlock(&entity->rq_lock); if (fifo) drm_sched_rq_update_fifo(entity, submit_ts); - drm_sched_wakeup_if_can_queue(entity->rq->sched); + drm_sched_wakeup_if_can_queue(sched); } } EXPORT_SYMBOL(drm_sched_entity_push_job); diff --git a/drivers/gpu/drm/scheduler/sched_fence.c b/drivers/gpu/drm/scheduler/sched_fence.c index 06cedfe4b486..f6b926f5e188 100644 --- a/drivers/gpu/drm/scheduler/sched_fence.c +++ b/drivers/gpu/drm/scheduler/sched_fence.c @@ -225,7 +225,7 @@ void drm_sched_fence_init(struct drm_sched_fence *fence, { unsigned seq; - fence->sched = entity->rq->sched; + fence->sched = drm_sched_entity_to_scheduler(entity); seq = atomic_inc_return(&entity->fence_seq); dma_fence_init(&fence->scheduled, &drm_sched_fence_ops_scheduled, &fence->lock, entity->fence_context, seq); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 545d5298c086..cede47afc800 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -32,7 +32,8 @@ * backend operations to the scheduler like submitting a job to hardware run queue, * returning the dependencies of a job etc. * - * The organisation of the scheduler is the following: + * The organisation of the scheduler is the following for scheduling policies + * DRM_SCHED_POLICY_RR and DRM_SCHED_POLICY_FIFO: * * 1. Each hw run queue has one scheduler * 2. Each scheduler has multiple run queues with different priorities @@ -43,6 +44,23 @@ * * The jobs in a entity are always scheduled in the order that they were pushed. * + * The organisation of the scheduler is the following for scheduling policy + * DRM_SCHED_POLICY_SINGLE_ENTITY: + * + * 1. One to one relationship between scheduler and entity + * 2. No priorities implemented per scheduler (single job queue) + * 3. No run queues in scheduler rather jobs are directly dequeued from entity + * 4. The entity maintains a queue of jobs that will be scheduled on the + * hardware + * + * The jobs in a entity are always scheduled in the order that they were pushed + * regardless of scheduling policy. + * + * A policy of DRM_SCHED_POLICY_RR or DRM_SCHED_POLICY_FIFO is expected to used + * when the KMD is scheduling directly on the hardware while a scheduling policy + * of DRM_SCHED_POLICY_SINGLE_ENTITY is expected to be used when there is a + * firmware scheduler. + * * Note that once a job was taken from the entities queue and pushed to the * hardware, i.e. the pending queue, the entity must not be referenced anymore * through the jobs entity pointer. @@ -96,6 +114,8 @@ static inline void drm_sched_rq_remove_fifo_locked(struct drm_sched_entity *enti void drm_sched_rq_update_fifo(struct drm_sched_entity *entity, ktime_t ts) { + WARN_ON(!!entity->single_sched); + /* * Both locks need to be grabbed, one to protect from entity->rq change * for entity from within concurrent drm_sched_entity_select_rq and the @@ -126,6 +146,8 @@ void drm_sched_rq_update_fifo(struct drm_sched_entity *entity, ktime_t ts) static void drm_sched_rq_init(struct drm_gpu_scheduler *sched, struct drm_sched_rq *rq) { + WARN_ON(sched->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + spin_lock_init(&rq->lock); INIT_LIST_HEAD(&rq->entities); rq->rb_tree_root = RB_ROOT_CACHED; @@ -144,6 +166,8 @@ static void drm_sched_rq_init(struct drm_gpu_scheduler *sched, void drm_sched_rq_add_entity(struct drm_sched_rq *rq, struct drm_sched_entity *entity) { + WARN_ON(!!entity->single_sched); + if (!list_empty(&entity->list)) return; @@ -166,6 +190,8 @@ void drm_sched_rq_add_entity(struct drm_sched_rq *rq, void drm_sched_rq_remove_entity(struct drm_sched_rq *rq, struct drm_sched_entity *entity) { + WARN_ON(!!entity->single_sched); + if (list_empty(&entity->list)) return; @@ -641,7 +667,7 @@ int drm_sched_job_init(struct drm_sched_job *job, struct drm_sched_entity *entity, void *owner) { - if (!entity->rq) + if (!entity->rq && !entity->single_sched) return -ENOENT; job->entity = entity; @@ -674,13 +700,16 @@ void drm_sched_job_arm(struct drm_sched_job *job) { struct drm_gpu_scheduler *sched; struct drm_sched_entity *entity = job->entity; + bool single_entity = !!entity->single_sched; BUG_ON(!entity); - drm_sched_entity_select_rq(entity); - sched = entity->rq->sched; + if (!single_entity) + drm_sched_entity_select_rq(entity); + sched = drm_sched_entity_to_scheduler(entity); job->sched = sched; - job->s_priority = entity->rq - sched->sched_rq; + if (!single_entity) + job->s_priority = entity->rq - sched->sched_rq; job->id = atomic64_inc_return(&sched->job_id_count); drm_sched_fence_init(job->s_fence, job->entity); @@ -896,6 +925,13 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched) if (!drm_sched_can_queue(sched)) return NULL; + if (sched->single_entity) { + if (drm_sched_entity_is_ready(sched->single_entity)) + return sched->single_entity; + + return NULL; + } + /* Kernel run queue has higher priority than normal run queue*/ for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { entity = sched->sched_policy == DRM_SCHED_POLICY_FIFO ? @@ -1091,6 +1127,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, return -EINVAL; sched->ops = ops; + sched->single_entity = NULL; sched->hw_submission_limit = hw_submission; sched->name = name; sched->submit_wq = submit_wq ? : system_wq; @@ -1103,7 +1140,9 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, sched->sched_policy = default_drm_sched_policy; else sched->sched_policy = sched_policy; - for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) + for (i = DRM_SCHED_PRIORITY_MIN; sched_policy != + DRM_SCHED_POLICY_SINGLE_ENTITY && i < DRM_SCHED_PRIORITY_COUNT; + i++) drm_sched_rq_init(sched, &sched->sched_rq[i]); init_waitqueue_head(&sched->job_scheduled); @@ -1135,7 +1174,15 @@ void drm_sched_fini(struct drm_gpu_scheduler *sched) drm_sched_submit_stop(sched); - for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { + if (sched->single_entity) { + spin_lock(&sched->single_entity->rq_lock); + sched->single_entity->stopped = true; + spin_unlock(&sched->single_entity->rq_lock); + } + + for (i = DRM_SCHED_PRIORITY_COUNT - 1; sched->sched_policy != + DRM_SCHED_POLICY_SINGLE_ENTITY && i >= DRM_SCHED_PRIORITY_MIN; + i--) { struct drm_sched_rq *rq = &sched->sched_rq[i]; spin_lock(&rq->lock); @@ -1176,6 +1223,8 @@ void drm_sched_increase_karma(struct drm_sched_job *bad) struct drm_sched_entity *entity; struct drm_gpu_scheduler *sched = bad->sched; + WARN_ON(sched->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + /* don't change @bad's karma if it's from KERNEL RQ, * because sometimes GPU hang would cause kernel jobs (like VM updating jobs) * corrupt but keep in mind that kernel jobs always considered good. diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 897d52a4ff4f..04eec2d7635f 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -79,6 +79,7 @@ enum drm_sched_policy { DRM_SCHED_POLICY_DEFAULT, DRM_SCHED_POLICY_RR, DRM_SCHED_POLICY_FIFO, + DRM_SCHED_POLICY_SINGLE_ENTITY, DRM_SCHED_POLICY_COUNT, }; @@ -112,6 +113,9 @@ struct drm_sched_entity { */ struct drm_sched_rq *rq; + /** @single_sched: Single scheduler */ + struct drm_gpu_scheduler *single_sched; + /** * @sched_list: * @@ -473,6 +477,7 @@ struct drm_sched_backend_ops { * struct drm_gpu_scheduler - scheduler instance-specific data * * @ops: backend operations provided by the driver. + * @single_entity: Single entity for the scheduler * @hw_submission_limit: the max size of the hardware queue. * @timeout: the time after which a job is removed from the scheduler. * @name: name of the ring for which this scheduler is being used. @@ -503,6 +508,7 @@ struct drm_sched_backend_ops { */ struct drm_gpu_scheduler { const struct drm_sched_backend_ops *ops; + struct drm_sched_entity *single_entity; uint32_t hw_submission_limit; long timeout; const char *name; @@ -585,6 +591,8 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list, atomic_t *guilty); +struct drm_gpu_scheduler * +drm_sched_entity_to_scheduler(struct drm_sched_entity *entity); long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout); void drm_sched_entity_fini(struct drm_sched_entity *entity); void drm_sched_entity_destroy(struct drm_sched_entity *entity); From patchwork Fri Aug 11 02:31:32 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13350011 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A987CC001E0 for ; Fri, 11 Aug 2023 02:31:57 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 2E89F10E632; Fri, 11 Aug 2023 02:31:47 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 439F410E623; Fri, 11 Aug 2023 02:31:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691721101; x=1723257101; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=y3aMEmMkj+6m2f6jDH+9+Z9CpZnqmjg7WCvsHkkchaw=; b=T9XyMcmsErCjdAvVNBCAFJiMW4vGwJWukEXaSdD5shuyBubggmcnfdYv RDkVDckFizDl+NbYKd6I4I3Zhw6Qvh9U7qZNe177rQmGbUNgpcb5xnRVR O6vRCJlPK3Wyl6foHMo2LhyMiBUvykT9AAD7GhUgCbgmfw7UxJmnUzwfH N+dOx07ytO62TkEPNbxSsnI6+WeoHhGaoukcPN88M4QsguJ8vWKFozrYO HN9zQVDBhya27p6A6W/mj62CEPtCJMmYDsKH3Jp992jDG6UwyFDHXZ5cW +VajyfprJdHq7wDTOTwxAggQCNIP/R6vy+PQycKnxR6U35yYQ/h5qO9dA A==; X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="361714820" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="361714820" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:40 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="797838403" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="797838403" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:40 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v2 4/9] drm/sched: Split free_job into own work item Date: Thu, 10 Aug 2023 19:31:32 -0700 Message-Id: <20230811023137.659037-5-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230811023137.659037-1-matthew.brost@intel.com> References: <20230811023137.659037-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, luben.tuikov@amd.com, lina@asahilina.net, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Rather than call free_job and run_job in same work item have a dedicated work item for each. This aligns with the design and intended use of work queues. Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 137 ++++++++++++++++++------- include/drm/gpu_scheduler.h | 8 +- 2 files changed, 106 insertions(+), 39 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index cede47afc800..b67469eac179 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -213,11 +213,12 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq, * drm_sched_rq_select_entity_rr - Select an entity which could provide a job to run * * @rq: scheduler run queue to check. + * @dequeue: dequeue selected entity * * Try to find a ready entity, returns NULL if none found. */ static struct drm_sched_entity * -drm_sched_rq_select_entity_rr(struct drm_sched_rq *rq) +drm_sched_rq_select_entity_rr(struct drm_sched_rq *rq, bool dequeue) { struct drm_sched_entity *entity; @@ -227,8 +228,10 @@ drm_sched_rq_select_entity_rr(struct drm_sched_rq *rq) if (entity) { list_for_each_entry_continue(entity, &rq->entities, list) { if (drm_sched_entity_is_ready(entity)) { - rq->current_entity = entity; - reinit_completion(&entity->entity_idle); + if (dequeue) { + rq->current_entity = entity; + reinit_completion(&entity->entity_idle); + } spin_unlock(&rq->lock); return entity; } @@ -238,8 +241,10 @@ drm_sched_rq_select_entity_rr(struct drm_sched_rq *rq) list_for_each_entry(entity, &rq->entities, list) { if (drm_sched_entity_is_ready(entity)) { - rq->current_entity = entity; - reinit_completion(&entity->entity_idle); + if (dequeue) { + rq->current_entity = entity; + reinit_completion(&entity->entity_idle); + } spin_unlock(&rq->lock); return entity; } @@ -257,11 +262,12 @@ drm_sched_rq_select_entity_rr(struct drm_sched_rq *rq) * drm_sched_rq_select_entity_fifo - Select an entity which provides a job to run * * @rq: scheduler run queue to check. + * @dequeue: dequeue selected entity * * Find oldest waiting ready entity, returns NULL if none found. */ static struct drm_sched_entity * -drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq) +drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq, bool dequeue) { struct rb_node *rb; @@ -271,8 +277,10 @@ drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq) entity = rb_entry(rb, struct drm_sched_entity, rb_tree_node); if (drm_sched_entity_is_ready(entity)) { - rq->current_entity = entity; - reinit_completion(&entity->entity_idle); + if (dequeue) { + rq->current_entity = entity; + reinit_completion(&entity->entity_idle); + } break; } } @@ -282,13 +290,54 @@ drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq) } /** - * drm_sched_submit_queue - scheduler queue submission + * drm_sched_run_job_queue - queue job submission * @sched: scheduler instance */ -static void drm_sched_submit_queue(struct drm_gpu_scheduler *sched) +static void drm_sched_run_job_queue(struct drm_gpu_scheduler *sched) { if (!READ_ONCE(sched->pause_submit)) - queue_work(sched->submit_wq, &sched->work_submit); + queue_work(sched->submit_wq, &sched->work_run_job); +} + +static struct drm_sched_entity * +drm_sched_select_entity(struct drm_gpu_scheduler *sched, bool dequeue); + +/** + * drm_sched_run_job_queue_if_ready - queue job submission if ready + * @sched: scheduler instance + */ +static void drm_sched_run_job_queue_if_ready(struct drm_gpu_scheduler *sched) +{ + if (drm_sched_select_entity(sched, false)) + drm_sched_run_job_queue(sched); +} + +/** + * drm_sched_free_job_queue - queue free job + * + * @sched: scheduler instance to queue free job + */ +static void drm_sched_free_job_queue(struct drm_gpu_scheduler *sched) +{ + if (!READ_ONCE(sched->pause_submit)) + queue_work(sched->submit_wq, &sched->work_free_job); +} + +/** + * drm_sched_free_job_queue_if_ready - queue free job if ready + * + * @sched: scheduler instance to queue free job + */ +static void drm_sched_free_job_queue_if_ready(struct drm_gpu_scheduler *sched) +{ + struct drm_sched_job *job; + + spin_lock(&sched->job_list_lock); + job = list_first_entry_or_null(&sched->pending_list, + struct drm_sched_job, list); + if (job && dma_fence_is_signaled(&job->s_fence->finished)) + drm_sched_free_job_queue(sched); + spin_unlock(&sched->job_list_lock); } /** @@ -310,7 +359,7 @@ static void drm_sched_job_done(struct drm_sched_job *s_job, int result) dma_fence_get(&s_fence->finished); drm_sched_fence_finished(s_fence, result); dma_fence_put(&s_fence->finished); - drm_sched_submit_queue(sched); + drm_sched_free_job_queue(sched); } /** @@ -906,18 +955,19 @@ static bool drm_sched_can_queue(struct drm_gpu_scheduler *sched) void drm_sched_wakeup_if_can_queue(struct drm_gpu_scheduler *sched) { if (drm_sched_can_queue(sched)) - drm_sched_submit_queue(sched); + drm_sched_run_job_queue(sched); } /** * drm_sched_select_entity - Select next entity to process * * @sched: scheduler instance + * @dequeue: dequeue selected entity * * Returns the entity to process or NULL if none are found. */ static struct drm_sched_entity * -drm_sched_select_entity(struct drm_gpu_scheduler *sched) +drm_sched_select_entity(struct drm_gpu_scheduler *sched, bool dequeue) { struct drm_sched_entity *entity; int i; @@ -935,8 +985,10 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched) /* Kernel run queue has higher priority than normal run queue*/ for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { entity = sched->sched_policy == DRM_SCHED_POLICY_FIFO ? - drm_sched_rq_select_entity_fifo(&sched->sched_rq[i]) : - drm_sched_rq_select_entity_rr(&sched->sched_rq[i]); + drm_sched_rq_select_entity_fifo(&sched->sched_rq[i], + dequeue) : + drm_sched_rq_select_entity_rr(&sched->sched_rq[i], + dequeue); if (entity) break; } @@ -1024,30 +1076,44 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list, EXPORT_SYMBOL(drm_sched_pick_best); /** - * drm_sched_main - main scheduler thread + * drm_sched_free_job_work - worker to call free_job * - * @param: scheduler instance + * @w: free job work */ -static void drm_sched_main(struct work_struct *w) +static void drm_sched_free_job_work(struct work_struct *w) { struct drm_gpu_scheduler *sched = - container_of(w, struct drm_gpu_scheduler, work_submit); - struct drm_sched_entity *entity; + container_of(w, struct drm_gpu_scheduler, work_free_job); struct drm_sched_job *cleanup_job; - int r; if (READ_ONCE(sched->pause_submit)) return; cleanup_job = drm_sched_get_cleanup_job(sched); - entity = drm_sched_select_entity(sched); + if (cleanup_job) { + sched->ops->free_job(cleanup_job); + + drm_sched_free_job_queue_if_ready(sched); + drm_sched_run_job_queue_if_ready(sched); + } +} - if (!entity && !cleanup_job) - return; /* No more work */ +/** + * drm_sched_run_job_work - worker to call run_job + * + * @w: run job work + */ +static void drm_sched_run_job_work(struct work_struct *w) +{ + struct drm_gpu_scheduler *sched = + container_of(w, struct drm_gpu_scheduler, work_run_job); + struct drm_sched_entity *entity; + int r; - if (cleanup_job) - sched->ops->free_job(cleanup_job); + if (READ_ONCE(sched->pause_submit)) + return; + entity = drm_sched_select_entity(sched, true); if (entity) { struct dma_fence *fence; struct drm_sched_fence *s_fence; @@ -1056,9 +1122,7 @@ static void drm_sched_main(struct work_struct *w) sched_job = drm_sched_entity_pop_job(entity); if (!sched_job) { complete_all(&entity->entity_idle); - if (!cleanup_job) - return; /* No more work */ - goto again; + return; /* No more work */ } s_fence = sched_job->s_fence; @@ -1088,10 +1152,8 @@ static void drm_sched_main(struct work_struct *w) } wake_up(&sched->job_scheduled); + drm_sched_run_job_queue_if_ready(sched); } - -again: - drm_sched_submit_queue(sched); } /** @@ -1150,7 +1212,8 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, spin_lock_init(&sched->job_list_lock); atomic_set(&sched->hw_rq_count, 0); INIT_DELAYED_WORK(&sched->work_tdr, drm_sched_job_timedout); - INIT_WORK(&sched->work_submit, drm_sched_main); + INIT_WORK(&sched->work_run_job, drm_sched_run_job_work); + INIT_WORK(&sched->work_free_job, drm_sched_free_job_work); atomic_set(&sched->_score, 0); atomic64_set(&sched->job_id_count, 0); sched->pause_submit = false; @@ -1275,7 +1338,8 @@ EXPORT_SYMBOL(drm_sched_submit_ready); void drm_sched_submit_stop(struct drm_gpu_scheduler *sched) { WRITE_ONCE(sched->pause_submit, true); - cancel_work_sync(&sched->work_submit); + cancel_work_sync(&sched->work_run_job); + cancel_work_sync(&sched->work_free_job); } EXPORT_SYMBOL(drm_sched_submit_stop); @@ -1287,6 +1351,7 @@ EXPORT_SYMBOL(drm_sched_submit_stop); void drm_sched_submit_start(struct drm_gpu_scheduler *sched) { WRITE_ONCE(sched->pause_submit, false); - queue_work(sched->submit_wq, &sched->work_submit); + queue_work(sched->submit_wq, &sched->work_run_job); + queue_work(sched->submit_wq, &sched->work_free_job); } EXPORT_SYMBOL(drm_sched_submit_start); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 04eec2d7635f..fbc083a92757 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -487,9 +487,10 @@ struct drm_sched_backend_ops { * finished. * @hw_rq_count: the number of jobs currently in the hardware queue. * @job_id_count: used to assign unique id to the each job. - * @submit_wq: workqueue used to queue @work_submit + * @submit_wq: workqueue used to queue @work_run_job and @work_free_job * @timeout_wq: workqueue used to queue @work_tdr - * @work_submit: schedules jobs and cleans up entities + * @work_run_job: schedules jobs + * @work_free_job: cleans up jobs * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the * timeout interval is over. * @pending_list: the list of jobs which are currently in the job queue. @@ -518,7 +519,8 @@ struct drm_gpu_scheduler { atomic64_t job_id_count; struct workqueue_struct *submit_wq; struct workqueue_struct *timeout_wq; - struct work_struct work_submit; + struct work_struct work_run_job; + struct work_struct work_free_job; struct delayed_work work_tdr; struct list_head pending_list; spinlock_t job_list_lock; From patchwork Fri Aug 11 02:31:33 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13350016 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 95456C04A94 for ; Fri, 11 Aug 2023 02:32:06 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 2D36D10E628; Fri, 11 Aug 2023 02:31:56 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id DA41A10E627; Fri, 11 Aug 2023 02:31:41 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691721101; x=1723257101; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Mgy+OkNGmOqugardTTwEk11ZEQmM/8hq+ngo7uZWH4g=; b=hX7gGQRCftMfN92HCS4tQcJoSAwN1c61558wMooILwS86eh3EDvo6ECz QKJLMLkrl/cO/JNhrNT6JjF3fGLNWGpCwux8lg5teI7cHarZuqMIArMCD j1vl9l8lyYEgMYyPcY3G6tGkFbXEFqw9O6F/5Q2oU/IIhKInxog6+y+Ym pxMTZVZIiRs7rS56QRvT7Rvn/UEtmRwNFaqauvX1hzq++8c4yQCsPUZW0 Y+2FNPR4dPLt4GEZYPHPQZ5VaQETa68i9huHEubDXA0a1lOVWdAZw54Er JbebdG97pYyuVttrzriLEBvNnLwvJO/CXrOYAJKVjR1cqPK/Yj7XPFUub A==; X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="361714829" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="361714829" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:41 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="797838407" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="797838407" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:40 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v2 5/9] drm/sched: Add generic scheduler message interface Date: Thu, 10 Aug 2023 19:31:33 -0700 Message-Id: <20230811023137.659037-6-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230811023137.659037-1-matthew.brost@intel.com> References: <20230811023137.659037-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, luben.tuikov@amd.com, lina@asahilina.net, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add generic schedule message interface which sends messages to backend from the drm_gpu_scheduler main submission thread. The idea is some of these messages modify some state in drm_sched_entity which is also modified during submission. By scheduling these messages and submission in the same thread their is not race changing states in drm_sched_entity. This interface will be used in Xe, new Intel GPU driver, to cleanup, suspend, resume, and change scheduling properties of a drm_sched_entity. The interface is designed to be generic and extendable with only the backend understanding the messages. v2: - (Christian) We dedicated work item Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 98 ++++++++++++++++++++++++++ include/drm/gpu_scheduler.h | 34 ++++++++- 2 files changed, 131 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index b67469eac179..fbd99f7e5b4a 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -340,6 +340,35 @@ static void drm_sched_free_job_queue_if_ready(struct drm_gpu_scheduler *sched) spin_unlock(&sched->job_list_lock); } +/** + * drm_sched_process_msg_queue - queue process msg worker + * + * @sched: scheduler instance to queue process_msg worker + */ +static void drm_sched_process_msg_queue(struct drm_gpu_scheduler *sched) +{ + if (!READ_ONCE(sched->pause_submit)) + queue_work(sched->submit_wq, &sched->work_process_msg); +} + +/** + * drm_sched_process_msg_queue_if_ready - queue process msg worker if ready + * + * @sched: scheduler instance to queue process_msg worker + */ +static void +drm_sched_process_msg_queue_if_ready(struct drm_gpu_scheduler *sched) +{ + struct drm_sched_msg *msg; + + spin_lock(&sched->job_list_lock); + msg = list_first_entry_or_null(&sched->msgs, + struct drm_sched_msg, link); + if (msg) + drm_sched_process_msg_queue(sched); + spin_unlock(&sched->job_list_lock); +} + /** * drm_sched_job_done - complete a job * @s_job: pointer to the job which is done @@ -1075,6 +1104,71 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list, } EXPORT_SYMBOL(drm_sched_pick_best); +/** + * drm_sched_add_msg - add scheduler message + * + * @sched: scheduler instance + * @msg: message to be added + * + * Can and will pass an jobs waiting on dependencies or in a runnable queue. + * Messages processing will stop if schedule run wq is stopped and resume when + * run wq is started. + */ +void drm_sched_add_msg(struct drm_gpu_scheduler *sched, + struct drm_sched_msg *msg) +{ + spin_lock(&sched->job_list_lock); + list_add_tail(&msg->link, &sched->msgs); + spin_unlock(&sched->job_list_lock); + + drm_sched_process_msg_queue(sched); +} +EXPORT_SYMBOL(drm_sched_add_msg); + +/** + * drm_sched_get_msg - get scheduler message + * + * @sched: scheduler instance + * + * Returns NULL or message + */ +static struct drm_sched_msg * +drm_sched_get_msg(struct drm_gpu_scheduler *sched) +{ + struct drm_sched_msg *msg; + + spin_lock(&sched->job_list_lock); + msg = list_first_entry_or_null(&sched->msgs, + struct drm_sched_msg, link); + if (msg) + list_del(&msg->link); + spin_unlock(&sched->job_list_lock); + + return msg; +} + +/** + * drm_sched_process_msg_work - worker to call process_msg + * + * @w: process msg work + */ +static void drm_sched_process_msg_work(struct work_struct *w) +{ + struct drm_gpu_scheduler *sched = + container_of(w, struct drm_gpu_scheduler, work_process_msg); + struct drm_sched_msg *msg; + + if (READ_ONCE(sched->pause_submit)) + return; + + msg = drm_sched_get_msg(sched); + if (msg) { + sched->ops->process_msg(msg); + + drm_sched_process_msg_queue_if_ready(sched); + } +} + /** * drm_sched_free_job_work - worker to call free_job * @@ -1209,11 +1303,13 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, init_waitqueue_head(&sched->job_scheduled); INIT_LIST_HEAD(&sched->pending_list); + INIT_LIST_HEAD(&sched->msgs); spin_lock_init(&sched->job_list_lock); atomic_set(&sched->hw_rq_count, 0); INIT_DELAYED_WORK(&sched->work_tdr, drm_sched_job_timedout); INIT_WORK(&sched->work_run_job, drm_sched_run_job_work); INIT_WORK(&sched->work_free_job, drm_sched_free_job_work); + INIT_WORK(&sched->work_process_msg, drm_sched_process_msg_work); atomic_set(&sched->_score, 0); atomic64_set(&sched->job_id_count, 0); sched->pause_submit = false; @@ -1340,6 +1436,7 @@ void drm_sched_submit_stop(struct drm_gpu_scheduler *sched) WRITE_ONCE(sched->pause_submit, true); cancel_work_sync(&sched->work_run_job); cancel_work_sync(&sched->work_free_job); + cancel_work_sync(&sched->work_process_msg); } EXPORT_SYMBOL(drm_sched_submit_stop); @@ -1353,5 +1450,6 @@ void drm_sched_submit_start(struct drm_gpu_scheduler *sched) WRITE_ONCE(sched->pause_submit, false); queue_work(sched->submit_wq, &sched->work_run_job); queue_work(sched->submit_wq, &sched->work_free_job); + queue_work(sched->submit_wq, &sched->work_process_msg); } EXPORT_SYMBOL(drm_sched_submit_start); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index fbc083a92757..5d753ecb5d71 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -394,6 +394,23 @@ enum drm_gpu_sched_stat { DRM_GPU_SCHED_STAT_ENODEV, }; +/** + * struct drm_sched_msg - an in-band (relative to GPU scheduler run queue) + * message + * + * Generic enough for backend defined messages, backend can expand if needed. + */ +struct drm_sched_msg { + /** @link: list link into the gpu scheduler list of messages */ + struct list_head link; + /** + * @private_data: opaque pointer to message private data (backend defined) + */ + void *private_data; + /** @opcode: opcode of message (backend defined) */ + unsigned int opcode; +}; + /** * struct drm_sched_backend_ops - Define the backend operations * called by the scheduler @@ -471,6 +488,12 @@ struct drm_sched_backend_ops { * and it's time to clean it up. */ void (*free_job)(struct drm_sched_job *sched_job); + + /** + * @process_msg: Process a message. Allowed to block, it is this + * function's responsibility to free message if dynamically allocated. + */ + void (*process_msg)(struct drm_sched_msg *msg); }; /** @@ -482,15 +505,18 @@ struct drm_sched_backend_ops { * @timeout: the time after which a job is removed from the scheduler. * @name: name of the ring for which this scheduler is being used. * @sched_rq: priority wise array of run queues. + * @msgs: list of messages to be processed in @work_process_msg * @job_scheduled: once @drm_sched_entity_do_release is called the scheduler * waits on this wait queue until all the scheduled jobs are * finished. * @hw_rq_count: the number of jobs currently in the hardware queue. * @job_id_count: used to assign unique id to the each job. - * @submit_wq: workqueue used to queue @work_run_job and @work_free_job + * @submit_wq: workqueue used to queue @work_run_job, @work_free_job, and + * @work_process_msg * @timeout_wq: workqueue used to queue @work_tdr * @work_run_job: schedules jobs * @work_free_job: cleans up jobs + * @work_process_msg: processes messages * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the * timeout interval is over. * @pending_list: the list of jobs which are currently in the job queue. @@ -502,6 +528,8 @@ struct drm_sched_backend_ops { * @sched_policy: Schedule policy for scheduler * @ready: marks if the underlying HW is ready to work * @free_guilty: A hit to time out handler to free the guilty job. + * @pause_submit: pause queuing of @work_run_job, @work_free_job, and + * @work_process_msg on @submit_wq * @pause_submit: pause queuing of @work_submit on @submit_wq * @dev: system &struct device * @@ -514,6 +542,7 @@ struct drm_gpu_scheduler { long timeout; const char *name; struct drm_sched_rq sched_rq[DRM_SCHED_PRIORITY_COUNT]; + struct list_head msgs; wait_queue_head_t job_scheduled; atomic_t hw_rq_count; atomic64_t job_id_count; @@ -521,6 +550,7 @@ struct drm_gpu_scheduler { struct workqueue_struct *timeout_wq; struct work_struct work_run_job; struct work_struct work_free_job; + struct work_struct work_process_msg; struct delayed_work work_tdr; struct list_head pending_list; spinlock_t job_list_lock; @@ -568,6 +598,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, void drm_sched_job_cleanup(struct drm_sched_job *job); void drm_sched_wakeup_if_can_queue(struct drm_gpu_scheduler *sched); +void drm_sched_add_msg(struct drm_gpu_scheduler *sched, + struct drm_sched_msg *msg); bool drm_sched_submit_ready(struct drm_gpu_scheduler *sched); void drm_sched_submit_stop(struct drm_gpu_scheduler *sched); void drm_sched_submit_start(struct drm_gpu_scheduler *sched); From patchwork Fri Aug 11 02:31:34 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13350015 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0BC5EC001E0 for ; Fri, 11 Aug 2023 02:32:05 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0095110E625; Fri, 11 Aug 2023 02:31:56 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 57DE810E628; Fri, 11 Aug 2023 02:31:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691721102; x=1723257102; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=HXyVsMrjZ/t+FVSLlj4E8K+Ry+4JKU9NSvqOpzcOeSA=; b=kAvHqEzwP7XdmtcqTinOR8YGbWMcyGtgZfVohNGF+aPYElNvTZGoGvrE pd5UZRIyukXmwVKH/sMZUv+qDif9SadEVqXzZ8MtaDPByr3O5sOCwgKG4 aTm1G44fLNp0PJRFmNpJsMRx45OpXppIRBuW66y3duGjYPhaG88//0KPM onOg9OaRd9YVnxSyvvxgAxLhwn/XO0HFfmf2LGnN8NLLXGrA2mEmWoc+v 94WwuvYWeCCYNerQ9E/VajF6Z0Jb+Sr8a/oCUYMS/ZJPE0TV410KM4XIx 5sIgzc0K768ysXyYtx1Khj+DgGdfsvJSvx7HFqPooqZEZtMcLyaW0DYH4 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="361714838" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="361714838" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="797838410" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="797838410" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:41 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v2 6/9] drm/sched: Add drm_sched_start_timeout_unlocked helper Date: Thu, 10 Aug 2023 19:31:34 -0700 Message-Id: <20230811023137.659037-7-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230811023137.659037-1-matthew.brost@intel.com> References: <20230811023137.659037-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, luben.tuikov@amd.com, lina@asahilina.net, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Also add a lockdep assert to drm_sched_start_timeout. Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index fbd99f7e5b4a..d5f6d86985c5 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -412,11 +412,20 @@ static void drm_sched_job_done_cb(struct dma_fence *f, struct dma_fence_cb *cb) */ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched) { + lockdep_assert_held(&sched->job_list_lock); + if (sched->timeout != MAX_SCHEDULE_TIMEOUT && !list_empty(&sched->pending_list)) queue_delayed_work(sched->timeout_wq, &sched->work_tdr, sched->timeout); } +static void drm_sched_start_timeout_unlocked(struct drm_gpu_scheduler *sched) +{ + spin_lock(&sched->job_list_lock); + drm_sched_start_timeout(sched); + spin_unlock(&sched->job_list_lock); +} + /** * drm_sched_fault - immediately start timeout handler * @@ -529,11 +538,8 @@ static void drm_sched_job_timedout(struct work_struct *work) spin_unlock(&sched->job_list_lock); } - if (status != DRM_GPU_SCHED_STAT_ENODEV) { - spin_lock(&sched->job_list_lock); - drm_sched_start_timeout(sched); - spin_unlock(&sched->job_list_lock); - } + if (status != DRM_GPU_SCHED_STAT_ENODEV) + drm_sched_start_timeout_unlocked(sched); } /** @@ -659,11 +665,8 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery) drm_sched_job_done(s_job, -ECANCELED); } - if (full_recovery) { - spin_lock(&sched->job_list_lock); - drm_sched_start_timeout(sched); - spin_unlock(&sched->job_list_lock); - } + if (full_recovery) + drm_sched_start_timeout_unlocked(sched); drm_sched_submit_start(sched); } From patchwork Fri Aug 11 02:31:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13350009 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2EB0CC04A94 for ; Fri, 11 Aug 2023 02:31:52 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D254F10E62C; Fri, 11 Aug 2023 02:31:44 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id A101510E627; Fri, 11 Aug 2023 02:31:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691721102; x=1723257102; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=uxEB+MKmGI5F3sY6+pI3RvHBcMNY2fAvnTbcMUC3SC0=; b=E8uDaDB6k7K7yNI0G2ItlsL3pWvcFJM9lnnX/0wikXuGjMoNW2MsCERw O1gMODnQDVl0OSm/1iOBxMwpArGI/B1pek/KJ1FjvLKSCPHLkJPJK4rFe JhH05P0K53otf8+bZse6C70d11fHbMHwSGBof+srCZTNpc4TfzB/rOgv/ 4TkbcsONkwaC/l7wGdM8CMxwGQpS7hOIKg6sc/XLBWZkuAVJcb5XD2b9g p4chYq57lc4b39TodWgBpojYiPW/2411QwLRZhf9JDXDl0rvmwjI/TY2i gA1v4SJHypyYRcszzzWEyAa3A7EG9Xkwi/S8TugsD70ujgNOsx8FrEFo6 g==; X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="361714848" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="361714848" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="797838413" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="797838413" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:41 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v2 7/9] drm/sched: Start run wq before TDR in drm_sched_start Date: Thu, 10 Aug 2023 19:31:35 -0700 Message-Id: <20230811023137.659037-8-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230811023137.659037-1-matthew.brost@intel.com> References: <20230811023137.659037-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, luben.tuikov@amd.com, lina@asahilina.net, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" If the TDR is set to a very small value it can fire before the run wq is started in the function drm_sched_start. The run wq is expected to running when the TDR fires, fix this ordering so this expectation is always met. Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index d5f6d86985c5..0e7d9e227a6a 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -665,10 +665,10 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery) drm_sched_job_done(s_job, -ECANCELED); } + drm_sched_submit_start(sched); + if (full_recovery) drm_sched_start_timeout_unlocked(sched); - - drm_sched_submit_start(sched); } EXPORT_SYMBOL(drm_sched_start); From patchwork Fri Aug 11 02:31:36 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13350012 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 93594C001DE for ; Fri, 11 Aug 2023 02:31:59 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id F138010E636; Fri, 11 Aug 2023 02:31:47 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 1B5EF10E627; Fri, 11 Aug 2023 02:31:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691721103; x=1723257103; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=cFrSbP44yEQ6+Q0dReeUFBOCkM4J4jR3u4//cL0MRSw=; b=jbI7EjkcpacBvqbpCZoOxofJ7IielHIhvitob5wP1RCMivHTwm7Lh/Xs +jBDqICNbpn6JO9XgpZW0CvBxF2J3UmyvqWHAi3p8eyWnJ5IR9+Kgjfig u6t2mhtH2dHBuuBYAgc5J0GBpkawJyqCg/QQ4sKYSe6pW/yAJYu3RRebM +kQnmLb2giCB1QO+toZyjAOx0zhjbURPl8MNidQkBfxsADbkxH4fQc8zz 3WbPvkYiRZlu5LGqTyfA2bxYZvzzFfFa+Emus/0qBeYpDVBaihD0i0U8m nA8ly6o8QzHtwEsdudbkmHCLAf2Uu9O3AY+4dlpGMNbkEKi7Oo9z0XgQy w==; X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="361714857" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="361714857" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:42 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="797838416" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="797838416" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:42 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v2 8/9] drm/sched: Submit job before starting TDR Date: Thu, 10 Aug 2023 19:31:36 -0700 Message-Id: <20230811023137.659037-9-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230811023137.659037-1-matthew.brost@intel.com> References: <20230811023137.659037-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, luben.tuikov@amd.com, lina@asahilina.net, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" If the TDR is set to a value, it can fire before a job is submitted in drm_sched_main. The job should be always be submitted before the TDR fires, fix this ordering. v2: - Add to pending list before run_job, start TDR after (Luben, Boris) Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 0e7d9e227a6a..6aa3a35f55dc 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -498,7 +498,6 @@ static void drm_sched_job_begin(struct drm_sched_job *s_job) spin_lock(&sched->job_list_lock); list_add_tail(&s_job->list, &sched->pending_list); - drm_sched_start_timeout(sched); spin_unlock(&sched->job_list_lock); } @@ -1231,6 +1230,7 @@ static void drm_sched_run_job_work(struct work_struct *w) fence = sched->ops->run_job(sched_job); complete_all(&entity->entity_idle); drm_sched_fence_scheduled(s_fence, fence); + drm_sched_start_timeout_unlocked(sched); if (!IS_ERR_OR_NULL(fence)) { /* Drop for original kref_init of the fence */ From patchwork Fri Aug 11 02:31:37 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13350010 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E71F8C001E0 for ; Fri, 11 Aug 2023 02:31:54 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0697F10E62F; Fri, 11 Aug 2023 02:31:47 +0000 (UTC) Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.20]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7691F10E627; Fri, 11 Aug 2023 02:31:43 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691721103; x=1723257103; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=nkr4lFWb5q586BumVDtISFqgzuwHjyNS9TAu3K1cihw=; b=TOdzuD8Y0fPlI19Uuu9ZsL7Ip4cTBXzsNpViqF1FH5Ur6j+U9hHpKUc+ 6X25/vIs44JK+Xlv4U/1Ru3NEnbeU7d239KZeS177rdABC44ukcB4vaon NpfNAcu8X91aN597G8sPbMl88DEeoD8zoSk6XJhZhl/DXmechgmI95YdG 3vMWrz87y/jLuTP0ieXZOzhFJzE2gfanZncGsx5CaEviiqVEuQRthBdCS v4olQyg/GSph5HUTHOqJ8nOtDGdTqxxVanadm8r1piqhR7NXtz3wOkkm2 KG4wGQmd0BiDtVXP8Ocosof+jcomDv26MyE1htwMlFDp2wceVIpq2UBQ7 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="361714866" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="361714866" Received: from fmsmga008.fm.intel.com ([10.253.24.58]) by orsmga101.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:43 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10798"; a="797838420" X-IronPort-AV: E=Sophos;i="6.01,164,1684825200"; d="scan'208";a="797838420" Received: from lstrano-desk.jf.intel.com ([10.54.39.91]) by fmsmga008-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 10 Aug 2023 19:31:42 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [PATCH v2 9/9] drm/sched: Add helper to set TDR timeout Date: Thu, 10 Aug 2023 19:31:37 -0700 Message-Id: <20230811023137.659037-10-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230811023137.659037-1-matthew.brost@intel.com> References: <20230811023137.659037-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, Matthew Brost , sarah.walker@imgtec.com, ketil.johnsen@arm.com, Liviu.Dudau@arm.com, luben.tuikov@amd.com, lina@asahilina.net, donald.robson@imgtec.com, boris.brezillon@collabora.com, christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add helper to set TDR timeout and restart the TDR with new timeout value. This will be used in XE, new Intel GPU driver, to trigger the TDR to cleanup drm_sched_entity that encounter errors. Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 18 ++++++++++++++++++ include/drm/gpu_scheduler.h | 1 + 2 files changed, 19 insertions(+) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 6aa3a35f55dc..67e0fb6e7d18 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -426,6 +426,24 @@ static void drm_sched_start_timeout_unlocked(struct drm_gpu_scheduler *sched) spin_unlock(&sched->job_list_lock); } +/** + * drm_sched_set_timeout - set timeout for reset worker + * + * @sched: scheduler instance to set and (re)-start the worker for + * @timeout: timeout period + * + * Set and (re)-start the timeout for the given scheduler. + */ +void drm_sched_set_timeout(struct drm_gpu_scheduler *sched, long timeout) +{ + spin_lock(&sched->job_list_lock); + sched->timeout = timeout; + cancel_delayed_work(&sched->work_tdr); + drm_sched_start_timeout(sched); + spin_unlock(&sched->job_list_lock); +} +EXPORT_SYMBOL(drm_sched_set_timeout); + /** * drm_sched_fault - immediately start timeout handler * diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 5d753ecb5d71..b7b818cd81b6 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -596,6 +596,7 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list); +void drm_sched_set_timeout(struct drm_gpu_scheduler *sched, long timeout); void drm_sched_job_cleanup(struct drm_sched_job *job); void drm_sched_wakeup_if_can_queue(struct drm_gpu_scheduler *sched); void drm_sched_add_msg(struct drm_gpu_scheduler *sched,