From patchwork Tue Apr 4 00:22:02 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13198938 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 0647AC77B6E for ; Tue, 4 Apr 2023 00:22:27 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B62FA10E08A; Tue, 4 Apr 2023 00:22:25 +0000 (UTC) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 335E810E048; Tue, 4 Apr 2023 00:22:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680567743; x=1712103743; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=wwnYER675pjMy4dMA1qWSvZlWrML7EZ5g0Owafj6GsQ=; b=WTKeB9vlsJ+wiuAFbUMDJF3kqYRDPpQXOmwuvr5jGhGn5FBxRjN3DM1t 1Rsoa2m01Ggyf7BP1SD2ZzfXwVKrCl7SwNXrYEXplgPM9TmBgtdXXm13h 69ZFKXSZY7AA1Xfuo31H17sCKjXhhY8/yE8jMk5HoDLFyFi26HgyJD3la s1L8w1oztNE3V1+/9AcZ9v403jOGwVxqIenZkqGGk3IOEXIbmpk9SwdE+ Pk5uh9ggisASR/1TlXn9UA1p78DfjgtO2WNSdJq+njWJ9dLXDn+MYCO9O ZxewPUaFmgDwcpbDMIUtLTzMejw4erjhK4syNG8eEOUBP/VrBZML4xod9 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="404810485" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="404810485" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="716460291" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="716460291" Received: from lstrano-desk.jf.intel.com ([10.24.89.184]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC PATCH 01/10] drm/sched: Convert drm scheduler to use a work queue rather than kthread Date: Mon, 3 Apr 2023 17:22:02 -0700 Message-Id: <20230404002211.3611376-2-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230404002211.3611376-1-matthew.brost@intel.com> References: <20230404002211.3611376-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, Matthew Brost , christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" In XE, the new Intel GPU driver, a choice has made to have a 1 to 1 mapping between a drm_gpu_scheduler and drm_sched_entity. At first this seems a bit odd but let us explain the reasoning below. 1. In XE the submission order from multiple drm_sched_entity is not guaranteed to be the same completion even if targeting the same hardware engine. This is because in XE we have a firmware scheduler, the GuC, which allowed to reorder, timeslice, and preempt submissions. If a using shared drm_gpu_scheduler across multiple drm_sched_entity, the TDR falls apart as the TDR expects submission order == completion order. Using a dedicated drm_gpu_scheduler per drm_sched_entity solve this problem. 2. In XE submissions are done via programming a ring buffer (circular buffer), a drm_gpu_scheduler provides a limit on number of jobs, if the limit of number jobs is set to RING_SIZE / MAX_SIZE_PER_JOB we get flow control on the ring for free. A problem with this design is currently a drm_gpu_scheduler uses a kthread for submission / job cleanup. This doesn't scale if a large number of drm_gpu_scheduler are used. To work around the scaling issue, use a worker rather than kthread for submission / job cleanup. v2: - (Rob Clark) Fix msm build - Pass in run work queue Signed-off-by: Matthew Brost --- drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c | 14 +-- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 14 +-- drivers/gpu/drm/etnaviv/etnaviv_sched.c | 2 +- drivers/gpu/drm/lima/lima_sched.c | 2 +- drivers/gpu/drm/msm/adreno/adreno_device.c | 6 +- drivers/gpu/drm/msm/msm_ringbuffer.c | 2 +- drivers/gpu/drm/panfrost/panfrost_job.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 126 ++++++++++++-------- drivers/gpu/drm/v3d/v3d_sched.c | 10 +- include/drm/gpu_scheduler.h | 14 ++- 10 files changed, 110 insertions(+), 82 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c index f60753f97ac5..9c2a10aeb0b3 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c @@ -1489,9 +1489,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused) for (i = 0; i < AMDGPU_MAX_RINGS; i++) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !ring->sched.ready) continue; - kthread_park(ring->sched.thread); + drm_sched_run_wq_stop(&ring->sched); } seq_printf(m, "run ib test:\n"); @@ -1505,9 +1505,9 @@ static int amdgpu_debugfs_test_ib_show(struct seq_file *m, void *unused) for (i = 0; i < AMDGPU_MAX_RINGS; i++) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !ring->sched.ready) continue; - kthread_unpark(ring->sched.thread); + drm_sched_run_wq_start(&ring->sched); } up_write(&adev->reset_domain->sem); @@ -1727,7 +1727,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val) ring = adev->rings[val]; - if (!ring || !ring->funcs->preempt_ib || !ring->sched.thread) + if (!ring || !ring->funcs->preempt_ib || !ring->sched.ready) return -EINVAL; /* the last preemption failed */ @@ -1745,7 +1745,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val) goto pro_end; /* stop the scheduler */ - kthread_park(ring->sched.thread); + drm_sched_run_wq_stop(&ring->sched); /* preempt the IB */ r = amdgpu_ring_preempt_ib(ring); @@ -1779,7 +1779,7 @@ static int amdgpu_debugfs_ib_preempt(void *data, u64 val) failure: /* restart the scheduler */ - kthread_unpark(ring->sched.thread); + drm_sched_run_wq_start(&ring->sched); up_read(&adev->reset_domain->sem); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index fac9312b1695..00c9c03c8f94 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -2364,7 +2364,7 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev) break; } - r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, + r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, NULL, ring->num_hw_submission, amdgpu_job_hang_limit, timeout, adev->reset_domain->wq, ring->sched_score, ring->name, @@ -4627,7 +4627,7 @@ bool amdgpu_device_has_job_running(struct amdgpu_device *adev) for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !ring->sched.ready) continue; spin_lock(&ring->sched.job_list_lock); @@ -4753,7 +4753,7 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !ring->sched.ready) continue; /*clear job fence from fence drv to avoid force_completion @@ -5294,7 +5294,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = tmp_adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !ring->sched.ready) continue; drm_sched_stop(&ring->sched, job ? &job->base : NULL); @@ -5369,7 +5369,7 @@ int amdgpu_device_gpu_recover(struct amdgpu_device *adev, for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = tmp_adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !ring->sched.ready) continue; drm_sched_start(&ring->sched, true); @@ -5696,7 +5696,7 @@ pci_ers_result_t amdgpu_pci_error_detected(struct pci_dev *pdev, pci_channel_sta for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !ring->sched.ready) continue; drm_sched_stop(&ring->sched, NULL); @@ -5824,7 +5824,7 @@ void amdgpu_pci_resume(struct pci_dev *pdev) for (i = 0; i < AMDGPU_MAX_RINGS; ++i) { struct amdgpu_ring *ring = adev->rings[i]; - if (!ring || !ring->sched.thread) + if (!ring || !ring->sched.ready) continue; drm_sched_start(&ring->sched, true); diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c index 1ae87dfd19c4..8486a2923f1b 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c @@ -133,7 +133,7 @@ int etnaviv_sched_init(struct etnaviv_gpu *gpu) { int ret; - ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, + ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, NULL, etnaviv_hw_jobs_limit, etnaviv_job_hang_limit, msecs_to_jiffies(500), NULL, NULL, dev_name(gpu->dev), gpu->dev); diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c index ff003403fbbc..54f53bece27c 100644 --- a/drivers/gpu/drm/lima/lima_sched.c +++ b/drivers/gpu/drm/lima/lima_sched.c @@ -488,7 +488,7 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name) INIT_WORK(&pipe->recover_work, lima_sched_recover_work); - return drm_sched_init(&pipe->base, &lima_sched_ops, 1, + return drm_sched_init(&pipe->base, &lima_sched_ops, NULL, 1, lima_job_hang_limit, msecs_to_jiffies(timeout), NULL, NULL, name, pipe->ldev->dev); diff --git a/drivers/gpu/drm/msm/adreno/adreno_device.c b/drivers/gpu/drm/msm/adreno/adreno_device.c index c5c4c93b3689..f76ce11a5384 100644 --- a/drivers/gpu/drm/msm/adreno/adreno_device.c +++ b/drivers/gpu/drm/msm/adreno/adreno_device.c @@ -662,7 +662,8 @@ static void suspend_scheduler(struct msm_gpu *gpu) */ for (i = 0; i < gpu->nr_rings; i++) { struct drm_gpu_scheduler *sched = &gpu->rb[i]->sched; - kthread_park(sched->thread); + + drm_sched_run_wq_stop(sched); } } @@ -672,7 +673,8 @@ static void resume_scheduler(struct msm_gpu *gpu) for (i = 0; i < gpu->nr_rings; i++) { struct drm_gpu_scheduler *sched = &gpu->rb[i]->sched; - kthread_unpark(sched->thread); + + drm_sched_run_wq_start(sched); } } diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.c b/drivers/gpu/drm/msm/msm_ringbuffer.c index 57a8e9564540..5879fc262047 100644 --- a/drivers/gpu/drm/msm/msm_ringbuffer.c +++ b/drivers/gpu/drm/msm/msm_ringbuffer.c @@ -95,7 +95,7 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, /* currently managing hangcheck ourselves: */ sched_timeout = MAX_SCHEDULE_TIMEOUT; - ret = drm_sched_init(&ring->sched, &msm_sched_ops, + ret = drm_sched_init(&ring->sched, &msm_sched_ops, NULL, num_hw_submissions, 0, sched_timeout, NULL, NULL, to_msm_bo(ring->bo)->name, gpu->dev->dev); if (ret) { diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index dbc597ab46fb..f48b07056a16 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -815,7 +815,7 @@ int panfrost_job_init(struct panfrost_device *pfdev) js->queue[j].fence_context = dma_fence_context_alloc(1); ret = drm_sched_init(&js->queue[j].sched, - &panfrost_sched_ops, + &panfrost_sched_ops, NULL, nentries, 0, msecs_to_jiffies(JOB_TIMEOUT_MS), pfdev->reset.wq, diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index a18c8f5e8cc0..808008990721 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -44,7 +44,6 @@ * The jobs in a entity are always scheduled in the order that they were pushed. */ -#include #include #include #include @@ -252,6 +251,53 @@ drm_sched_rq_select_entity_fifo(struct drm_sched_rq *rq) return rb ? rb_entry(rb, struct drm_sched_entity, rb_tree_node) : NULL; } +/** + * drm_sched_run_wq_stop - stop scheduler run worker + * + * @sched: scheduler instance to stop run worker + */ +void drm_sched_run_wq_stop(struct drm_gpu_scheduler *sched) +{ + sched->pause_run_wq = true; + smp_wmb(); + + cancel_work_sync(&sched->work_run); +} +EXPORT_SYMBOL(drm_sched_run_wq_stop); + +/** + * drm_sched_run_wq_start - start scheduler run worker + * + * @sched: scheduler instance to start run worker + */ +void drm_sched_run_wq_start(struct drm_gpu_scheduler *sched) +{ + sched->pause_run_wq = false; + smp_wmb(); + + queue_work(sched->run_wq, &sched->work_run); +} +EXPORT_SYMBOL(drm_sched_run_wq_start); + +/** + * drm_sched_run_wq_queue - queue scheduler run worker + * + * @sched: scheduler instance to queue run worker + */ +static void drm_sched_run_wq_queue(struct drm_gpu_scheduler *sched) +{ + smp_rmb(); + + /* + * Try not to schedule work if pause_run_wq set but not the end of world + * if we do as either it will be cancelled by the above + * cancel_work_sync, or drm_sched_main turns into a NOP while + * pause_run_wq is set. + */ + if (!sched->pause_run_wq) + queue_work(sched->run_wq, &sched->work_run); +} + /** * drm_sched_job_done - complete a job * @s_job: pointer to the job which is done @@ -271,7 +317,7 @@ static void drm_sched_job_done(struct drm_sched_job *s_job) dma_fence_get(&s_fence->finished); drm_sched_fence_finished(s_fence); dma_fence_put(&s_fence->finished); - wake_up_interruptible(&sched->wake_up_worker); + drm_sched_run_wq_queue(sched); } /** @@ -434,7 +480,7 @@ void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad) { struct drm_sched_job *s_job, *tmp; - kthread_park(sched->thread); + drm_sched_run_wq_stop(sched); /* * Reinsert back the bad job here - now it's safe as @@ -547,7 +593,7 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery) spin_unlock(&sched->job_list_lock); } - kthread_unpark(sched->thread); + drm_sched_run_wq_start(sched); } EXPORT_SYMBOL(drm_sched_start); @@ -864,7 +910,7 @@ static bool drm_sched_ready(struct drm_gpu_scheduler *sched) void drm_sched_wakeup(struct drm_gpu_scheduler *sched) { if (drm_sched_ready(sched)) - wake_up_interruptible(&sched->wake_up_worker); + drm_sched_run_wq_queue(sched); } /** @@ -974,60 +1020,42 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list, } EXPORT_SYMBOL(drm_sched_pick_best); -/** - * drm_sched_blocked - check if the scheduler is blocked - * - * @sched: scheduler instance - * - * Returns true if blocked, otherwise false. - */ -static bool drm_sched_blocked(struct drm_gpu_scheduler *sched) -{ - if (kthread_should_park()) { - kthread_parkme(); - return true; - } - - return false; -} - /** * drm_sched_main - main scheduler thread * * @param: scheduler instance - * - * Returns 0. */ -static int drm_sched_main(void *param) +static void drm_sched_main(struct work_struct *w) { - struct drm_gpu_scheduler *sched = (struct drm_gpu_scheduler *)param; + struct drm_gpu_scheduler *sched = + container_of(w, struct drm_gpu_scheduler, work_run); int r; - sched_set_fifo_low(current); - - while (!kthread_should_stop()) { - struct drm_sched_entity *entity = NULL; + while (!READ_ONCE(sched->pause_run_wq)) { + struct drm_sched_entity *entity; struct drm_sched_fence *s_fence; struct drm_sched_job *sched_job; struct dma_fence *fence; - struct drm_sched_job *cleanup_job = NULL; + struct drm_sched_job *cleanup_job; - wait_event_interruptible(sched->wake_up_worker, - (cleanup_job = drm_sched_get_cleanup_job(sched)) || - (!drm_sched_blocked(sched) && - (entity = drm_sched_select_entity(sched))) || - kthread_should_stop()); + cleanup_job = drm_sched_get_cleanup_job(sched); + entity = drm_sched_select_entity(sched); if (cleanup_job) sched->ops->free_job(cleanup_job); - if (!entity) + if (!entity) { + if (!cleanup_job) + break; continue; + } sched_job = drm_sched_entity_pop_job(entity); if (!sched_job) { complete_all(&entity->entity_idle); + if (!cleanup_job) + break; continue; } @@ -1055,14 +1083,14 @@ static int drm_sched_main(void *param) r); } else { if (IS_ERR(fence)) - dma_fence_set_error(&s_fence->finished, PTR_ERR(fence)); + dma_fence_set_error(&s_fence->finished, + PTR_ERR(fence)); drm_sched_job_done(sched_job); } wake_up(&sched->job_scheduled); } - return 0; } /** @@ -1070,6 +1098,7 @@ static int drm_sched_main(void *param) * * @sched: scheduler instance * @ops: backend operations for this scheduler + * @run_wq: workqueue to use for run work. If NULL, the system_wq is used * @hw_submission: number of hw submissions that can be in flight * @hang_limit: number of times to allow a job to hang before dropping it * @timeout: timeout value in jiffies for the scheduler @@ -1083,14 +1112,16 @@ static int drm_sched_main(void *param) */ int drm_sched_init(struct drm_gpu_scheduler *sched, const struct drm_sched_backend_ops *ops, + struct workqueue_struct *run_wq, unsigned hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, atomic_t *score, const char *name, struct device *dev) { - int i, ret; + int i; sched->ops = ops; sched->hw_submission_limit = hw_submission; sched->name = name; + sched->run_wq = run_wq ? : system_wq; sched->timeout = timeout; sched->timeout_wq = timeout_wq ? : system_wq; sched->hang_limit = hang_limit; @@ -1099,23 +1130,15 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) drm_sched_rq_init(sched, &sched->sched_rq[i]); - init_waitqueue_head(&sched->wake_up_worker); init_waitqueue_head(&sched->job_scheduled); INIT_LIST_HEAD(&sched->pending_list); spin_lock_init(&sched->job_list_lock); atomic_set(&sched->hw_rq_count, 0); INIT_DELAYED_WORK(&sched->work_tdr, drm_sched_job_timedout); + INIT_WORK(&sched->work_run, drm_sched_main); atomic_set(&sched->_score, 0); atomic64_set(&sched->job_id_count, 0); - - /* Each scheduler will run on a seperate kernel thread */ - sched->thread = kthread_run(drm_sched_main, sched, sched->name); - if (IS_ERR(sched->thread)) { - ret = PTR_ERR(sched->thread); - sched->thread = NULL; - DRM_DEV_ERROR(sched->dev, "Failed to create scheduler for %s.\n", name); - return ret; - } + sched->pause_run_wq = false; sched->ready = true; return 0; @@ -1134,8 +1157,7 @@ void drm_sched_fini(struct drm_gpu_scheduler *sched) struct drm_sched_entity *s_entity; int i; - if (sched->thread) - kthread_stop(sched->thread); + drm_sched_run_wq_stop(sched); for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { struct drm_sched_rq *rq = &sched->sched_rq[i]; diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index 06238e6d7f5c..38e092ea41e6 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -388,7 +388,7 @@ v3d_sched_init(struct v3d_dev *v3d) int ret; ret = drm_sched_init(&v3d->queue[V3D_BIN].sched, - &v3d_bin_sched_ops, + &v3d_bin_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_bin", v3d->drm.dev); @@ -396,7 +396,7 @@ v3d_sched_init(struct v3d_dev *v3d) return ret; ret = drm_sched_init(&v3d->queue[V3D_RENDER].sched, - &v3d_render_sched_ops, + &v3d_render_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_render", v3d->drm.dev); @@ -404,7 +404,7 @@ v3d_sched_init(struct v3d_dev *v3d) goto fail; ret = drm_sched_init(&v3d->queue[V3D_TFU].sched, - &v3d_tfu_sched_ops, + &v3d_tfu_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_tfu", v3d->drm.dev); @@ -413,7 +413,7 @@ v3d_sched_init(struct v3d_dev *v3d) if (v3d_has_csd(v3d)) { ret = drm_sched_init(&v3d->queue[V3D_CSD].sched, - &v3d_csd_sched_ops, + &v3d_csd_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_csd", v3d->drm.dev); @@ -421,7 +421,7 @@ v3d_sched_init(struct v3d_dev *v3d) goto fail; ret = drm_sched_init(&v3d->queue[V3D_CACHE_CLEAN].sched, - &v3d_cache_clean_sched_ops, + &v3d_cache_clean_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, NULL, "v3d_cache_clean", v3d->drm.dev); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index c0586d832260..98fb5f85eba6 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -473,17 +473,16 @@ struct drm_sched_backend_ops { * @timeout: the time after which a job is removed from the scheduler. * @name: name of the ring for which this scheduler is being used. * @sched_rq: priority wise array of run queues. - * @wake_up_worker: the wait queue on which the scheduler sleeps until a job - * is ready to be scheduled. * @job_scheduled: once @drm_sched_entity_do_release is called the scheduler * waits on this wait queue until all the scheduled jobs are * finished. * @hw_rq_count: the number of jobs currently in the hardware queue. * @job_id_count: used to assign unique id to the each job. + * @run_wq: workqueue used to queue @work_run * @timeout_wq: workqueue used to queue @work_tdr + * @work_run: schedules jobs and cleans up entities * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the * timeout interval is over. - * @thread: the kthread on which the scheduler which run. * @pending_list: the list of jobs which are currently in the job queue. * @job_list_lock: lock to protect the pending_list. * @hang_limit: once the hangs by a job crosses this limit then it is marked @@ -492,6 +491,7 @@ struct drm_sched_backend_ops { * @_score: score used when the driver doesn't provide one * @ready: marks if the underlying HW is ready to work * @free_guilty: A hit to time out handler to free the guilty job. + * @pause_run_wq: pause queuing of @work_run on @run_wq * @dev: system &struct device * * One scheduler is implemented for each hardware ring. @@ -502,13 +502,13 @@ struct drm_gpu_scheduler { long timeout; const char *name; struct drm_sched_rq sched_rq[DRM_SCHED_PRIORITY_COUNT]; - wait_queue_head_t wake_up_worker; wait_queue_head_t job_scheduled; atomic_t hw_rq_count; atomic64_t job_id_count; + struct workqueue_struct *run_wq; struct workqueue_struct *timeout_wq; + struct work_struct work_run; struct delayed_work work_tdr; - struct task_struct *thread; struct list_head pending_list; spinlock_t job_list_lock; int hang_limit; @@ -516,11 +516,13 @@ struct drm_gpu_scheduler { atomic_t _score; bool ready; bool free_guilty; + bool pause_run_wq; struct device *dev; }; int drm_sched_init(struct drm_gpu_scheduler *sched, const struct drm_sched_backend_ops *ops, + struct workqueue_struct *run_wq, uint32_t hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, atomic_t *score, const char *name, struct device *dev); @@ -550,6 +552,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, void drm_sched_job_cleanup(struct drm_sched_job *job); void drm_sched_wakeup(struct drm_gpu_scheduler *sched); +void drm_sched_run_wq_stop(struct drm_gpu_scheduler *sched); +void drm_sched_run_wq_start(struct drm_gpu_scheduler *sched); void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad); void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery); void drm_sched_resubmit_jobs(struct drm_gpu_scheduler *sched); From patchwork Tue Apr 4 00:22:03 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13198945 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DD6A2C76188 for ; Tue, 4 Apr 2023 00:22:52 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 37DA210E1B8; Tue, 4 Apr 2023 00:22:29 +0000 (UTC) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id BB02E10E086; Tue, 4 Apr 2023 00:22:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680567743; x=1712103743; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=i//o76H+6IJY5z6VKRLFR7jKKkXZLITW7cnM9UaKkl8=; b=c9e6aNgBjNrft1OPiqIz5Vo0d3I8g7lZA58pNElB3leX2+WrUMpZAd1h 8QpVKs8fSqOoqSCKH03iOT7a43e/ylJLYk4wwf6OIgSh4ILjgrQqhbWpQ 8K3N5fI00q4DUTWknrRjkOHrNnCUAJu5igYLAEfZWkBes69s8bFQfBCaJ poZ6bKwUt1wUbVkSBQPLYGxOlPjYPt0XREFq1QODbYMRiWqVLzaat6ELO hp5sBZECuR0OkSkRMAZb1wqSPFBGViqUxjFv5DOHyXAu/Er4fi71/SN1g qRQqNHMQ4to82xrH7lDBqGiHAADbwh/Et5B/fVb0CQP59PDRk61RqakfD g==; X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="404810493" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="404810493" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="716460294" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="716460294" Received: from lstrano-desk.jf.intel.com ([10.24.89.184]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC PATCH 02/10] drm/sched: Move schedule policy to scheduler / entity Date: Mon, 3 Apr 2023 17:22:03 -0700 Message-Id: <20230404002211.3611376-3-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230404002211.3611376-1-matthew.brost@intel.com> References: <20230404002211.3611376-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, Matthew Brost , christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Rather than a global modparam for scheduling policy, move the scheduling policy to scheduler / entity so user can control each scheduler / entity policy. Signed-off-by: Matthew Brost --- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 1 + drivers/gpu/drm/etnaviv/etnaviv_sched.c | 3 ++- drivers/gpu/drm/lima/lima_sched.c | 3 ++- drivers/gpu/drm/msm/msm_ringbuffer.c | 3 ++- drivers/gpu/drm/panfrost/panfrost_job.c | 3 ++- drivers/gpu/drm/scheduler/sched_entity.c | 25 ++++++++++++++++++---- drivers/gpu/drm/scheduler/sched_main.c | 21 +++++++++++++----- drivers/gpu/drm/v3d/v3d_sched.c | 15 ++++++++----- include/drm/gpu_scheduler.h | 23 ++++++++++++++------ 9 files changed, 73 insertions(+), 24 deletions(-) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 00c9c03c8f94..4df0fca5a74c 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -2368,6 +2368,7 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev) ring->num_hw_submission, amdgpu_job_hang_limit, timeout, adev->reset_domain->wq, ring->sched_score, ring->name, + DRM_SCHED_POLICY_DEFAULT, adev->dev); if (r) { DRM_ERROR("Failed to create scheduler on ring %s.\n", diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c index 8486a2923f1b..61204a3f8b0b 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c @@ -136,7 +136,8 @@ int etnaviv_sched_init(struct etnaviv_gpu *gpu) ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, NULL, etnaviv_hw_jobs_limit, etnaviv_job_hang_limit, msecs_to_jiffies(500), NULL, NULL, - dev_name(gpu->dev), gpu->dev); + dev_name(gpu->dev), DRM_SCHED_POLICY_DEFAULT, + gpu->dev); if (ret) return ret; diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c index 54f53bece27c..33042ba6ae93 100644 --- a/drivers/gpu/drm/lima/lima_sched.c +++ b/drivers/gpu/drm/lima/lima_sched.c @@ -491,7 +491,8 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name) return drm_sched_init(&pipe->base, &lima_sched_ops, NULL, 1, lima_job_hang_limit, msecs_to_jiffies(timeout), NULL, - NULL, name, pipe->ldev->dev); + NULL, name, DRM_SCHED_POLICY_DEFAULT, + pipe->ldev->dev); } void lima_sched_pipe_fini(struct lima_sched_pipe *pipe) diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.c b/drivers/gpu/drm/msm/msm_ringbuffer.c index 5879fc262047..f408a9097315 100644 --- a/drivers/gpu/drm/msm/msm_ringbuffer.c +++ b/drivers/gpu/drm/msm/msm_ringbuffer.c @@ -97,7 +97,8 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, ret = drm_sched_init(&ring->sched, &msm_sched_ops, NULL, num_hw_submissions, 0, sched_timeout, - NULL, NULL, to_msm_bo(ring->bo)->name, gpu->dev->dev); + NULL, NULL, to_msm_bo(ring->bo)->name, + DRM_SCHED_POLICY_DEFAULT, gpu->dev->dev); if (ret) { goto fail; } diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index f48b07056a16..effa48b33dce 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -819,7 +819,8 @@ int panfrost_job_init(struct panfrost_device *pfdev) nentries, 0, msecs_to_jiffies(JOB_TIMEOUT_MS), pfdev->reset.wq, - NULL, "pan_js", pfdev->dev); + NULL, "pan_js", DRM_SCHED_POLICY_DEFAULT, + pfdev->dev); if (ret) { dev_err(pfdev->dev, "Failed to create scheduler: %d.", ret); goto err_sched; diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index 15d04a0ec623..f1299e51860b 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -33,6 +33,20 @@ #define to_drm_sched_job(sched_job) \ container_of((sched_job), struct drm_sched_job, queue_node) +static bool bad_policies(struct drm_gpu_scheduler **sched_list, + unsigned int num_sched_list) +{ + enum drm_sched_policy sched_policy = sched_list[0]->sched_policy; + unsigned int i; + + /* All scdedule policies must match */ + for (i = 1; i < num_sched_list; ++i) + if (sched_policy != sched_list[i]->sched_policy) + return true; + + return false; +} + /** * drm_sched_entity_init - Init a context entity used by scheduler when * submit to HW ring. @@ -62,7 +76,8 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, unsigned int num_sched_list, atomic_t *guilty) { - if (!(entity && sched_list && (num_sched_list == 0 || sched_list[0]))) + if (!(entity && sched_list && (num_sched_list == 0 || sched_list[0])) || + bad_policies(sched_list, num_sched_list)) return -EINVAL; memset(entity, 0, sizeof(struct drm_sched_entity)); @@ -75,8 +90,10 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, entity->last_scheduled = NULL; RB_CLEAR_NODE(&entity->rb_tree_node); - if(num_sched_list) + if(num_sched_list) { entity->rq = &sched_list[0]->sched_rq[entity->priority]; + entity->sched_policy = sched_list[0]->sched_policy; + } init_completion(&entity->entity_idle); @@ -440,7 +457,7 @@ struct drm_sched_job *drm_sched_entity_pop_job(struct drm_sched_entity *entity) * Update the entity's location in the min heap according to * the timestamp of the next job, if any. */ - if (drm_sched_policy == DRM_SCHED_POLICY_FIFO) { + if (entity->sched_policy == DRM_SCHED_POLICY_FIFO) { struct drm_sched_job *next; next = to_drm_sched_job(spsc_queue_peek(&entity->job_queue)); @@ -528,7 +545,7 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) drm_sched_rq_add_entity(entity->rq, entity); spin_unlock(&entity->rq_lock); - if (drm_sched_policy == DRM_SCHED_POLICY_FIFO) + if (entity->sched_policy == DRM_SCHED_POLICY_FIFO) drm_sched_rq_update_fifo(entity, sched_job->submit_ts); drm_sched_wakeup(entity->rq->sched); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 808008990721..77894976fa55 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -62,14 +62,14 @@ #define to_drm_sched_job(sched_job) \ container_of((sched_job), struct drm_sched_job, queue_node) -int drm_sched_policy = DRM_SCHED_POLICY_FIFO; +int default_drm_sched_policy = DRM_SCHED_POLICY_FIFO; /** * DOC: sched_policy (int) * Used to override default entities scheduling policy in a run queue. */ MODULE_PARM_DESC(sched_policy, "Specify the scheduling policy for entities on a run-queue, " __stringify(DRM_SCHED_POLICY_RR) " = Round Robin, " __stringify(DRM_SCHED_POLICY_FIFO) " = FIFO (default)."); -module_param_named(sched_policy, drm_sched_policy, int, 0444); +module_param_named(sched_policy, default_drm_sched_policy, int, 0444); static __always_inline bool drm_sched_entity_compare_before(struct rb_node *a, const struct rb_node *b) @@ -173,7 +173,7 @@ void drm_sched_rq_remove_entity(struct drm_sched_rq *rq, if (rq->current_entity == entity) rq->current_entity = NULL; - if (drm_sched_policy == DRM_SCHED_POLICY_FIFO) + if (entity->sched_policy == DRM_SCHED_POLICY_FIFO) drm_sched_rq_remove_fifo_locked(entity); spin_unlock(&rq->lock); @@ -931,7 +931,7 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched) /* Kernel run queue has higher priority than normal run queue*/ for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { - entity = drm_sched_policy == DRM_SCHED_POLICY_FIFO ? + entity = sched->sched_policy == DRM_SCHED_POLICY_FIFO ? drm_sched_rq_select_entity_fifo(&sched->sched_rq[i]) : drm_sched_rq_select_entity_rr(&sched->sched_rq[i]); if (entity) @@ -1106,6 +1106,7 @@ static void drm_sched_main(struct work_struct *w) * used * @score: optional score atomic shared with other schedulers * @name: name used for debugging + * @sched_policy: schedule policy * @dev: target &struct device * * Return 0 on success, otherwise error code. @@ -1115,9 +1116,15 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, struct workqueue_struct *run_wq, unsigned hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, - atomic_t *score, const char *name, struct device *dev) + atomic_t *score, const char *name, + enum drm_sched_policy sched_policy, + struct device *dev) { int i; + + if (sched_policy >= DRM_SCHED_POLICY_MAX) + return -EINVAL; + sched->ops = ops; sched->hw_submission_limit = hw_submission; sched->name = name; @@ -1127,6 +1134,10 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, sched->hang_limit = hang_limit; sched->score = score ? score : &sched->_score; sched->dev = dev; + if (sched_policy == DRM_SCHED_POLICY_DEFAULT) + sched->sched_policy = default_drm_sched_policy; + else + sched->sched_policy = sched_policy; for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) drm_sched_rq_init(sched, &sched->sched_rq[i]); diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index 38e092ea41e6..5e3fe77fa991 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -391,7 +391,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_bin_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_bin", v3d->drm.dev); + NULL, "v3d_bin", DRM_SCHED_POLICY_DEFAULT, + v3d->drm.dev); if (ret) return ret; @@ -399,7 +400,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_render_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_render", v3d->drm.dev); + ULL, "v3d_render", DRM_SCHED_POLICY_DEFAULT, + v3d->drm.dev); if (ret) goto fail; @@ -407,7 +409,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_tfu_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_tfu", v3d->drm.dev); + NULL, "v3d_tfu", DRM_SCHED_POLICY_DEFAULT, + v3d->drm.dev); if (ret) goto fail; @@ -416,7 +419,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_csd_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_csd", v3d->drm.dev); + NULL, "v3d_csd", DRM_SCHED_POLICY_DEFAULT, + v3d->drm.dev); if (ret) goto fail; @@ -424,7 +428,8 @@ v3d_sched_init(struct v3d_dev *v3d) &v3d_cache_clean_sched_ops, NULL, hw_jobs_limit, job_hang_limit, msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_cache_clean", v3d->drm.dev); + NULL, "v3d_cache_clean", + DRM_SCHED_POLICY_DEFAULT, v3d->drm.dev); if (ret) goto fail; } diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 98fb5f85eba6..39cb72b7fe5d 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -72,11 +72,15 @@ enum drm_sched_priority { DRM_SCHED_PRIORITY_UNSET = -2 }; -/* Used to chose between FIFO and RR jobs scheduling */ -extern int drm_sched_policy; - -#define DRM_SCHED_POLICY_RR 0 -#define DRM_SCHED_POLICY_FIFO 1 +/* Used to chose default scheduling policy*/ +extern int default_drm_sched_policy; + +enum drm_sched_policy { + DRM_SCHED_POLICY_DEFAULT, + DRM_SCHED_POLICY_RR, + DRM_SCHED_POLICY_FIFO, + DRM_SCHED_POLICY_MAX, +}; /** * struct drm_sched_entity - A wrapper around a job queue (typically @@ -217,6 +221,9 @@ struct drm_sched_entity { */ bool stopped; + /** @sched_policy: Schedule policy for entity */ + enum drm_sched_policy sched_policy; + /** * @entity_idle: * @@ -489,6 +496,7 @@ struct drm_sched_backend_ops { * guilty and it will no longer be considered for scheduling. * @score: score to help loadbalancer pick a idle sched * @_score: score used when the driver doesn't provide one + * @sched_policy: Schedule policy for scheduler * @ready: marks if the underlying HW is ready to work * @free_guilty: A hit to time out handler to free the guilty job. * @pause_run_wq: pause queuing of @work_run on @run_wq @@ -514,6 +522,7 @@ struct drm_gpu_scheduler { int hang_limit; atomic_t *score; atomic_t _score; + enum drm_sched_policy sched_policy; bool ready; bool free_guilty; bool pause_run_wq; @@ -525,7 +534,9 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, struct workqueue_struct *run_wq, uint32_t hw_submission, unsigned hang_limit, long timeout, struct workqueue_struct *timeout_wq, - atomic_t *score, const char *name, struct device *dev); + atomic_t *score, const char *name, + enum drm_sched_policy sched_policy, + struct device *dev); void drm_sched_fini(struct drm_gpu_scheduler *sched); int drm_sched_job_init(struct drm_sched_job *job, From patchwork Tue Apr 4 00:22:04 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13198941 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63540C76188 for ; Tue, 4 Apr 2023 00:22:44 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 47BD210E1A4; Tue, 4 Apr 2023 00:22:28 +0000 (UTC) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id B425810E048; Tue, 4 Apr 2023 00:22:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680567743; x=1712103743; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=qyWrzvF3FMBXCP0ybyOQKWSpVZz3YfV8OGfDUOvJNuA=; b=Zui7y/Pfo90bXsLswLwxZJOq6fNkY9+TJBC/MrIeTHpbqMDiOQPLbUx0 5fWWiBOZOXsPC/9erg1R1CRxLkEHnVs5FC2Gu6rokY1B5uqMhSTT0z3kk aKI+kgLM9RN4can0LPfuMoi3cQ0cuWRDxg0M/YPmM5em37qqzvOWTONUL 5BavJEn0zIBIHtvE4UrPwICi8ZYvBQh3yd7SwMI02IF9YNi0ttbRtaV0Z d1uZTSVaFvw5McGxyxhvgmL4pHmMBQTEq2nEORT7cAvPSCA/p8cBRBQFr fInt81SmR+SAqcPhVHCpxgSyUhSRrOg/Q/16lxJrD53kDYk4J8z+ukqXS A==; X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="404810501" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="404810501" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="716460297" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="716460297" Received: from lstrano-desk.jf.intel.com ([10.24.89.184]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC PATCH 03/10] drm/sched: Add DRM_SCHED_POLICY_SINGLE_ENTITY scheduling policy Date: Mon, 3 Apr 2023 17:22:04 -0700 Message-Id: <20230404002211.3611376-4-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230404002211.3611376-1-matthew.brost@intel.com> References: <20230404002211.3611376-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, Matthew Brost , christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" DRM_SCHED_POLICY_SINGLE_ENTITY creates a 1 to 1 relationship between scheduler and entity. No priorities or run queue used in this mode. Intended for devices with firmware schedulers. Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_entity.c | 58 +++++++++++++++++---- drivers/gpu/drm/scheduler/sched_fence.c | 2 +- drivers/gpu/drm/scheduler/sched_main.c | 64 +++++++++++++++++++++--- include/drm/gpu_scheduler.h | 29 +++++++---- 4 files changed, 123 insertions(+), 30 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index f1299e51860b..ccea4d079d0f 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -91,8 +91,15 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, RB_CLEAR_NODE(&entity->rb_tree_node); if(num_sched_list) { - entity->rq = &sched_list[0]->sched_rq[entity->priority]; entity->sched_policy = sched_list[0]->sched_policy; + if (entity->sched_policy != DRM_SCHED_POLICY_SINGLE_ENTITY) { + entity->rq = &sched_list[0]->sched_rq[entity->priority]; + } else { + if (num_sched_list != 1 || sched_list[0]->single_entity) + return -EINVAL; + sched_list[0]->single_entity = entity; + entity->single_sched = sched_list[0]; + } } init_completion(&entity->entity_idle); @@ -126,7 +133,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list) { - WARN_ON(!num_sched_list || !sched_list); + WARN_ON(!num_sched_list || !sched_list || + entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); entity->sched_list = sched_list; entity->num_sched_list = num_sched_list; @@ -196,13 +204,16 @@ static void drm_sched_entity_kill(struct drm_sched_entity *entity) { struct drm_sched_job *job; struct dma_fence *prev; + bool single_entity = + entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY; - if (!entity->rq) + if (!entity->rq && !single_entity) return; spin_lock(&entity->rq_lock); entity->stopped = true; - drm_sched_rq_remove_entity(entity->rq, entity); + if (!single_entity) + drm_sched_rq_remove_entity(entity->rq, entity); spin_unlock(&entity->rq_lock); /* Make sure this entity is not used by the scheduler at the moment */ @@ -224,6 +235,21 @@ static void drm_sched_entity_kill(struct drm_sched_entity *entity) dma_fence_put(prev); } +/** + * drm_sched_entity_to_scheduler - Schedule entity to GPU scheduler + * @entity: scheduler entity + * + * Returns GPU scheduler for the entity + */ +struct drm_gpu_scheduler * +drm_sched_entity_to_scheduler(struct drm_sched_entity *entity) +{ + bool single_entity = + entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY; + + return single_entity ? entity->single_sched : entity->rq->sched; +} + /** * drm_sched_entity_flush - Flush a context entity * @@ -241,11 +267,13 @@ long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout) struct drm_gpu_scheduler *sched; struct task_struct *last_user; long ret = timeout; + bool single_entity = + entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY; - if (!entity->rq) + if (!entity->rq && !single_entity) return 0; - sched = entity->rq->sched; + sched = drm_sched_entity_to_scheduler(entity); /** * The client will not queue more IBs during this fini, consume existing * queued IBs or discard them on SIGKILL @@ -338,7 +366,7 @@ static void drm_sched_entity_wakeup(struct dma_fence *f, container_of(cb, struct drm_sched_entity, cb); drm_sched_entity_clear_dep(f, cb); - drm_sched_wakeup(entity->rq->sched); + drm_sched_wakeup(drm_sched_entity_to_scheduler(entity)); } /** @@ -352,6 +380,8 @@ static void drm_sched_entity_wakeup(struct dma_fence *f, void drm_sched_entity_set_priority(struct drm_sched_entity *entity, enum drm_sched_priority priority) { + WARN_ON(entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + spin_lock(&entity->rq_lock); entity->priority = priority; spin_unlock(&entity->rq_lock); @@ -364,7 +394,7 @@ EXPORT_SYMBOL(drm_sched_entity_set_priority); */ static bool drm_sched_entity_add_dependency_cb(struct drm_sched_entity *entity) { - struct drm_gpu_scheduler *sched = entity->rq->sched; + struct drm_gpu_scheduler *sched = drm_sched_entity_to_scheduler(entity); struct dma_fence *fence = entity->dependency; struct drm_sched_fence *s_fence; @@ -474,6 +504,8 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity) struct drm_gpu_scheduler *sched; struct drm_sched_rq *rq; + WARN_ON(entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + /* single possible engine and already selected */ if (!entity->sched_list) return; @@ -523,10 +555,13 @@ void drm_sched_entity_select_rq(struct drm_sched_entity *entity) void drm_sched_entity_push_job(struct drm_sched_job *sched_job) { struct drm_sched_entity *entity = sched_job->entity; + bool single_entity = + entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY; bool first; trace_drm_sched_job(sched_job, entity); - atomic_inc(entity->rq->sched->score); + if (!single_entity) + atomic_inc(entity->rq->sched->score); WRITE_ONCE(entity->last_user, current->group_leader); first = spsc_queue_push(&entity->job_queue, &sched_job->queue_node); sched_job->submit_ts = ktime_get(); @@ -542,13 +577,14 @@ void drm_sched_entity_push_job(struct drm_sched_job *sched_job) return; } - drm_sched_rq_add_entity(entity->rq, entity); + if (!single_entity) + drm_sched_rq_add_entity(entity->rq, entity); spin_unlock(&entity->rq_lock); if (entity->sched_policy == DRM_SCHED_POLICY_FIFO) drm_sched_rq_update_fifo(entity, sched_job->submit_ts); - drm_sched_wakeup(entity->rq->sched); + drm_sched_wakeup(drm_sched_entity_to_scheduler(entity)); } } EXPORT_SYMBOL(drm_sched_entity_push_job); diff --git a/drivers/gpu/drm/scheduler/sched_fence.c b/drivers/gpu/drm/scheduler/sched_fence.c index fe9c6468e440..d7cfc0441885 100644 --- a/drivers/gpu/drm/scheduler/sched_fence.c +++ b/drivers/gpu/drm/scheduler/sched_fence.c @@ -213,7 +213,7 @@ void drm_sched_fence_init(struct drm_sched_fence *fence, { unsigned seq; - fence->sched = entity->rq->sched; + fence->sched = drm_sched_entity_to_scheduler(entity); seq = atomic_inc_return(&entity->fence_seq); dma_fence_init(&fence->scheduled, &drm_sched_fence_ops_scheduled, &fence->lock, entity->fence_context, seq); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 77894976fa55..2795021efe7b 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -32,7 +32,8 @@ * backend operations to the scheduler like submitting a job to hardware run queue, * returning the dependencies of a job etc. * - * The organisation of the scheduler is the following: + * The organisation of the scheduler is the following for scheduling policies + * DRM_SCHED_POLICY_RR and DRM_SCHED_POLICY_FIFO: * * 1. Each hw run queue has one scheduler * 2. Each scheduler has multiple run queues with different priorities @@ -41,7 +42,22 @@ * 4. Entities themselves maintain a queue of jobs that will be scheduled on * the hardware. * - * The jobs in a entity are always scheduled in the order that they were pushed. + * The organisation of the scheduler is the following for scheduling policy + * DRM_SCHED_POLICY_SINGLE_ENTITY: + * + * 1. One to one relationship between scheduler and entity + * 2. No priorities implemented per scheduler (single job queue) + * 3. No run queues in scheduler rather jobs are directly dequeued from entity + * 4. The entity maintains a queue of jobs that will be scheduled on the + * hardware + * + * The jobs in a entity are always scheduled in the order that they were pushed + * regardless of scheduling policy. + * + * A policy of DRM_SCHED_POLICY_RR or DRM_SCHED_POLICY_FIFO is expected to used + * when the KMD is scheduling directly on the hardware while a scheduling policy + * of DRM_SCHED_POLICY_SINGLE_ENTITY is expected to be used when there is a + * firmare scheduler. */ #include @@ -92,6 +108,8 @@ static inline void drm_sched_rq_remove_fifo_locked(struct drm_sched_entity *enti void drm_sched_rq_update_fifo(struct drm_sched_entity *entity, ktime_t ts) { + WARN_ON(entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + /* * Both locks need to be grabbed, one to protect from entity->rq change * for entity from within concurrent drm_sched_entity_select_rq and the @@ -122,6 +140,8 @@ void drm_sched_rq_update_fifo(struct drm_sched_entity *entity, ktime_t ts) static void drm_sched_rq_init(struct drm_gpu_scheduler *sched, struct drm_sched_rq *rq) { + WARN_ON(sched->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + spin_lock_init(&rq->lock); INIT_LIST_HEAD(&rq->entities); rq->rb_tree_root = RB_ROOT_CACHED; @@ -140,6 +160,8 @@ static void drm_sched_rq_init(struct drm_gpu_scheduler *sched, void drm_sched_rq_add_entity(struct drm_sched_rq *rq, struct drm_sched_entity *entity) { + WARN_ON(entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + if (!list_empty(&entity->list)) return; @@ -162,6 +184,8 @@ void drm_sched_rq_add_entity(struct drm_sched_rq *rq, void drm_sched_rq_remove_entity(struct drm_sched_rq *rq, struct drm_sched_entity *entity) { + WARN_ON(entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + if (list_empty(&entity->list)) return; @@ -673,7 +697,7 @@ int drm_sched_job_init(struct drm_sched_job *job, struct drm_sched_entity *entity, void *owner) { - if (!entity->rq) + if (!entity->rq && !entity->single_sched) return -ENOENT; job->entity = entity; @@ -706,13 +730,17 @@ void drm_sched_job_arm(struct drm_sched_job *job) { struct drm_gpu_scheduler *sched; struct drm_sched_entity *entity = job->entity; + bool single_entity = + entity->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY; BUG_ON(!entity); - drm_sched_entity_select_rq(entity); - sched = entity->rq->sched; + if (!single_entity) + drm_sched_entity_select_rq(entity); + sched = drm_sched_entity_to_scheduler(entity); job->sched = sched; - job->s_priority = entity->rq - sched->sched_rq; + if (!single_entity) + job->s_priority = entity->rq - sched->sched_rq; job->id = atomic64_inc_return(&sched->job_id_count); drm_sched_fence_init(job->s_fence, job->entity); @@ -929,6 +957,13 @@ drm_sched_select_entity(struct drm_gpu_scheduler *sched) if (!drm_sched_ready(sched)) return NULL; + if (sched->single_entity) { + if (drm_sched_entity_is_ready(sched->single_entity)) + return sched->single_entity; + + return NULL; + } + /* Kernel run queue has higher priority than normal run queue*/ for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { entity = sched->sched_policy == DRM_SCHED_POLICY_FIFO ? @@ -1126,6 +1161,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, return -EINVAL; sched->ops = ops; + sched->single_entity = NULL; sched->hw_submission_limit = hw_submission; sched->name = name; sched->run_wq = run_wq ? : system_wq; @@ -1138,7 +1174,9 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, sched->sched_policy = default_drm_sched_policy; else sched->sched_policy = sched_policy; - for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) + for (i = DRM_SCHED_PRIORITY_MIN; sched_policy != + DRM_SCHED_POLICY_SINGLE_ENTITY && i < DRM_SCHED_PRIORITY_COUNT; + i++) drm_sched_rq_init(sched, &sched->sched_rq[i]); init_waitqueue_head(&sched->job_scheduled); @@ -1170,7 +1208,15 @@ void drm_sched_fini(struct drm_gpu_scheduler *sched) drm_sched_run_wq_stop(sched); - for (i = DRM_SCHED_PRIORITY_COUNT - 1; i >= DRM_SCHED_PRIORITY_MIN; i--) { + if (sched->single_entity) { + spin_lock(&sched->single_entity->rq_lock); + sched->single_entity->stopped = true; + spin_unlock(&sched->single_entity->rq_lock); + } + + for (i = DRM_SCHED_PRIORITY_COUNT - 1; sched->sched_policy != + DRM_SCHED_POLICY_SINGLE_ENTITY && i >= DRM_SCHED_PRIORITY_MIN; + i--) { struct drm_sched_rq *rq = &sched->sched_rq[i]; if (!rq) @@ -1214,6 +1260,8 @@ void drm_sched_increase_karma(struct drm_sched_job *bad) struct drm_sched_entity *entity; struct drm_gpu_scheduler *sched = bad->sched; + WARN_ON(sched->sched_policy == DRM_SCHED_POLICY_SINGLE_ENTITY); + /* don't change @bad's karma if it's from KERNEL RQ, * because sometimes GPU hang would cause kernel jobs (like VM updating jobs) * corrupt but keep in mind that kernel jobs always considered good. diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 39cb72b7fe5d..3e421f5a710c 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -79,6 +79,7 @@ enum drm_sched_policy { DRM_SCHED_POLICY_DEFAULT, DRM_SCHED_POLICY_RR, DRM_SCHED_POLICY_FIFO, + DRM_SCHED_POLICY_SINGLE_ENTITY, DRM_SCHED_POLICY_MAX, }; @@ -101,16 +102,20 @@ struct drm_sched_entity { */ struct list_head list; - /** - * @rq: - * - * Runqueue on which this entity is currently scheduled. - * - * FIXME: Locking is very unclear for this. Writers are protected by - * @rq_lock, but readers are generally lockless and seem to just race - * with not even a READ_ONCE. - */ - struct drm_sched_rq *rq; + union { + /** + * @rq: + * + * Runqueue on which this entity is currently scheduled. + * + * FIXME: Locking is very unclear for this. Writers are + * protected by @rq_lock, but readers are generally lockless and + * seem to just race with not even a READ_ONCE. + */ + struct drm_sched_rq *rq; + /** @single_sched: Single scheduler */ + struct drm_gpu_scheduler *single_sched; + }; /** * @sched_list: @@ -476,6 +481,7 @@ struct drm_sched_backend_ops { * struct drm_gpu_scheduler - scheduler instance-specific data * * @ops: backend operations provided by the driver. + * @single_entity: Single entity for the scheduler * @hw_submission_limit: the max size of the hardware queue. * @timeout: the time after which a job is removed from the scheduler. * @name: name of the ring for which this scheduler is being used. @@ -506,6 +512,7 @@ struct drm_sched_backend_ops { */ struct drm_gpu_scheduler { const struct drm_sched_backend_ops *ops; + struct drm_sched_entity *single_entity; uint32_t hw_submission_limit; long timeout; const char *name; @@ -587,6 +594,8 @@ int drm_sched_entity_init(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list, atomic_t *guilty); +struct drm_gpu_scheduler * +drm_sched_entity_to_scheduler(struct drm_sched_entity *entity); long drm_sched_entity_flush(struct drm_sched_entity *entity, long timeout); void drm_sched_entity_fini(struct drm_sched_entity *entity); void drm_sched_entity_destroy(struct drm_sched_entity *entity); From patchwork Tue Apr 4 00:22:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13198944 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 24617C761AF for ; Tue, 4 Apr 2023 00:22:51 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DC89510E1A7; Tue, 4 Apr 2023 00:22:28 +0000 (UTC) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0647610E08A; Tue, 4 Apr 2023 00:22:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680567744; x=1712103744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=zd+u+hxhIrladvT+HfImaD13Rb9MNMxFiIuWwPxgDlw=; b=Vh7Q8lUo6kxzNqu/yzlbKNVaQ6+bcx7MZZnDNd5Eghuj16y+LYsAQcqk yDy8rxwrQwjTsRkHd1jF86EGjQDavCAuo4ZgpL3brFE6CnZXvzKpn5RZe GoQ2WFjMgBtGz3GSXHCgjq/FDKRNbFJv34RHF5Yl6fFf5oHOzBUjybgN6 vLr2tkuSgWUjBiuEzsftudT7S5APclJIKz0MZvNAKwJ1MvtKqB0/5IUn5 T/bMLbVNnSz6RNwDkSgPmvUnU3mMe7caXNk1eEuRRw4Jctp3wHMbfdWWb qr9VUrV9DYp3YqE3fXHGqXx2A/aWHcFUSgdx/HK4Y6QlExiRpSzjGzVuf w==; X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="404810510" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="404810510" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="716460300" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="716460300" Received: from lstrano-desk.jf.intel.com ([10.24.89.184]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC PATCH 04/10] drm/sched: Add generic scheduler message interface Date: Mon, 3 Apr 2023 17:22:05 -0700 Message-Id: <20230404002211.3611376-5-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230404002211.3611376-1-matthew.brost@intel.com> References: <20230404002211.3611376-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, Matthew Brost , christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add generic schedule message interface which sends messages to backend from the drm_gpu_scheduler main submission thread. The idea is some of these messages modify some state in drm_sched_entity which is also modified during submission. By scheduling these messages and submission in the same thread their is not race changing states in drm_sched_entity. This interface will be used in XE, new Intel GPU driver, to cleanup, suspend, resume, and change scheduling properties of a drm_sched_entity. The interface is designed to be generic and extendable with only the backend understanding the messages. Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 58 +++++++++++++++++++++++++- include/drm/gpu_scheduler.h | 29 ++++++++++++- 2 files changed, 84 insertions(+), 3 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 2795021efe7b..9dc3378e9c5e 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -1055,6 +1055,54 @@ drm_sched_pick_best(struct drm_gpu_scheduler **sched_list, } EXPORT_SYMBOL(drm_sched_pick_best); +/** + * drm_sched_add_msg - add scheduler message + * + * @sched: scheduler instance + * @msg: message to be added + * + * Can and will pass an jobs waiting on dependencies or in a runnable queue. + * Messages processing will stop if schedule run wq is stopped and resume when + * run wq is started. + */ +void drm_sched_add_msg(struct drm_gpu_scheduler *sched, + struct drm_sched_msg *msg) +{ + spin_lock(&sched->job_list_lock); + list_add_tail(&msg->link, &sched->msgs); + spin_unlock(&sched->job_list_lock); + + /* + * Same as above in drm_sched_run_wq_queue, try to kick worker if + * paused, harmless if this races + */ + if (!sched->pause_run_wq) + queue_work(sched->run_wq, &sched->work_run); +} +EXPORT_SYMBOL(drm_sched_add_msg); + +/** + * drm_sched_get_msg - get scheduler message + * + * @sched: scheduler instance + * + * Returns NULL or message + */ +static struct drm_sched_msg * +drm_sched_get_msg(struct drm_gpu_scheduler *sched) +{ + struct drm_sched_msg *msg; + + spin_lock(&sched->job_list_lock); + msg = list_first_entry_or_null(&sched->msgs, + struct drm_sched_msg, link); + if (msg) + list_del(&msg->link); + spin_unlock(&sched->job_list_lock); + + return msg; +} + /** * drm_sched_main - main scheduler thread * @@ -1068,6 +1116,7 @@ static void drm_sched_main(struct work_struct *w) while (!READ_ONCE(sched->pause_run_wq)) { struct drm_sched_entity *entity; + struct drm_sched_msg *msg; struct drm_sched_fence *s_fence; struct drm_sched_job *sched_job; struct dma_fence *fence; @@ -1075,12 +1124,16 @@ static void drm_sched_main(struct work_struct *w) cleanup_job = drm_sched_get_cleanup_job(sched); entity = drm_sched_select_entity(sched); + msg = drm_sched_get_msg(sched); if (cleanup_job) sched->ops->free_job(cleanup_job); + if (msg) + sched->ops->process_msg(msg); + if (!entity) { - if (!cleanup_job) + if (!cleanup_job && !msg) break; continue; } @@ -1089,7 +1142,7 @@ static void drm_sched_main(struct work_struct *w) if (!sched_job) { complete_all(&entity->entity_idle); - if (!cleanup_job) + if (!cleanup_job && !msg) break; continue; } @@ -1181,6 +1234,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, init_waitqueue_head(&sched->job_scheduled); INIT_LIST_HEAD(&sched->pending_list); + INIT_LIST_HEAD(&sched->msgs); spin_lock_init(&sched->job_list_lock); atomic_set(&sched->hw_rq_count, 0); INIT_DELAYED_WORK(&sched->work_tdr, drm_sched_job_timedout); diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 3e421f5a710c..18172ae63ab7 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -398,6 +398,23 @@ enum drm_gpu_sched_stat { DRM_GPU_SCHED_STAT_ENODEV, }; +/** + * struct drm_sched_msg - an in-band (relative to GPU scheduler run queue) + * message + * + * Generic enough for backend defined messages, backend can expand if needed. + */ +struct drm_sched_msg { + /** @link: list link into the gpu scheduler list of messages */ + struct list_head link; + /** + * @private_data: opaque pointer to message private data (backend defined) + */ + void *private_data; + /** @opcode: opcode of message (backend defined) */ + unsigned int opcode; +}; + /** * struct drm_sched_backend_ops - Define the backend operations * called by the scheduler @@ -475,6 +492,12 @@ struct drm_sched_backend_ops { * and it's time to clean it up. */ void (*free_job)(struct drm_sched_job *sched_job); + + /** + * @process_msg: Process a message. Allowed to block, it is this + * function's responsibility to free message if dynamically allocated. + */ + void (*process_msg)(struct drm_sched_msg *msg); }; /** @@ -486,6 +509,7 @@ struct drm_sched_backend_ops { * @timeout: the time after which a job is removed from the scheduler. * @name: name of the ring for which this scheduler is being used. * @sched_rq: priority wise array of run queues. + * @msgs: list of messages to be processed in @work_run * @job_scheduled: once @drm_sched_entity_do_release is called the scheduler * waits on this wait queue until all the scheduled jobs are * finished. @@ -493,7 +517,7 @@ struct drm_sched_backend_ops { * @job_id_count: used to assign unique id to the each job. * @run_wq: workqueue used to queue @work_run * @timeout_wq: workqueue used to queue @work_tdr - * @work_run: schedules jobs and cleans up entities + * @work_run: schedules jobs, cleans up jobs, and processes messages * @work_tdr: schedules a delayed call to @drm_sched_job_timedout after the * timeout interval is over. * @pending_list: the list of jobs which are currently in the job queue. @@ -517,6 +541,7 @@ struct drm_gpu_scheduler { long timeout; const char *name; struct drm_sched_rq sched_rq[DRM_SCHED_PRIORITY_COUNT]; + struct list_head msgs; wait_queue_head_t job_scheduled; atomic_t hw_rq_count; atomic64_t job_id_count; @@ -570,6 +595,8 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, void drm_sched_job_cleanup(struct drm_sched_job *job); void drm_sched_wakeup(struct drm_gpu_scheduler *sched); +void drm_sched_add_msg(struct drm_gpu_scheduler *sched, + struct drm_sched_msg *msg); void drm_sched_run_wq_stop(struct drm_gpu_scheduler *sched); void drm_sched_run_wq_start(struct drm_gpu_scheduler *sched); void drm_sched_stop(struct drm_gpu_scheduler *sched, struct drm_sched_job *bad); From patchwork Tue Apr 4 00:22:06 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13198947 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4B2F8C76188 for ; Tue, 4 Apr 2023 00:22:56 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id A242910E1CA; Tue, 4 Apr 2023 00:22:29 +0000 (UTC) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 0598210E05F; Tue, 4 Apr 2023 00:22:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680567744; x=1712103744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=pM87jfqrMf8vXEWTm6fPP8Tl1MKGUh+f7ud0diOflQc=; b=fMhJYgOWm7uIGMUOp/2WvKGkyo1U3K411nA3kCUWXpfHOpOCkNjH6uJE 9LgmVoIttkWB/ehqnobpklL3ZZtEDs5napqQCAZeud1Ru65dB1zo3zhmz VFR9qJvZgWF6d5fQUam6Mx939FpyfOKufIG78iNG4jdGqz7ri++g1RYL9 t9C6T8epLEpvx3TLC6GorPeN7vejMkTrLdJdd93CWYZqQzVFztLxjEty2 JB+QFK8jw6xBzBuDpq0fOKNYNkc2wX2FII06gdc5X8HCjBkVB4lelzV9a fhEfsxZnv+JgINeLW9rxPNaxC25dZ2Zm9FV5WL8F5/3hI4rdjIbDrdkxk A==; X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="404810517" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="404810517" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="716460303" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="716460303" Received: from lstrano-desk.jf.intel.com ([10.24.89.184]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC PATCH 05/10] drm/sched: Start run wq before TDR in drm_sched_start Date: Mon, 3 Apr 2023 17:22:06 -0700 Message-Id: <20230404002211.3611376-6-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230404002211.3611376-1-matthew.brost@intel.com> References: <20230404002211.3611376-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, Matthew Brost , christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" If the TDR is set to a very small value it can fire before the run wq is started in the function drm_sched_start. The run wq is expected to running when the TDR fires, fix this ordering so this expectation is always met. Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 9dc3378e9c5e..6ae710017024 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -611,13 +611,13 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery) drm_sched_job_done(s_job); } + drm_sched_run_wq_start(sched); + if (full_recovery) { spin_lock(&sched->job_list_lock); drm_sched_start_timeout(sched); spin_unlock(&sched->job_list_lock); } - - drm_sched_run_wq_start(sched); } EXPORT_SYMBOL(drm_sched_start); From patchwork Tue Apr 4 00:22:07 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13198939 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2E9B0C77B62 for ; Tue, 4 Apr 2023 00:22:37 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id B62CC10E048; Tue, 4 Apr 2023 00:22:25 +0000 (UTC) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7068C10E05F; Tue, 4 Apr 2023 00:22:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680567744; x=1712103744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=S1slQD3OeKnMaC9hsbtA7WJFpQywRZAU0l5Iq4exbig=; b=FY5Z2nrvyO4exbKTVCIVPpxt7Gp383dR8YkPc5ZPU6qmp6beBSJ/gxO+ LQNyqyZ2+0ZhwAkXDZbXJmUz9rF2cRyKSzcbeodb3ULrPgiTa9pnjCM9X ssSDja8szzWZX715Sr6koxpZQPTiaitPNDVTtatJh5A2tMstl0lI7VVg0 DlUWrMWYw3dfJD8nhTyjoBwend/rsURNkCHd68LwC2D+6Uh9GUSz5WTMA /Mtsu9Iy5W1kqqXKhX822dId4gd0CInEIdzhvlsjoSJ8ehEu5d7CyQ6zc jT0FGM3QUy47Auum9EncpUQTQrTlLLGM+5n1LceTDlV0MOsyTEevfg4S9 Q==; X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="404810527" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="404810527" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="716460306" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="716460306" Received: from lstrano-desk.jf.intel.com ([10.24.89.184]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC PATCH 06/10] drm/sched: Submit job before starting TDR Date: Mon, 3 Apr 2023 17:22:07 -0700 Message-Id: <20230404002211.3611376-7-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230404002211.3611376-1-matthew.brost@intel.com> References: <20230404002211.3611376-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, Matthew Brost , christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" If the TDR is set to a value, it can fire before a job is submitted in drm_sched_main. The job should be always be submitted before the TDR fires, fix this ordering. Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 6ae710017024..4eac02d212c1 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -1150,10 +1150,10 @@ static void drm_sched_main(struct work_struct *w) s_fence = sched_job->s_fence; atomic_inc(&sched->hw_rq_count); - drm_sched_job_begin(sched_job); trace_drm_run_job(sched_job, entity); fence = sched->ops->run_job(sched_job); + drm_sched_job_begin(sched_job); complete_all(&entity->entity_idle); drm_sched_fence_scheduled(s_fence); From patchwork Tue Apr 4 00:22:08 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13198942 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 14F5DC761AF for ; Tue, 4 Apr 2023 00:22:47 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4998010E1B6; Tue, 4 Apr 2023 00:22:28 +0000 (UTC) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 3921410E048; Tue, 4 Apr 2023 00:22:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680567744; x=1712103744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=dcXnSMvcGNfZZWnYrnn67phCKax1D1Ipxk/D6bzwRVk=; b=Jn6P4RmXZvdbvYdTe7mcpYyveY6w5nDj5cb7vPrVqViziiL7+hB0fbZL s6sqh9zqZhxYD9P0EJOiKk89lBPRJAY2NhiR2I04ISoGqoGwrC91P0b3p TdG4LotSjH6TnFFDjJeABZghyw0Fk5PZgK6Uyh4MOwX+QpQyM0WgJyXjk PT3iklUaYvLtllCZjzMaGwsFY90+L/gh39RIis5O7oEhqRuKZoZbckhAi GLp0DjDlIGuuR3dw6g9vxURN1QZEGdi+sAZu/WfBSkBvg67RtB+VQYDu6 f8WiyJgCEzysCo/tyO/Ov2DxwEdcU3I9Pr46QbDVwLP/TOB9xINsXrDtg g==; X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="404810536" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="404810536" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="716460309" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="716460309" Received: from lstrano-desk.jf.intel.com ([10.24.89.184]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC PATCH 07/10] drm/sched: Add helper to set TDR timeout Date: Mon, 3 Apr 2023 17:22:08 -0700 Message-Id: <20230404002211.3611376-8-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230404002211.3611376-1-matthew.brost@intel.com> References: <20230404002211.3611376-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, Matthew Brost , christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Add helper to set TDR timeout and restart the TDR with new timeout value. This will be used in XE, new Intel GPU driver, to trigger the TDR to cleanup drm_sched_entity that encounter errors. Signed-off-by: Matthew Brost --- drivers/gpu/drm/scheduler/sched_main.c | 18 ++++++++++++++++++ include/drm/gpu_scheduler.h | 1 + 2 files changed, 19 insertions(+) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 4eac02d212c1..d61880315d8d 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -370,6 +370,24 @@ static void drm_sched_start_timeout(struct drm_gpu_scheduler *sched) queue_delayed_work(sched->timeout_wq, &sched->work_tdr, sched->timeout); } +/** + * drm_sched_set_timeout - set timeout for reset worker + * + * @sched: scheduler instance to set and (re)-start the worker for + * @timeout: timeout period + * + * Set and (re)-start the timeout for the given scheduler. + */ +void drm_sched_set_timeout(struct drm_gpu_scheduler *sched, long timeout) +{ + spin_lock(&sched->job_list_lock); + sched->timeout = timeout; + cancel_delayed_work(&sched->work_tdr); + drm_sched_start_timeout(sched); + spin_unlock(&sched->job_list_lock); +} +EXPORT_SYMBOL(drm_sched_set_timeout); + /** * drm_sched_fault - immediately start timeout handler * diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 18172ae63ab7..6258e324bd7c 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -593,6 +593,7 @@ void drm_sched_entity_modify_sched(struct drm_sched_entity *entity, struct drm_gpu_scheduler **sched_list, unsigned int num_sched_list); +void drm_sched_set_timeout(struct drm_gpu_scheduler *sched, long timeout); void drm_sched_job_cleanup(struct drm_sched_job *job); void drm_sched_wakeup(struct drm_gpu_scheduler *sched); void drm_sched_add_msg(struct drm_gpu_scheduler *sched, From patchwork Tue Apr 4 00:22:09 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13198940 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 56396C76196 for ; Tue, 4 Apr 2023 00:22:41 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 460AD10E0E3; Tue, 4 Apr 2023 00:22:26 +0000 (UTC) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id A138C10E048; Tue, 4 Apr 2023 00:22:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680567744; x=1712103744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=PkFDbYuDEsMLPA63PNlLCfLAaP0jp+1ED1yaUoTrNYQ=; b=DTXU4MiaMA6ZGzx3FF2408oQ/1zB79ycN9JyPUHziphyq3yn7/ZJ+ZMm nxpGEh5hKTRgOcxY78AJAzzDiaSdlh7xaqMTwOSUvko5yqaYZMDLBQl6Z Lao2G1QFwHwraP6UYmVz+0fb4wMhPMb/z3MqAzXyudOIx6f9XIydUUUvZ fvEzj2GYepaOkKeI5nb3ghG5avyU+jdgMjUw8TfLeWtQW3XsmqyQNRWPD 7gh3TiJmTr753RRQ/PUiHBa+wVH0AMM8N5uQ4+Yx428wx8mDYfDuhZ2Wk /MnHRrYq5+AWFRepPC7QWY8mjjH41S7AGzJAXc+OVZ1NkWlHGhjtlMQye w==; X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="404810549" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="404810549" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="716460312" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="716460312" Received: from lstrano-desk.jf.intel.com ([10.24.89.184]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC PATCH 08/10] dma-buf/dma-fence: Introduce long-running completion fences Date: Mon, 3 Apr 2023 17:22:09 -0700 Message-Id: <20230404002211.3611376-9-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230404002211.3611376-1-matthew.brost@intel.com> References: <20230404002211.3611376-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, Matthew Brost , christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Thomas Hellström For long-running workloads, drivers either need to open-code completion waits, invent their own synchronization primitives or internally use dma-fences that do not obey the cross-driver dma-fence protocol, but without any lockdep annotation all these approaches are error prone. So since for example the drm scheduler uses dma-fences it is desirable for a driver to be able to use it for throttling and error handling also with internal dma-fences tha do not obey the cros-driver dma-fence protocol. Introduce long-running completion fences in form of dma-fences, and add lockdep annotation for them. In particular: * Do not allow waiting under any memory management locks. * Do not allow to attach them to a dma-resv object. * Introduce a new interface for adding callbacks making the helper adding a callback sign off on that it is aware that the dma-fence may not complete anytime soon. Typically this will be the scheduler chaining a new long-running fence on another one. Signed-off-by: Matthew Brost Signed-off-by: Thomas Hellström --- drivers/dma-buf/dma-fence.c | 142 ++++++++++++++++++++++++++---------- drivers/dma-buf/dma-resv.c | 5 ++ include/linux/dma-fence.h | 55 +++++++++++++- 3 files changed, 160 insertions(+), 42 deletions(-) diff --git a/drivers/dma-buf/dma-fence.c b/drivers/dma-buf/dma-fence.c index f177c56269bb..9726b2a3c67d 100644 --- a/drivers/dma-buf/dma-fence.c +++ b/drivers/dma-buf/dma-fence.c @@ -111,6 +111,20 @@ static atomic64_t dma_fence_context_counter = ATOMIC64_INIT(1); * drivers/gpu should ever call dma_fence_wait() in such contexts. */ +/** + * DOC: Long-Running (lr) dma-fences. + * + * * Long-running dma-fences are NOT required to complete in reasonable time. + * Typically they signal completion of user-space controlled workloads and + * as such, need to never be part of a cross-driver contract, never waited + * for inside a kernel lock, nor attached to a dma-resv. There are helpers + * and warnings in place to help facilitate that that never happens. + * + * * The motivation for their existense is that helpers that are intended to + * be used by drivers may use dma-fences that, given the workloads mentioned + * above, become long-running. + */ + static const char *dma_fence_stub_get_name(struct dma_fence *fence) { return "stub"; @@ -284,6 +298,34 @@ static struct lockdep_map dma_fence_lockdep_map = { .name = "dma_fence_map" }; +static struct lockdep_map dma_fence_lr_lockdep_map = { + .name = "dma_fence_lr_map" +}; + +static bool __dma_fence_begin_signalling(struct lockdep_map *map) +{ + /* explicitly nesting ... */ + if (lock_is_held_type(map, 1)) + return true; + + /* rely on might_sleep check for soft/hardirq locks */ + if (in_atomic()) + return true; + + /* ... and non-recursive readlock */ + lock_acquire(map, 0, 0, 1, 1, NULL, _RET_IP_); + + return false; +} + +static void __dma_fence_end_signalling(bool cookie, struct lockdep_map *map) +{ + if (cookie) + return; + + lock_release(map, _RET_IP_); +} + /** * dma_fence_begin_signalling - begin a critical DMA fence signalling section * @@ -300,18 +342,7 @@ static struct lockdep_map dma_fence_lockdep_map = { */ bool dma_fence_begin_signalling(void) { - /* explicitly nesting ... */ - if (lock_is_held_type(&dma_fence_lockdep_map, 1)) - return true; - - /* rely on might_sleep check for soft/hardirq locks */ - if (in_atomic()) - return true; - - /* ... and non-recursive readlock */ - lock_acquire(&dma_fence_lockdep_map, 0, 0, 1, 1, NULL, _RET_IP_); - - return false; + return __dma_fence_begin_signalling(&dma_fence_lockdep_map); } EXPORT_SYMBOL(dma_fence_begin_signalling); @@ -323,25 +354,61 @@ EXPORT_SYMBOL(dma_fence_begin_signalling); */ void dma_fence_end_signalling(bool cookie) { - if (cookie) - return; - - lock_release(&dma_fence_lockdep_map, _RET_IP_); + __dma_fence_end_signalling(cookie, &dma_fence_lockdep_map); } EXPORT_SYMBOL(dma_fence_end_signalling); -void __dma_fence_might_wait(void) +/** + * dma_fence_lr begin_signalling - begin a critical long-running DMA fence + * signalling section + * + * Drivers should use this to annotate the beginning of any code section + * required to eventually complete &dma_fence by calling dma_fence_signal(). + * + * The end of these critical sections are annotated with + * dma_fence_lr_end_signalling(). Ideally the section should encompass all + * locks that are ever required to signal a long-running dma-fence. + * + * Return: An opaque cookie needed by the implementation, which needs to be + * passed to dma_fence_lr end_signalling(). + */ +bool dma_fence_lr_begin_signalling(void) +{ + return __dma_fence_begin_signalling(&dma_fence_lr_lockdep_map); +} +EXPORT_SYMBOL(dma_fence_lr_begin_signalling); + +/** + * dma_fence_lr_end_signalling - end a critical DMA fence signalling section + * @cookie: opaque cookie from dma_fence_lr_begin_signalling() + * + * Closes a critical section annotation opened by + * dma_fence_lr_begin_signalling(). + */ +void dma_fence_lr_end_signalling(bool cookie) +{ + __dma_fence_end_signalling(cookie, &dma_fence_lr_lockdep_map); +} +EXPORT_SYMBOL(dma_fence_lr_end_signalling); + +static void ___dma_fence_might_wait(struct lockdep_map *map) { bool tmp; - tmp = lock_is_held_type(&dma_fence_lockdep_map, 1); + tmp = lock_is_held_type(map, 1); if (tmp) - lock_release(&dma_fence_lockdep_map, _THIS_IP_); - lock_map_acquire(&dma_fence_lockdep_map); - lock_map_release(&dma_fence_lockdep_map); + lock_release(map, _THIS_IP_); + lock_map_acquire(map); + lock_map_release(map); if (tmp) - lock_acquire(&dma_fence_lockdep_map, 0, 0, 1, 1, NULL, _THIS_IP_); + lock_acquire(map, 0, 0, 1, 1, NULL, _THIS_IP_); +} + +void __dma_fence_might_wait(void) +{ + ___dma_fence_might_wait(&dma_fence_lockdep_map); } + #endif @@ -506,7 +573,11 @@ dma_fence_wait_timeout(struct dma_fence *fence, bool intr, signed long timeout) might_sleep(); - __dma_fence_might_wait(); +#ifdef CONFIG_LOCKDEP + ___dma_fence_might_wait(dma_fence_is_lr(fence) ? + &dma_fence_lr_lockdep_map : + &dma_fence_lockdep_map); +#endif dma_fence_enable_sw_signaling(fence); @@ -618,29 +689,22 @@ void dma_fence_enable_sw_signaling(struct dma_fence *fence) EXPORT_SYMBOL(dma_fence_enable_sw_signaling); /** - * dma_fence_add_callback - add a callback to be called when the fence + * dma_fence_lr_add_callback - add a callback to be called when the fence * is signaled * @fence: the fence to wait on * @cb: the callback to register * @func: the function to call * - * Add a software callback to the fence. The caller should keep a reference to - * the fence. - * - * @cb will be initialized by dma_fence_add_callback(), no initialization - * by the caller is required. Any number of callbacks can be registered - * to a fence, but a callback can only be registered to one fence at a time. - * - * If fence is already signaled, this function will return -ENOENT (and - * *not* call the callback). - * - * Note that the callback can be called from an atomic context or irq context. + * This function is identical to dma_fence_add_callback() but allows adding + * callbacks also to lr dma-fences. The naming helps annotating the fact that + * we're adding a callback to a a lr fence and that the callback might therefore + * not be called within a reasonable amount of time. * - * Returns 0 in case of success, -ENOENT if the fence is already signaled + * Return: 0 in case of success, -ENOENT if the fence is already signaled * and -EINVAL in case of error. */ -int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, - dma_fence_func_t func) +int dma_fence_lr_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, + dma_fence_func_t func) { unsigned long flags; int ret = 0; @@ -667,7 +731,7 @@ int dma_fence_add_callback(struct dma_fence *fence, struct dma_fence_cb *cb, return ret; } -EXPORT_SYMBOL(dma_fence_add_callback); +EXPORT_SYMBOL(dma_fence_lr_add_callback); /** * dma_fence_get_status - returns the status upon completion diff --git a/drivers/dma-buf/dma-resv.c b/drivers/dma-buf/dma-resv.c index 2a594b754af1..fa0210c1442e 100644 --- a/drivers/dma-buf/dma-resv.c +++ b/drivers/dma-buf/dma-resv.c @@ -292,6 +292,7 @@ void dma_resv_add_fence(struct dma_resv *obj, struct dma_fence *fence, * individually. */ WARN_ON(dma_fence_is_container(fence)); + WARN_ON_ONCE(dma_fence_is_lr(fence)); fobj = dma_resv_fences_list(obj); count = fobj->num_fences; @@ -340,6 +341,7 @@ void dma_resv_replace_fences(struct dma_resv *obj, uint64_t context, unsigned int i; dma_resv_assert_held(obj); + WARN_ON_ONCE(dma_fence_is_lr(replacement)); list = dma_resv_fences_list(obj); for (i = 0; list && i < list->num_fences; ++i) { @@ -764,6 +766,7 @@ static int __init dma_resv_lockdep(void) struct ww_acquire_ctx ctx; struct dma_resv obj; struct address_space mapping; + bool lr_cookie; int ret; if (!mm) @@ -772,6 +775,7 @@ static int __init dma_resv_lockdep(void) dma_resv_init(&obj); address_space_init_once(&mapping); + lr_cookie = dma_fence_lr_begin_signalling(); mmap_read_lock(mm); ww_acquire_init(&ctx, &reservation_ww_class); ret = dma_resv_lock(&obj, &ctx); @@ -792,6 +796,7 @@ static int __init dma_resv_lockdep(void) ww_mutex_unlock(&obj.lock); ww_acquire_fini(&ctx); mmap_read_unlock(mm); + dma_fence_lr_end_signalling(lr_cookie); mmput(mm); diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h index d54b595a0fe0..08d21e26782b 100644 --- a/include/linux/dma-fence.h +++ b/include/linux/dma-fence.h @@ -99,6 +99,7 @@ enum dma_fence_flag_bits { DMA_FENCE_FLAG_SIGNALED_BIT, DMA_FENCE_FLAG_TIMESTAMP_BIT, DMA_FENCE_FLAG_ENABLE_SIGNAL_BIT, + DMA_FENCE_FLAG_LR_BIT, DMA_FENCE_FLAG_USER_BITS, /* must always be last member */ }; @@ -279,6 +280,11 @@ struct dma_fence_ops { void (*set_deadline)(struct dma_fence *fence, ktime_t deadline); }; +static inline bool dma_fence_is_lr(const struct dma_fence *fence) +{ + return test_bit(DMA_FENCE_FLAG_LR_BIT, &fence->flags); +} + void dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, spinlock_t *lock, u64 context, u64 seqno); @@ -377,13 +383,23 @@ dma_fence_get_rcu_safe(struct dma_fence __rcu **fencep) #ifdef CONFIG_LOCKDEP bool dma_fence_begin_signalling(void); void dma_fence_end_signalling(bool cookie); +bool dma_fence_lr_begin_signalling(void); +void dma_fence_lr_end_signalling(bool cookie); void __dma_fence_might_wait(void); #else + static inline bool dma_fence_begin_signalling(void) { return true; } + static inline void dma_fence_end_signalling(bool cookie) {} +static inline bool dma_fence_lr_begin_signalling(void) +{ + return true; +} + +static inline void dma_fence_lr_end_signalling(bool cookie) {} static inline void __dma_fence_might_wait(void) {} #endif @@ -394,9 +410,42 @@ int dma_fence_signal_timestamp_locked(struct dma_fence *fence, ktime_t timestamp); signed long dma_fence_default_wait(struct dma_fence *fence, bool intr, signed long timeout); -int dma_fence_add_callback(struct dma_fence *fence, - struct dma_fence_cb *cb, - dma_fence_func_t func); + +int dma_fence_lr_add_callback(struct dma_fence *fence, + struct dma_fence_cb *cb, + dma_fence_func_t func); + +/** + * dma_fence_add_callback - add a callback to be called when the fence + * is signaled + * @fence: the fence to wait on + * @cb: the callback to register + * @func: the function to call + * + * Add a software callback to the fence. The caller should keep a reference to + * the fence. + * + * @cb will be initialized by dma_fence_add_callback(), no initialization + * by the caller is required. Any number of callbacks can be registered + * to a fence, but a callback can only be registered to one fence at a time. + * + * If fence is already signaled, this function will return -ENOENT (and + * *not* call the callback). + * + * Note that the callback can be called from an atomic context or irq context. + * + * Returns 0 in case of success, -ENOENT if the fence is already signaled + * and -EINVAL in case of error. + */ +static inline int dma_fence_add_callback(struct dma_fence *fence, + struct dma_fence_cb *cb, + dma_fence_func_t func) +{ + WARN_ON(IS_ENABLED(CONFIG_LOCKDEP) && dma_fence_is_lr(fence)); + + return dma_fence_lr_add_callback(fence, cb, func); +} + bool dma_fence_remove_callback(struct dma_fence *fence, struct dma_fence_cb *cb); void dma_fence_enable_sw_signaling(struct dma_fence *fence); From patchwork Tue Apr 4 00:22:10 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13198946 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 87FBBC761AF for ; Tue, 4 Apr 2023 00:22:54 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id BE11610E08C; Tue, 4 Apr 2023 00:22:29 +0000 (UTC) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 70CE310E08A; Tue, 4 Apr 2023 00:22:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680567744; x=1712103744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=bCWnPihC2yKjoD/PHDXw9txgKVWfGootNC5roVYYdgw=; b=YKE2cZiUhr3rpYqJFn6dZdBPZEik/cTwF136pOD+/MKBTL1zy4mNUCMr zzd39GkHRKf3IUw+g1oGZywMY1TpYLbykR5LUUoTA8DDMCNN73oBD9nbK hQTjeJwAq2kujSq9IoCWHHWcwIBYkX53eSotJ0RqqdTrrsbIdII3zCEVE woJ9+IDSE9gsFXfP5KRZb61ETUukxXjgWp/sSPuvGpCD5QvDbQHotMsmI yy+Y+xotClFFwYpFlG7lI1y4v+sUcqlcZ1IlVxicBVYWD1oc3EKVMQHaz 4n6nd8XJvIYNFuSmmLSfyI5Au5o0eA6RPxq3i9Er0892QWm/LRJUQU6Tn A==; X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="404810551" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="404810551" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="716460315" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="716460315" Received: from lstrano-desk.jf.intel.com ([10.24.89.184]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC PATCH 09/10] drm/sched: Support long-running sched entities Date: Mon, 3 Apr 2023 17:22:10 -0700 Message-Id: <20230404002211.3611376-10-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230404002211.3611376-1-matthew.brost@intel.com> References: <20230404002211.3611376-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, airlied@linux.ie, lina@asahilina.net, boris.brezillon@collabora.com, Matthew Brost , christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Thomas Hellström Make the drm scheduler aware of long-running dma fences by * Enable marking a sched entity as producing long-running fences. * Disallowing long-running fences as dependencies for non-long-running sched entities, while long-running sched entities allow those. Signed-off-by: Matthew Brost Signed-off-by: Thomas Hellström --- drivers/gpu/drm/scheduler/sched_entity.c | 44 +++++++++++++++++++----- drivers/gpu/drm/scheduler/sched_fence.c | 4 +++ drivers/gpu/drm/scheduler/sched_main.c | 9 ++--- include/drm/gpu_scheduler.h | 36 +++++++++++++++++++ include/linux/dma-fence.h | 5 +++ 5 files changed, 86 insertions(+), 12 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_entity.c b/drivers/gpu/drm/scheduler/sched_entity.c index ccea4d079d0f..0640fc9d4491 100644 --- a/drivers/gpu/drm/scheduler/sched_entity.c +++ b/drivers/gpu/drm/scheduler/sched_entity.c @@ -174,6 +174,32 @@ static void drm_sched_entity_kill_jobs_work(struct work_struct *wrk) job->sched->ops->free_job(job); } +/** + * drm_sched_entity_add_fence_cb() - Helper to add a fence callback + * @entity: The sched entity + * @f: The possbily long-running dma-fence on which to add a callback + * @cb: The struct dma_fence_cb to use for the callback + * @func: The callback function. + * + * This function calls the proper dma_fence add callback function + * depending on whether @entity is marked as long-running or not. If it + * is not, this will make sure we get a warning if trying to add a + * callback on a long-running dma-fence. + * + * Return: Zero on success, -ENOENT if already signaled and -EINVAL in case + * of error. + */ +int drm_sched_entity_add_fence_cb(struct drm_sched_entity *entity, + struct dma_fence *f, + struct dma_fence_cb *cb, + dma_fence_func_t func) +{ + if (drm_sched_entity_is_lr(entity)) + return dma_fence_lr_add_callback(f, cb, func); + + return dma_fence_add_callback(f, cb, func); +} + /* Signal the scheduler finished fence when the entity in question is killed. */ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, struct dma_fence_cb *cb) @@ -187,8 +213,8 @@ static void drm_sched_entity_kill_jobs_cb(struct dma_fence *f, /* Wait for all dependencies to avoid data corruptions */ while (!xa_empty(&job->dependencies)) { f = xa_erase(&job->dependencies, job->last_dependency++); - r = dma_fence_add_callback(f, &job->finish_cb, - drm_sched_entity_kill_jobs_cb); + r = drm_sched_entity_add_fence_cb(job->entity, f, &job->finish_cb, + drm_sched_entity_kill_jobs_cb); if (!r) return; @@ -226,8 +252,9 @@ static void drm_sched_entity_kill(struct drm_sched_entity *entity) dma_fence_set_error(&s_fence->finished, -ESRCH); dma_fence_get(&s_fence->finished); - if (!prev || dma_fence_add_callback(prev, &job->finish_cb, - drm_sched_entity_kill_jobs_cb)) + if (!prev || drm_sched_entity_add_fence_cb(job->entity, prev, + &job->finish_cb, + drm_sched_entity_kill_jobs_cb)) drm_sched_entity_kill_jobs_cb(NULL, &job->finish_cb); prev = &s_fence->finished; @@ -420,8 +447,8 @@ static bool drm_sched_entity_add_dependency_cb(struct drm_sched_entity *entity) fence = dma_fence_get(&s_fence->scheduled); dma_fence_put(entity->dependency); entity->dependency = fence; - if (!dma_fence_add_callback(fence, &entity->cb, - drm_sched_entity_clear_dep)) + if (!drm_sched_entity_add_fence_cb(entity, fence, &entity->cb, + drm_sched_entity_clear_dep)) return true; /* Ignore it when it is already scheduled */ @@ -429,8 +456,9 @@ static bool drm_sched_entity_add_dependency_cb(struct drm_sched_entity *entity) return false; } - if (!dma_fence_add_callback(entity->dependency, &entity->cb, - drm_sched_entity_wakeup)) + if (!drm_sched_entity_add_fence_cb(entity, entity->dependency, + &entity->cb, + drm_sched_entity_wakeup)) return true; dma_fence_put(entity->dependency); diff --git a/drivers/gpu/drm/scheduler/sched_fence.c b/drivers/gpu/drm/scheduler/sched_fence.c index d7cfc0441885..a566723ecc2c 100644 --- a/drivers/gpu/drm/scheduler/sched_fence.c +++ b/drivers/gpu/drm/scheduler/sched_fence.c @@ -217,8 +217,12 @@ void drm_sched_fence_init(struct drm_sched_fence *fence, seq = atomic_inc_return(&entity->fence_seq); dma_fence_init(&fence->scheduled, &drm_sched_fence_ops_scheduled, &fence->lock, entity->fence_context, seq); + if (drm_sched_entity_is_lr(entity)) + dma_fence_set_lr(&fence->scheduled); dma_fence_init(&fence->finished, &drm_sched_fence_ops_finished, &fence->lock, entity->fence_context + 1, seq); + if (drm_sched_entity_is_lr(entity)) + dma_fence_set_lr(&fence->finished); } module_init(drm_sched_fence_slab_init); diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index d61880315d8d..76336a31aa82 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -618,8 +618,8 @@ void drm_sched_start(struct drm_gpu_scheduler *sched, bool full_recovery) continue; if (fence) { - r = dma_fence_add_callback(fence, &s_job->cb, - drm_sched_job_done_cb); + r = drm_sched_entity_add_fence_cb(s_job->entity, fence, + &s_job->cb, drm_sched_job_done_cb); if (r == -ENOENT) drm_sched_job_done(s_job); else if (r) @@ -1180,8 +1180,9 @@ static void drm_sched_main(struct work_struct *w) /* Drop for original kref_init of the fence */ dma_fence_put(fence); - r = dma_fence_add_callback(fence, &sched_job->cb, - drm_sched_job_done_cb); + r = drm_sched_entity_add_fence_cb(sched_job->entity, fence, + &sched_job->cb, + drm_sched_job_done_cb); if (r == -ENOENT) drm_sched_job_done(sched_job); else if (r) diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 6258e324bd7c..546507852771 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -142,6 +142,16 @@ struct drm_sched_entity { */ unsigned int num_sched_list; + /** + * @flags: Flags to govern the behaviour: + * + * DRM_SCHED_ENTITY_LR: The entity handles long-running jobs and + * produces long-running completion fences, as well as accepts + * long-running dependency fences. + */ + u32 flags; +#define DRM_SCHED_ENTITY_LR BIT(0) + /** * @priority: * @@ -253,6 +263,32 @@ struct drm_sched_entity { }; +/** + * drm_sched_entity_is_lr() - Whether the entity manages long-running jobs. + * @entity: The entity. + * + * Return: true if managing long-running jobs. Otherwise false. + */ +static inline bool drm_sched_entity_is_lr(const struct drm_sched_entity *entity) +{ + return entity->flags & DRM_SCHED_ENTITY_LR; +} + +/** + * drm_sched_entity_set_lr() - Mark the entity as managing long-running jobs. + * @entity: The entity. + * + */ +static inline void drm_sched_entity_set_lr(struct drm_sched_entity *entity) +{ + entity->flags |= DRM_SCHED_ENTITY_LR; +} + +int drm_sched_entity_add_fence_cb(struct drm_sched_entity *entity, + struct dma_fence *f, + struct dma_fence_cb *cb, + dma_fence_func_t func); + /** * struct drm_sched_rq - queue of entities to be scheduled. * diff --git a/include/linux/dma-fence.h b/include/linux/dma-fence.h index 08d21e26782b..b513811ce536 100644 --- a/include/linux/dma-fence.h +++ b/include/linux/dma-fence.h @@ -285,6 +285,11 @@ static inline bool dma_fence_is_lr(const struct dma_fence *fence) return test_bit(DMA_FENCE_FLAG_LR_BIT, &fence->flags); } +static inline void dma_fence_set_lr(struct dma_fence *fence) +{ + __set_bit(DMA_FENCE_FLAG_LR_BIT, &fence->flags); +} + void dma_fence_init(struct dma_fence *fence, const struct dma_fence_ops *ops, spinlock_t *lock, u64 context, u64 seqno); From patchwork Tue Apr 4 00:22:11 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Brost X-Patchwork-Id: 13198948 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D79EDC761AF for ; Tue, 4 Apr 2023 00:22:57 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 3996C10E1BF; Tue, 4 Apr 2023 00:22:33 +0000 (UTC) Received: from mga06.intel.com (mga06b.intel.com [134.134.136.31]) by gabe.freedesktop.org (Postfix) with ESMTPS id 39ADF10E086; Tue, 4 Apr 2023 00:22:24 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1680567744; x=1712103744; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=rI+mt1YKYp3orUuX569X3FxbQOR31uaY8xNpPR6u4lY=; b=mDTCZx++h/HokRhhuHiGVtluwwEMEkgnGYUZYPV0VlRLS5itHYymY8d0 qGYwp8GU27ESvbrnqKG+nQmNoGpbHQX8CEXytwm5fm1uC1plmrdXlo8Hs c2jRF0AQJIQIvVc+zgiVwM2OYp5qirNKMZuMUCT1LXHeh9MsQpnQRIE8w hMn5zprn5T1KUwz3sjfhd8RDD0V4WsH2W+byZy35oJpJKUqx2wYWCOm/m iGxBRNy69G/0zsewkyhlzQf3Tmp16RSr87DrICgrUG4zHep9uSWI83mw5 zM7LtaautEaP+UZbaNXATbp7wTm9k5RFSyXljPlaY7dyLoT6J0l1kC099 w==; X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="404810544" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="404810544" Received: from orsmga008.jf.intel.com ([10.7.209.65]) by orsmga104.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:22 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10669"; a="716460318" X-IronPort-AV: E=Sophos;i="5.98,316,1673942400"; d="scan'208";a="716460318" Received: from lstrano-desk.jf.intel.com ([10.24.89.184]) by orsmga008-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 03 Apr 2023 17:22:21 -0700 From: Matthew Brost To: dri-devel@lists.freedesktop.org, intel-xe@lists.freedesktop.org Subject: [RFC PATCH 10/10] drm/syncobj: Warn on long running dma-fences Date: Mon, 3 Apr 2023 17:22:11 -0700 Message-Id: <20230404002211.3611376-11-matthew.brost@intel.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230404002211.3611376-1-matthew.brost@intel.com> References: <20230404002211.3611376-1-matthew.brost@intel.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: robdclark@chromium.org, thomas.hellstrom@linux.intel.com, airlied@linux.ie, lina@asahilina.net, Matthew Brost , boris.brezillon@collabora.com, Matthew Brost , christian.koenig@amd.com, faith.ekstrand@collabora.com Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" Long running dma-fences are not allowed to be exported, a drm_syncobj is designed to be exported to the user, so add a warn if drm_syncobj install long running dna-fences as this is not allowed. Signed-off-by: Matthew Brost --- drivers/gpu/drm/drm_syncobj.c | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/drivers/gpu/drm/drm_syncobj.c b/drivers/gpu/drm/drm_syncobj.c index 0c2be8360525..7c304cd7d037 100644 --- a/drivers/gpu/drm/drm_syncobj.c +++ b/drivers/gpu/drm/drm_syncobj.c @@ -291,6 +291,7 @@ void drm_syncobj_add_point(struct drm_syncobj *syncobj, struct syncobj_wait_entry *cur, *tmp; struct dma_fence *prev; + WARN_ON_ONCE(dma_fence_is_lr(fence)); dma_fence_get(fence); spin_lock(&syncobj->lock); @@ -325,8 +326,10 @@ void drm_syncobj_replace_fence(struct drm_syncobj *syncobj, struct dma_fence *old_fence; struct syncobj_wait_entry *cur, *tmp; - if (fence) + if (fence) { + WARN_ON_ONCE(dma_fence_is_lr(fence)); dma_fence_get(fence); + } spin_lock(&syncobj->lock);