From patchwork Wed Jan 8 18:35:25 2025 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Tvrtko Ursulin X-Patchwork-Id: 13931479 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2A7AAE7719C for ; Wed, 8 Jan 2025 18:35:50 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4A57B10EC6F; Wed, 8 Jan 2025 18:35:47 +0000 (UTC) Authentication-Results: gabe.freedesktop.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=igalia.com header.i=@igalia.com header.b="h2Sdo0Tf"; dkim-atps=neutral Received: from fanzine2.igalia.com (fanzine.igalia.com [178.60.130.6]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5FB8810EC64 for ; Wed, 8 Jan 2025 18:35:45 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=igalia.com; s=20170329; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=NJN8pscg2xyCc2iQW/9xPHj0/Iqw2W4o/rLLK7ZNz/c=; b=h2Sdo0TfO5kGDnAV7Bn8LVqkfo hozaaFsklV0FHeGk6SUa/cpe8YZCE6PsdUHAgsUgdu3lJChm841i0jgrXUlUSvWsqruB8ICi83+p6 4zmKjx2XVeURG/ncnGt96k/3i0Z4NBEA0W+MTzyx3peqCrtq3DQ9s++bHtLoZ+0JBIYbT6t9JRJdc u+ytuQjeeCTVaPaqjnNA1f8iF6J9xZGd4W2iJi2WS6/gbS7moTILs74d14PPmB2IDAEYzThKiPfNa 2CcqGPQlCCW8FT4RJwnKKWwmH0wpnFo8icbv7bNYyye+EzqtVEQJda/bfdt0UxL1Ltd8b9/yda9rQ tNZVZSew==; Received: from [90.241.98.187] (helo=localhost) by fanzine2.igalia.com with esmtpsa (Cipher TLS1.3:ECDHE_SECP256R1__RSA_PSS_RSAE_SHA256__AES_256_GCM:256) (Exim) id 1tVau7-00DFGZ-KY; Wed, 08 Jan 2025 19:35:43 +0100 From: Tvrtko Ursulin To: dri-devel@lists.freedesktop.org Cc: kernel-dev@igalia.com, Tvrtko Ursulin , =?utf-8?q?Christian_K=C3=B6nig?= , Danilo Krummrich , Matthew Brost , Philipp Stanner Subject: [RFC 15/18] drm/sched: Queue all free credits in one worker invocation Date: Wed, 8 Jan 2025 18:35:25 +0000 Message-ID: <20250108183528.41007-16-tvrtko.ursulin@igalia.com> X-Mailer: git-send-email 2.47.1 In-Reply-To: <20250108183528.41007-1-tvrtko.ursulin@igalia.com> References: <20250108183528.41007-1-tvrtko.ursulin@igalia.com> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" There is no reason to queue just a single job if scheduler can take more and re-queue the worker to queue more. We can simply feed the hardware with as much as it can take in one go and hopefully win some latency. Signed-off-by: Tvrtko Ursulin Cc: Christian König Cc: Danilo Krummrich Cc: Matthew Brost Cc: Philipp Stanner --- drivers/gpu/drm/scheduler/sched_main.c | 109 ++++++++++++------------- drivers/gpu/drm/scheduler/sched_rq.c | 17 +--- include/drm/gpu_scheduler.h | 2 - 3 files changed, 54 insertions(+), 74 deletions(-) diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index f748dcd06774..e3d885678b9b 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -98,33 +98,6 @@ static u32 drm_sched_available_credits(struct drm_gpu_scheduler *sched) return credits; } -/** - * drm_sched_can_queue -- Can we queue more to the hardware? - * @sched: scheduler instance - * @entity: the scheduler entity - * - * Return true if we can push at least one more job from @entity, false - * otherwise. - */ -bool drm_sched_can_queue(struct drm_gpu_scheduler *sched, - struct drm_sched_entity *entity) -{ - struct drm_sched_job *s_job; - - s_job = to_drm_sched_job(spsc_queue_peek(&entity->job_queue)); - if (!s_job) - return false; - - /* If a job exceeds the credit limit, truncate it to the credit limit - * itself to guarantee forward progress. - */ - if (drm_WARN(sched, s_job->credits > sched->credit_limit, - "Jobs may not exceed the credit limit, truncate.\n")) - s_job->credits = sched->credit_limit; - - return drm_sched_available_credits(sched) >= s_job->credits; -} - /** * drm_sched_run_job_queue - enqueue run-job work * @sched: scheduler instance @@ -928,48 +901,68 @@ static void drm_sched_run_job_work(struct work_struct *w) { struct drm_gpu_scheduler *sched = container_of(w, struct drm_gpu_scheduler, work_run_job); + u32 job_credits, submitted_credits = 0; struct drm_sched_entity *entity; - struct dma_fence *fence; struct drm_sched_job *sched_job; - int r; + struct dma_fence *fence; - /* Find entity with a ready job */ - entity = drm_sched_rq_select_entity(sched, sched->rq); - if (IS_ERR_OR_NULL(entity)) - return; /* No more work */ + for (;;) { + /* Find entity with a ready job */ + entity = drm_sched_rq_select_entity(sched, sched->rq); + if (!entity) + break; /* No more work */ - sched_job = drm_sched_entity_pop_job(entity); - if (!sched_job) { + /* + * If a job exceeds the credit limit truncate it to guarantee + * forward progress. + */ + sched_job = to_drm_sched_job(spsc_queue_peek(&entity->job_queue)); + job_credits = sched_job->credits; + if (drm_WARN_ONCE(sched, job_credits > sched->credit_limit, + "Jobs may not exceed the credit limit, truncating.\n")) + job_credits = sched_job->credits = sched->credit_limit; + + if (job_credits > drm_sched_available_credits(sched)) { + complete_all(&entity->entity_idle); + break; + } + + sched_job = drm_sched_entity_pop_job(entity); complete_all(&entity->entity_idle); - drm_sched_run_job_queue(sched); - return; - } + if (!sched_job) { + /* Top entity is not yet runnable after all */ + continue; + } + + drm_sched_job_begin(sched_job); + trace_drm_run_job(sched_job, entity); + submitted_credits += job_credits; + atomic_add(job_credits, &sched->credit_count); - atomic_add(sched_job->credits, &sched->credit_count); - drm_sched_job_begin(sched_job); + fence = sched->ops->run_job(sched_job); + drm_sched_fence_scheduled(sched_job->s_fence, fence); - trace_drm_run_job(sched_job, entity); - fence = sched->ops->run_job(sched_job); - complete_all(&entity->entity_idle); - drm_sched_fence_scheduled(sched_job->s_fence, fence); + if (!IS_ERR_OR_NULL(fence)) { + int r; - if (!IS_ERR_OR_NULL(fence)) { - /* Drop for original kref_init of the fence */ - dma_fence_put(fence); + /* Drop for original kref_init of the fence */ + dma_fence_put(fence); - r = dma_fence_add_callback(fence, &sched_job->cb, - drm_sched_job_done_cb); - if (r == -ENOENT) - drm_sched_job_done(sched_job, fence->error); - else if (r) - DRM_DEV_ERROR(sched->dev, "fence add callback failed (%d)\n", r); - } else { - drm_sched_job_done(sched_job, IS_ERR(fence) ? - PTR_ERR(fence) : 0); + r = dma_fence_add_callback(fence, &sched_job->cb, + drm_sched_job_done_cb); + if (r == -ENOENT) + drm_sched_job_done(sched_job, fence->error); + else if (r) + DRM_DEV_ERROR(sched->dev, + "fence add callback failed (%d)\n", r); + } else { + drm_sched_job_done(sched_job, IS_ERR(fence) ? + PTR_ERR(fence) : 0); + } } - wake_up(&sched->job_scheduled); - drm_sched_run_job_queue(sched); + if (submitted_credits) + wake_up(&sched->job_scheduled); } /** diff --git a/drivers/gpu/drm/scheduler/sched_rq.c b/drivers/gpu/drm/scheduler/sched_rq.c index d9c854b2e495..647734d3ed73 100644 --- a/drivers/gpu/drm/scheduler/sched_rq.c +++ b/drivers/gpu/drm/scheduler/sched_rq.c @@ -153,9 +153,7 @@ void drm_sched_rq_pop_entity(struct drm_sched_rq *rq, * * Find oldest waiting ready entity. * - * Return an entity if one is found; return an error-pointer (!NULL) if an - * entity was ready, but the scheduler had insufficient credits to accommodate - * its job; return NULL, if no ready entity was found. + * Return an entity if one is found or NULL if no ready entity was found. */ struct drm_sched_entity * drm_sched_rq_select_entity(struct drm_gpu_scheduler *sched, @@ -174,17 +172,8 @@ drm_sched_rq_select_entity(struct drm_gpu_scheduler *sched, } spin_unlock(&rq->lock); - if (!entity) - return NULL; - - /* - * If scheduler cannot take more jobs signal the caller to not consider - * lower priority queues. - */ - if (!drm_sched_can_queue(sched, entity)) - return ERR_PTR(-ENOSPC); - - reinit_completion(&entity->entity_idle); + if (entity) + reinit_completion(&entity->entity_idle); return entity; } diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index 9f952574efe2..6d3a38772e72 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -543,8 +543,6 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, atomic_t *score, const char *name, struct device *dev); void drm_sched_fini(struct drm_gpu_scheduler *sched); -bool drm_sched_can_queue(struct drm_gpu_scheduler *sched, - struct drm_sched_entity *entity); int drm_sched_job_init(struct drm_sched_job *job, struct drm_sched_entity *entity,