From patchwork Thu Jan 19 13:09:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikko Perttunen X-Patchwork-Id: 13107952 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 14D75C00A5A for ; Thu, 19 Jan 2023 13:09:39 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 653DC10E242; Thu, 19 Jan 2023 13:09:39 +0000 (UTC) Received: from mail.kapsi.fi (mail.kapsi.fi [IPv6:2001:67c:1be8::25]) by gabe.freedesktop.org (Postfix) with ESMTPS id BFE6E10E91F for ; Thu, 19 Jan 2023 13:09:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kapsi.fi; s=20161220; h=Content-Transfer-Encoding:MIME-Version:Message-Id:Date:Subject: Cc:To:From:Sender:Reply-To:Content-Type:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: In-Reply-To:References:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=yxvZRuDgdikFRbFohfPeXUHNucrqEQrrcSHLXLhGx5M=; b=pdwcZbTBVlxfZAqOGr7GGCgL+i WgW7WYE0epvjb79N2y/a1lUQqx15OlOBplsaYuXVhGqP4CeK1CsCCybkIlxElOBZu2GN2BRcOrBnl jO4bfOTcaX8WVXbZ855ybtzJDTtZGrqhRurrApdstsCUp6uarJRzTphR4KERTzCWACYcpBlGfFeYM 8LfF3iqftlKeLK+2ccJEevhs6Y5s5UbAz3E5WW8q/g1RWLG6uWe0yK1gumTEniaNURkskFjEfQN20 46lIjccFirWjgTwscxZt5hb+OwVP6pSE0gD9UsHYPHz7rlLyzU0LjaBaBW3ogV23o+46L77V0ODgS cuUd2QBw==; Received: from 91-158-25-70.elisa-laajakaista.fi ([91.158.25.70] helo=toshino.localdomain) by mail.kapsi.fi with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pIUfe-000zLh-Mx; Thu, 19 Jan 2023 15:09:34 +0200 From: Mikko Perttunen To: Thierry Reding , Jonathan Hunter Subject: [PATCH 1/4] gpu: host1x: Implement syncpoint wait using DMA fences Date: Thu, 19 Jan 2023 15:09:18 +0200 Message-Id: <20230119130921.1882602-1-cyndis@kapsi.fi> X-Mailer: git-send-email 2.39.0 MIME-Version: 1.0 X-SA-Exim-Connect-IP: 91.158.25.70 X-SA-Exim-Mail-From: cyndis@kapsi.fi X-SA-Exim-Scanned: No (on mail.kapsi.fi); SAEximRunCond expanded to false X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-tegra@vger.kernel.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, Mikko Perttunen Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Mikko Perttunen In anticipation of removal of the intr API, move host1x_syncpt_wait to use DMA fences instead. As of this patch, this means that waits have a 30 second maximum timeout because of the implicit timeout we have with fences, but that will be lifted in a follow-up patch. Signed-off-by: Mikko Perttunen --- drivers/gpu/host1x/syncpt.c | 96 ++++++++----------------------------- 1 file changed, 20 insertions(+), 76 deletions(-) diff --git a/drivers/gpu/host1x/syncpt.c b/drivers/gpu/host1x/syncpt.c index f87a8705f518..75f58ec2ae23 100644 --- a/drivers/gpu/host1x/syncpt.c +++ b/drivers/gpu/host1x/syncpt.c @@ -7,6 +7,7 @@ #include #include +#include #include #include @@ -209,17 +210,6 @@ int host1x_syncpt_incr(struct host1x_syncpt *sp) } EXPORT_SYMBOL(host1x_syncpt_incr); -/* - * Updated sync point form hardware, and returns true if syncpoint is expired, - * false if we may need to wait - */ -static bool syncpt_load_min_is_expired(struct host1x_syncpt *sp, u32 thresh) -{ - host1x_hw_syncpt_load(sp->host, sp); - - return host1x_syncpt_is_expired(sp, thresh); -} - /** * host1x_syncpt_wait() - wait for a syncpoint to reach a given value * @sp: host1x syncpoint @@ -230,10 +220,10 @@ static bool syncpt_load_min_is_expired(struct host1x_syncpt *sp, u32 thresh) int host1x_syncpt_wait(struct host1x_syncpt *sp, u32 thresh, long timeout, u32 *value) { - DECLARE_WAIT_QUEUE_HEAD_ONSTACK(wq); - void *ref; - struct host1x_waitlist *waiter; - int err = 0, check_count = 0; + struct dma_fence *fence; + long wait_err; + + host1x_hw_syncpt_load(sp->host, sp); if (value) *value = host1x_syncpt_load(sp); @@ -241,73 +231,27 @@ int host1x_syncpt_wait(struct host1x_syncpt *sp, u32 thresh, long timeout, if (host1x_syncpt_is_expired(sp, thresh)) return 0; - if (!timeout) { - err = -EAGAIN; - goto done; - } - - /* allocate a waiter */ - waiter = kzalloc(sizeof(*waiter), GFP_KERNEL); - if (!waiter) { - err = -ENOMEM; - goto done; - } - - /* schedule a wakeup when the syncpoint value is reached */ - err = host1x_intr_add_action(sp->host, sp, thresh, - HOST1X_INTR_ACTION_WAKEUP_INTERRUPTIBLE, - &wq, waiter, &ref); - if (err) - goto done; - - err = -EAGAIN; - /* Caller-specified timeout may be impractically low */ if (timeout < 0) timeout = LONG_MAX; + else if (timeout == 0) + return -EAGAIN; - /* wait for the syncpoint, or timeout, or signal */ - while (timeout) { - long check = min_t(long, SYNCPT_CHECK_PERIOD, timeout); - int remain; - - remain = wait_event_interruptible_timeout(wq, - syncpt_load_min_is_expired(sp, thresh), - check); - if (remain > 0 || host1x_syncpt_is_expired(sp, thresh)) { - if (value) - *value = host1x_syncpt_load(sp); + fence = host1x_fence_create(sp, thresh); + if (IS_ERR(fence)) + return PTR_ERR(fence); - err = 0; + wait_err = dma_fence_wait_timeout(fence, true, timeout); + dma_fence_put(fence); - break; - } - - if (remain < 0) { - err = remain; - break; - } - - timeout -= check; - - if (timeout && check_count <= MAX_STUCK_CHECK_COUNT) { - dev_warn(sp->host->dev, - "%s: syncpoint id %u (%s) stuck waiting %d, timeout=%ld\n", - current->comm, sp->id, sp->name, - thresh, timeout); - - host1x_debug_dump_syncpts(sp->host); - - if (check_count == MAX_STUCK_CHECK_COUNT) - host1x_debug_dump(sp->host); - - check_count++; - } - } - - host1x_intr_put_ref(sp->host, sp->id, ref, true); + if (value) + *value = host1x_syncpt_load(sp); -done: - return err; + if (wait_err == 0) + return -EAGAIN; + else if (wait_err < 0) + return wait_err; + else + return 0; } EXPORT_SYMBOL(host1x_syncpt_wait); From patchwork Thu Jan 19 13:09:19 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikko Perttunen X-Patchwork-Id: 13107955 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5915AC00A5A for ; Thu, 19 Jan 2023 13:09:53 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 0B2F110E926; Thu, 19 Jan 2023 13:09:41 +0000 (UTC) Received: from mail.kapsi.fi (mail.kapsi.fi [IPv6:2001:67c:1be8::25]) by gabe.freedesktop.org (Postfix) with ESMTPS id C0C2710E923 for ; Thu, 19 Jan 2023 13:09:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kapsi.fi; s=20161220; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=TcKOFs0Yh57RnWuxPcj7FheziyDoms7tarPYcXv91uw=; b=aZ3OO57IVX+BXPBDWSs2BuaEsc 7e7izMk+YkqDqseGyi9Kw0QJ2y2RBitk1MXYCLrmJnpKJ32WStKO8DySqTJOpGt9xP8aNKvRcGvOv uJFjbqbBz0RTpos5ONOIy50SCcDD0WCLfAzYhvZyMQvpzQ1sELt/1hAMBkcnnajrqmbBwm4y38aJC N0uNM8sHG5rmJQ0icbA0fIGh71b3W638/4T1YIu2Q8Pal/8Ivqf2+xXUcbcCqc1JpU86ykO0TK7PS Zobm67WwOkVfbMHyN03UmtgbytoFwfStjFWj44LCpC0MdKSrNy/DuBvXEc7UoWwAVT8CB2U6QoENT F9JvgYag==; Received: from 91-158-25-70.elisa-laajakaista.fi ([91.158.25.70] helo=toshino.localdomain) by mail.kapsi.fi with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pIUfe-000zLh-Vz; Thu, 19 Jan 2023 15:09:35 +0200 From: Mikko Perttunen To: Thierry Reding , Jonathan Hunter Subject: [PATCH 2/4] gpu: host1x: Implement job tracking using DMA fences Date: Thu, 19 Jan 2023 15:09:19 +0200 Message-Id: <20230119130921.1882602-2-cyndis@kapsi.fi> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230119130921.1882602-1-cyndis@kapsi.fi> References: <20230119130921.1882602-1-cyndis@kapsi.fi> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 91.158.25.70 X-SA-Exim-Mail-From: cyndis@kapsi.fi X-SA-Exim-Scanned: No (on mail.kapsi.fi); SAEximRunCond expanded to false X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-tegra@vger.kernel.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, Mikko Perttunen Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Mikko Perttunen In anticipation of removal of the intr API, implement job tracking using DMA fences instead. The main two things about this are making cdma_update schedule the work since fence completion can now be called from interrupt context, and some complication in ensuring the callback is not running when we free the fence. Signed-off-by: Mikko Perttunen --- drivers/gpu/host1x/cdma.c | 14 +++++++-- drivers/gpu/host1x/cdma.h | 2 ++ drivers/gpu/host1x/hw/channel_hw.c | 48 +++++++++++++++++------------- drivers/gpu/host1x/job.c | 12 ++++++-- include/linux/host1x.h | 6 ++-- 5 files changed, 53 insertions(+), 29 deletions(-) diff --git a/drivers/gpu/host1x/cdma.c b/drivers/gpu/host1x/cdma.c index 103fda055394..bc821b0ed908 100644 --- a/drivers/gpu/host1x/cdma.c +++ b/drivers/gpu/host1x/cdma.c @@ -490,6 +490,15 @@ void host1x_cdma_update_sync_queue(struct host1x_cdma *cdma, host1x_hw_cdma_resume(host1x, cdma, restart_addr); } +static void cdma_update_work(struct work_struct *work) +{ + struct host1x_cdma *cdma = container_of(work, struct host1x_cdma, update_work); + + mutex_lock(&cdma->lock); + update_cdma_locked(cdma); + mutex_unlock(&cdma->lock); +} + /* * Create a cdma */ @@ -499,6 +508,7 @@ int host1x_cdma_init(struct host1x_cdma *cdma) mutex_init(&cdma->lock); init_completion(&cdma->complete); + INIT_WORK(&cdma->update_work, cdma_update_work); INIT_LIST_HEAD(&cdma->sync_queue); @@ -679,7 +689,5 @@ void host1x_cdma_end(struct host1x_cdma *cdma, */ void host1x_cdma_update(struct host1x_cdma *cdma) { - mutex_lock(&cdma->lock); - update_cdma_locked(cdma); - mutex_unlock(&cdma->lock); + schedule_work(&cdma->update_work); } diff --git a/drivers/gpu/host1x/cdma.h b/drivers/gpu/host1x/cdma.h index 12c4327c4df0..7fd8168af4f9 100644 --- a/drivers/gpu/host1x/cdma.h +++ b/drivers/gpu/host1x/cdma.h @@ -11,6 +11,7 @@ #include #include #include +#include struct host1x_syncpt; struct host1x_userctx_timeout; @@ -69,6 +70,7 @@ struct host1x_cdma { struct buffer_timeout timeout; /* channel's timeout state/wq */ bool running; bool torndown; + struct work_struct update_work; }; #define cdma_to_channel(cdma) container_of(cdma, struct host1x_channel, cdma) diff --git a/drivers/gpu/host1x/hw/channel_hw.c b/drivers/gpu/host1x/hw/channel_hw.c index 732abe0750ff..8a3119fc5a77 100644 --- a/drivers/gpu/host1x/hw/channel_hw.c +++ b/drivers/gpu/host1x/hw/channel_hw.c @@ -278,6 +278,14 @@ static void channel_program_cdma(struct host1x_job *job) #endif } +static void job_complete_callback(struct dma_fence *fence, struct dma_fence_cb *cb) +{ + struct host1x_job *job = container_of(cb, struct host1x_job, fence_cb); + + /* Schedules CDMA update. */ + host1x_cdma_update(&job->channel->cdma); +} + static int channel_submit(struct host1x_job *job) { struct host1x_channel *ch = job->channel; @@ -285,7 +293,6 @@ static int channel_submit(struct host1x_job *job) u32 prev_max = 0; u32 syncval; int err; - struct host1x_waitlist *completed_waiter = NULL; struct host1x *host = dev_get_drvdata(ch->dev->parent); trace_host1x_channel_submit(dev_name(ch->dev), @@ -298,14 +305,7 @@ static int channel_submit(struct host1x_job *job) /* get submit lock */ err = mutex_lock_interruptible(&ch->submitlock); if (err) - goto error; - - completed_waiter = kzalloc(sizeof(*completed_waiter), GFP_KERNEL); - if (!completed_waiter) { - mutex_unlock(&ch->submitlock); - err = -ENOMEM; - goto error; - } + return err; host1x_channel_set_streamid(ch); host1x_enable_gather_filter(ch); @@ -315,31 +315,37 @@ static int channel_submit(struct host1x_job *job) err = host1x_cdma_begin(&ch->cdma, job); if (err) { mutex_unlock(&ch->submitlock); - goto error; + return err; } channel_program_cdma(job); syncval = host1x_syncpt_read_max(sp); + /* + * Create fence before submitting job to HW to avoid job completing + * before the fence is set up. + */ + job->fence = host1x_fence_create(sp, syncval); + if (WARN(IS_ERR(job->fence), "Failed to create submit complete fence")) { + job->fence = NULL; + } else { + err = dma_fence_add_callback(job->fence, &job->fence_cb, + job_complete_callback); + } + /* end CDMA submit & stash pinned hMems into sync queue */ host1x_cdma_end(&ch->cdma, job); trace_host1x_channel_submitted(dev_name(ch->dev), prev_max, syncval); - /* schedule a submit complete interrupt */ - err = host1x_intr_add_action(host, sp, syncval, - HOST1X_INTR_ACTION_SUBMIT_COMPLETE, ch, - completed_waiter, &job->waiter); - completed_waiter = NULL; - WARN(err, "Failed to set submit complete interrupt"); - mutex_unlock(&ch->submitlock); - return 0; + if (err == -ENOENT) + host1x_cdma_update(&ch->cdma); + else + WARN(err, "Failed to set submit complete interrupt"); -error: - kfree(completed_waiter); - return err; + return 0; } static int host1x_channel_init(struct host1x_channel *ch, struct host1x *dev, diff --git a/drivers/gpu/host1x/job.c b/drivers/gpu/host1x/job.c index b2761aa03b95..3ed49e1fd933 100644 --- a/drivers/gpu/host1x/job.c +++ b/drivers/gpu/host1x/job.c @@ -88,9 +88,15 @@ static void job_free(struct kref *ref) if (job->release) job->release(job); - if (job->waiter) - host1x_intr_put_ref(job->syncpt->host, job->syncpt->id, - job->waiter, false); + if (job->fence) { + /* + * remove_callback is atomic w.r.t. fence signaling, so + * after the call returns, we know that the callback is not + * in execution, and the fence can be safely freed. + */ + dma_fence_remove_callback(job->fence, &job->fence_cb); + dma_fence_put(job->fence); + } if (job->syncpt) host1x_syncpt_put(job->syncpt); diff --git a/include/linux/host1x.h b/include/linux/host1x.h index dc55d9d3b94f..db6cf6f34361 100644 --- a/include/linux/host1x.h +++ b/include/linux/host1x.h @@ -8,6 +8,7 @@ #include #include +#include #include #include @@ -288,8 +289,9 @@ struct host1x_job { u32 syncpt_incrs; u32 syncpt_end; - /* Completion waiter ref */ - void *waiter; + /* Completion fence for job tracking */ + struct dma_fence *fence; + struct dma_fence_cb fence_cb; /* Maximum time to wait for this job */ unsigned int timeout; From patchwork Thu Jan 19 13:09:20 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikko Perttunen X-Patchwork-Id: 13107953 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id F0604C00A5A for ; Thu, 19 Jan 2023 13:09:47 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 2D18210E923; Thu, 19 Jan 2023 13:09:40 +0000 (UTC) Received: from mail.kapsi.fi (mail.kapsi.fi [IPv6:2001:67c:1be8::25]) by gabe.freedesktop.org (Postfix) with ESMTPS id C261010E924 for ; Thu, 19 Jan 2023 13:09:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kapsi.fi; s=20161220; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=ZnC1ql9ot76j997v/IKUEMj0kO5bAhz7bOeizc9L4wU=; b=aNPH+AiRaNL5zTzcCAHkZaP4Ga PxTuBUo/pYRwFCY7ezScVs5qil72N5gZZqdcRRyzFDSzV8a8nbpq4ap8lo3vkoCk/I1A+NaG7xl95 mWmZUACmM+wzjgXYlmQDbX23DQp1R2N+/B1BKyUXykFJFFkbMnn96dXjmugwAunludKz2pZK6iBjh /pAliwWfArkuc26CyVX0viqglJjBFrxSvQRlKmOyJ14hsQvZrXA4lT4HbswFWn0e5vc+MFy4lqIQX qFrC5f2YS3WFul0ndJOctUGbXxf6hHNyhf6tQf5nVA1GVmS44g1K+uR1l7D0C1q5abiRRG1MguIJ9 wD5H1I9Q==; Received: from 91-158-25-70.elisa-laajakaista.fi ([91.158.25.70] helo=toshino.localdomain) by mail.kapsi.fi with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pIUff-000zLh-9z; Thu, 19 Jan 2023 15:09:35 +0200 From: Mikko Perttunen To: Thierry Reding , Jonathan Hunter Subject: [PATCH 3/4] gpu: host1x: Rewrite syncpoint interrupt handling Date: Thu, 19 Jan 2023 15:09:20 +0200 Message-Id: <20230119130921.1882602-3-cyndis@kapsi.fi> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230119130921.1882602-1-cyndis@kapsi.fi> References: <20230119130921.1882602-1-cyndis@kapsi.fi> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 91.158.25.70 X-SA-Exim-Mail-From: cyndis@kapsi.fi X-SA-Exim-Scanned: No (on mail.kapsi.fi); SAEximRunCond expanded to false X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-tegra@vger.kernel.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, Mikko Perttunen Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Mikko Perttunen Move from the old, complex intr handling code to a new implementation based on dma_fences. While there is a fair bit of churn to get there, the new implementation is much simpler and likely faster as well due to allowing signaling directly from interrupt context. Signed-off-by: Mikko Perttunen --- drivers/gpu/host1x/debug.c | 7 +- drivers/gpu/host1x/dev.c | 4 +- drivers/gpu/host1x/dev.h | 10 +- drivers/gpu/host1x/fence.c | 96 ++++----- drivers/gpu/host1x/fence.h | 18 +- drivers/gpu/host1x/hw/intr_hw.c | 74 ++----- drivers/gpu/host1x/intr.c | 334 ++++++-------------------------- drivers/gpu/host1x/intr.h | 83 +------- drivers/gpu/host1x/syncpt.h | 3 +- 9 files changed, 149 insertions(+), 480 deletions(-) diff --git a/drivers/gpu/host1x/debug.c b/drivers/gpu/host1x/debug.c index 6649b04b7131..a18cc8d8caf5 100644 --- a/drivers/gpu/host1x/debug.c +++ b/drivers/gpu/host1x/debug.c @@ -77,6 +77,7 @@ static int show_channel(struct host1x_channel *ch, void *data, bool show_fifo) static void show_syncpts(struct host1x *m, struct output *o, bool show_all) { + unsigned long irqflags; struct list_head *pos; unsigned int i; int err; @@ -92,10 +93,10 @@ static void show_syncpts(struct host1x *m, struct output *o, bool show_all) u32 min = host1x_syncpt_load(m->syncpt + i); unsigned int waiters = 0; - spin_lock(&m->syncpt[i].intr.lock); - list_for_each(pos, &m->syncpt[i].intr.wait_head) + spin_lock_irqsave(&m->syncpt[i].fences.lock, irqflags); + list_for_each(pos, &m->syncpt[i].fences.list) waiters++; - spin_unlock(&m->syncpt[i].intr.lock); + spin_unlock_irqrestore(&m->syncpt[i].fences.lock, irqflags); if (!kref_read(&m->syncpt[i].ref)) continue; diff --git a/drivers/gpu/host1x/dev.c b/drivers/gpu/host1x/dev.c index f31039aca03c..4872d183d860 100644 --- a/drivers/gpu/host1x/dev.c +++ b/drivers/gpu/host1x/dev.c @@ -516,7 +516,7 @@ static int host1x_probe(struct platform_device *pdev) return PTR_ERR(host->regs); } - syncpt_irq = platform_get_irq(pdev, 0); + host->syncpt_irq = platform_get_irq(pdev, 0); if (syncpt_irq < 0) return syncpt_irq; @@ -578,7 +578,7 @@ static int host1x_probe(struct platform_device *pdev) goto free_contexts; } - err = host1x_intr_init(host, syncpt_irq); + err = host1x_intr_init(host); if (err) { dev_err(&pdev->dev, "failed to initialize interrupts\n"); goto deinit_syncpt; diff --git a/drivers/gpu/host1x/dev.h b/drivers/gpu/host1x/dev.h index 920e5548cfbc..75de50fe03d0 100644 --- a/drivers/gpu/host1x/dev.h +++ b/drivers/gpu/host1x/dev.h @@ -74,8 +74,7 @@ struct host1x_syncpt_ops { }; struct host1x_intr_ops { - int (*init_host_sync)(struct host1x *host, u32 cpm, - void (*syncpt_thresh_work)(struct work_struct *work)); + int (*init_host_sync)(struct host1x *host, u32 cpm); void (*set_syncpt_threshold)( struct host1x *host, unsigned int id, u32 thresh); void (*enable_syncpt_intr)(struct host1x *host, unsigned int id); @@ -125,6 +124,7 @@ struct host1x { void __iomem *regs; void __iomem *hv_regs; /* hypervisor region */ void __iomem *common_regs; + int syncpt_irq; struct host1x_syncpt *syncpt; struct host1x_syncpt_base *bases; struct device *dev; @@ -138,7 +138,6 @@ struct host1x { dma_addr_t iova_end; struct mutex intr_mutex; - int intr_syncpt_irq; const struct host1x_syncpt_ops *syncpt_op; const struct host1x_intr_ops *intr_op; @@ -216,10 +215,9 @@ static inline void host1x_hw_syncpt_enable_protection(struct host1x *host) return host->syncpt_op->enable_protection(host); } -static inline int host1x_hw_intr_init_host_sync(struct host1x *host, u32 cpm, - void (*syncpt_thresh_work)(struct work_struct *)) +static inline int host1x_hw_intr_init_host_sync(struct host1x *host, u32 cpm) { - return host->intr_op->init_host_sync(host, cpm, syncpt_thresh_work); + return host->intr_op->init_host_sync(host, cpm); } static inline void host1x_hw_intr_set_syncpt_threshold(struct host1x *host, diff --git a/drivers/gpu/host1x/fence.c b/drivers/gpu/host1x/fence.c index df428bcbae69..df5b56692d2c 100644 --- a/drivers/gpu/host1x/fence.c +++ b/drivers/gpu/host1x/fence.c @@ -15,22 +15,6 @@ #include "intr.h" #include "syncpt.h" -static DEFINE_SPINLOCK(lock); - -struct host1x_syncpt_fence { - struct dma_fence base; - - atomic_t signaling; - - struct host1x_syncpt *sp; - u32 threshold; - - struct host1x_waitlist *waiter; - void *waiter_ref; - - struct delayed_work timeout_work; -}; - static const char *host1x_syncpt_fence_get_driver_name(struct dma_fence *f) { return "host1x"; @@ -49,11 +33,12 @@ static struct host1x_syncpt_fence *to_host1x_fence(struct dma_fence *f) static bool host1x_syncpt_fence_enable_signaling(struct dma_fence *f) { struct host1x_syncpt_fence *sf = to_host1x_fence(f); - int err; if (host1x_syncpt_is_expired(sf->sp, sf->threshold)) return false; + /* One reference for interrupt path, one for timeout path. */ + dma_fence_get(f); dma_fence_get(f); /* @@ -61,24 +46,13 @@ static bool host1x_syncpt_fence_enable_signaling(struct dma_fence *f) * reference to any fences for which 'enable_signaling' has been * called (and that have not been signalled). * - * We provide a userspace API to create arbitrary syncpoint fences, - * so we cannot normally guarantee that all fences get signalled. + * We cannot (for now) normally guarantee that all fences get signalled. * As such, setup a timeout, so that long-lasting fences will get * reaped eventually. */ schedule_delayed_work(&sf->timeout_work, msecs_to_jiffies(30000)); - err = host1x_intr_add_action(sf->sp->host, sf->sp, sf->threshold, - HOST1X_INTR_ACTION_SIGNAL_FENCE, f, - sf->waiter, &sf->waiter_ref); - if (err) { - cancel_delayed_work_sync(&sf->timeout_work); - dma_fence_put(f); - return false; - } - - /* intr framework takes ownership of waiter */ - sf->waiter = NULL; + host1x_intr_add_fence_locked(sf->sp->host, sf); /* * The fence may get signalled at any time after the above call, @@ -89,37 +63,32 @@ static bool host1x_syncpt_fence_enable_signaling(struct dma_fence *f) return true; } -static void host1x_syncpt_fence_release(struct dma_fence *f) -{ - struct host1x_syncpt_fence *sf = to_host1x_fence(f); - - if (sf->waiter) - kfree(sf->waiter); - - dma_fence_free(f); -} - static const struct dma_fence_ops host1x_syncpt_fence_ops = { .get_driver_name = host1x_syncpt_fence_get_driver_name, .get_timeline_name = host1x_syncpt_fence_get_timeline_name, .enable_signaling = host1x_syncpt_fence_enable_signaling, - .release = host1x_syncpt_fence_release, }; void host1x_fence_signal(struct host1x_syncpt_fence *f) { - if (atomic_xchg(&f->signaling, 1)) + if (atomic_xchg(&f->signaling, 1)) { + /* + * Already on timeout path, but we removed the fence before + * timeout path could, so drop interrupt path reference. + */ + dma_fence_put(&f->base); return; + } - /* - * Cancel pending timeout work - if it races, it will - * not get 'f->signaling' and return. - */ - cancel_delayed_work_sync(&f->timeout_work); - - host1x_intr_put_ref(f->sp->host, f->sp->id, f->waiter_ref, false); + if (cancel_delayed_work(&f->timeout_work)) { + /* + * We know that the timeout path will not be entered. + * Safe to drop the timeout path's reference now. + */ + dma_fence_put(&f->base); + } - dma_fence_signal(&f->base); + dma_fence_signal_locked(&f->base); dma_fence_put(&f->base); } @@ -129,17 +98,24 @@ static void do_fence_timeout(struct work_struct *work) struct host1x_syncpt_fence *f = container_of(dwork, struct host1x_syncpt_fence, timeout_work); - if (atomic_xchg(&f->signaling, 1)) + if (atomic_xchg(&f->signaling, 1)) { + /* Already on interrupt path, drop timeout path reference. */ + dma_fence_put(&f->base); return; + } - /* - * Cancel pending timeout work - if it races, it will - * not get 'f->signaling' and return. - */ - host1x_intr_put_ref(f->sp->host, f->sp->id, f->waiter_ref, true); + if (host1x_intr_remove_fence(f->sp->host, f)) { + /* + * Managed to remove fence from queue, so it's safe to drop + * the interrupt path's reference. + */ + dma_fence_put(&f->base); + } dma_fence_set_error(&f->base, -ETIMEDOUT); dma_fence_signal(&f->base); + + /* Drop timeout path reference. */ dma_fence_put(&f->base); } @@ -151,16 +127,10 @@ struct dma_fence *host1x_fence_create(struct host1x_syncpt *sp, u32 threshold) if (!fence) return ERR_PTR(-ENOMEM); - fence->waiter = kzalloc(sizeof(*fence->waiter), GFP_KERNEL); - if (!fence->waiter) { - kfree(fence); - return ERR_PTR(-ENOMEM); - } - fence->sp = sp; fence->threshold = threshold; - dma_fence_init(&fence->base, &host1x_syncpt_fence_ops, &lock, + dma_fence_init(&fence->base, &host1x_syncpt_fence_ops, &sp->fences.lock, dma_fence_context_alloc(1), 0); INIT_DELAYED_WORK(&fence->timeout_work, do_fence_timeout); diff --git a/drivers/gpu/host1x/fence.h b/drivers/gpu/host1x/fence.h index 70c91de82f14..4352d046f92c 100644 --- a/drivers/gpu/host1x/fence.h +++ b/drivers/gpu/host1x/fence.h @@ -6,7 +6,23 @@ #ifndef HOST1X_FENCE_H #define HOST1X_FENCE_H -struct host1x_syncpt_fence; +struct host1x_syncpt_fence { + struct dma_fence base; + + atomic_t signaling; + + struct host1x_syncpt *sp; + u32 threshold; + + struct delayed_work timeout_work; + + struct list_head list; +}; + +struct host1x_fence_list { + spinlock_t lock; + struct list_head list; +}; void host1x_fence_signal(struct host1x_syncpt_fence *fence); diff --git a/drivers/gpu/host1x/hw/intr_hw.c b/drivers/gpu/host1x/hw/intr_hw.c index 9acccdb139e6..b915ef7d0348 100644 --- a/drivers/gpu/host1x/hw/intr_hw.c +++ b/drivers/gpu/host1x/hw/intr_hw.c @@ -13,23 +13,6 @@ #include "../intr.h" #include "../dev.h" -/* - * Sync point threshold interrupt service function - * Handles sync point threshold triggers, in interrupt context - */ -static void host1x_intr_syncpt_handle(struct host1x_syncpt *syncpt) -{ - unsigned int id = syncpt->id; - struct host1x *host = syncpt->host; - - host1x_sync_writel(host, BIT(id % 32), - HOST1X_SYNC_SYNCPT_THRESH_INT_DISABLE(id / 32)); - host1x_sync_writel(host, BIT(id % 32), - HOST1X_SYNC_SYNCPT_THRESH_CPU0_INT_STATUS(id / 32)); - - schedule_work(&syncpt->intr.work); -} - static irqreturn_t syncpt_thresh_isr(int irq, void *dev_id) { struct host1x *host = dev_id; @@ -39,17 +22,20 @@ static irqreturn_t syncpt_thresh_isr(int irq, void *dev_id) for (i = 0; i < DIV_ROUND_UP(host->info->nb_pts, 32); i++) { reg = host1x_sync_readl(host, HOST1X_SYNC_SYNCPT_THRESH_CPU0_INT_STATUS(i)); - for_each_set_bit(id, ®, 32) { - struct host1x_syncpt *syncpt = - host->syncpt + (i * 32 + id); - host1x_intr_syncpt_handle(syncpt); - } + + host1x_sync_writel(host, reg, + HOST1X_SYNC_SYNCPT_THRESH_INT_DISABLE(i)); + host1x_sync_writel(host, reg, + HOST1X_SYNC_SYNCPT_THRESH_CPU0_INT_STATUS(i)); + + for_each_set_bit(id, ®, 32) + host1x_intr_handle_interrupt(host, i * 32 + id); } return IRQ_HANDLED; } -static void _host1x_intr_disable_all_syncpt_intrs(struct host1x *host) +static void host1x_intr_disable_all_syncpt_intrs(struct host1x *host) { unsigned int i; @@ -90,45 +76,38 @@ static void intr_hw_init(struct host1x *host, u32 cpm) } static int -_host1x_intr_init_host_sync(struct host1x *host, u32 cpm, - void (*syncpt_thresh_work)(struct work_struct *)) +host1x_intr_init_host_sync(struct host1x *host, u32 cpm) { - unsigned int i; int err; host1x_hw_intr_disable_all_syncpt_intrs(host); - for (i = 0; i < host->info->nb_pts; i++) - INIT_WORK(&host->syncpt[i].intr.work, syncpt_thresh_work); - - err = devm_request_irq(host->dev, host->intr_syncpt_irq, + err = devm_request_irq(host->dev, host->syncpt_irq, syncpt_thresh_isr, IRQF_SHARED, "host1x_syncpt", host); - if (err < 0) { - WARN_ON(1); + if (err < 0) return err; - } intr_hw_init(host, cpm); return 0; } -static void _host1x_intr_set_syncpt_threshold(struct host1x *host, +static void host1x_intr_set_syncpt_threshold(struct host1x *host, unsigned int id, u32 thresh) { host1x_sync_writel(host, thresh, HOST1X_SYNC_SYNCPT_INT_THRESH(id)); } -static void _host1x_intr_enable_syncpt_intr(struct host1x *host, +static void host1x_intr_enable_syncpt_intr(struct host1x *host, unsigned int id) { host1x_sync_writel(host, BIT(id % 32), HOST1X_SYNC_SYNCPT_THRESH_INT_ENABLE_CPU0(id / 32)); } -static void _host1x_intr_disable_syncpt_intr(struct host1x *host, +static void host1x_intr_disable_syncpt_intr(struct host1x *host, unsigned int id) { host1x_sync_writel(host, BIT(id % 32), @@ -137,23 +116,10 @@ static void _host1x_intr_disable_syncpt_intr(struct host1x *host, HOST1X_SYNC_SYNCPT_THRESH_CPU0_INT_STATUS(id / 32)); } -static int _host1x_free_syncpt_irq(struct host1x *host) -{ - unsigned int i; - - devm_free_irq(host->dev, host->intr_syncpt_irq, host); - - for (i = 0; i < host->info->nb_pts; i++) - cancel_work_sync(&host->syncpt[i].intr.work); - - return 0; -} - static const struct host1x_intr_ops host1x_intr_ops = { - .init_host_sync = _host1x_intr_init_host_sync, - .set_syncpt_threshold = _host1x_intr_set_syncpt_threshold, - .enable_syncpt_intr = _host1x_intr_enable_syncpt_intr, - .disable_syncpt_intr = _host1x_intr_disable_syncpt_intr, - .disable_all_syncpt_intrs = _host1x_intr_disable_all_syncpt_intrs, - .free_syncpt_irq = _host1x_free_syncpt_irq, + .init_host_sync = host1x_intr_init_host_sync, + .set_syncpt_threshold = host1x_intr_set_syncpt_threshold, + .enable_syncpt_intr = host1x_intr_enable_syncpt_intr, + .disable_syncpt_intr = host1x_intr_disable_syncpt_intr, + .disable_all_syncpt_intrs = host1x_intr_disable_all_syncpt_intrs, }; diff --git a/drivers/gpu/host1x/intr.c b/drivers/gpu/host1x/intr.c index 965ba21818b1..995bfa980837 100644 --- a/drivers/gpu/host1x/intr.c +++ b/drivers/gpu/host1x/intr.c @@ -2,299 +2,113 @@ /* * Tegra host1x Interrupt Management * - * Copyright (c) 2010-2013, NVIDIA Corporation. + * Copyright (c) 2010-2021, NVIDIA Corporation. */ #include -#include -#include -#include -#include -#include "channel.h" #include "dev.h" #include "fence.h" #include "intr.h" -/* Wait list management */ - -enum waitlist_state { - WLS_PENDING, - WLS_REMOVED, - WLS_CANCELLED, - WLS_HANDLED -}; - -static void waiter_release(struct kref *kref) -{ - kfree(container_of(kref, struct host1x_waitlist, refcount)); -} - -/* - * add a waiter to a waiter queue, sorted by threshold - * returns true if it was added at the head of the queue - */ -static bool add_waiter_to_queue(struct host1x_waitlist *waiter, - struct list_head *queue) -{ - struct host1x_waitlist *pos; - u32 thresh = waiter->thresh; - - list_for_each_entry_reverse(pos, queue, list) - if ((s32)(pos->thresh - thresh) <= 0) { - list_add(&waiter->list, &pos->list); - return false; - } - - list_add(&waiter->list, queue); - return true; -} - -/* - * run through a waiter queue for a single sync point ID - * and gather all completed waiters into lists by actions - */ -static void remove_completed_waiters(struct list_head *head, u32 sync, - struct list_head completed[HOST1X_INTR_ACTION_COUNT]) +static void host1x_intr_add_fence_to_list(struct host1x_fence_list *list, + struct host1x_syncpt_fence *fence) { - struct list_head *dest; - struct host1x_waitlist *waiter, *next, *prev; - - list_for_each_entry_safe(waiter, next, head, list) { - if ((s32)(waiter->thresh - sync) > 0) - break; + struct host1x_syncpt_fence *fence_in_list; - dest = completed + waiter->action; - - /* consolidate submit cleanups */ - if (waiter->action == HOST1X_INTR_ACTION_SUBMIT_COMPLETE && - !list_empty(dest)) { - prev = list_entry(dest->prev, - struct host1x_waitlist, list); - if (prev->data == waiter->data) { - prev->count++; - dest = NULL; - } + list_for_each_entry_reverse(fence_in_list, &list->list, list) { + if ((s32)(fence_in_list->threshold - fence->threshold) <= 0) { + /* Fence in list is before us, we can insert here */ + list_add(&fence->list, &fence_in_list->list); + return; } - - /* PENDING->REMOVED or CANCELLED->HANDLED */ - if (atomic_inc_return(&waiter->state) == WLS_HANDLED || !dest) { - list_del(&waiter->list); - kref_put(&waiter->refcount, waiter_release); - } else - list_move_tail(&waiter->list, dest); } -} - -static void reset_threshold_interrupt(struct host1x *host, - struct list_head *head, - unsigned int id) -{ - u32 thresh = - list_first_entry(head, struct host1x_waitlist, list)->thresh; - - host1x_hw_intr_set_syncpt_threshold(host, id, thresh); - host1x_hw_intr_enable_syncpt_intr(host, id); -} - -static void action_submit_complete(struct host1x_waitlist *waiter) -{ - struct host1x_channel *channel = waiter->data; - - host1x_cdma_update(&channel->cdma); - - /* Add nr_completed to trace */ - trace_host1x_channel_submit_complete(dev_name(channel->dev), - waiter->count, waiter->thresh); -} -static void action_wakeup(struct host1x_waitlist *waiter) -{ - wait_queue_head_t *wq = waiter->data; - - wake_up(wq); -} - -static void action_wakeup_interruptible(struct host1x_waitlist *waiter) -{ - wait_queue_head_t *wq = waiter->data; - - wake_up_interruptible(wq); + /* Add as first in list */ + list_add(&fence->list, &list->list); } -static void action_signal_fence(struct host1x_waitlist *waiter) +static void host1x_intr_update_hw_state(struct host1x *host, struct host1x_syncpt *sp) { - struct host1x_syncpt_fence *f = waiter->data; - - host1x_fence_signal(f); -} + struct host1x_syncpt_fence *fence; -typedef void (*action_handler)(struct host1x_waitlist *waiter); + if (!list_empty(&sp->fences.list)) { + fence = list_first_entry(&sp->fences.list, struct host1x_syncpt_fence, list); -static const action_handler action_handlers[HOST1X_INTR_ACTION_COUNT] = { - action_submit_complete, - action_wakeup, - action_wakeup_interruptible, - action_signal_fence, -}; - -static void run_handlers(struct list_head completed[HOST1X_INTR_ACTION_COUNT]) -{ - struct list_head *head = completed; - unsigned int i; - - for (i = 0; i < HOST1X_INTR_ACTION_COUNT; ++i, ++head) { - action_handler handler = action_handlers[i]; - struct host1x_waitlist *waiter, *next; - - list_for_each_entry_safe(waiter, next, head, list) { - list_del(&waiter->list); - handler(waiter); - WARN_ON(atomic_xchg(&waiter->state, WLS_HANDLED) != - WLS_REMOVED); - kref_put(&waiter->refcount, waiter_release); - } + host1x_hw_intr_set_syncpt_threshold(host, sp->id, fence->threshold); + host1x_hw_intr_enable_syncpt_intr(host, sp->id); + } else { + host1x_hw_intr_disable_syncpt_intr(host, sp->id); } } -/* - * Remove & handle all waiters that have completed for the given syncpt - */ -static int process_wait_list(struct host1x *host, - struct host1x_syncpt *syncpt, - u32 threshold) +void host1x_intr_add_fence_locked(struct host1x *host, struct host1x_syncpt_fence *fence) { - struct list_head completed[HOST1X_INTR_ACTION_COUNT]; - unsigned int i; - int empty; - - for (i = 0; i < HOST1X_INTR_ACTION_COUNT; ++i) - INIT_LIST_HEAD(completed + i); - - spin_lock(&syncpt->intr.lock); - - remove_completed_waiters(&syncpt->intr.wait_head, threshold, - completed); - - empty = list_empty(&syncpt->intr.wait_head); - if (empty) - host1x_hw_intr_disable_syncpt_intr(host, syncpt->id); - else - reset_threshold_interrupt(host, &syncpt->intr.wait_head, - syncpt->id); - - spin_unlock(&syncpt->intr.lock); - - run_handlers(completed); - - return empty; -} + struct host1x_fence_list *fence_list = &fence->sp->fences; -/* - * Sync point threshold interrupt service thread function - * Handles sync point threshold triggers, in thread context - */ + INIT_LIST_HEAD(&fence->list); -static void syncpt_thresh_work(struct work_struct *work) -{ - struct host1x_syncpt_intr *syncpt_intr = - container_of(work, struct host1x_syncpt_intr, work); - struct host1x_syncpt *syncpt = - container_of(syncpt_intr, struct host1x_syncpt, intr); - unsigned int id = syncpt->id; - struct host1x *host = syncpt->host; - - (void)process_wait_list(host, syncpt, - host1x_syncpt_load(host->syncpt + id)); + host1x_intr_add_fence_to_list(fence_list, fence); + host1x_intr_update_hw_state(host, fence->sp); } -int host1x_intr_add_action(struct host1x *host, struct host1x_syncpt *syncpt, - u32 thresh, enum host1x_intr_action action, - void *data, struct host1x_waitlist *waiter, - void **ref) +bool host1x_intr_remove_fence(struct host1x *host, struct host1x_syncpt_fence *fence) { - int queue_was_empty; - - if (waiter == NULL) { - pr_warn("%s: NULL waiter\n", __func__); - return -EINVAL; - } - - /* initialize a new waiter */ - INIT_LIST_HEAD(&waiter->list); - kref_init(&waiter->refcount); - if (ref) - kref_get(&waiter->refcount); - waiter->thresh = thresh; - waiter->action = action; - atomic_set(&waiter->state, WLS_PENDING); - waiter->data = data; - waiter->count = 1; - - spin_lock(&syncpt->intr.lock); + struct host1x_fence_list *fence_list = &fence->sp->fences; + unsigned long irqflags; - queue_was_empty = list_empty(&syncpt->intr.wait_head); + spin_lock_irqsave(&fence_list->lock, irqflags); - if (add_waiter_to_queue(waiter, &syncpt->intr.wait_head)) { - /* added at head of list - new threshold value */ - host1x_hw_intr_set_syncpt_threshold(host, syncpt->id, thresh); - - /* added as first waiter - enable interrupt */ - if (queue_was_empty) - host1x_hw_intr_enable_syncpt_intr(host, syncpt->id); + if (list_empty(&fence->list)) { + spin_unlock_irqrestore(&fence_list->lock, irqflags); + return false; } - if (ref) - *ref = waiter; + list_del_init(&fence->list); + host1x_intr_update_hw_state(host, fence->sp); - spin_unlock(&syncpt->intr.lock); + spin_unlock_irqrestore(&fence_list->lock, irqflags); - return 0; + return true; } -void host1x_intr_put_ref(struct host1x *host, unsigned int id, void *ref, - bool flush) +void host1x_intr_handle_interrupt(struct host1x *host, unsigned int id) { - struct host1x_waitlist *waiter = ref; - struct host1x_syncpt *syncpt; + struct host1x_syncpt *sp = &host->syncpt[id]; + struct host1x_syncpt_fence *fence, *tmp; + unsigned int value; - atomic_cmpxchg(&waiter->state, WLS_PENDING, WLS_CANCELLED); + value = host1x_syncpt_load(sp); - syncpt = host->syncpt + id; + spin_lock(&sp->fences.lock); - spin_lock(&syncpt->intr.lock); - if (atomic_cmpxchg(&waiter->state, WLS_CANCELLED, WLS_HANDLED) == - WLS_CANCELLED) { - list_del(&waiter->list); - kref_put(&waiter->refcount, waiter_release); - } - spin_unlock(&syncpt->intr.lock); + list_for_each_entry_safe(fence, tmp, &sp->fences.list, list) { + if (((value - fence->threshold) & 0x80000000U) != 0U) { + /* Fence is not yet expired, we are done */ + break; + } - if (flush) { - /* Wait until any concurrently executing handler has finished. */ - while (atomic_read(&waiter->state) != WLS_HANDLED) - schedule(); + list_del_init(&fence->list); + host1x_fence_signal(fence); } - kref_put(&waiter->refcount, waiter_release); + /* Re-enable interrupt if necessary */ + host1x_intr_update_hw_state(host, sp); + + spin_unlock(&sp->fences.lock); } -int host1x_intr_init(struct host1x *host, unsigned int irq_sync) +int host1x_intr_init(struct host1x *host) { unsigned int id; - u32 nb_pts = host1x_syncpt_nb_pts(host); mutex_init(&host->intr_mutex); - host->intr_syncpt_irq = irq_sync; - for (id = 0; id < nb_pts; ++id) { - struct host1x_syncpt *syncpt = host->syncpt + id; + for (id = 0; id < host1x_syncpt_nb_pts(host); ++id) { + struct host1x_syncpt *syncpt = &host->syncpt[id]; - spin_lock_init(&syncpt->intr.lock); - INIT_LIST_HEAD(&syncpt->intr.wait_head); - snprintf(syncpt->intr.thresh_irq_name, - sizeof(syncpt->intr.thresh_irq_name), - "host1x_sp_%02u", id); + spin_lock_init(&syncpt->fences.lock); + INIT_LIST_HEAD(&syncpt->fences.list); } return 0; @@ -310,8 +124,7 @@ void host1x_intr_start(struct host1x *host) int err; mutex_lock(&host->intr_mutex); - err = host1x_hw_intr_init_host_sync(host, DIV_ROUND_UP(hz, 1000000), - syncpt_thresh_work); + err = host1x_hw_intr_init_host_sync(host, DIV_ROUND_UP(hz, 1000000)); if (err) { mutex_unlock(&host->intr_mutex); return; @@ -321,36 +134,5 @@ void host1x_intr_start(struct host1x *host) void host1x_intr_stop(struct host1x *host) { - unsigned int id; - struct host1x_syncpt *syncpt = host->syncpt; - u32 nb_pts = host1x_syncpt_nb_pts(host); - - mutex_lock(&host->intr_mutex); - host1x_hw_intr_disable_all_syncpt_intrs(host); - - for (id = 0; id < nb_pts; ++id) { - struct host1x_waitlist *waiter, *next; - - list_for_each_entry_safe(waiter, next, - &syncpt[id].intr.wait_head, list) { - if (atomic_cmpxchg(&waiter->state, - WLS_CANCELLED, WLS_HANDLED) == WLS_CANCELLED) { - list_del(&waiter->list); - kref_put(&waiter->refcount, waiter_release); - } - } - - if (!list_empty(&syncpt[id].intr.wait_head)) { - /* output diagnostics */ - mutex_unlock(&host->intr_mutex); - pr_warn("%s cannot stop syncpt intr id=%u\n", - __func__, id); - return; - } - } - - host1x_hw_intr_free_syncpt_irq(host); - - mutex_unlock(&host->intr_mutex); } diff --git a/drivers/gpu/host1x/intr.h b/drivers/gpu/host1x/intr.h index e4c346099273..3b5610b525e5 100644 --- a/drivers/gpu/host1x/intr.h +++ b/drivers/gpu/host1x/intr.h @@ -2,87 +2,17 @@ /* * Tegra host1x Interrupt Management * - * Copyright (c) 2010-2013, NVIDIA Corporation. + * Copyright (c) 2010-2021, NVIDIA Corporation. */ #ifndef __HOST1X_INTR_H #define __HOST1X_INTR_H -#include -#include - -struct host1x_syncpt; struct host1x; - -enum host1x_intr_action { - /* - * Perform cleanup after a submit has completed. - * 'data' points to a channel - */ - HOST1X_INTR_ACTION_SUBMIT_COMPLETE = 0, - - /* - * Wake up a task. - * 'data' points to a wait_queue_head_t - */ - HOST1X_INTR_ACTION_WAKEUP, - - /* - * Wake up a interruptible task. - * 'data' points to a wait_queue_head_t - */ - HOST1X_INTR_ACTION_WAKEUP_INTERRUPTIBLE, - - HOST1X_INTR_ACTION_SIGNAL_FENCE, - - HOST1X_INTR_ACTION_COUNT -}; - -struct host1x_syncpt_intr { - spinlock_t lock; - struct list_head wait_head; - char thresh_irq_name[12]; - struct work_struct work; -}; - -struct host1x_waitlist { - struct list_head list; - struct kref refcount; - u32 thresh; - enum host1x_intr_action action; - atomic_t state; - void *data; - int count; -}; - -/* - * Schedule an action to be taken when a sync point reaches the given threshold. - * - * @id the sync point - * @thresh the threshold - * @action the action to take - * @data a pointer to extra data depending on action, see above - * @waiter waiter structure - assumes ownership - * @ref must be passed if cancellation is possible, else NULL - * - * This is a non-blocking api. - */ -int host1x_intr_add_action(struct host1x *host, struct host1x_syncpt *syncpt, - u32 thresh, enum host1x_intr_action action, - void *data, struct host1x_waitlist *waiter, - void **ref); - -/* - * Unreference an action submitted to host1x_intr_add_action(). - * You must call this if you passed non-NULL as ref. - * @ref the ref returned from host1x_intr_add_action() - * @flush wait until any pending handlers have completed before returning. - */ -void host1x_intr_put_ref(struct host1x *host, unsigned int id, void *ref, - bool flush); +struct host1x_syncpt_fence; /* Initialize host1x sync point interrupt */ -int host1x_intr_init(struct host1x *host, unsigned int irq_sync); +int host1x_intr_init(struct host1x *host); /* Deinitialize host1x sync point interrupt */ void host1x_intr_deinit(struct host1x *host); @@ -93,5 +23,10 @@ void host1x_intr_start(struct host1x *host); /* Disable host1x sync point interrupt */ void host1x_intr_stop(struct host1x *host); -irqreturn_t host1x_syncpt_thresh_fn(void *dev_id); +void host1x_intr_handle_interrupt(struct host1x *host, unsigned int id); + +void host1x_intr_add_fence_locked(struct host1x *host, struct host1x_syncpt_fence *fence); + +bool host1x_intr_remove_fence(struct host1x *host, struct host1x_syncpt_fence *fence); + #endif diff --git a/drivers/gpu/host1x/syncpt.h b/drivers/gpu/host1x/syncpt.h index 95cd29b79d6d..4c3f3b2f0e9c 100644 --- a/drivers/gpu/host1x/syncpt.h +++ b/drivers/gpu/host1x/syncpt.h @@ -14,6 +14,7 @@ #include #include +#include "fence.h" #include "intr.h" struct host1x; @@ -39,7 +40,7 @@ struct host1x_syncpt { struct host1x_syncpt_base *base; /* interrupt data */ - struct host1x_syncpt_intr intr; + struct host1x_fence_list fences; /* * If a submission incrementing this syncpoint fails, lock it so that From patchwork Thu Jan 19 13:09:21 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mikko Perttunen X-Patchwork-Id: 13107954 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 254FDC004D4 for ; Thu, 19 Jan 2023 13:09:51 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 5131410E924; Thu, 19 Jan 2023 13:09:40 +0000 (UTC) Received: from mail.kapsi.fi (mail.kapsi.fi [IPv6:2001:67c:1be8::25]) by gabe.freedesktop.org (Postfix) with ESMTPS id 099D810E242 for ; Thu, 19 Jan 2023 13:09:38 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kapsi.fi; s=20161220; h=Content-Transfer-Encoding:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To:Content-Type:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=3MUFW1CfWqbN3yWzOzLyDN6ndFRfS8BtwjxRAMF39DU=; b=LWDSh2RF65bmmQFQT5tr3uLhRb jNNBvSKwR3ffyKVZRdPfblu2B4UYAGNp2wPbJG30nHsw+MuqvmFaN2hDJ2PSd6Q42i0K6yHaMCxn1 AwGYab4uV7+ZcitFO3MbyrIuSo1Gju+XY+9Yd7eQgC99p3HJFfQkumHtXGdTreawImq5wb8WB2kd+ yIXSMRkvskllX4YTKvkyrX9SOdDELnTNgR6QSmcuucZZnFLPWFymc26UumavAH+5PcPJv8XeeXIhg eIuL5E8Qh8Oao9w2kSlD7E4XxfxOqmQzEzJXTGFcNTXgbUL7Qbzr4WO2o2eQK0SZJZQgovX/od/Iq FP7nJkpA==; Received: from 91-158-25-70.elisa-laajakaista.fi ([91.158.25.70] helo=toshino.localdomain) by mail.kapsi.fi with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1pIUff-000zLh-NW; Thu, 19 Jan 2023 15:09:35 +0200 From: Mikko Perttunen To: Thierry Reding , Jonathan Hunter Subject: [PATCH 4/4] gpu: host1x: External timeout/cancellation for fences Date: Thu, 19 Jan 2023 15:09:21 +0200 Message-Id: <20230119130921.1882602-4-cyndis@kapsi.fi> X-Mailer: git-send-email 2.39.0 In-Reply-To: <20230119130921.1882602-1-cyndis@kapsi.fi> References: <20230119130921.1882602-1-cyndis@kapsi.fi> MIME-Version: 1.0 X-SA-Exim-Connect-IP: 91.158.25.70 X-SA-Exim-Mail-From: cyndis@kapsi.fi X-SA-Exim-Scanned: No (on mail.kapsi.fi); SAEximRunCond expanded to false X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: linux-tegra@vger.kernel.org, linux-kernel@vger.kernel.org, dri-devel@lists.freedesktop.org, Mikko Perttunen Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Mikko Perttunen Currently all fences have a 30 second timeout to ensure they are cleaned up if the fence never completes otherwise. However, this one size fits all solution doesn't actually fit in every case, such as syncpoint waiting where we want to be able to have timeouts longer than 30 seconds. As such, we want to be able to give control over fence cancellation to the caller (and maybe eventually get rid of the internal timeout altogether). Here we add this cancellation mechanism by essentially adding a function for entering the timeout path by function call, and changing the syncpoint wait function to use it. Signed-off-by: Mikko Perttunen --- drivers/gpu/drm/tegra/submit.c | 2 +- drivers/gpu/host1x/fence.c | 40 ++++++++++++++++++++---------- drivers/gpu/host1x/fence.h | 1 + drivers/gpu/host1x/hw/channel_hw.c | 2 +- drivers/gpu/host1x/syncpt.c | 4 ++- include/linux/host1x.h | 4 ++- 6 files changed, 36 insertions(+), 17 deletions(-) diff --git a/drivers/gpu/drm/tegra/submit.c b/drivers/gpu/drm/tegra/submit.c index 066f88564169..f4688fcafe93 100644 --- a/drivers/gpu/drm/tegra/submit.c +++ b/drivers/gpu/drm/tegra/submit.c @@ -654,7 +654,7 @@ int tegra_drm_ioctl_channel_submit(struct drm_device *drm, void *data, args->syncpt.value = job->syncpt_end; if (syncobj) { - struct dma_fence *fence = host1x_fence_create(job->syncpt, job->syncpt_end); + struct dma_fence *fence = host1x_fence_create(job->syncpt, job->syncpt_end, true); if (IS_ERR(fence)) { err = PTR_ERR(fence); SUBMIT_ERR(context, "failed to create postfence: %d", err); diff --git a/drivers/gpu/host1x/fence.c b/drivers/gpu/host1x/fence.c index df5b56692d2c..139ad1afd935 100644 --- a/drivers/gpu/host1x/fence.c +++ b/drivers/gpu/host1x/fence.c @@ -37,8 +37,7 @@ static bool host1x_syncpt_fence_enable_signaling(struct dma_fence *f) if (host1x_syncpt_is_expired(sf->sp, sf->threshold)) return false; - /* One reference for interrupt path, one for timeout path. */ - dma_fence_get(f); + /* Reference for interrupt path. */ dma_fence_get(f); /* @@ -46,11 +45,15 @@ static bool host1x_syncpt_fence_enable_signaling(struct dma_fence *f) * reference to any fences for which 'enable_signaling' has been * called (and that have not been signalled). * - * We cannot (for now) normally guarantee that all fences get signalled. - * As such, setup a timeout, so that long-lasting fences will get - * reaped eventually. + * We cannot currently always guarantee that all fences get signalled + * or cancelled. As such, for such situations, set up a timeout, so + * that long-lasting fences will get reaped eventually. */ - schedule_delayed_work(&sf->timeout_work, msecs_to_jiffies(30000)); + if (sf->timeout) { + /* Reference for timeout path. */ + dma_fence_get(f); + schedule_delayed_work(&sf->timeout_work, msecs_to_jiffies(30000)); + } host1x_intr_add_fence_locked(sf->sp->host, sf); @@ -80,7 +83,7 @@ void host1x_fence_signal(struct host1x_syncpt_fence *f) return; } - if (cancel_delayed_work(&f->timeout_work)) { + if (f->timeout && cancel_delayed_work(&f->timeout_work)) { /* * We know that the timeout path will not be entered. * Safe to drop the timeout path's reference now. @@ -99,8 +102,9 @@ static void do_fence_timeout(struct work_struct *work) container_of(dwork, struct host1x_syncpt_fence, timeout_work); if (atomic_xchg(&f->signaling, 1)) { - /* Already on interrupt path, drop timeout path reference. */ - dma_fence_put(&f->base); + /* Already on interrupt path, drop timeout path reference if any. */ + if (f->timeout) + dma_fence_put(&f->base); return; } @@ -114,12 +118,12 @@ static void do_fence_timeout(struct work_struct *work) dma_fence_set_error(&f->base, -ETIMEDOUT); dma_fence_signal(&f->base); - - /* Drop timeout path reference. */ - dma_fence_put(&f->base); + if (f->timeout) + dma_fence_put(&f->base); } -struct dma_fence *host1x_fence_create(struct host1x_syncpt *sp, u32 threshold) +struct dma_fence *host1x_fence_create(struct host1x_syncpt *sp, u32 threshold, + bool timeout) { struct host1x_syncpt_fence *fence; @@ -129,6 +133,7 @@ struct dma_fence *host1x_fence_create(struct host1x_syncpt *sp, u32 threshold) fence->sp = sp; fence->threshold = threshold; + fence->timeout = timeout; dma_fence_init(&fence->base, &host1x_syncpt_fence_ops, &sp->fences.lock, dma_fence_context_alloc(1), 0); @@ -138,3 +143,12 @@ struct dma_fence *host1x_fence_create(struct host1x_syncpt *sp, u32 threshold) return &fence->base; } EXPORT_SYMBOL(host1x_fence_create); + +void host1x_fence_cancel(struct dma_fence *f) +{ + struct host1x_syncpt_fence *sf = to_host1x_fence(f); + + schedule_delayed_work(&sf->timeout_work, 0); + flush_delayed_work(&sf->timeout_work); +} +EXPORT_SYMBOL(host1x_fence_cancel); diff --git a/drivers/gpu/host1x/fence.h b/drivers/gpu/host1x/fence.h index 4352d046f92c..f3c644c73cad 100644 --- a/drivers/gpu/host1x/fence.h +++ b/drivers/gpu/host1x/fence.h @@ -13,6 +13,7 @@ struct host1x_syncpt_fence { struct host1x_syncpt *sp; u32 threshold; + bool timeout; struct delayed_work timeout_work; diff --git a/drivers/gpu/host1x/hw/channel_hw.c b/drivers/gpu/host1x/hw/channel_hw.c index 8a3119fc5a77..0a528573a792 100644 --- a/drivers/gpu/host1x/hw/channel_hw.c +++ b/drivers/gpu/host1x/hw/channel_hw.c @@ -325,7 +325,7 @@ static int channel_submit(struct host1x_job *job) * Create fence before submitting job to HW to avoid job completing * before the fence is set up. */ - job->fence = host1x_fence_create(sp, syncval); + job->fence = host1x_fence_create(sp, syncval, true); if (WARN(IS_ERR(job->fence), "Failed to create submit complete fence")) { job->fence = NULL; } else { diff --git a/drivers/gpu/host1x/syncpt.c b/drivers/gpu/host1x/syncpt.c index 75f58ec2ae23..2d2007760eac 100644 --- a/drivers/gpu/host1x/syncpt.c +++ b/drivers/gpu/host1x/syncpt.c @@ -236,11 +236,13 @@ int host1x_syncpt_wait(struct host1x_syncpt *sp, u32 thresh, long timeout, else if (timeout == 0) return -EAGAIN; - fence = host1x_fence_create(sp, thresh); + fence = host1x_fence_create(sp, thresh, false); if (IS_ERR(fence)) return PTR_ERR(fence); wait_err = dma_fence_wait_timeout(fence, true, timeout); + if (wait_err == 0) + host1x_fence_cancel(fence); dma_fence_put(fence); if (value) diff --git a/include/linux/host1x.h b/include/linux/host1x.h index db6cf6f34361..9a9de4b97a25 100644 --- a/include/linux/host1x.h +++ b/include/linux/host1x.h @@ -222,7 +222,9 @@ u32 host1x_syncpt_base_id(struct host1x_syncpt_base *base); void host1x_syncpt_release_vblank_reservation(struct host1x_client *client, u32 syncpt_id); -struct dma_fence *host1x_fence_create(struct host1x_syncpt *sp, u32 threshold); +struct dma_fence *host1x_fence_create(struct host1x_syncpt *sp, u32 threshold, + bool timeout); +void host1x_fence_cancel(struct dma_fence *fence); /* * host1x channel