From patchwork Mon Sep 18 14:31:49 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oded Gabbay X-Patchwork-Id: 13389713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 88B4FCD37B0 for ; Mon, 18 Sep 2023 14:32:10 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DD0A210E04D; Mon, 18 Sep 2023 14:32:09 +0000 (UTC) Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by gabe.freedesktop.org (Postfix) with ESMTPS id 6583210E04D for ; Mon, 18 Sep 2023 14:32:07 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 678B8B80E3D; Mon, 18 Sep 2023 14:32:05 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1A4CFC32788; Mon, 18 Sep 2023 14:32:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695047524; bh=KWXNEqRTOD4JwIUMnjqhect4NpZR1NVv+98WQbVwsYM=; h=From:To:Cc:Subject:Date:From; b=Be4/FI1GjUVv0ePGd9MD0fqS7DYnI7O4cOlqFGAy0jnWuJDdTOwI8UnTFu6vN1oSB CxSsF98S0A31rIVV4DevEUaWCvE3nxvIbmsm61QWps3p7ShJ0JOZcHc55NCmrQSipy mqrftwWk0xzG9rUAXGR7+JruTKph0VlD/xPRvwRc4OrbqvNPsCB6EOld+KkYM2REBW O1vEXg+SXn7Xw/CtZOwhV3BwmCTCoCYbMNV1EvzGwqNLo7wwgbMJRLIe9FdMbbQ1Gt Tfc1rbmQQ7qwA3kMAzEh+7ri68IjoOGnwFSK48Z+mmPxIcXVq2gq/sthS/j2AY7Nb8 MVXeCGFPx90xw== From: Oded Gabbay To: dri-devel@lists.freedesktop.org Subject: [PATCH 01/10] accel/habanalabs: fix bug in timestamp interrupt handling Date: Mon, 18 Sep 2023 17:31:49 +0300 Message-Id: <20230918143158.903207-1-ogabbay@kernel.org> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: farah kassabri Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: farah kassabri There is a potential race between user thread seeking to re-use a timestamp record with new interrupt id, while this record is still in the middle of interrupt handling and it is about to be freed. Imagine the driver set the record in_use to 0 and only then fill the free_node information. This might lead to unpleasant scenario where the new registration thread detects the record as free to use, and change the cq buff address. That will cause the free_node to get the wrong buffer address to put refcount to. Signed-off-by: farah kassabri Reviewed-by: Oded Gabbay Signed-off-by: Oded Gabbay --- drivers/accel/habanalabs/common/irq.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/drivers/accel/habanalabs/common/irq.c b/drivers/accel/habanalabs/common/irq.c index f6b6c54bc868..058f27040805 100644 --- a/drivers/accel/habanalabs/common/irq.c +++ b/drivers/accel/habanalabs/common/irq.c @@ -259,8 +259,6 @@ static int handle_registration_node(struct hl_device *hdev, struct hl_user_pendi dev_dbg(hdev->dev, "Irq handle: Timestamp record (%p) ts cb address (%p), interrupt_id: %u\n", pend, pend->ts_reg_info.timestamp_kernel_addr, interrupt_id); - /* Mark kernel CB node as free */ - pend->ts_reg_info.in_use = false; list_del(&pend->wait_list_node); /* Putting the refcount for ts_buff and cq_cb objects will be handled @@ -270,6 +268,9 @@ static int handle_registration_node(struct hl_device *hdev, struct hl_user_pendi free_node->cq_cb = pend->ts_reg_info.cq_cb; list_add(&free_node->free_objects_node, *free_list); + /* Mark TS record as free */ + pend->ts_reg_info.in_use = false; + return 0; } From patchwork Mon Sep 18 14:31:50 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oded Gabbay X-Patchwork-Id: 13389715 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6DBC3C46CA1 for ; Mon, 18 Sep 2023 14:32:17 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 81E1B10E18C; Mon, 18 Sep 2023 14:32:13 +0000 (UTC) Received: from ams.source.kernel.org (ams.source.kernel.org [IPv6:2604:1380:4601:e00::1]) by gabe.freedesktop.org (Postfix) with ESMTPS id D6D9210E04D for ; Mon, 18 Sep 2023 14:32:08 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 35B68B80E9F; Mon, 18 Sep 2023 14:32:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 83A16C3278A; Mon, 18 Sep 2023 14:32:04 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695047525; bh=0S4QcskDYEILA7KkaPHsta8Hjw18p8gI5+R+R6/O6/0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=WaAGEHsVGmlJpGKlK2ZtrnOqC4tjuvBlaavjfkgkld+iYoxbty5AYFDLUq78QzxFY 7vA8ijJsYQoVSmZyPPhFhUw5WUpk1jQdRjwt6wr0hDT8SGrJRmuY2s821gqUiHodWJ iPGv/SjLyv5s6MhMkVzlISU7OlKRiVArxmxfXlozqBare8YX43fIj5NuTE86/V1q3+ cF2sWlvPn8+Zg5S9k2e+Tfni8RU8rWmB2sz6M4OxUrFQeimCd0Q3OZ/sH5olVGta4l cG4faBIjfukYMbU0O6V04nifo8Z8l2uhf31xcC8qJ2/B7BpqDWhq8Ud5ET4LzvK83T tsqhNKkS9L5pw== From: Oded Gabbay To: dri-devel@lists.freedesktop.org Subject: [PATCH 02/10] accel/habanalabs: optimize timestamp registration handler Date: Mon, 18 Sep 2023 17:31:50 +0300 Message-Id: <20230918143158.903207-2-ogabbay@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230918143158.903207-1-ogabbay@kernel.org> References: <20230918143158.903207-1-ogabbay@kernel.org> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tomer Tayar , farah kassabri Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: farah kassabri Currently we use dynamic allocation inside the irq handler in order to allocate free node to be used for the free jobs. This operation is expensive, especially when we deal with large burst of events records that get released at the same time. The alternative is to have pre allocated pool of free nodes and just fetch nodes from this pool at irq handling time instead of allocating them. In case the pool becomes full, then the driver will fallback to dynamic allocations. As part of the optimization also update the unregister flow upon re-using a timestamp record, by making the operation much simpler and quicker. We already have the record in the registration flow and now we just seek to re-use with different interrupt. Therefore, no need to look for buffer according to the user handle. Signed-off-by: farah kassabri Reviewed-by: Tomer Tayar Signed-off-by: Oded Gabbay --- .../habanalabs/common/command_submission.c | 83 +++++++------------ drivers/accel/habanalabs/common/device.c | 59 ++++++++++++- drivers/accel/habanalabs/common/habanalabs.h | 27 ++++++ drivers/accel/habanalabs/common/irq.c | 77 ++++++++++++++--- 4 files changed, 179 insertions(+), 67 deletions(-) diff --git a/drivers/accel/habanalabs/common/command_submission.c b/drivers/accel/habanalabs/common/command_submission.c index daaa3cbe0b12..02049bd26356 100644 --- a/drivers/accel/habanalabs/common/command_submission.c +++ b/drivers/accel/habanalabs/common/command_submission.c @@ -3246,60 +3246,29 @@ static int validate_and_get_ts_record(struct device *dev, return 0; } -static int unregister_timestamp_node(struct hl_device *hdev, struct hl_ctx *ctx, - struct hl_mem_mgr *mmg, u64 ts_handle, u64 ts_offset, - struct hl_user_interrupt *interrupt) +static void unregister_timestamp_node(struct hl_device *hdev, + struct hl_user_pending_interrupt *record, bool need_lock) { - struct hl_user_pending_interrupt *req_event_record, *pend, *temp_pend; - struct hl_mmap_mem_buf *buff; - struct hl_ts_buff *ts_buff; + struct hl_user_interrupt *interrupt = record->ts_reg_info.interrupt; bool ts_rec_found = false; - int rc; - - buff = hl_mmap_mem_buf_get(mmg, ts_handle); - if (!buff) { - dev_err(hdev->dev, "invalid TS buff handle!\n"); - return -EINVAL; - } - - ts_buff = buff->private; - - rc = validate_and_get_ts_record(hdev->dev, ts_buff, ts_offset, &req_event_record); - if (rc) - goto put_buf; - /* - * Note: we don't use the ts in_use field here, but we rather scan the list - * because we cannot rely on the user to keep the order of register/unregister calls - * and since we might have races here all the time between the irq and register/unregister - * calls so it safer to lock the list and scan it to find the node. - * If the node found on the list we mark it as not in use and delete it from the list, - * if it's not here then the node was handled already in the irq before we get into - * this ioctl. - */ - spin_lock(&interrupt->wait_list_lock); + if (need_lock) + spin_lock(&interrupt->wait_list_lock); - list_for_each_entry_safe(pend, temp_pend, &interrupt->wait_list_head, wait_list_node) { - if (pend == req_event_record) { - pend->ts_reg_info.in_use = false; - list_del(&pend->wait_list_node); - ts_rec_found = true; - break; - } + if (record->ts_reg_info.in_use) { + record->ts_reg_info.in_use = false; + list_del(&record->wait_list_node); + ts_rec_found = true; } - spin_unlock(&interrupt->wait_list_lock); + if (need_lock) + spin_unlock(&interrupt->wait_list_lock); /* Put refcounts that were taken when we registered the event */ if (ts_rec_found) { - hl_mmap_mem_buf_put(pend->ts_reg_info.buf); - hl_cb_put(pend->ts_reg_info.cq_cb); + hl_mmap_mem_buf_put(record->ts_reg_info.buf); + hl_cb_put(record->ts_reg_info.cq_cb); } - -put_buf: - hl_mmap_mem_buf_put(buff); - - return rc; } static int ts_get_and_handle_kernel_record(struct hl_device *hdev, struct hl_ctx *ctx, @@ -3308,6 +3277,7 @@ static int ts_get_and_handle_kernel_record(struct hl_device *hdev, struct hl_ctx { struct hl_user_pending_interrupt *req_offset_record; struct hl_ts_buff *ts_buff = data->buf->private; + bool need_lock = false; int rc; rc = validate_and_get_ts_record(data->buf->mmg->dev, ts_buff, data->ts_offset, @@ -3315,21 +3285,30 @@ static int ts_get_and_handle_kernel_record(struct hl_device *hdev, struct hl_ctx if (rc) return rc; - /* In case the node already registered, need to unregister first then re-use*/ + /* In case the node already registered, need to unregister first then re-use */ if (req_offset_record->ts_reg_info.in_use) { dev_dbg(data->buf->mmg->dev, - "Requested ts offset(%llx) is in use, unregister first\n", - data->ts_offset); + "Requested record %p is in use on irq: %u ts addr: %p, unregister first then put on irq: %u\n", + req_offset_record, + req_offset_record->ts_reg_info.interrupt->interrupt_id, + req_offset_record->ts_reg_info.timestamp_kernel_addr, + data->interrupt->interrupt_id); /* * Since interrupt here can be different than the one the node currently registered - * on, and we don't wan't to lock two lists while we're doing unregister, so + * on, and we don't want to lock two lists while we're doing unregister, so * unlock the new interrupt wait list here and acquire the lock again after you done */ - spin_unlock_irqrestore(&data->interrupt->wait_list_lock, data->flags); + if (data->interrupt->interrupt_id != + req_offset_record->ts_reg_info.interrupt->interrupt_id) { - unregister_timestamp_node(hdev, ctx, data->mmg, data->ts_handle, - data->ts_offset, req_offset_record->ts_reg_info.interrupt); - spin_lock_irqsave(&data->interrupt->wait_list_lock, data->flags); + need_lock = true; + spin_unlock(&data->interrupt->wait_list_lock); + } + + unregister_timestamp_node(hdev, req_offset_record, need_lock); + + if (need_lock) + spin_lock(&data->interrupt->wait_list_lock); } /* Fill up the new registration node info and add it to the list */ diff --git a/drivers/accel/habanalabs/common/device.c b/drivers/accel/habanalabs/common/device.c index 13f14b80a7d4..d31f228533c0 100644 --- a/drivers/accel/habanalabs/common/device.c +++ b/drivers/accel/habanalabs/common/device.c @@ -2033,7 +2033,9 @@ void hl_notifier_event_send_all(struct hl_device *hdev, u64 event_mask) int hl_device_init(struct hl_device *hdev) { int i, rc, cq_cnt, user_interrupt_cnt, cq_ready_cnt; + struct hl_ts_free_jobs *free_jobs_data; bool expose_interfaces_on_err = false; + void *p; /* Initialize ASIC function pointers and perform early init */ rc = device_early_init(hdev); @@ -2050,15 +2052,43 @@ int hl_device_init(struct hl_device *hdev) rc = -ENOMEM; goto early_fini; } + + /* Timestamp records supported only if CQ supported in device */ + if (hdev->asic_prop.first_available_cq[0] != USHRT_MAX) { + for (i = 0 ; i < user_interrupt_cnt ; i++) { + p = vzalloc(TIMESTAMP_FREE_NODES_NUM * + sizeof(struct timestamp_reg_free_node)); + if (!p) { + rc = -ENOMEM; + goto free_usr_intr_mem; + } + free_jobs_data = &hdev->user_interrupt[i].ts_free_jobs_data; + free_jobs_data->free_nodes_pool = p; + free_jobs_data->free_nodes_length = TIMESTAMP_FREE_NODES_NUM; + free_jobs_data->next_avail_free_node_idx = 0; + } + } + } + + free_jobs_data = &hdev->common_user_cq_interrupt.ts_free_jobs_data; + p = vzalloc(TIMESTAMP_FREE_NODES_NUM * + sizeof(struct timestamp_reg_free_node)); + if (!p) { + rc = -ENOMEM; + goto free_usr_intr_mem; } + free_jobs_data->free_nodes_pool = p; + free_jobs_data->free_nodes_length = TIMESTAMP_FREE_NODES_NUM; + free_jobs_data->next_avail_free_node_idx = 0; + /* * Start calling ASIC initialization. First S/W then H/W and finally * late init */ rc = hdev->asic_funcs->sw_init(hdev); if (rc) - goto free_usr_intr_mem; + goto free_common_usr_intr_mem; /* initialize completion structure for multi CS wait */ @@ -2297,8 +2327,17 @@ int hl_device_init(struct hl_device *hdev) hl_hw_queues_destroy(hdev); sw_fini: hdev->asic_funcs->sw_fini(hdev); +free_common_usr_intr_mem: + vfree(hdev->common_user_cq_interrupt.ts_free_jobs_data.free_nodes_pool); free_usr_intr_mem: - kfree(hdev->user_interrupt); + if (user_interrupt_cnt) { + for (i = 0 ; i < user_interrupt_cnt ; i++) { + if (!hdev->user_interrupt[i].ts_free_jobs_data.free_nodes_pool) + break; + vfree(hdev->user_interrupt[i].ts_free_jobs_data.free_nodes_pool); + } + kfree(hdev->user_interrupt); + } early_fini: device_early_fini(hdev); out_disabled: @@ -2323,6 +2362,7 @@ int hl_device_init(struct hl_device *hdev) */ void hl_device_fini(struct hl_device *hdev) { + u32 user_interrupt_cnt; bool device_in_reset; ktime_t timeout; u64 reset_sec; @@ -2449,7 +2489,20 @@ void hl_device_fini(struct hl_device *hdev) for (i = 0 ; i < hdev->asic_prop.completion_queues_count ; i++) hl_cq_fini(hdev, &hdev->completion_queue[i]); kfree(hdev->completion_queue); - kfree(hdev->user_interrupt); + + user_interrupt_cnt = hdev->asic_prop.user_dec_intr_count + + hdev->asic_prop.user_interrupt_count; + + if (user_interrupt_cnt) { + if (hdev->asic_prop.first_available_cq[0] != USHRT_MAX) { + for (i = 0 ; i < user_interrupt_cnt ; i++) + vfree(hdev->user_interrupt[i].ts_free_jobs_data.free_nodes_pool); + } + + kfree(hdev->user_interrupt); + } + + vfree(hdev->common_user_cq_interrupt.ts_free_jobs_data.free_nodes_pool); hl_hw_queues_destroy(hdev); diff --git a/drivers/accel/habanalabs/common/habanalabs.h b/drivers/accel/habanalabs/common/habanalabs.h index 4c5d55c9109d..1342686d0ce5 100644 --- a/drivers/accel/habanalabs/common/habanalabs.h +++ b/drivers/accel/habanalabs/common/habanalabs.h @@ -106,6 +106,8 @@ struct hl_fpriv; /* MMU */ #define MMU_HASH_TABLE_BITS 7 /* 1 << 7 buckets */ +#define TIMESTAMP_FREE_NODES_NUM 512 + /** * enum hl_mmu_page_table_location - mmu page table location * @MMU_DR_PGT: page-table is located on device DRAM. @@ -1104,9 +1106,26 @@ enum hl_user_interrupt_type { HL_USR_INTERRUPT_UNEXPECTED }; +/** + * struct hl_ts_free_jobs - holds user interrupt ts free nodes related data + * @free_nodes_pool: pool of nodes to be used for free timestamp jobs + * @free_nodes_length: number of nodes in free_nodes_pool + * @next_avail_free_node_idx: index of the next free node in the pool + * + * the free nodes pool must be protected by the user interrupt lock + * to avoid race between different interrupts which are using the same + * ts buffer with different offsets. + */ +struct hl_ts_free_jobs { + struct timestamp_reg_free_node *free_nodes_pool; + u32 free_nodes_length; + u32 next_avail_free_node_idx; +}; + /** * struct hl_user_interrupt - holds user interrupt information * @hdev: pointer to the device structure + * @ts_free_jobs_data: timestamp free jobs related data * @type: user interrupt type * @wait_list_head: head to the list of user threads pending on this interrupt * @wait_list_lock: protects wait_list_head @@ -1115,6 +1134,7 @@ enum hl_user_interrupt_type { */ struct hl_user_interrupt { struct hl_device *hdev; + struct hl_ts_free_jobs ts_free_jobs_data; enum hl_user_interrupt_type type; struct list_head wait_list_head; spinlock_t wait_list_lock; @@ -1127,11 +1147,15 @@ struct hl_user_interrupt { * @free_objects_node: node in the list free_obj_jobs * @cq_cb: pointer to cq command buffer to be freed * @buf: pointer to timestamp buffer to be freed + * @in_use: indicates whether the node still in use in workqueue thread. + * @dynamic_alloc: indicates whether the node was allocated dynamically in the interrupt handler */ struct timestamp_reg_free_node { struct list_head free_objects_node; struct hl_cb *cq_cb; struct hl_mmap_mem_buf *buf; + atomic_t in_use; + u8 dynamic_alloc; }; /* struct timestamp_reg_work_obj - holds the timestamp registration free objects job @@ -1140,11 +1164,14 @@ struct timestamp_reg_free_node { * @free_obj: workqueue object to free timestamp registration node objects * @hdev: pointer to the device structure * @free_obj_head: list of free jobs nodes (node type timestamp_reg_free_node) + * @dynamic_alloc_free_obj_head: list of free jobs nodes which were dynamically allocated in the + * interrupt handler. */ struct timestamp_reg_work_obj { struct work_struct free_obj; struct hl_device *hdev; struct list_head *free_obj_head; + struct list_head *dynamic_alloc_free_obj_head; }; /* struct timestamp_reg_info - holds the timestamp registration related data. diff --git a/drivers/accel/habanalabs/common/irq.c b/drivers/accel/habanalabs/common/irq.c index 058f27040805..0947d286a5ab 100644 --- a/drivers/accel/habanalabs/common/irq.c +++ b/drivers/accel/habanalabs/common/irq.c @@ -204,8 +204,10 @@ static void hl_ts_free_objects(struct work_struct *work) { struct timestamp_reg_work_obj *job = container_of(work, struct timestamp_reg_work_obj, free_obj); + struct list_head *dynamic_alloc_free_list_head = job->dynamic_alloc_free_obj_head; struct timestamp_reg_free_node *free_obj, *temp_free_obj; struct list_head *free_list_head = job->free_obj_head; + struct hl_device *hdev = job->hdev; list_for_each_entry_safe(free_obj, temp_free_obj, free_list_head, free_objects_node) { @@ -215,10 +217,28 @@ static void hl_ts_free_objects(struct work_struct *work) hl_mmap_mem_buf_put(free_obj->buf); hl_cb_put(free_obj->cq_cb); - kfree(free_obj); + atomic_set(&free_obj->in_use, 0); } kfree(free_list_head); + + if (dynamic_alloc_free_list_head) { + list_for_each_entry_safe(free_obj, temp_free_obj, dynamic_alloc_free_list_head, + free_objects_node) { + dev_dbg(hdev->dev, + "Dynamic_Alloc list: About to put refcount to buf (%p) cq_cb(%p)\n", + free_obj->buf, + free_obj->cq_cb); + + hl_mmap_mem_buf_put(free_obj->buf); + hl_cb_put(free_obj->cq_cb); + list_del(&free_obj->free_objects_node); + kfree(free_obj); + } + + kfree(dynamic_alloc_free_list_head); + } + kfree(job); } @@ -233,12 +253,18 @@ static void hl_ts_free_objects(struct work_struct *work) * list to a dedicated workqueue to do the actual put. */ static int handle_registration_node(struct hl_device *hdev, struct hl_user_pending_interrupt *pend, - struct list_head **free_list, ktime_t now, - u32 interrupt_id) + struct list_head **free_list, + struct list_head **dynamic_alloc_list, + struct hl_user_interrupt *intr) { + struct hl_ts_free_jobs *ts_free_jobs_data; struct timestamp_reg_free_node *free_node; + u32 free_node_index; u64 timestamp; + ts_free_jobs_data = &intr->ts_free_jobs_data; + free_node_index = ts_free_jobs_data->next_avail_free_node_idx; + if (!(*free_list)) { /* Alloc/Init the timestamp registration free objects list */ *free_list = kmalloc(sizeof(struct list_head), GFP_ATOMIC); @@ -248,16 +274,35 @@ static int handle_registration_node(struct hl_device *hdev, struct hl_user_pendi INIT_LIST_HEAD(*free_list); } - free_node = kmalloc(sizeof(*free_node), GFP_ATOMIC); - if (!free_node) - return -ENOMEM; + free_node = &ts_free_jobs_data->free_nodes_pool[free_node_index]; + if (atomic_cmpxchg(&free_node->in_use, 0, 1)) { + dev_dbg(hdev->dev, + "Timestamp free node pool is full, buff: %p, record: %p, irq: %u\n", + pend->ts_reg_info.buf, + pend, + intr->interrupt_id); - timestamp = ktime_to_ns(now); + if (!(*dynamic_alloc_list)) { + *dynamic_alloc_list = kmalloc(sizeof(struct list_head), GFP_ATOMIC); + if (!(*dynamic_alloc_list)) + return -ENOMEM; + + INIT_LIST_HEAD(*dynamic_alloc_list); + } + + free_node = kmalloc(sizeof(struct timestamp_reg_free_node), GFP_ATOMIC); + if (!free_node) + return -ENOMEM; + + free_node->dynamic_alloc = 1; + } + + timestamp = ktime_to_ns(intr->timestamp); *pend->ts_reg_info.timestamp_kernel_addr = timestamp; dev_dbg(hdev->dev, "Irq handle: Timestamp record (%p) ts cb address (%p), interrupt_id: %u\n", - pend, pend->ts_reg_info.timestamp_kernel_addr, interrupt_id); + pend, pend->ts_reg_info.timestamp_kernel_addr, intr->interrupt_id); list_del(&pend->wait_list_node); @@ -266,7 +311,14 @@ static int handle_registration_node(struct hl_device *hdev, struct hl_user_pendi */ free_node->buf = pend->ts_reg_info.buf; free_node->cq_cb = pend->ts_reg_info.cq_cb; - list_add(&free_node->free_objects_node, *free_list); + + if (free_node->dynamic_alloc) { + list_add(&free_node->free_objects_node, *dynamic_alloc_list); + } else { + ts_free_jobs_data->next_avail_free_node_idx = + (++free_node_index) % ts_free_jobs_data->free_nodes_length; + list_add(&free_node->free_objects_node, *free_list); + } /* Mark TS record as free */ pend->ts_reg_info.in_use = false; @@ -276,8 +328,8 @@ static int handle_registration_node(struct hl_device *hdev, struct hl_user_pendi static void handle_user_interrupt(struct hl_device *hdev, struct hl_user_interrupt *intr) { + struct list_head *ts_reg_free_list_head = NULL, *dynamic_alloc_list_head = NULL; struct hl_user_pending_interrupt *pend, *temp_pend; - struct list_head *ts_reg_free_list_head = NULL; struct timestamp_reg_work_obj *job; bool reg_node_handle_fail = false; int rc; @@ -303,8 +355,8 @@ static void handle_user_interrupt(struct hl_device *hdev, struct hl_user_interru if (pend->ts_reg_info.buf) { if (!reg_node_handle_fail) { rc = handle_registration_node(hdev, pend, - &ts_reg_free_list_head, intr->timestamp, - intr->interrupt_id); + &ts_reg_free_list_head, + &dynamic_alloc_list_head, intr); if (rc) reg_node_handle_fail = true; } @@ -320,6 +372,7 @@ static void handle_user_interrupt(struct hl_device *hdev, struct hl_user_interru if (ts_reg_free_list_head) { INIT_WORK(&job->free_obj, hl_ts_free_objects); job->free_obj_head = ts_reg_free_list_head; + job->dynamic_alloc_free_obj_head = dynamic_alloc_list_head; job->hdev = hdev; queue_work(hdev->ts_free_obj_wq, &job->free_obj); } else { From patchwork Mon Sep 18 14:31:51 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oded Gabbay X-Patchwork-Id: 13389714 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 31213CD37B0 for ; Mon, 18 Sep 2023 14:32:14 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DC50210E0A8; Mon, 18 Sep 2023 14:32:12 +0000 (UTC) Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by gabe.freedesktop.org (Postfix) with ESMTPS id DC63F10E0A8 for ; Mon, 18 Sep 2023 14:32:10 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 321C4B80E3D; Mon, 18 Sep 2023 14:32:09 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 6DC2AC32788; Mon, 18 Sep 2023 14:32:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695047527; bh=XKkDjv0e/eMNWjG9jRQocyFRrh7/5+36OBy3yTELA+U=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=m6M5A2UvQEKxKx/YnfHcVQRwpDbpl883sUgnsDZlx1ywJ9Nzgi9yZqFucLD2HJCTU OoAuNlFDRtOG9vqaLaMQZ6dX/CvL/p+uV1wxW+UF7yvU9GlBxNcQUnvQJOlLkF0Kno IuXoc5NKnX9XK6AWr45kJGatIK/2mkFcAflQgjsAZBvt/5SJGN2L+qqCVU6TAB98OY pPfT9mIZXBmAOmb5I81UFZQpMgHdKnqOb1KWywHInNl5anlvmXsKjJhtmokE5DDVt1 xCrmL4VcIEG2uaxfaKOYcRXbA/Ehxj21zKUcwOKCz0DddgvahdpEO4gt18jmAtSs9i GCHZ5or2RnvOA== From: Oded Gabbay To: dri-devel@lists.freedesktop.org Subject: [PATCH 03/10] accel/habanalabs: split user interrupts pending list Date: Mon, 18 Sep 2023 17:31:51 +0300 Message-Id: <20230918143158.903207-3-ogabbay@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230918143158.903207-1-ogabbay@kernel.org> References: <20230918143158.903207-1-ogabbay@kernel.org> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tomer Tayar , farah kassabri Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: farah kassabri Currently driver maintain one list for both pending user interrupts which seeks to wait till CQ reaches it's target value and also the ones that seeks to get timestamp records when the CQ reaches it's target value. This causes delay in handling the waiters which gets higher priority than the timestamp records. In order to solve this, let's split the list into two, one for each case and each one is protected by it's own spinlock. Waiters will be handled within the interrupt context first, then the timestamp records will be set. Freeing the timestamp related memory will be handled in a workqueue. Signed-off-by: farah kassabri Reviewed-by: Tomer Tayar Signed-off-by: Oded Gabbay --- .../habanalabs/common/command_submission.c | 235 ++++++++++-------- drivers/accel/habanalabs/common/habanalabs.h | 12 +- drivers/accel/habanalabs/common/irq.c | 89 ++++--- drivers/accel/habanalabs/gaudi2/gaudi2.c | 20 +- 4 files changed, 209 insertions(+), 147 deletions(-) diff --git a/drivers/accel/habanalabs/common/command_submission.c b/drivers/accel/habanalabs/common/command_submission.c index 02049bd26356..751d2c7d3fb8 100644 --- a/drivers/accel/habanalabs/common/command_submission.c +++ b/drivers/accel/habanalabs/common/command_submission.c @@ -1098,19 +1098,22 @@ static void wake_pending_user_interrupt_threads(struct hl_user_interrupt *interrupt) { struct hl_user_pending_interrupt *pend, *temp; + unsigned long flags; - spin_lock(&interrupt->wait_list_lock); - list_for_each_entry_safe(pend, temp, &interrupt->wait_list_head, wait_list_node) { - if (pend->ts_reg_info.buf) { - list_del(&pend->wait_list_node); - hl_mmap_mem_buf_put(pend->ts_reg_info.buf); - hl_cb_put(pend->ts_reg_info.cq_cb); - } else { - pend->fence.error = -EIO; - complete_all(&pend->fence.completion); - } + spin_lock_irqsave(&interrupt->wait_list_lock, flags); + list_for_each_entry_safe(pend, temp, &interrupt->wait_list_head, list_node) { + pend->fence.error = -EIO; + complete_all(&pend->fence.completion); } - spin_unlock(&interrupt->wait_list_lock); + spin_unlock_irqrestore(&interrupt->wait_list_lock, flags); + + spin_lock_irqsave(&interrupt->ts_list_lock, flags); + list_for_each_entry_safe(pend, temp, &interrupt->ts_list_head, list_node) { + list_del(&pend->list_node); + hl_mmap_mem_buf_put(pend->ts_reg_info.buf); + hl_cb_put(pend->ts_reg_info.cq_cb); + } + spin_unlock_irqrestore(&interrupt->ts_list_lock, flags); } void hl_release_pending_user_interrupts(struct hl_device *hdev) @@ -3251,18 +3254,19 @@ static void unregister_timestamp_node(struct hl_device *hdev, { struct hl_user_interrupt *interrupt = record->ts_reg_info.interrupt; bool ts_rec_found = false; + unsigned long flags; if (need_lock) - spin_lock(&interrupt->wait_list_lock); + spin_lock_irqsave(&interrupt->ts_list_lock, flags); if (record->ts_reg_info.in_use) { record->ts_reg_info.in_use = false; - list_del(&record->wait_list_node); + list_del(&record->list_node); ts_rec_found = true; } if (need_lock) - spin_unlock(&interrupt->wait_list_lock); + spin_unlock_irqrestore(&interrupt->ts_list_lock, flags); /* Put refcounts that were taken when we registered the event */ if (ts_rec_found) { @@ -3272,7 +3276,7 @@ static void unregister_timestamp_node(struct hl_device *hdev, } static int ts_get_and_handle_kernel_record(struct hl_device *hdev, struct hl_ctx *ctx, - struct wait_interrupt_data *data, + struct wait_interrupt_data *data, unsigned long *flags, struct hl_user_pending_interrupt **pend) { struct hl_user_pending_interrupt *req_offset_record; @@ -3302,13 +3306,13 @@ static int ts_get_and_handle_kernel_record(struct hl_device *hdev, struct hl_ctx req_offset_record->ts_reg_info.interrupt->interrupt_id) { need_lock = true; - spin_unlock(&data->interrupt->wait_list_lock); + spin_unlock_irqrestore(&data->interrupt->ts_list_lock, *flags); } unregister_timestamp_node(hdev, req_offset_record, need_lock); if (need_lock) - spin_lock(&data->interrupt->wait_list_lock); + spin_lock_irqsave(&data->interrupt->ts_list_lock, *flags); } /* Fill up the new registration node info and add it to the list */ @@ -3325,18 +3329,14 @@ static int ts_get_and_handle_kernel_record(struct hl_device *hdev, struct hl_ctx return rc; } -static int _hl_interrupt_wait_ioctl(struct hl_device *hdev, struct hl_ctx *ctx, +static int _hl_interrupt_ts_reg_ioctl(struct hl_device *hdev, struct hl_ctx *ctx, struct wait_interrupt_data *data, - bool register_ts_record, u32 *status, u64 *timestamp) { struct hl_user_pending_interrupt *pend; - unsigned long timeout; - long completion_rc; + unsigned long flags; int rc = 0; - timeout = hl_usecs64_to_jiffies(data->intr_timeout_us); - hl_ctx_get(ctx); data->cq_cb = hl_cb_get(data->mmg, data->cq_handle); @@ -3352,61 +3352,109 @@ static int _hl_interrupt_wait_ioctl(struct hl_device *hdev, struct hl_ctx *ctx, goto put_cq_cb; } - if (register_ts_record) { - dev_dbg(hdev->dev, "Timestamp registration: interrupt id: %u, handle: 0x%llx, ts offset: %llu, cq_offset: %llu\n", - data->interrupt->interrupt_id, data->ts_handle, - data->ts_offset, data->cq_offset); + dev_dbg(hdev->dev, "Timestamp registration: interrupt id: %u, handle: 0x%llx, ts offset: %llu, cq_offset: %llu\n", + data->interrupt->interrupt_id, data->ts_handle, + data->ts_offset, data->cq_offset); - data->buf = hl_mmap_mem_buf_get(data->mmg, data->ts_handle); - if (!data->buf) { - rc = -EINVAL; - goto put_cq_cb; - } + data->buf = hl_mmap_mem_buf_get(data->mmg, data->ts_handle); + if (!data->buf) { + rc = -EINVAL; + goto put_cq_cb; + } - spin_lock_irqsave(&data->interrupt->wait_list_lock, data->flags); + spin_lock_irqsave(&data->interrupt->ts_list_lock, flags); - /* get ts buffer record */ - rc = ts_get_and_handle_kernel_record(hdev, ctx, data, &pend); - if (rc) { - spin_unlock_irqrestore(&data->interrupt->wait_list_lock, data->flags); - goto put_ts_buff; - } - } else { - pend = kzalloc(sizeof(*pend), GFP_KERNEL); - if (!pend) { - rc = -ENOMEM; - goto put_cq_cb; - } - hl_fence_init(&pend->fence, ULONG_MAX); - pend->cq_kernel_addr = (u64 *) data->cq_cb->kernel_address + data->cq_offset; - pend->cq_target_value = data->target_value; - spin_lock_irqsave(&data->interrupt->wait_list_lock, data->flags); + /* get ts buffer record */ + rc = ts_get_and_handle_kernel_record(hdev, ctx, data, &flags, &pend); + if (rc) { + spin_unlock_irqrestore(&data->interrupt->ts_list_lock, flags); + goto put_ts_buff; } /* We check for completion value as interrupt could have been received - * before we add the wait/timestamp node to the wait list. + * before we add the timestamp node to the ts list. */ if (*pend->cq_kernel_addr >= data->target_value) { - spin_unlock_irqrestore(&data->interrupt->wait_list_lock, data->flags); + spin_unlock_irqrestore(&data->interrupt->ts_list_lock, flags); - if (register_ts_record) { - dev_dbg(hdev->dev, "Target value already reached release ts record: pend: %p, offset: %llu, interrupt: %u\n", - pend, data->ts_offset, data->interrupt->interrupt_id); - pend->ts_reg_info.in_use = false; - } + dev_dbg(hdev->dev, "Target value already reached release ts record: pend: %p, offset: %llu, interrupt: %u\n", + pend, data->ts_offset, data->interrupt->interrupt_id); + pend->ts_reg_info.in_use = 0; *status = HL_WAIT_CS_STATUS_COMPLETED; + *pend->ts_reg_info.timestamp_kernel_addr = ktime_get_ns(); + + goto put_ts_buff; + } + + list_add_tail(&pend->list_node, &data->interrupt->ts_list_head); + spin_unlock_irqrestore(&data->interrupt->ts_list_lock, flags); + + rc = *status = HL_WAIT_CS_STATUS_COMPLETED; + + hl_ctx_put(ctx); + + return rc; + +put_ts_buff: + hl_mmap_mem_buf_put(data->buf); +put_cq_cb: + hl_cb_put(data->cq_cb); +put_ctx: + hl_ctx_put(ctx); + + return rc; +} + +static int _hl_interrupt_wait_ioctl(struct hl_device *hdev, struct hl_ctx *ctx, + struct wait_interrupt_data *data, + u32 *status, u64 *timestamp) +{ + struct hl_user_pending_interrupt *pend; + unsigned long timeout, flags; + long completion_rc; + int rc = 0; + + timeout = hl_usecs64_to_jiffies(data->intr_timeout_us); + + hl_ctx_get(ctx); + + data->cq_cb = hl_cb_get(data->mmg, data->cq_handle); + if (!data->cq_cb) { + rc = -EINVAL; + goto put_ctx; + } + + /* Validate the cq offset */ + if (((u64 *) data->cq_cb->kernel_address + data->cq_offset) >= + ((u64 *) data->cq_cb->kernel_address + (data->cq_cb->size / sizeof(u64)))) { + rc = -EINVAL; + goto put_cq_cb; + } + + pend = kzalloc(sizeof(*pend), GFP_KERNEL); + if (!pend) { + rc = -ENOMEM; + goto put_cq_cb; + } + + hl_fence_init(&pend->fence, ULONG_MAX); + pend->cq_kernel_addr = (u64 *) data->cq_cb->kernel_address + data->cq_offset; + pend->cq_target_value = data->target_value; + spin_lock_irqsave(&data->interrupt->wait_list_lock, flags); + + + /* We check for completion value as interrupt could have been received + * before we add the wait node to the wait list. + */ + if (*pend->cq_kernel_addr >= data->target_value || (!data->intr_timeout_us)) { + spin_unlock_irqrestore(&data->interrupt->wait_list_lock, flags); + + if (*pend->cq_kernel_addr >= data->target_value) + *status = HL_WAIT_CS_STATUS_COMPLETED; + else + *status = HL_WAIT_CS_STATUS_BUSY; - if (register_ts_record) { - *pend->ts_reg_info.timestamp_kernel_addr = ktime_get_ns(); - goto put_ts_buff; - } else { - pend->fence.timestamp = ktime_get(); - goto set_timestamp; - } - } else if (!data->intr_timeout_us) { - spin_unlock_irqrestore(&data->interrupt->wait_list_lock, data->flags); - *status = HL_WAIT_CS_STATUS_BUSY; pend->fence.timestamp = ktime_get(); goto set_timestamp; } @@ -3417,13 +3465,8 @@ static int _hl_interrupt_wait_ioctl(struct hl_device *hdev, struct hl_ctx *ctx, * in order to shorten the list pass loop, since * same list could have nodes for different cq counter handle. */ - list_add_tail(&pend->wait_list_node, &data->interrupt->wait_list_head); - spin_unlock_irqrestore(&data->interrupt->wait_list_lock, data->flags); - - if (register_ts_record) { - rc = *status = HL_WAIT_CS_STATUS_COMPLETED; - goto ts_registration_exit; - } + list_add_tail(&pend->list_node, &data->interrupt->wait_list_head); + spin_unlock_irqrestore(&data->interrupt->wait_list_lock, flags); /* Wait for interrupt handler to signal completion */ completion_rc = wait_for_completion_interruptible_timeout(&pend->fence.completion, @@ -3462,21 +3505,18 @@ static int _hl_interrupt_wait_ioctl(struct hl_device *hdev, struct hl_ctx *ctx, * for ts record, the node will be deleted in the irq handler after * we reach the target value. */ - spin_lock_irqsave(&data->interrupt->wait_list_lock, data->flags); - list_del(&pend->wait_list_node); - spin_unlock_irqrestore(&data->interrupt->wait_list_lock, data->flags); + spin_lock_irqsave(&data->interrupt->wait_list_lock, flags); + list_del(&pend->list_node); + spin_unlock_irqrestore(&data->interrupt->wait_list_lock, flags); set_timestamp: *timestamp = ktime_to_ns(pend->fence.timestamp); kfree(pend); hl_cb_put(data->cq_cb); -ts_registration_exit: hl_ctx_put(ctx); return rc; -put_ts_buff: - hl_mmap_mem_buf_put(data->buf); put_cq_cb: hl_cb_put(data->cq_cb); put_ctx: @@ -3513,7 +3553,7 @@ static int _hl_interrupt_wait_ioctl_user_addr(struct hl_device *hdev, struct hl_ * handler to monitor */ spin_lock(&interrupt->wait_list_lock); - list_add_tail(&pend->wait_list_node, &interrupt->wait_list_head); + list_add_tail(&pend->list_node, &interrupt->wait_list_head); spin_unlock(&interrupt->wait_list_lock); /* We check for completion value as interrupt could have been received @@ -3590,7 +3630,7 @@ static int _hl_interrupt_wait_ioctl_user_addr(struct hl_device *hdev, struct hl_ remove_pending_user_interrupt: spin_lock(&interrupt->wait_list_lock); - list_del(&pend->wait_list_node); + list_del(&pend->list_node); spin_unlock(&interrupt->wait_list_lock); *timestamp = ktime_to_ns(pend->fence.timestamp); @@ -3649,16 +3689,6 @@ static int hl_interrupt_wait_ioctl(struct hl_fpriv *hpriv, void *data) return -EINVAL; } - /* - * Allow only one registration at a time. this is needed in order to prevent issues - * while handling the flow of re-use of the same offset. - * Since the registration flow is protected only by the interrupt lock, re-use flow - * might request to move ts node to another interrupt list, and in such case we're - * not protected. - */ - if (args->in.flags & HL_WAIT_CS_FLAGS_REGISTER_INTERRUPT) - mutex_lock(&hpriv->ctx->ts_reg_lock); - if (args->in.flags & HL_WAIT_CS_FLAGS_INTERRUPT_KERNEL_CQ) { struct wait_interrupt_data wait_intr_data = {0}; @@ -3671,9 +3701,23 @@ static int hl_interrupt_wait_ioctl(struct hl_fpriv *hpriv, void *data) wait_intr_data.target_value = args->in.target; wait_intr_data.intr_timeout_us = args->in.interrupt_timeout_us; - rc = _hl_interrupt_wait_ioctl(hdev, hpriv->ctx, &wait_intr_data, - !!(args->in.flags & HL_WAIT_CS_FLAGS_REGISTER_INTERRUPT), - &status, ×tamp); + if (args->in.flags & HL_WAIT_CS_FLAGS_REGISTER_INTERRUPT) { + /* + * Allow only one registration at a time. this is needed in order to prevent + * issues while handling the flow of re-use of the same offset. + * Since the registration flow is protected only by the interrupt lock, + * re-use flow might request to move ts node to another interrupt list, + * and in such case we're not protected. + */ + mutex_lock(&hpriv->ctx->ts_reg_lock); + + rc = _hl_interrupt_ts_reg_ioctl(hdev, hpriv->ctx, &wait_intr_data, + &status, ×tamp); + + mutex_unlock(&hpriv->ctx->ts_reg_lock); + } else + rc = _hl_interrupt_wait_ioctl(hdev, hpriv->ctx, &wait_intr_data, + &status, ×tamp); } else { rc = _hl_interrupt_wait_ioctl_user_addr(hdev, hpriv->ctx, args->in.interrupt_timeout_us, args->in.addr, @@ -3681,9 +3725,6 @@ static int hl_interrupt_wait_ioctl(struct hl_fpriv *hpriv, void *data) ×tamp); } - if (args->in.flags & HL_WAIT_CS_FLAGS_REGISTER_INTERRUPT) - mutex_unlock(&hpriv->ctx->ts_reg_lock); - if (rc) return rc; diff --git a/drivers/accel/habanalabs/common/habanalabs.h b/drivers/accel/habanalabs/common/habanalabs.h index 1342686d0ce5..7c2da8cfe844 100644 --- a/drivers/accel/habanalabs/common/habanalabs.h +++ b/drivers/accel/habanalabs/common/habanalabs.h @@ -1128,7 +1128,9 @@ struct hl_ts_free_jobs { * @ts_free_jobs_data: timestamp free jobs related data * @type: user interrupt type * @wait_list_head: head to the list of user threads pending on this interrupt + * @ts_list_head: head to the list of timestamp records * @wait_list_lock: protects wait_list_head + * @ts_list_lock: protects ts_list_head * @timestamp: last timestamp taken upon interrupt * @interrupt_id: msix interrupt id */ @@ -1137,7 +1139,9 @@ struct hl_user_interrupt { struct hl_ts_free_jobs ts_free_jobs_data; enum hl_user_interrupt_type type; struct list_head wait_list_head; + struct list_head ts_list_head; spinlock_t wait_list_lock; + spinlock_t ts_list_lock; ktime_t timestamp; u32 interrupt_id; }; @@ -1199,7 +1203,7 @@ struct timestamp_reg_info { * struct hl_user_pending_interrupt - holds a context to a user thread * pending on an interrupt * @ts_reg_info: holds the timestamps registration nodes info - * @wait_list_node: node in the list of user threads pending on an interrupt + * @list_node: node in the list of user threads pending on an interrupt or timestamp * @fence: hl fence object for interrupt completion * @cq_target_value: CQ target value * @cq_kernel_addr: CQ kernel address, to be used in the cq interrupt @@ -1207,7 +1211,7 @@ struct timestamp_reg_info { */ struct hl_user_pending_interrupt { struct timestamp_reg_info ts_reg_info; - struct list_head wait_list_node; + struct list_head list_node; struct hl_fence fence; u64 cq_target_value; u64 *cq_kernel_addr; @@ -2742,6 +2746,8 @@ void hl_wreg(struct hl_device *hdev, u32 reg, u32 val); usr_intr.type = intr_type; \ INIT_LIST_HEAD(&usr_intr.wait_list_head); \ spin_lock_init(&usr_intr.wait_list_lock); \ + INIT_LIST_HEAD(&usr_intr.ts_list_head); \ + spin_lock_init(&usr_intr.ts_list_lock); \ }) struct hwmon_chip_info; @@ -3712,7 +3718,7 @@ void hl_eq_reset(struct hl_device *hdev, struct hl_eq *q); irqreturn_t hl_irq_handler_cq(int irq, void *arg); irqreturn_t hl_irq_handler_eq(int irq, void *arg); irqreturn_t hl_irq_handler_dec_abnrm(int irq, void *arg); -irqreturn_t hl_irq_handler_user_interrupt(int irq, void *arg); +irqreturn_t hl_irq_user_interrupt_handler(int irq, void *arg); irqreturn_t hl_irq_user_interrupt_thread_handler(int irq, void *arg); irqreturn_t hl_irq_eq_error_interrupt_thread_handler(int irq, void *arg); u32 hl_cq_inc_ptr(u32 ptr); diff --git a/drivers/accel/habanalabs/common/irq.c b/drivers/accel/habanalabs/common/irq.c index 0947d286a5ab..978b7f4d5eeb 100644 --- a/drivers/accel/habanalabs/common/irq.c +++ b/drivers/accel/habanalabs/common/irq.c @@ -304,7 +304,7 @@ static int handle_registration_node(struct hl_device *hdev, struct hl_user_pendi dev_dbg(hdev->dev, "Irq handle: Timestamp record (%p) ts cb address (%p), interrupt_id: %u\n", pend, pend->ts_reg_info.timestamp_kernel_addr, intr->interrupt_id); - list_del(&pend->wait_list_node); + list_del(&pend->list_node); /* Putting the refcount for ts_buff and cq_cb objects will be handled * in workqueue context, just add job to free_list. @@ -326,12 +326,13 @@ static int handle_registration_node(struct hl_device *hdev, struct hl_user_pendi return 0; } -static void handle_user_interrupt(struct hl_device *hdev, struct hl_user_interrupt *intr) +static void handle_user_interrupt_ts_list(struct hl_device *hdev, struct hl_user_interrupt *intr) { struct list_head *ts_reg_free_list_head = NULL, *dynamic_alloc_list_head = NULL; struct hl_user_pending_interrupt *pend, *temp_pend; struct timestamp_reg_work_obj *job; bool reg_node_handle_fail = false; + unsigned long flags; int rc; /* For registration nodes: @@ -340,34 +341,27 @@ static void handle_user_interrupt(struct hl_device *hdev, struct hl_user_interru * or in irq handler context at all (since release functions are long and * might sleep), so we will need to handle that part in workqueue context. * To avoid handling kmalloc failure which compels us rolling back actions - * and move nodes hanged on the free list back to the interrupt wait list + * and move nodes hanged on the free list back to the interrupt ts list * we always alloc the job of the WQ at the beginning. */ job = kmalloc(sizeof(*job), GFP_ATOMIC); if (!job) return; - spin_lock(&intr->wait_list_lock); - - list_for_each_entry_safe(pend, temp_pend, &intr->wait_list_head, wait_list_node) { + spin_lock_irqsave(&intr->ts_list_lock, flags); + list_for_each_entry_safe(pend, temp_pend, &intr->ts_list_head, list_node) { if ((pend->cq_kernel_addr && *(pend->cq_kernel_addr) >= pend->cq_target_value) || !pend->cq_kernel_addr) { - if (pend->ts_reg_info.buf) { - if (!reg_node_handle_fail) { - rc = handle_registration_node(hdev, pend, - &ts_reg_free_list_head, - &dynamic_alloc_list_head, intr); - if (rc) - reg_node_handle_fail = true; - } - } else { - /* Handle wait target value node */ - pend->fence.timestamp = intr->timestamp; - complete_all(&pend->fence.completion); + if (!reg_node_handle_fail) { + rc = handle_registration_node(hdev, pend, + &ts_reg_free_list_head, + &dynamic_alloc_list_head, intr); + if (rc) + reg_node_handle_fail = true; } } } - spin_unlock(&intr->wait_list_lock); + spin_unlock_irqrestore(&intr->ts_list_lock, flags); if (ts_reg_free_list_head) { INIT_WORK(&job->free_obj, hl_ts_free_objects); @@ -380,6 +374,23 @@ static void handle_user_interrupt(struct hl_device *hdev, struct hl_user_interru } } +static void handle_user_interrupt_wait_list(struct hl_device *hdev, struct hl_user_interrupt *intr) +{ + struct hl_user_pending_interrupt *pend, *temp_pend; + unsigned long flags; + + spin_lock_irqsave(&intr->wait_list_lock, flags); + list_for_each_entry_safe(pend, temp_pend, &intr->wait_list_head, list_node) { + if ((pend->cq_kernel_addr && *(pend->cq_kernel_addr) >= pend->cq_target_value) || + !pend->cq_kernel_addr) { + /* Handle wait target value node */ + pend->fence.timestamp = intr->timestamp; + complete_all(&pend->fence.completion); + } + } + spin_unlock_irqrestore(&intr->wait_list_lock, flags); +} + static void handle_tpc_interrupt(struct hl_device *hdev) { u64 event_mask; @@ -401,19 +412,38 @@ static void handle_unexpected_user_interrupt(struct hl_device *hdev) } /** - * hl_irq_handler_user_interrupt - irq handler for user interrupts + * hl_irq_user_interrupt_handler - irq handler for user interrupts. * * @irq: irq number * @arg: pointer to user interrupt structure - * */ -irqreturn_t hl_irq_handler_user_interrupt(int irq, void *arg) +irqreturn_t hl_irq_user_interrupt_handler(int irq, void *arg) { struct hl_user_interrupt *user_int = arg; + struct hl_device *hdev = user_int->hdev; user_int->timestamp = ktime_get(); + switch (user_int->type) { + case HL_USR_INTERRUPT_CQ: + /* First handle user waiters threads */ + handle_user_interrupt_wait_list(hdev, &hdev->common_user_cq_interrupt); + handle_user_interrupt_wait_list(hdev, user_int); + + /* Second handle user timestamp registrations */ + handle_user_interrupt_ts_list(hdev, &hdev->common_user_cq_interrupt); + handle_user_interrupt_ts_list(hdev, user_int); + break; + case HL_USR_INTERRUPT_DECODER: + handle_user_interrupt_wait_list(hdev, &hdev->common_decoder_interrupt); + + /* Handle decoder interrupt registered on this specific irq */ + handle_user_interrupt_wait_list(hdev, user_int); + break; + default: + break; + } - return IRQ_WAKE_THREAD; + return IRQ_HANDLED; } /** @@ -429,19 +459,8 @@ irqreturn_t hl_irq_user_interrupt_thread_handler(int irq, void *arg) struct hl_user_interrupt *user_int = arg; struct hl_device *hdev = user_int->hdev; + user_int->timestamp = ktime_get(); switch (user_int->type) { - case HL_USR_INTERRUPT_CQ: - handle_user_interrupt(hdev, &hdev->common_user_cq_interrupt); - - /* Handle user cq interrupt registered on this specific irq */ - handle_user_interrupt(hdev, user_int); - break; - case HL_USR_INTERRUPT_DECODER: - handle_user_interrupt(hdev, &hdev->common_decoder_interrupt); - - /* Handle decoder interrupt registered on this specific irq */ - handle_user_interrupt(hdev, user_int); - break; case HL_USR_INTERRUPT_TPC: handle_tpc_interrupt(hdev); break; diff --git a/drivers/accel/habanalabs/gaudi2/gaudi2.c b/drivers/accel/habanalabs/gaudi2/gaudi2.c index b0ba62b691ec..867175431418 100644 --- a/drivers/accel/habanalabs/gaudi2/gaudi2.c +++ b/drivers/accel/habanalabs/gaudi2/gaudi2.c @@ -4227,9 +4227,7 @@ static int gaudi2_dec_enable_msix(struct hl_device *hdev) rc = request_irq(irq, hl_irq_handler_dec_abnrm, 0, gaudi2_irq_name(i), (void *) dec); } else { - rc = request_threaded_irq(irq, hl_irq_handler_user_interrupt, - hl_irq_user_interrupt_thread_handler, IRQF_ONESHOT, - gaudi2_irq_name(i), + rc = request_irq(irq, hl_irq_user_interrupt_handler, 0, gaudi2_irq_name(i), (void *) &hdev->user_interrupt[dec->core_id]); } @@ -4287,17 +4285,17 @@ static int gaudi2_enable_msix(struct hl_device *hdev) } irq = pci_irq_vector(hdev->pdev, GAUDI2_IRQ_NUM_TPC_ASSERT); - rc = request_threaded_irq(irq, hl_irq_handler_user_interrupt, - hl_irq_user_interrupt_thread_handler, IRQF_ONESHOT, - gaudi2_irq_name(GAUDI2_IRQ_NUM_TPC_ASSERT), &hdev->tpc_interrupt); + rc = request_threaded_irq(irq, NULL, hl_irq_user_interrupt_thread_handler, IRQF_ONESHOT, + gaudi2_irq_name(GAUDI2_IRQ_NUM_TPC_ASSERT), + &hdev->tpc_interrupt); if (rc) { dev_err(hdev->dev, "Failed to request IRQ %d", irq); goto free_dec_irq; } irq = pci_irq_vector(hdev->pdev, GAUDI2_IRQ_NUM_UNEXPECTED_ERROR); - rc = request_irq(irq, hl_irq_handler_user_interrupt, 0, - gaudi2_irq_name(GAUDI2_IRQ_NUM_UNEXPECTED_ERROR), + rc = request_threaded_irq(irq, NULL, hl_irq_user_interrupt_thread_handler, IRQF_ONESHOT, + gaudi2_irq_name(GAUDI2_IRQ_NUM_UNEXPECTED_ERROR), &hdev->unexpected_error_interrupt); if (rc) { dev_err(hdev->dev, "Failed to request IRQ %d", irq); @@ -4309,10 +4307,8 @@ static int gaudi2_enable_msix(struct hl_device *hdev) i++, j++, user_irq_init_cnt++) { irq = pci_irq_vector(hdev->pdev, i); - rc = request_threaded_irq(irq, hl_irq_handler_user_interrupt, - hl_irq_user_interrupt_thread_handler, IRQF_ONESHOT, - gaudi2_irq_name(i), &hdev->user_interrupt[j]); - + rc = request_irq(irq, hl_irq_user_interrupt_handler, 0, gaudi2_irq_name(i), + &hdev->user_interrupt[j]); if (rc) { dev_err(hdev->dev, "Failed to request IRQ %d", irq); goto free_user_irq; From patchwork Mon Sep 18 14:31:52 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oded Gabbay X-Patchwork-Id: 13389721 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59F0BC46CA1 for ; Mon, 18 Sep 2023 14:32:35 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id D3F3910E0AC; Mon, 18 Sep 2023 14:32:34 +0000 (UTC) Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2C9F810E0A8 for ; Mon, 18 Sep 2023 14:32:12 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 9BA27B80EA0; Mon, 18 Sep 2023 14:32:10 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 62728C32789; Mon, 18 Sep 2023 14:32:08 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695047529; bh=vGUOt9KcUO/z+qvU+mkl7UIBpqBXYYdyCaU0w8vD7zU=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=ohQ64hBuen7t2T3SdkUaYZ+OtRz/T/m5gYYm3HCeqLKImVzBUCAjnBbclXVoQ0CdP EMjazYl5Vyd67uo0yvJ7KK7BYrzLbNmUlWZqvwwnO7g3G0m4KEKj8Ck/YkBGeN+Z28 4a+kf5ybkYpym+z8dIt+DuawU7Nc3H3/lKSlqFEb9XoXjDXi1zIZRL9AOB4ffJ1rF5 3yTb/ObJDV00ofVXawPpXimD492MJLGCJ9FrCXe3wZOfPtntAM6fJBWpubC2OwGCu7 7cl7IYZQLsJSQXsu486ZjUta/Qo3F50TF1e+xirkH9SdKomvVXRUpPlUmOuwViAnsl fsFtvXnrTabfg== From: Oded Gabbay To: dri-devel@lists.freedesktop.org Subject: [PATCH 04/10] accel/habanalabs: fix SG table creation for dma-buf mapping Date: Mon, 18 Sep 2023 17:31:52 +0300 Message-Id: <20230918143158.903207-4-ogabbay@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230918143158.903207-1-ogabbay@kernel.org> References: <20230918143158.903207-1-ogabbay@kernel.org> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tomer Tayar Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Tomer Tayar In some cases the calculated number of required entries for the dma-buf SG table is wrong. For example, if the page size is larger than both the dma max segment size of the importer device and from the exported side, or if the exported size is part of a phys_pg_pack that is composed of several pages. In these cases, redundant entries will be added to the SG table. Modify the method that the number of entries is calculated, and the way they are prepared. Signed-off-by: Tomer Tayar Reviewed-by: Oded Gabbay Signed-off-by: Oded Gabbay --- drivers/accel/habanalabs/common/memory.c | 199 ++++++++++++----------- 1 file changed, 104 insertions(+), 95 deletions(-) diff --git a/drivers/accel/habanalabs/common/memory.c b/drivers/accel/habanalabs/common/memory.c index d0edbe4b4210..5c1e98e73a47 100644 --- a/drivers/accel/habanalabs/common/memory.c +++ b/drivers/accel/habanalabs/common/memory.c @@ -1535,21 +1535,17 @@ static struct sg_table *alloc_sgt_from_device_pages(struct hl_device *hdev, u64 u64 page_size, u64 exported_size, struct device *dev, enum dma_data_direction dir) { - u64 chunk_size, bar_address, dma_max_seg_size, cur_size_to_export, cur_npages; - struct asic_fixed_properties *prop; - int rc, i, j, nents, cur_page; + u64 dma_max_seg_size, curr_page, size, chunk_size, left_size_to_export, left_size_in_page, + left_size_in_dma_seg, device_address, bar_address; + struct asic_fixed_properties *prop = &hdev->asic_prop; struct scatterlist *sg; + unsigned int nents, i; struct sg_table *sgt; + bool next_sg_entry; + int rc; - prop = &hdev->asic_prop; - - dma_max_seg_size = dma_get_max_seg_size(dev); - - /* We would like to align the max segment size to PAGE_SIZE, so the - * SGL will contain aligned addresses that can be easily mapped to - * an MMU - */ - dma_max_seg_size = ALIGN_DOWN(dma_max_seg_size, PAGE_SIZE); + /* Align max segment size to PAGE_SIZE to fit the minimal IOMMU mapping granularity */ + dma_max_seg_size = ALIGN_DOWN(dma_get_max_seg_size(dev), PAGE_SIZE); if (dma_max_seg_size < PAGE_SIZE) { dev_err_ratelimited(hdev->dev, "dma_max_seg_size %llu can't be smaller than PAGE_SIZE\n", @@ -1561,120 +1557,133 @@ static struct sg_table *alloc_sgt_from_device_pages(struct hl_device *hdev, u64 if (!sgt) return ERR_PTR(-ENOMEM); - cur_size_to_export = exported_size; + /* Calculate the required number of entries for the SG table */ + curr_page = 0; + nents = 1; + left_size_to_export = exported_size; + left_size_in_page = page_size; + left_size_in_dma_seg = dma_max_seg_size; + next_sg_entry = false; + + while (true) { + size = min3(left_size_to_export, left_size_in_page, left_size_in_dma_seg); + left_size_to_export -= size; + left_size_in_page -= size; + left_size_in_dma_seg -= size; + + if (!left_size_to_export) + break; - /* If the size of each page is larger than the dma max segment size, - * then we can't combine pages and the number of entries in the SGL - * will just be the - * * - */ - if (page_size > dma_max_seg_size) { - /* we should limit number of pages according to the exported size */ - cur_npages = DIV_ROUND_UP_SECTOR_T(cur_size_to_export, page_size); - nents = cur_npages * DIV_ROUND_UP_SECTOR_T(page_size, dma_max_seg_size); - } else { - cur_npages = npages; - - /* Get number of non-contiguous chunks */ - for (i = 1, nents = 1, chunk_size = page_size ; i < cur_npages ; i++) { - if (pages[i - 1] + page_size != pages[i] || - chunk_size + page_size > dma_max_seg_size) { - nents++; - chunk_size = page_size; - continue; - } + if (!left_size_in_page) { + /* left_size_to_export is not zero so there must be another page */ + if (pages[curr_page] + page_size != pages[curr_page + 1]) + next_sg_entry = true; + + ++curr_page; + left_size_in_page = page_size; + } - chunk_size += page_size; + if (!left_size_in_dma_seg) { + next_sg_entry = true; + left_size_in_dma_seg = dma_max_seg_size; + } + + if (next_sg_entry) { + ++nents; + next_sg_entry = false; } } rc = sg_alloc_table(sgt, nents, GFP_KERNEL | __GFP_ZERO); if (rc) - goto error_free; - - cur_page = 0; - - if (page_size > dma_max_seg_size) { - u64 size_left, cur_device_address = 0; + goto err_free_sgt; - size_left = page_size; + /* Prepare the SG table entries */ + curr_page = 0; + device_address = pages[curr_page]; + left_size_to_export = exported_size; + left_size_in_page = page_size; + left_size_in_dma_seg = dma_max_seg_size; + next_sg_entry = false; - /* Need to split each page into the number of chunks of - * dma_max_seg_size - */ - for_each_sgtable_dma_sg(sgt, sg, i) { - if (size_left == page_size) - cur_device_address = - pages[cur_page] - prop->dram_base_address; - else - cur_device_address += dma_max_seg_size; - - /* make sure not to export over exported size */ - chunk_size = min3(size_left, dma_max_seg_size, cur_size_to_export); - - bar_address = hdev->dram_pci_bar_start + cur_device_address; - - rc = set_dma_sg(sg, bar_address, chunk_size, dev, dir); - if (rc) - goto error_unmap; + for_each_sgtable_dma_sg(sgt, sg, i) { + bar_address = hdev->dram_pci_bar_start + (device_address - prop->dram_base_address); + chunk_size = 0; + + for ( ; curr_page < npages ; ++curr_page) { + size = min3(left_size_to_export, left_size_in_page, left_size_in_dma_seg); + chunk_size += size; + left_size_to_export -= size; + left_size_in_page -= size; + left_size_in_dma_seg -= size; + + if (!left_size_to_export) + break; + + if (!left_size_in_page) { + /* left_size_to_export is not zero so there must be another page */ + if (pages[curr_page] + page_size != pages[curr_page + 1]) { + device_address = pages[curr_page + 1]; + next_sg_entry = true; + } + + left_size_in_page = page_size; + } - cur_size_to_export -= chunk_size; + if (!left_size_in_dma_seg) { + /* + * Skip setting a new device address if already moving to a page + * which is not contiguous with the current page. + */ + if (!next_sg_entry) { + device_address += chunk_size; + next_sg_entry = true; + } + + left_size_in_dma_seg = dma_max_seg_size; + } - if (size_left > dma_max_seg_size) { - size_left -= dma_max_seg_size; - } else { - cur_page++; - size_left = page_size; + if (next_sg_entry) { + next_sg_entry = false; + break; } } - } else { - /* Merge pages and put them into the scatterlist */ - for_each_sgtable_dma_sg(sgt, sg, i) { - chunk_size = page_size; - for (j = cur_page + 1 ; j < cur_npages ; j++) { - if (pages[j - 1] + page_size != pages[j] || - chunk_size + page_size > dma_max_seg_size) - break; - - chunk_size += page_size; - } - - bar_address = hdev->dram_pci_bar_start + - (pages[cur_page] - prop->dram_base_address); - /* make sure not to export over exported size */ - chunk_size = min(chunk_size, cur_size_to_export); - rc = set_dma_sg(sg, bar_address, chunk_size, dev, dir); - if (rc) - goto error_unmap; + rc = set_dma_sg(sg, bar_address, chunk_size, dev, dir); + if (rc) + goto err_unmap; + } - cur_size_to_export -= chunk_size; - cur_page = j; - } + /* There should be nothing left to export exactly after looping over all SG elements */ + if (left_size_to_export) { + dev_err(hdev->dev, + "left size to export %#llx after initializing %u SG elements\n", + left_size_to_export, sgt->nents); + rc = -ENOMEM; + goto err_unmap; } - /* Because we are not going to include a CPU list we want to have some - * chance that other users will detect this by setting the orig_nents - * to 0 and using only nents (length of DMA list) when going over the - * sgl + /* + * Because we are not going to include a CPU list, we want to have some chance that other + * users will detect this when going over SG table, by setting the orig_nents to 0 and using + * only nents (length of DMA list). */ sgt->orig_nents = 0; return sgt; -error_unmap: +err_unmap: for_each_sgtable_dma_sg(sgt, sg, i) { if (!sg_dma_len(sg)) continue; - dma_unmap_resource(dev, sg_dma_address(sg), - sg_dma_len(sg), dir, + dma_unmap_resource(dev, sg_dma_address(sg), sg_dma_len(sg), dir, DMA_ATTR_SKIP_CPU_SYNC); } sg_free_table(sgt); -error_free: +err_free_sgt: kfree(sgt); return ERR_PTR(rc); } From patchwork Mon Sep 18 14:31:53 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oded Gabbay X-Patchwork-Id: 13389718 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 55418C46CA1 for ; Mon, 18 Sep 2023 14:32:24 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C2ED110E285; Mon, 18 Sep 2023 14:32:18 +0000 (UTC) Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9264D10E190 for ; Mon, 18 Sep 2023 14:32:13 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 1BC3AB80E9F; Mon, 18 Sep 2023 14:32:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id D2A76C32788; Mon, 18 Sep 2023 14:32:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695047530; bh=ClWgUvldec1UbLGX8X9VNPwRuqMyICgQEez2cg1rCt0=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=snRL8w4bHqnC1czai84Fckh8urs0Fimie0t6JZQGs0FIDsqJuF/rylJjfAVxavL+B tLm+rk5qYtAdrLOQxS1/8zOR8eu0wEGlJWHxI7NY5+tzQ/5I42GwetZAa2fzDm3f9t uHsHfM68ms95h25rXHUGGDn49AOywJw0F4ArTbNxYYDV9TXpN+B67Q+OFHO84dvTpG 3XImCX+tRAzdkQjC5gj6fLiKzdwiL2R1TqHawbn7zoF+IVYu1cou9mpR/7FcZudgX5 uU9ctQHw/9fBys+GRxTikaG2yNM6N5j1gXLBh6vi/j4eqmc691k4pDjFFACEh+WxRB 2xQKPN4cO7lIQ== From: Oded Gabbay To: dri-devel@lists.freedesktop.org Subject: [PATCH 05/10] accel/habanalabs: set hl_dmabuf_priv.device_address only when needed Date: Mon, 18 Sep 2023 17:31:53 +0300 Message-Id: <20230918143158.903207-5-ogabbay@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230918143158.903207-1-ogabbay@kernel.org> References: <20230918143158.903207-1-ogabbay@kernel.org> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tomer Tayar Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Tomer Tayar The device_address member of 'struct hl_dmabuf_priv' is used only when virtual device memory is not supported and dma-buf is exported from address. Set the value of this field only when it is relevant, and add "phys" to its name so it would be clearer that it can't be a device virtual address. Signed-off-by: Tomer Tayar Reviewed-by: Oded Gabbay Signed-off-by: Oded Gabbay --- drivers/accel/habanalabs/common/habanalabs.h | 8 ++++---- drivers/accel/habanalabs/common/memory.c | 6 +++--- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/drivers/accel/habanalabs/common/habanalabs.h b/drivers/accel/habanalabs/common/habanalabs.h index 7c2da8cfe844..43e682cdeb75 100644 --- a/drivers/accel/habanalabs/common/habanalabs.h +++ b/drivers/accel/habanalabs/common/habanalabs.h @@ -1811,16 +1811,16 @@ struct hl_cs_counters_atomic { * @phys_pg_pack: pointer to physical page pack if the dma-buf was exported * where virtual memory is supported. * @memhash_hnode: pointer to the memhash node. this object holds the export count. - * @device_address: physical address of the device's memory. Relevant only - * if phys_pg_pack is NULL (dma-buf was exported from address). - * The total size can be taken from the dmabuf object. + * device_phys_addr: physical address of the device's memory. Relevant only + * if phys_pg_pack is NULL (dma-buf was exported from address). + * The total size can be taken from the dmabuf object. */ struct hl_dmabuf_priv { struct dma_buf *dmabuf; struct hl_ctx *ctx; struct hl_vm_phys_pg_pack *phys_pg_pack; struct hl_vm_hash_node *memhash_hnode; - uint64_t device_address; + u64 device_phys_addr; }; #define HL_CS_OUTCOME_HISTORY_LEN 256 diff --git a/drivers/accel/habanalabs/common/memory.c b/drivers/accel/habanalabs/common/memory.c index 5c1e98e73a47..dc5ff3e74c39 100644 --- a/drivers/accel/habanalabs/common/memory.c +++ b/drivers/accel/habanalabs/common/memory.c @@ -1730,7 +1730,7 @@ static struct sg_table *hl_map_dmabuf(struct dma_buf_attachment *attachment, npages = phys_pg_pack->npages; page_size = phys_pg_pack->page_size; } else { - pages = &hl_dmabuf->device_address; + pages = &hl_dmabuf->device_phys_addr; npages = 1; page_size = hl_dmabuf->dmabuf->size; } @@ -2063,9 +2063,9 @@ static int export_dmabuf_from_addr(struct hl_ctx *ctx, u64 addr, u64 size, u64 o rc = validate_export_params_no_mmu(hdev, export_addr, size); if (rc) goto err_free_dmabuf_wrapper; - } - hl_dmabuf->device_address = export_addr; + hl_dmabuf->device_phys_addr = export_addr; + } rc = export_dmabuf(ctx, hl_dmabuf, size, flags, dmabuf_fd); if (rc) From patchwork Mon Sep 18 14:31:54 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oded Gabbay X-Patchwork-Id: 13389716 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id C6919CD13D2 for ; Mon, 18 Sep 2023 14:32:19 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 4063610E190; Mon, 18 Sep 2023 14:32:18 +0000 (UTC) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by gabe.freedesktop.org (Postfix) with ESMTPS id 974CC10E1A4 for ; Mon, 18 Sep 2023 14:32:13 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id C7D9861140; Mon, 18 Sep 2023 14:32:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 4F271C32789; Mon, 18 Sep 2023 14:32:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695047532; bh=Cto32TG6gL6s//uxAZ/DB7cW9t2kgyOg4eawDG+DdXs=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=a7Xk/v4ZcwxIsl5/nejB/vQLxEoJCTKuJshfw8TtfIjPWXeDRQWiqbn+WyEkeG96O E8r2fXzbHND+tdpxn/8rjEg/zT8fTQc97QNf4e94OVdApxQEmIrBMEhzpyFmDw1aCl brpmoARqwMNiQictfWcAhEkFJ4XE7E3P9yCHbQCpB9f+DuRbUHvRd7RCt8TJv5Lha9 qzOH+ZeQxmH/JtvyIm/vazC4qSozPjoAL8lbQ3cLrsz00365Oz6dQll3f9Z6c4ZQec 3cT/DkV8VylT+61CHE44Fgzaenh1bsDxTM9UGzDmIO2c+amr+1epQBtlDaKt4Kgc4t FVxlftvdRpqtA== From: Oded Gabbay To: dri-devel@lists.freedesktop.org Subject: [PATCH 06/10] accel/habanalabs: add missing offset handling for dma-buf Date: Mon, 18 Sep 2023 17:31:54 +0300 Message-Id: <20230918143158.903207-6-ogabbay@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230918143158.903207-1-ogabbay@kernel.org> References: <20230918143158.903207-1-ogabbay@kernel.org> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tomer Tayar Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Tomer Tayar On devices with virtual device memory (Gaudi2 onwards), user can provide an offset within an allocated device memory from which he wants to export a dma-buf object. The offset value is verified by driver, but it is not taken into consideration when the importer driver maps the dma-buf and the SG table it prepared. Add the missing offset handling. Signed-off-by: Tomer Tayar Reviewed-by: Oded Gabbay Signed-off-by: Oded Gabbay --- drivers/accel/habanalabs/common/habanalabs.h | 3 + drivers/accel/habanalabs/common/memory.c | 75 +++++++++++--------- 2 files changed, 43 insertions(+), 35 deletions(-) diff --git a/drivers/accel/habanalabs/common/habanalabs.h b/drivers/accel/habanalabs/common/habanalabs.h index 43e682cdeb75..874ae76cbd78 100644 --- a/drivers/accel/habanalabs/common/habanalabs.h +++ b/drivers/accel/habanalabs/common/habanalabs.h @@ -1811,6 +1811,8 @@ struct hl_cs_counters_atomic { * @phys_pg_pack: pointer to physical page pack if the dma-buf was exported * where virtual memory is supported. * @memhash_hnode: pointer to the memhash node. this object holds the export count. + * @offset: the offset into the buffer from which the memory is exported. + * Relevant only if virtual memory is supported and phys_pg_pack is being used. * device_phys_addr: physical address of the device's memory. Relevant only * if phys_pg_pack is NULL (dma-buf was exported from address). * The total size can be taken from the dmabuf object. @@ -1820,6 +1822,7 @@ struct hl_dmabuf_priv { struct hl_ctx *ctx; struct hl_vm_phys_pg_pack *phys_pg_pack; struct hl_vm_hash_node *memhash_hnode; + u64 offset; u64 device_phys_addr; }; diff --git a/drivers/accel/habanalabs/common/memory.c b/drivers/accel/habanalabs/common/memory.c index dc5ff3e74c39..00e73b5573be 100644 --- a/drivers/accel/habanalabs/common/memory.c +++ b/drivers/accel/habanalabs/common/memory.c @@ -1532,11 +1532,11 @@ static int set_dma_sg(struct scatterlist *sg, u64 bar_address, u64 chunk_size, } static struct sg_table *alloc_sgt_from_device_pages(struct hl_device *hdev, u64 *pages, u64 npages, - u64 page_size, u64 exported_size, + u64 page_size, u64 exported_size, u64 offset, struct device *dev, enum dma_data_direction dir) { u64 dma_max_seg_size, curr_page, size, chunk_size, left_size_to_export, left_size_in_page, - left_size_in_dma_seg, device_address, bar_address; + left_size_in_dma_seg, device_address, bar_address, start_page; struct asic_fixed_properties *prop = &hdev->asic_prop; struct scatterlist *sg; unsigned int nents, i; @@ -1557,11 +1557,20 @@ static struct sg_table *alloc_sgt_from_device_pages(struct hl_device *hdev, u64 if (!sgt) return ERR_PTR(-ENOMEM); + /* Use the offset to move to the actual first page that is exported */ + for (start_page = 0 ; start_page < npages ; ++start_page) { + if (offset < page_size) + break; + + /* The offset value was validated so there can't be an underflow */ + offset -= page_size; + } + /* Calculate the required number of entries for the SG table */ - curr_page = 0; + curr_page = start_page; nents = 1; left_size_to_export = exported_size; - left_size_in_page = page_size; + left_size_in_page = page_size - offset; left_size_in_dma_seg = dma_max_seg_size; next_sg_entry = false; @@ -1599,10 +1608,10 @@ static struct sg_table *alloc_sgt_from_device_pages(struct hl_device *hdev, u64 goto err_free_sgt; /* Prepare the SG table entries */ - curr_page = 0; - device_address = pages[curr_page]; + curr_page = start_page; + device_address = pages[curr_page] + offset; left_size_to_export = exported_size; - left_size_in_page = page_size; + left_size_in_page = page_size - offset; left_size_in_dma_seg = dma_max_seg_size; next_sg_entry = false; @@ -1708,7 +1717,7 @@ static int hl_dmabuf_attach(struct dma_buf *dmabuf, static struct sg_table *hl_map_dmabuf(struct dma_buf_attachment *attachment, enum dma_data_direction dir) { - u64 *pages, npages, page_size, exported_size; + u64 *pages, npages, page_size, exported_size, offset; struct dma_buf *dma_buf = attachment->dmabuf; struct hl_vm_phys_pg_pack *phys_pg_pack; struct hl_dmabuf_priv *hl_dmabuf; @@ -1723,6 +1732,7 @@ static struct sg_table *hl_map_dmabuf(struct dma_buf_attachment *attachment, hl_dmabuf = dma_buf->priv; hdev = hl_dmabuf->ctx->hdev; exported_size = hl_dmabuf->dmabuf->size; + offset = hl_dmabuf->offset; phys_pg_pack = hl_dmabuf->phys_pg_pack; if (phys_pg_pack) { @@ -1735,7 +1745,7 @@ static struct sg_table *hl_map_dmabuf(struct dma_buf_attachment *attachment, page_size = hl_dmabuf->dmabuf->size; } - sgt = alloc_sgt_from_device_pages(hdev, pages, npages, page_size, exported_size, + sgt = alloc_sgt_from_device_pages(hdev, pages, npages, page_size, exported_size, offset, attachment->dev, dir); if (IS_ERR(sgt)) dev_err(hdev->dev, "failed (%ld) to initialize sgt for dmabuf\n", PTR_ERR(sgt)); @@ -1881,12 +1891,12 @@ static int export_dmabuf(struct hl_ctx *ctx, return rc; } -static int validate_export_params_common(struct hl_device *hdev, u64 device_addr, u64 size) +static int validate_export_params_common(struct hl_device *hdev, u64 addr, u64 size, u64 offset) { - if (!PAGE_ALIGNED(device_addr)) { + if (!PAGE_ALIGNED(addr)) { dev_dbg(hdev->dev, "exported device memory address 0x%llx should be aligned to PAGE_SIZE 0x%lx\n", - device_addr, PAGE_SIZE); + addr, PAGE_SIZE); return -EINVAL; } @@ -1897,6 +1907,13 @@ static int validate_export_params_common(struct hl_device *hdev, u64 device_addr return -EINVAL; } + if (!PAGE_ALIGNED(offset)) { + dev_dbg(hdev->dev, + "exported device memory offset %llu should be a multiple of PAGE_SIZE %lu\n", + offset, PAGE_SIZE); + return -EINVAL; + } + return 0; } @@ -1906,13 +1923,13 @@ static int validate_export_params_no_mmu(struct hl_device *hdev, u64 device_addr u64 bar_address; int rc; - rc = validate_export_params_common(hdev, device_addr, size); + rc = validate_export_params_common(hdev, device_addr, size, 0); if (rc) return rc; if (device_addr < prop->dram_user_base_address || - (device_addr + size) > prop->dram_end_address || - (device_addr + size) < device_addr) { + (device_addr + size) > prop->dram_end_address || + (device_addr + size) < device_addr) { dev_dbg(hdev->dev, "DRAM memory range 0x%llx (+0x%llx) is outside of DRAM boundaries\n", device_addr, size); @@ -1939,36 +1956,26 @@ static int validate_export_params(struct hl_device *hdev, u64 device_addr, u64 s u64 bar_address; int i, rc; - rc = validate_export_params_common(hdev, device_addr, size); + rc = validate_export_params_common(hdev, device_addr, size, offset); if (rc) return rc; - if (!PAGE_ALIGNED(offset)) { - dev_dbg(hdev->dev, - "exported device memory offset %llu should be a multiple of PAGE_SIZE %lu\n", - offset, PAGE_SIZE); - return -EINVAL; - } - if ((offset + size) > phys_pg_pack->total_size) { dev_dbg(hdev->dev, "offset %#llx and size %#llx exceed total map size %#llx\n", - offset, size, phys_pg_pack->total_size); + offset, size, phys_pg_pack->total_size); return -EINVAL; } for (i = 0 ; i < phys_pg_pack->npages ; i++) { - bar_address = hdev->dram_pci_bar_start + - (phys_pg_pack->pages[i] - prop->dram_base_address); + (phys_pg_pack->pages[i] - prop->dram_base_address); if ((bar_address + phys_pg_pack->page_size) > (hdev->dram_pci_bar_start + prop->dram_pci_bar_size) || (bar_address + phys_pg_pack->page_size) < bar_address) { dev_dbg(hdev->dev, "DRAM memory range 0x%llx (+0x%x) is outside of PCI BAR boundaries\n", - phys_pg_pack->pages[i], - phys_pg_pack->page_size); - + phys_pg_pack->pages[i], phys_pg_pack->page_size); return -EINVAL; } } @@ -2024,7 +2031,6 @@ static int export_dmabuf_from_addr(struct hl_ctx *ctx, u64 addr, u64 size, u64 o struct asic_fixed_properties *prop; struct hl_dmabuf_priv *hl_dmabuf; struct hl_device *hdev; - u64 export_addr; int rc; hdev = ctx->hdev; @@ -2036,8 +2042,6 @@ static int export_dmabuf_from_addr(struct hl_ctx *ctx, u64 addr, u64 size, u64 o return -EINVAL; } - export_addr = addr + offset; - hl_dmabuf = kzalloc(sizeof(*hl_dmabuf), GFP_KERNEL); if (!hl_dmabuf) return -ENOMEM; @@ -2053,18 +2057,19 @@ static int export_dmabuf_from_addr(struct hl_ctx *ctx, u64 addr, u64 size, u64 o rc = PTR_ERR(phys_pg_pack); goto dec_memhash_export_cnt; } - rc = validate_export_params(hdev, export_addr, size, offset, phys_pg_pack); + rc = validate_export_params(hdev, addr, size, offset, phys_pg_pack); if (rc) goto dec_memhash_export_cnt; hl_dmabuf->phys_pg_pack = phys_pg_pack; hl_dmabuf->memhash_hnode = hnode; + hl_dmabuf->offset = offset; } else { - rc = validate_export_params_no_mmu(hdev, export_addr, size); + rc = validate_export_params_no_mmu(hdev, addr, size); if (rc) goto err_free_dmabuf_wrapper; - hl_dmabuf->device_phys_addr = export_addr; + hl_dmabuf->device_phys_addr = addr; } rc = export_dmabuf(ctx, hl_dmabuf, size, flags, dmabuf_fd); From patchwork Mon Sep 18 14:31:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oded Gabbay X-Patchwork-Id: 13389722 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 42A17CD37B0 for ; Mon, 18 Sep 2023 14:32:36 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 954A110E290; Mon, 18 Sep 2023 14:32:35 +0000 (UTC) Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by gabe.freedesktop.org (Postfix) with ESMTPS id 7B33110E190 for ; Mon, 18 Sep 2023 14:32:15 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 037ADB80E3D; Mon, 18 Sep 2023 14:32:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BE07BC32788; Mon, 18 Sep 2023 14:32:12 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695047533; bh=H+SspTrUvoR2QTMJuVAGKvR7+leUmqiHwlWLm0zDOc4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=dZgpsW+oahqq0e59ohFRHXyoMNIr0+t1VL3dMNee+JmKLPT+P6X0jqTBGfMnAHKjO gA/jzz6N+UBOvY1bAStC3lra2taBnBzR+18xSL9j3HSViiVPRhprmEsyVx4IIz9cug 7yLitHWq/GClSEzVuUJOuO3WAfPZCkBpfEbx/fDQzWJmO5AomM126N8TxuIE8RMWRU LLriBtTiLiw5rN4rMOXsn4B1Q0MYGa/T+yi9l8R35Iefpl6xQaf8zWog0HoKvr+E8f EqZd3z0nJfDWXJj+aXr2gUA5thWI0BwavEKUE6jZH5CF8FZtg2f+o5lQZUvpVyeksE 61hMRyGe/Sf3Q== From: Oded Gabbay To: dri-devel@lists.freedesktop.org Subject: [PATCH 07/10] accel/habanalabs: add debug prints to dump content of SG table for dma-buf Date: Mon, 18 Sep 2023 17:31:55 +0300 Message-Id: <20230918143158.903207-7-ogabbay@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230918143158.903207-1-ogabbay@kernel.org> References: <20230918143158.903207-1-ogabbay@kernel.org> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tomer Tayar Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Tomer Tayar Add debug prints to dump the content of the SG table which is prepared when the dma-buf map op is called. Signed-off-by: Tomer Tayar Reviewed-by: Oded Gabbay Signed-off-by: Oded Gabbay --- drivers/accel/habanalabs/common/memory.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/drivers/accel/habanalabs/common/memory.c b/drivers/accel/habanalabs/common/memory.c index 00e73b5573be..33fb78aba812 100644 --- a/drivers/accel/habanalabs/common/memory.c +++ b/drivers/accel/habanalabs/common/memory.c @@ -1679,6 +1679,13 @@ static struct sg_table *alloc_sgt_from_device_pages(struct hl_device *hdev, u64 */ sgt->orig_nents = 0; + dev_dbg(hdev->dev, "prepared SG table with %u entries for importer %s\n", + nents, dev_name(dev)); + for_each_sgtable_dma_sg(sgt, sg, i) + dev_dbg(hdev->dev, + "SG entry %d: address %#llx, length %#x\n", + i, sg_dma_address(sg), sg_dma_len(sg)); + return sgt; err_unmap: From patchwork Mon Sep 18 14:31:56 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oded Gabbay X-Patchwork-Id: 13389717 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 60491CD37B0 for ; Mon, 18 Sep 2023 14:32:22 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 7B73F10E1A4; Mon, 18 Sep 2023 14:32:18 +0000 (UTC) Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by gabe.freedesktop.org (Postfix) with ESMTPS id 4789B10E190 for ; Mon, 18 Sep 2023 14:32:16 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B8D7B61140; Mon, 18 Sep 2023 14:32:15 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 395EDC32789; Mon, 18 Sep 2023 14:32:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695047535; bh=sq1acTii1lIjGqy8tMjofVj9BdiUZdcszq3xd7MNZRA=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=MO1bz00QYbdgOgflrqOnQuIe2kburzb/X2OhV76Cp4uIL1A6kz/j5rjliYQBmCKuP U8xApsMIrdz8GtQI5eGmMn+SqVxDw/ScbRp5aqxzSEH5tvtg7cMvfgKPvxPJjTe/2X G3Bn+kPCL+uu5L1HSCfg31NivZNobO4wIycHtYH+oqslkilMgZu5D737v6BFWRb9dA ntPuVTJ3i8LkTLPb8YDDN3sDzxYh96NAKHuzSrn0awzQIdaBigIqZGB5PvEHaqbNrH MdSIDAWu2bzymxBQV7XMYxxkLyHP0aBdQCPSAGUIsxDWKRmIfylnMP6XkLnIQ3yu1q AOaGRqlc0Py1A== From: Oded Gabbay To: dri-devel@lists.freedesktop.org Subject: [PATCH 08/10] accel/habanalabs: add fw status SHUTDOWN_PREP Date: Mon, 18 Sep 2023 17:31:56 +0300 Message-Id: <20230918143158.903207-8-ogabbay@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230918143158.903207-1-ogabbay@kernel.org> References: <20230918143158.903207-1-ogabbay@kernel.org> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Dafna Hirschfeld Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Dafna Hirschfeld update hl_boot_if.h from specs to include CPU_BOOT_STATUS_FW_SHUTDOWN_PREP Signed-off-by: Dafna Hirschfeld Reviewed-by: Oded Gabbay Signed-off-by: Oded Gabbay --- include/linux/habanalabs/hl_boot_if.h | 2 ++ 1 file changed, 2 insertions(+) diff --git a/include/linux/habanalabs/hl_boot_if.h b/include/linux/habanalabs/hl_boot_if.h index 7de8a5786a36..93366d5621fd 100644 --- a/include/linux/habanalabs/hl_boot_if.h +++ b/include/linux/habanalabs/hl_boot_if.h @@ -394,6 +394,8 @@ enum cpu_boot_status { CPU_BOOT_STATUS_WAITING_FOR_BOOT_FIT = 16, /* Internal Security has been initialized, device can be accessed */ CPU_BOOT_STATUS_SECURITY_READY = 17, + /* FW component is preparing to shutdown and communication with host is not available */ + CPU_BOOT_STATUS_FW_SHUTDOWN_PREP = 18, }; enum kmd_msg { From patchwork Mon Sep 18 14:31:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oded Gabbay X-Patchwork-Id: 13389719 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7AAA9CD37B0 for ; Mon, 18 Sep 2023 14:32:30 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id DEE6A10E289; Mon, 18 Sep 2023 14:32:29 +0000 (UTC) Received: from ams.source.kernel.org (ams.source.kernel.org [145.40.68.75]) by gabe.freedesktop.org (Postfix) with ESMTPS id 506E710E286 for ; Mon, 18 Sep 2023 14:32:19 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id F0E45B80EA0; Mon, 18 Sep 2023 14:32:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id B4443C32788; Mon, 18 Sep 2023 14:32:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695047536; bh=ir8WshTvQBQW8S1rqaBHbhJZZDNdSS7ityBkYcfQyks=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=OP6af9s51LO+joEtzrveAFTPSzv1fv0ChtD2FsZL2H66rymnZiFv38SWQuhgJIEkt xilfNwAB/nNPpHNPRxPJPGJCyGxeBbKQZWZNJ8VAdTrWs59BXg07OFY/SS+eRhNR9b CEdbSC7Kw2wubTytx6+j5Oxin5ARFozAtJh2NYBemJj0YvaKqUZPX5tavTFpNVgq/M UZyZWL5KTF9jVDAkr9qVj6OJdZviKFs8bwUll4w/i1tlLEsTcHuFesBihGdbleljZ2 YHsCSWyjmoPmUAXyp07Gc5O9N0KQJhb8Jzi80p+yXVyaN0nQfo0qpbn3uYhrT4HfBm aM7djyIsN2H/Q== From: Oded Gabbay To: dri-devel@lists.freedesktop.org Subject: [PATCH 09/10] accel/habanalabs: extend preboot timeout when preboot might take longer Date: Mon, 18 Sep 2023 17:31:57 +0300 Message-Id: <20230918143158.903207-9-ogabbay@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230918143158.903207-1-ogabbay@kernel.org> References: <20230918143158.903207-1-ogabbay@kernel.org> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Dafna Hirschfeld Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Dafna Hirschfeld There are cases such when FW runs MBIST, that preboot is expected to take longer than the usual. In such cases the firmware reports status SECURITY_READY/IN_PREBOOT and we extend the timeout waiting for it. This is currently implemented for Gaudi2 only. Signed-off-by: Dafna Hirschfeld Reviewed-by: Oded Gabbay Signed-off-by: Oded Gabbay --- drivers/accel/habanalabs/common/firmware_if.c | 26 ++++++++++++++++--- drivers/accel/habanalabs/common/habanalabs.h | 3 +++ drivers/accel/habanalabs/gaudi2/gaudi2.c | 2 ++ drivers/accel/habanalabs/gaudi2/gaudi2P.h | 1 + 4 files changed, 29 insertions(+), 3 deletions(-) diff --git a/drivers/accel/habanalabs/common/firmware_if.c b/drivers/accel/habanalabs/common/firmware_if.c index 2a6dfea3d27d..0fb360cca9cc 100644 --- a/drivers/accel/habanalabs/common/firmware_if.c +++ b/drivers/accel/habanalabs/common/firmware_if.c @@ -1474,8 +1474,9 @@ static void detect_cpu_boot_status(struct hl_device *hdev, u32 status) int hl_fw_wait_preboot_ready(struct hl_device *hdev) { struct pre_fw_load_props *pre_fw_load = &hdev->fw_loader.pre_fw_load; - u32 status; - int rc; + u32 status = 0, timeout; + int rc, tries = 1; + bool preboot_still_runs; /* Need to check two possible scenarios: * @@ -1485,6 +1486,8 @@ int hl_fw_wait_preboot_ready(struct hl_device *hdev) * All other status values - for older firmwares where the uboot was * loaded from the FLASH */ + timeout = pre_fw_load->wait_for_preboot_timeout; +retry: rc = hl_poll_timeout( hdev, pre_fw_load->cpu_boot_status_reg, @@ -1493,7 +1496,24 @@ int hl_fw_wait_preboot_ready(struct hl_device *hdev) (status == CPU_BOOT_STATUS_READY_TO_BOOT) || (status == CPU_BOOT_STATUS_WAITING_FOR_BOOT_FIT), hdev->fw_poll_interval_usec, - pre_fw_load->wait_for_preboot_timeout); + timeout); + /* + * if F/W reports "security-ready" it means preboot might take longer. + * If the field 'wait_for_preboot_extended_timeout' is non 0 we wait again + * with that timeout + */ + preboot_still_runs = (status == CPU_BOOT_STATUS_SECURITY_READY || + status == CPU_BOOT_STATUS_IN_PREBOOT || + status == CPU_BOOT_STATUS_FW_SHUTDOWN_PREP || + status == CPU_BOOT_STATUS_DRAM_RDY); + + if (rc && tries && preboot_still_runs) { + tries--; + if (pre_fw_load->wait_for_preboot_extended_timeout) { + timeout = pre_fw_load->wait_for_preboot_extended_timeout; + goto retry; + } + } if (rc) { detect_cpu_boot_status(hdev, status); diff --git a/drivers/accel/habanalabs/common/habanalabs.h b/drivers/accel/habanalabs/common/habanalabs.h index 874ae76cbd78..3c224942a758 100644 --- a/drivers/accel/habanalabs/common/habanalabs.h +++ b/drivers/accel/habanalabs/common/habanalabs.h @@ -1410,6 +1410,8 @@ struct dynamic_fw_load_mgr { * @boot_err0_reg: boot_err0 register address * @boot_err1_reg: boot_err1 register address * @wait_for_preboot_timeout: timeout to poll for preboot ready + * @wait_for_preboot_extended_timeout: timeout to pull for preboot ready in case where we know + * preboot needs longer time. */ struct pre_fw_load_props { u32 cpu_boot_status_reg; @@ -1418,6 +1420,7 @@ struct pre_fw_load_props { u32 boot_err0_reg; u32 boot_err1_reg; u32 wait_for_preboot_timeout; + u32 wait_for_preboot_extended_timeout; }; /** diff --git a/drivers/accel/habanalabs/gaudi2/gaudi2.c b/drivers/accel/habanalabs/gaudi2/gaudi2.c index 867175431418..35db02e5010b 100644 --- a/drivers/accel/habanalabs/gaudi2/gaudi2.c +++ b/drivers/accel/habanalabs/gaudi2/gaudi2.c @@ -4825,6 +4825,8 @@ static void gaudi2_init_firmware_preload_params(struct hl_device *hdev) pre_fw_load->boot_err0_reg = mmCPU_BOOT_ERR0; pre_fw_load->boot_err1_reg = mmCPU_BOOT_ERR1; pre_fw_load->wait_for_preboot_timeout = GAUDI2_PREBOOT_REQ_TIMEOUT_USEC; + pre_fw_load->wait_for_preboot_extended_timeout = + GAUDI2_PREBOOT_EXTENDED_REQ_TIMEOUT_USEC; } static void gaudi2_init_firmware_loader(struct hl_device *hdev) diff --git a/drivers/accel/habanalabs/gaudi2/gaudi2P.h b/drivers/accel/habanalabs/gaudi2/gaudi2P.h index 14e281fd9895..9b9eef0d97d6 100644 --- a/drivers/accel/habanalabs/gaudi2/gaudi2P.h +++ b/drivers/accel/habanalabs/gaudi2/gaudi2P.h @@ -84,6 +84,7 @@ #define CORESIGHT_TIMEOUT_USEC 100000 /* 100 ms */ #define GAUDI2_PREBOOT_REQ_TIMEOUT_USEC 25000000 /* 25s */ +#define GAUDI2_PREBOOT_EXTENDED_REQ_TIMEOUT_USEC 85000000 /* 85s */ #define GAUDI2_BOOT_FIT_REQ_TIMEOUT_USEC 10000000 /* 10s */ From patchwork Mon Sep 18 14:31:58 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Oded Gabbay X-Patchwork-Id: 13389720 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5EA4EC46CA1 for ; Mon, 18 Sep 2023 14:32:32 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 9D84610E2A8; Mon, 18 Sep 2023 14:32:31 +0000 (UTC) Received: from sin.source.kernel.org (sin.source.kernel.org [145.40.73.55]) by gabe.freedesktop.org (Postfix) with ESMTPS id 9288A10E289 for ; Mon, 18 Sep 2023 14:32:23 +0000 (UTC) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits)) (No client certificate requested) by sin.source.kernel.org (Postfix) with ESMTPS id ED1C5CE0FEA; Mon, 18 Sep 2023 14:32:20 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3692FC32789; Mon, 18 Sep 2023 14:32:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1695047538; bh=YMXU418eyrpH/EuRlc9SrcB0SbdT9/4ds3DNyrVCxH4=; h=From:To:Cc:Subject:Date:In-Reply-To:References:From; b=n3rAifWa3IozPiJUk6DWlVyF2Uk+kX9bmM7Z8JPQq7BAN9Z2Qz+e2+j7/wNWoNRro m/tyfGkNQwcE56GkS+0IcJGRxon2PM0maScIMN0cEb3tCVy24nsDCLLLf+KikrujAP +GFz9Z/VXv4/IOVMSB8Wju5gtpQ7BP6+9jHrKK/5EEB5fablCjkexGNwtJQnv8B0jK qyF+wmtofCMVsV8ZqqWyt87CNrNzBNzPGu8nkvx4lZdb/6yJdwIW2EvKfSA4g9laF8 G0l59CEupBxbSgjih043d1FbsDEFXHCVp8G/MlDCWwR9cfTtMbkxEWIFCRHk6NDrLP bop/6PQ3PN60A== From: Oded Gabbay To: dri-devel@lists.freedesktop.org Subject: [PATCH 10/10] accel/habanalabs: update boot status print Date: Mon, 18 Sep 2023 17:31:58 +0300 Message-Id: <20230918143158.903207-10-ogabbay@kernel.org> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20230918143158.903207-1-ogabbay@kernel.org> References: <20230918143158.903207-1-ogabbay@kernel.org> MIME-Version: 1.0 X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Ariel Suller Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" From: Ariel Suller FW shutdown preparation status was added to spec. Signed-off-by: Ariel Suller Reviewed-by: Oded Gabbay Signed-off-by: Oded Gabbay --- drivers/accel/habanalabs/common/firmware_if.c | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/drivers/accel/habanalabs/common/firmware_if.c b/drivers/accel/habanalabs/common/firmware_if.c index 0fb360cca9cc..47e8384134aa 100644 --- a/drivers/accel/habanalabs/common/firmware_if.c +++ b/drivers/accel/habanalabs/common/firmware_if.c @@ -1464,6 +1464,10 @@ static void detect_cpu_boot_status(struct hl_device *hdev, u32 status) dev_err(hdev->dev, "Device boot progress - Stuck in preboot after security initialization\n"); break; + case CPU_BOOT_STATUS_FW_SHUTDOWN_PREP: + dev_err(hdev->dev, + "Device boot progress - Stuck in preparation for shutdown\n"); + break; default: dev_err(hdev->dev, "Device boot progress - Invalid or unexpected status code %d\n", status);