From patchwork Fri Aug 21 13:54:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 11729533 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6502C138C for ; Fri, 21 Aug 2020 14:00:01 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3DC6820720 for ; Fri, 21 Aug 2020 14:00:01 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Y2oHNH/t" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3DC6820720 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=MqsmTjJriQ9z8lMfBHVllMxHn7RIMaxJZ7BC1HcvWLs=; b=Y2oHNH/tw44mH+1YxwGm00W85 kEj+ZAlo999T9o2Y1D4TCBqbLWZqdFbAScCNNQBz8AwTcl+pQ4yROSqkbhxQvm1uTOGqevZY9LJ46 p157lcbS3CxSODJ7z4zc9tOihExcIhssG1CFProTrWZ3vfvS27x4TnQ46vNBwYs6p25d1VXiqD+dM Rv+sqJhG7wZr1IxRgBxQeAyEsMuIp2QyNaJKrotp0SczCZKKTMwNW+VcjE+HMDw8bD6zLPJ6cs0Lw SbMFkV8eykONN75bC2kv3hJS2/1PRstvx9+5gOApF6UAVkS2G/+7ivsbHlxvCndI4PJjLPjFflwTR dYIsag/Eg==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k97Z1-0005Sg-RF; Fri, 21 Aug 2020 13:58:40 +0000 Received: from szxga04-in.huawei.com ([45.249.212.190] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k97Yp-0005KX-Ba for linux-arm-kernel@lists.infradead.org; Fri, 21 Aug 2020 13:58:29 +0000 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 1F74097332D8AA97487D; Fri, 21 Aug 2020 21:58:15 +0800 (CST) Received: from localhost.localdomain (10.69.192.58) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.487.0; Fri, 21 Aug 2020 21:58:04 +0800 From: John Garry To: , Subject: [PATCH v2 1/2] iommu/arm-smmu-v3: Calculate max commands per batch Date: Fri, 21 Aug 2020 21:54:21 +0800 Message-ID: <1598018062-175608-2-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1598018062-175608-1-git-send-email-john.garry@huawei.com> References: <1598018062-175608-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200821_095827_701434_4CC28FFE X-CRM114-Status: GOOD ( 14.60 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.190 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.190 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: maz@kernel.org, joro@8bytes.org, John Garry , linux-kernel@vger.kernel.org, linuxarm@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Calculate the batch size limit such that all CPUs in the system cannot issue so many commands as to fill the command queue. Signed-off-by: John Garry --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 46 +++++++++++++++++++-- 1 file changed, 42 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 7196207be7ea..a705fa3e18ea 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -532,6 +532,7 @@ struct arm_smmu_ll_queue { u8 __pad[SMP_CACHE_BYTES]; } ____cacheline_aligned_in_smp; u32 max_n_shift; + u32 max_cmd_per_batch; }; struct arm_smmu_queue { @@ -1515,7 +1516,10 @@ static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu, struct arm_smmu_cmdq_batch *cmds, struct arm_smmu_cmdq_ent *cmd) { - if (cmds->num == CMDQ_BATCH_ENTRIES) { + struct arm_smmu_cmdq *q = &smmu->cmdq; + struct arm_smmu_ll_queue *llq = &q->q.llq; + + if (cmds->num == llq->max_cmd_per_batch) { arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmds, cmds->num, false); cmds->num = 0; } @@ -3177,6 +3181,41 @@ static int arm_smmu_init_one_queue(struct arm_smmu_device *smmu, return 0; } +static int arm_smmu_init_cmd_queue(struct arm_smmu_device *smmu, + struct arm_smmu_queue *q, + unsigned long prod_off, + unsigned long cons_off, + size_t dwords) +{ + u32 cpus = num_possible_cpus(), entries_for_prod; + int ret; + + ret = arm_smmu_init_one_queue(smmu, q, prod_off, cons_off, dwords, + "cmdq"); + if (ret) + return ret; + + entries_for_prod = 1 << q->llq.max_n_shift; + + /* + * We need at least 2 commands in a batch (1 x CMD_SYNC and 1 x + * whatever else). + */ + if (entries_for_prod < 2 * cpus) { + dev_err(smmu->dev, "command queue size too small, suggest reduce #CPUs\n"); + return -ENXIO; + } + + /* + * When finding max_cmd_per_batch, deduct 1 entry per batch to take + * account of a CMD_SYNC being issued also. + */ + q->llq.max_cmd_per_batch = min((entries_for_prod / cpus) - 1, + (u32)CMDQ_BATCH_ENTRIES); + + return 0; +} + static void arm_smmu_cmdq_free_bitmap(void *data) { unsigned long *bitmap = data; @@ -3210,9 +3249,8 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu) int ret; /* cmdq */ - ret = arm_smmu_init_one_queue(smmu, &smmu->cmdq.q, ARM_SMMU_CMDQ_PROD, - ARM_SMMU_CMDQ_CONS, CMDQ_ENT_DWORDS, - "cmdq"); + ret = arm_smmu_init_cmd_queue(smmu, &smmu->cmdq.q, ARM_SMMU_CMDQ_PROD, + ARM_SMMU_CMDQ_CONS, CMDQ_ENT_DWORDS); if (ret) return ret; From patchwork Fri Aug 21 13:54:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 11729529 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 00ACC739 for ; Fri, 21 Aug 2020 13:58:47 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id C1C7C22B4E for ; Fri, 21 Aug 2020 13:58:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="b1JuBWOn" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C1C7C22B4E Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=pKRhOraXKvuG8ruD/LQPfgGDo8uLUGJmicSK6ZbWKjI=; b=b1JuBWOnvuRfWJjZ/rzpBx+nq PIqB2dnoKjBXTgJn5RlwyYqRB467M8g4evawVbZxjF2L+ApC1Kw2BPxRUwjny5Xtuww1qMWogKa72 WAnDWe29+DtZJuBRQ5fQ72UlkTr7bpYTXzwl0SgUL7RsoMqRT7pao1KWdrks6b822bGTotbQOp+Ca HR9720YKFrpz4k0f1CY0roBX6ephNTohklBQWMl44te2CcLgLO1T6Z3LhzqgCbl23jPOuR9zkeZRt T3X8TJIbMElRNg6RugU30quKzEkTvddIT9yKTkiQCO+IIUIrDDVeVoMwYITtQabJFs1+UA0IS+n8Y xnQRNzUAQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k97Yw-0005RC-PT; Fri, 21 Aug 2020 13:58:34 +0000 Received: from szxga04-in.huawei.com ([45.249.212.190] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k97Yk-0005KY-8F for linux-arm-kernel@lists.infradead.org; Fri, 21 Aug 2020 13:58:26 +0000 Received: from DGGEMS411-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id 18C9F5937A24790A5672; Fri, 21 Aug 2020 21:58:15 +0800 (CST) Received: from localhost.localdomain (10.69.192.58) by DGGEMS411-HUB.china.huawei.com (10.3.19.211) with Microsoft SMTP Server id 14.3.487.0; Fri, 21 Aug 2020 21:58:05 +0800 From: John Garry To: , Subject: [PATCH v2 2/2] iommu/arm-smmu-v3: Remove cmpxchg() in arm_smmu_cmdq_issue_cmdlist() Date: Fri, 21 Aug 2020 21:54:22 +0800 Message-ID: <1598018062-175608-3-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1598018062-175608-1-git-send-email-john.garry@huawei.com> References: <1598018062-175608-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200821_095822_866019_5B580A42 X-CRM114-Status: GOOD ( 28.28 ) X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.190 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.190 listed in list.dnswl.org] -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 SPF_PASS SPF: sender matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: maz@kernel.org, joro@8bytes.org, John Garry , linux-kernel@vger.kernel.org, linuxarm@huawei.com, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org It has been shown that the cmpxchg() for finding space in the cmdq can be a bottleneck: - for more CPUs contending the cmdq, the cmpxchg() will fail more often - since the software-maintained cons pointer is updated on the same 64b memory region, the chance of cmpxchg() failure increases again The cmpxchg() is removed as part of 2 related changes: - Update prod and cmdq owner in a single atomic64 add operation. The prod value is updated in one 32b region, while the "owner" is updated in another 32b region. For the "owner", we now count this value, instead of setting a flag. Similar to before, once the owner has finished gathering, it will clear a mask. As such, a CPU declares itself as the "owner" when it reads zero for this region. The owner is now responsible for all cmdq locking to avoid possible deadlock. The owner will lock the cmdq for all non-owers it has gathered when they have space in the queue and have written their entries. - Check for space in the cmdq after the prod pointer has been assigned. We don't bother checking for space in the cmdq before assigning the prod pointer, as this would be super racy. So since the prod pointer is updated unconditionally, it would be common for no space to be available in the cmdq when prod is assigned - that is, according the software-maintained prod and cons pointer. So now it must be ensured that the entries are not yet written and not until there is space. How the prod pointer is maintained also leads to a strange condition where the prod pointer can wrap past the cons pointer. We can detect this condition, and report no space here. However, a prod pointer progressed twice past the cons pointer cannot be detected. But it can be ensured that this that this scenario does not occur, as we limit the amount of commands any CPU can issue at any given time, such that we cannot progress prod pointer further. Signed-off-by: John Garry --- drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c | 120 ++++++++++++-------- 1 file changed, 72 insertions(+), 48 deletions(-) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index a705fa3e18ea..d3c1400c644c 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -521,16 +521,17 @@ struct arm_smmu_cmdq_ent { struct arm_smmu_ll_queue { union { u64 val; + atomic64_t atomic; struct { + struct { + u16 sync; + u16 count; + }; u32 prod; - u32 cons; }; - struct { - atomic_t prod; - atomic_t cons; - } atomic; u8 __pad[SMP_CACHE_BYTES]; } ____cacheline_aligned_in_smp; + u32 cons; u32 max_n_shift; u32 max_cmd_per_batch; }; @@ -771,10 +772,19 @@ static bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) prod = Q_IDX(q, q->prod); cons = Q_IDX(q, q->cons); - if (Q_WRP(q, q->prod) == Q_WRP(q, q->cons)) + if (Q_WRP(q, q->prod) == Q_WRP(q, q->cons)) { + /* check if we have wrapped, meaning definitely no space */ + if (cons > prod) + return false; + space = (1 << q->max_n_shift) - (prod - cons); - else + } else { + /* similar check to above */ + if (prod > cons) + return false; + space = cons - prod; + } return space >= n; } @@ -1072,7 +1082,7 @@ static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu) * fails if the caller appears to be the last lock holder (yes, this is * racy). All successful UNLOCK routines have RELEASE semantics. */ -static void arm_smmu_cmdq_shared_lock(struct arm_smmu_cmdq *cmdq) +static void arm_smmu_cmdq_shared_lock(struct arm_smmu_cmdq *cmdq, int count) { int val; @@ -1082,12 +1092,12 @@ static void arm_smmu_cmdq_shared_lock(struct arm_smmu_cmdq *cmdq) * to INT_MIN so these increments won't hurt as the value will remain * negative. */ - if (atomic_fetch_inc_relaxed(&cmdq->lock) >= 0) + if (atomic_fetch_add_relaxed(count, &cmdq->lock) >= 0) return; do { val = atomic_cond_read_relaxed(&cmdq->lock, VAL >= 0); - } while (atomic_cmpxchg_relaxed(&cmdq->lock, val, val + 1) != val); + } while (atomic_cmpxchg_relaxed(&cmdq->lock, val, val + count) != val); } static void arm_smmu_cmdq_shared_unlock(struct arm_smmu_cmdq *cmdq) @@ -1236,13 +1246,14 @@ static int arm_smmu_cmdq_poll_until_not_full(struct arm_smmu_device *smmu, if (arm_smmu_cmdq_exclusive_trylock_irqsave(cmdq, flags)) { WRITE_ONCE(cmdq->q.llq.cons, readl_relaxed(cmdq->q.cons_reg)); arm_smmu_cmdq_exclusive_unlock_irqrestore(cmdq, flags); - llq->val = READ_ONCE(cmdq->q.llq.val); + llq->cons = READ_ONCE(cmdq->q.llq.cons); return 0; } queue_poll_init(smmu, &qp); do { - llq->val = READ_ONCE(smmu->cmdq.q.llq.val); + llq->prod = READ_ONCE(smmu->cmdq.q.llq.prod); + llq->cons = READ_ONCE(smmu->cmdq.q.llq.cons); if (!queue_full(llq)) break; @@ -1289,7 +1300,7 @@ static int __arm_smmu_cmdq_poll_until_consumed(struct arm_smmu_device *smmu, int ret = 0; queue_poll_init(smmu, &qp); - llq->val = READ_ONCE(smmu->cmdq.q.llq.val); + llq->cons = READ_ONCE(smmu->cmdq.q.llq.cons); do { if (queue_consumed(llq, prod)) break; @@ -1383,53 +1394,46 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, struct arm_smmu_cmdq *cmdq = &smmu->cmdq; struct arm_smmu_ll_queue llq = { .max_n_shift = cmdq->q.llq.max_n_shift, - }, head = llq; + }, head = llq, space = { + .max_n_shift = cmdq->q.llq.max_n_shift, + /* We would need 2^16 CPUs gathered for overflow ... */ + .count = 1, + .sync = sync, + .prod = n + sync, + }; + u32 prod_mask = GENMASK(cmdq->q.llq.max_n_shift, 0); int ret = 0; /* 1. Allocate some space in the queue */ local_irq_save(flags); - llq.val = READ_ONCE(cmdq->q.llq.val); - do { - u64 old; - while (!queue_has_space(&llq, n + sync)) { - local_irq_restore(flags); - if (arm_smmu_cmdq_poll_until_not_full(smmu, &llq)) - dev_err_ratelimited(smmu->dev, "CMDQ timeout\n"); - local_irq_save(flags); - } + llq.val = atomic64_fetch_add(space.val, &cmdq->q.llq.atomic); + llq.prod &= prod_mask; + owner = !llq.count; + head.prod = queue_inc_prod_n(&llq, n + sync); - head.cons = llq.cons; - head.prod = queue_inc_prod_n(&llq, n + sync) | - CMDQ_PROD_OWNED_FLAG; - - old = cmpxchg_relaxed(&cmdq->q.llq.val, llq.val, head.val); - if (old == llq.val) - break; + /* + * Ensure it's safe to write the entries. For this, we need to ensure + * that there is space in the queue from our prod pointer. + */ + space.cons = READ_ONCE(cmdq->q.llq.cons); + space.prod = llq.prod; + while (!queue_has_space(&space, n + sync)) { + if (arm_smmu_cmdq_poll_until_not_full(smmu, &space)) + dev_err_ratelimited(smmu->dev, "CMDQ timeout\n"); - llq.val = old; - } while (1); - owner = !(llq.prod & CMDQ_PROD_OWNED_FLAG); - head.prod &= ~CMDQ_PROD_OWNED_FLAG; - llq.prod &= ~CMDQ_PROD_OWNED_FLAG; + space.prod = llq.prod; + } /* * 2. Write our commands into the queue - * Dependency ordering from the cmpxchg() loop above. + * Dependency ordering from the space-checking while loop, above. */ arm_smmu_cmdq_write_entries(cmdq, cmds, llq.prod, n); if (sync) { prod = queue_inc_prod_n(&llq, n); arm_smmu_cmdq_build_sync_cmd(cmd_sync, smmu, prod); queue_write(Q_ENT(&cmdq->q, prod), cmd_sync, CMDQ_ENT_DWORDS); - - /* - * In order to determine completion of our CMD_SYNC, we must - * ensure that the queue can't wrap twice without us noticing. - * We achieve that by taking the cmdq lock as shared before - * marking our slot as valid. - */ - arm_smmu_cmdq_shared_lock(cmdq); } /* 3. Mark our slots as valid, ensuring commands are visible first */ @@ -1438,13 +1442,20 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, /* 4. If we are the owner, take control of the SMMU hardware */ if (owner) { + struct arm_smmu_ll_queue tmp = { + .sync = ~0, + .count = ~0, + .prod = ~prod_mask, + }; + int sync_count; /* a. Wait for previous owner to finish */ atomic_cond_read_relaxed(&cmdq->owner_prod, VAL == llq.prod); - /* b. Stop gathering work by clearing the owned flag */ - prod = atomic_fetch_andnot_relaxed(CMDQ_PROD_OWNED_FLAG, - &cmdq->q.llq.atomic.prod); - prod &= ~CMDQ_PROD_OWNED_FLAG; + /* b. Stop gathering work by clearing the owned mask */ + tmp.val = atomic64_fetch_andnot_relaxed(tmp.val, + &cmdq->q.llq.atomic); + prod = Q_WRP(&llq, tmp.prod) | Q_IDX(&llq, tmp.prod); + sync_count = tmp.sync; /* * c. Wait for any gathered work to be written to the queue. @@ -1453,6 +1464,19 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, */ arm_smmu_cmdq_poll_valid_map(cmdq, llq.prod, prod); + /* + * In order to determine completion of the CMD_SYNCs, we must + * ensure that the queue can't wrap twice without us noticing. + * We achieve that by taking the cmdq lock as shared before + * progressing the prod pointer. + * The owner does this for all the non-owners it has gathered. + * Otherwise, some non-owner(s) may lock the cmdq, blocking the + * cons being updating. This could be when the cmdq has just + * become full. In this case, other sibling non-owners could be + * blocked from progressing, leading to deadlock. + */ + arm_smmu_cmdq_shared_lock(cmdq, sync_count); + /* * d. Advance the hardware prod pointer * Control dependency ordering from the entries becoming valid.