From patchwork Mon Jun 22 17:28:37 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 11618543 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AEE81618 for ; Mon, 22 Jun 2020 17:35:19 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8828520732 for ; Mon, 22 Jun 2020 17:35:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="Kzp9HqmC" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8828520732 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=SaP4qq2+Gpn+9rklI+8vsj8bdBfrdJLklbBXh9I8I1c=; b=Kzp9HqmC0B/8FMx+iciELilHJ B8fDBQWicWqdLJtaKf5vG8Xj5tZWCxq0wWoDJf1HT6C+Py85AnRqT10xz25RZeUtaSLnCVGeNYDw/ 9V7dg85e6ZOUqSogjyiuEdE075JKKnsNGzJlXuIngzluaxl26+BdGpVy1cZnBBWpGkAs+UHd1qtoC PBI4JoYzGOwyKrP05c6oIUks5ibpMp1dS6Clszd5oUFSKLW+c92IRJ3eiRpk8ZwNitt0EI6+wApWX F8y0XojK0hY/8N9yYwJH01aAIYS1epVcvUqp+aao32XnNb31af4ISPMiiwHLbYy/0CtIIqwn3cGlI znnw9IgbQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnQKA-0004n3-LW; Mon, 22 Jun 2020 17:33:38 +0000 Received: from szxga05-in.huawei.com ([45.249.212.191] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnQK4-0004iu-7G for linux-arm-kernel@lists.infradead.org; Mon, 22 Jun 2020 17:33:34 +0000 Received: from DGGEMS409-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 905A44100360CC9089D5; Tue, 23 Jun 2020 01:32:53 +0800 (CST) Received: from localhost.localdomain (10.69.192.58) by DGGEMS409-HUB.china.huawei.com (10.3.19.209) with Microsoft SMTP Server id 14.3.487.0; Tue, 23 Jun 2020 01:32:42 +0800 From: John Garry To: , Subject: [PATCH 1/4] iommu/arm-smmu-v3: Fix trivial typo Date: Tue, 23 Jun 2020 01:28:37 +0800 Message-ID: <1592846920-45338-2-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1592846920-45338-1-git-send-email-john.garry@huawei.com> References: <1592846920-45338-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-CFilter-Loop: Reflected X-Spam-Note: CRM114 invocation failed X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.191 listed in wl.mailspike.net] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.191 listed in list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: trivial@kernel.org, maz@kernel.org, joro@8bytes.org, John Garry , linuxarm@huawei.com, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Set "cmq" -> "cmdq". Signed-off-by: John Garry --- drivers/iommu/arm-smmu-v3.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index f578677a5c41..a8e814c652fe 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -1479,7 +1479,7 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, } /* - * Try to unlock the cmq lock. This will fail if we're the last + * Try to unlock the cmdq lock. This will fail if we're the last * reader, in which case we can safely update cmdq->q.llq.cons */ if (!arm_smmu_cmdq_shared_tryunlock(cmdq)) { From patchwork Mon Jun 22 17:28:38 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 11618533 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 048A0138C for ; Mon, 22 Jun 2020 17:34:49 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D132120656 for ; Mon, 22 Jun 2020 17:34:48 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="0eEy/TaI" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D132120656 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=mdbA4GoPvCvtzgxhR5tbvMZaQ42ayi5Q2GZApswvh1s=; b=0eEy/TaIbyd6iupF7CxwFcP27 SO0TuuQMgsFNBxlQzRsFvv0KTulANLjwL5rCvQg5i4f53rGxww3A09LWana1GKbBUDmBnRltxhRol vM6O4sIbs9V+eE0eG/iwvtEdFFy2jhjauO3fjdCnTWyeqtlzp3aL59kFg0xNnef31HLBjDEh48TBi f4XLn8Do3rFt6U2RzEjytEG6KxaDqY4d/6qTXmMsze4qGBfByg5zbP8ZsgkTvVEeNb4jw4cxMCNPh ffydBF1MB22YbkcPdQ7cr59iu62h3S1qDcMmMeikdkxKASaEH/fL6kO+V1xBAaChzzCCeHSiksR+C jDmfbab0A==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnQJg-0004ao-MU; Mon, 22 Jun 2020 17:33:08 +0000 Received: from szxga05-in.huawei.com ([45.249.212.191] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnQJb-0004Y7-Kr for linux-arm-kernel@lists.infradead.org; Mon, 22 Jun 2020 17:33:05 +0000 Received: from DGGEMS409-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 89B9DE93CC20475B994A; Tue, 23 Jun 2020 01:32:53 +0800 (CST) Received: from localhost.localdomain (10.69.192.58) by DGGEMS409-HUB.china.huawei.com (10.3.19.209) with Microsoft SMTP Server id 14.3.487.0; Tue, 23 Jun 2020 01:32:42 +0800 From: John Garry To: , Subject: [PATCH 2/4] iommu/arm-smmu-v3: Calculate bits for prod and owner Date: Tue, 23 Jun 2020 01:28:38 +0800 Message-ID: <1592846920-45338-3-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1592846920-45338-1-git-send-email-john.garry@huawei.com> References: <1592846920-45338-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-CFilter-Loop: Reflected X-Spam-Note: CRM114 invocation failed X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.191 listed in list.dnswl.org] 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.191 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: trivial@kernel.org, maz@kernel.org, joro@8bytes.org, John Garry , linuxarm@huawei.com, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org Since the arm_smmu_ll_queue.prod will be for counting the "owner" value and also HW prod pointer, calculate how many bits are available for and used by each. This is based on the number of possible CPUs in the system. And we require that each CPU can issue a minimum of 2 commands per batch - 1 x CMD_SYNC and at least 1 x other. Ignoring limits of HW max_n_shift and HW cmdq memory allocation, approx 32K is the max supported CPUs. For this, max_n_shift would be 15. Signed-off-by: John Garry --- drivers/iommu/arm-smmu-v3.c | 64 ++++++++++++++++++++++++++++++++++--- 1 file changed, 60 insertions(+), 4 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index a8e814c652fe..4e9677b066f1 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -532,6 +532,8 @@ struct arm_smmu_ll_queue { u8 __pad[SMP_CACHE_BYTES]; } ____cacheline_aligned_in_smp; u32 max_n_shift; + u32 max_cmd_per_batch; + u32 owner_count_shift; }; struct arm_smmu_queue { @@ -1515,7 +1517,10 @@ static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu, struct arm_smmu_cmdq_batch *cmds, struct arm_smmu_cmdq_ent *cmd) { - if (cmds->num == CMDQ_BATCH_ENTRIES) { + struct arm_smmu_cmdq *q = &smmu->cmdq; + struct arm_smmu_ll_queue *llq = &q->q.llq; + + if (cmds->num == llq->max_cmd_per_batch) { arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmds, cmds->num, false); cmds->num = 0; } @@ -3177,6 +3182,58 @@ static int arm_smmu_init_one_queue(struct arm_smmu_device *smmu, return 0; } +static int arm_smmu_init_cmd_queue(struct arm_smmu_device *smmu, + struct arm_smmu_queue *q, + unsigned long prod_off, + unsigned long cons_off, + size_t dwords) +{ + u32 cpus = num_possible_cpus(), bits_for_cmdq_owner, + bits_available_for_prod, entries_for_prod; + int ret; + + /* + * We can get the number of bits required for owner counting by + * log2(nr possible cpus) + 1 + */ + bits_for_cmdq_owner = ilog2(cpus) + 1; + + /* + * Add an extra bit to ensure prod(+wrap) do not overflow into + * owner count. + */ + bits_available_for_prod = 32 - 1 - bits_for_cmdq_owner; + + if (bits_available_for_prod < 1) /* How many CPUs??? */ + return -ENOMEM; + + q->llq.max_n_shift = min(q->llq.max_n_shift, bits_available_for_prod); + + ret = arm_smmu_init_one_queue(smmu, q, prod_off, cons_off, dwords, + "cmdq"); + if (ret) + return ret; + + entries_for_prod = 1 << q->llq.max_n_shift; + + /* + * We need at least 2 commands in a batch (1 x CMD_SYNC and 1 x + * whatever else). + */ + if (entries_for_prod < 2 * cpus) + return -ENOMEM; + + /* + * When finding max_cmd_per_batch, deduct 1 entry per batch to take + * account of CMD_SYNC + */ + q->llq.max_cmd_per_batch = min((entries_for_prod / cpus) - 1, + (u32)CMDQ_BATCH_ENTRIES); + q->llq.owner_count_shift = q->llq.max_n_shift + 2; + + return 0; +} + static void arm_smmu_cmdq_free_bitmap(void *data) { unsigned long *bitmap = data; @@ -3210,9 +3267,8 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu) int ret; /* cmdq */ - ret = arm_smmu_init_one_queue(smmu, &smmu->cmdq.q, ARM_SMMU_CMDQ_PROD, - ARM_SMMU_CMDQ_CONS, CMDQ_ENT_DWORDS, - "cmdq"); + ret = arm_smmu_init_cmd_queue(smmu, &smmu->cmdq.q, ARM_SMMU_CMDQ_PROD, + ARM_SMMU_CMDQ_CONS, CMDQ_ENT_DWORDS); if (ret) return ret; From patchwork Mon Jun 22 17:28:39 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 11618539 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 98112618 for ; Mon, 22 Jun 2020 17:35:10 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 716A720732 for ; Mon, 22 Jun 2020 17:35:10 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="MPRbKsIA"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="OjIIO3sJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 716A720732 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=U5Os8rXp5KNQ6U/eq/r8PiF+bG+CsSPwb+0j8cXiYcI=; b=MPRbKsIA6ba94+yUzpCpQ1tly KBEKKlWUeBpsA033g3W0J+cqHjXV0BE0AnBkBQ0F8yBOxiade3kByUb5iGUieb21gmsgamF0h1NEG z/gkTUObNtlLdsEY+q3jQ8zKiFOwgFReIQwi3Z20uDN3xXpGJ8l3ASoH27BYl7W4EAz3dlzv4UTll Okb3eDejAAAjHDQ9TifJFAaiQ1jNVInsZMoq8zVlY4HK20zCxTQgTxNKTNKvxIyYIUPgN6CIxaghS lWvythn/tk+dTCahYQLU3FcCtkReeCpABZQWR3FJHRe41UZcGxZrU+0sD2Jz1W7RZWZs1+fsd9XlP 2C9gylCcA==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnQJu-0004gY-MU; Mon, 22 Jun 2020 17:33:22 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnQJs-0004fT-7l for linux-arm-kernel@merlin.infradead.org; Mon, 22 Jun 2020 17:33:20 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Date:Subject:CC:To:From:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=DJt+CF08Nz7nSMVNZYOEQU7k+JPPjOeP9nDuL4ZI/Cg=; b=OjIIO3sJdsZOvbQUkMOI5isOmt Z0kJq03peCxTUvaybjyvqW518Y//fVsuJVfFj+UBo+q4zinevKzJgEL5bpXXjhpO2OFmGuz5aL9z4 +Q95Fcuq1gdRT+qgxsYHbf6qasaQfLJJ3Nd3kGoN6s2vU3/wZ8dMNMu+xj6vLvClRmxIO8JImWBb8 GAdyAzBKliUkmfR0evFK0sGIIKMUGjaZw5EVaIjdUev6JFMzMn4ooe7TFwQsV8JfyW/Uiu/eUKllG zAOcWYyRy7WhLN4OMDdGCLI187g/a3RZBZxdLTdeOoIQxS61BK5idtJTMh01TvTtEGawmW7w0OcOY n/8JWvcg==; Received: from szxga05-in.huawei.com ([45.249.212.191] helo=huawei.com) by casper.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnQJn-0002Ia-9h for linux-arm-kernel@lists.infradead.org; Mon, 22 Jun 2020 17:33:19 +0000 Received: from DGGEMS409-HUB.china.huawei.com (unknown [172.30.72.59]) by Forcepoint Email with ESMTP id 9675D3F52B4B178862C3; Tue, 23 Jun 2020 01:32:53 +0800 (CST) Received: from localhost.localdomain (10.69.192.58) by DGGEMS409-HUB.china.huawei.com (10.3.19.209) with Microsoft SMTP Server id 14.3.487.0; Tue, 23 Jun 2020 01:32:43 +0800 From: John Garry To: , Subject: [PATCH 3/4] iommu/arm-smmu-v3: Always issue a CMD_SYNC per batch Date: Tue, 23 Jun 2020 01:28:39 +0800 Message-ID: <1592846920-45338-4-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1592846920-45338-1-git-send-email-john.garry@huawei.com> References: <1592846920-45338-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200622_183316_862840_87F4753E X-CRM114-Status: GOOD ( 18.71 ) X-Spam-Score: -4.2 (----) X-Spam-Report: SpamAssassin version 3.4.4 on casper.infradead.org summary: Content analysis details: (-4.2 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.191 listed in list.dnswl.org] -0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.191 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record -0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: trivial@kernel.org, maz@kernel.org, joro@8bytes.org, John Garry , linuxarm@huawei.com, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org To ensure that a CPU does not send more than a permitted amount of commands to the cmdq, ensure that each batch includes a CMD_SYNC. When issuing a CMD_SYNC, we always wait for the consumption of its batch of commands - as such, we guarantee that any CPU will not issue more than its permitted amount. Signed-off-by: John Garry --- drivers/iommu/arm-smmu-v3.c | 86 +++++++++++++++++-------------------- 1 file changed, 39 insertions(+), 47 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 4e9677b066f1..45a39ccaf455 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -1373,11 +1373,15 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq, u64 *cmds, * - Command insertion is totally ordered, so if two CPUs each race to * insert their own list of commands then all of the commands from one * CPU will appear before any of the commands from the other CPU. + * + * - A CMD_SYNC is always inserted, ensuring that any CPU does not issue + * more than the permitted amount commands at once. */ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, - u64 *cmds, int n, bool sync) + u64 *cmds, int n) { u64 cmd_sync[CMDQ_ENT_DWORDS]; + const int sync = 1; u32 prod; unsigned long flags; bool owner; @@ -1419,19 +1423,17 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, * Dependency ordering from the cmpxchg() loop above. */ arm_smmu_cmdq_write_entries(cmdq, cmds, llq.prod, n); - if (sync) { - prod = queue_inc_prod_n(&llq, n); - arm_smmu_cmdq_build_sync_cmd(cmd_sync, smmu, prod); - queue_write(Q_ENT(&cmdq->q, prod), cmd_sync, CMDQ_ENT_DWORDS); - /* - * In order to determine completion of our CMD_SYNC, we must - * ensure that the queue can't wrap twice without us noticing. - * We achieve that by taking the cmdq lock as shared before - * marking our slot as valid. - */ - arm_smmu_cmdq_shared_lock(cmdq); - } + prod = queue_inc_prod_n(&llq, n); + arm_smmu_cmdq_build_sync_cmd(cmd_sync, smmu, prod); + queue_write(Q_ENT(&cmdq->q, prod), cmd_sync, CMDQ_ENT_DWORDS); + + /* + * In order to determine completion of our CMD_SYNC, we must ensure + * that the queue can't wrap twice without us noticing. We achieve that + * by taking the cmdq lock as shared before marking our slot as valid. + */ + arm_smmu_cmdq_shared_lock(cmdq); /* 3. Mark our slots as valid, ensuring commands are visible first */ dma_wmb(); @@ -1468,26 +1470,21 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, atomic_set_release(&cmdq->owner_prod, prod); } - /* 5. If we are inserting a CMD_SYNC, we must wait for it to complete */ - if (sync) { - llq.prod = queue_inc_prod_n(&llq, n); - ret = arm_smmu_cmdq_poll_until_sync(smmu, &llq); - if (ret) { - dev_err_ratelimited(smmu->dev, - "CMD_SYNC timeout at 0x%08x [hwprod 0x%08x, hwcons 0x%08x]\n", - llq.prod, - readl_relaxed(cmdq->q.prod_reg), - readl_relaxed(cmdq->q.cons_reg)); - } + /* 5. Since we always insert a CMD_SYNC, we must wait for it to complete */ + llq.prod = queue_inc_prod_n(&llq, n); + ret = arm_smmu_cmdq_poll_until_sync(smmu, &llq); + if (ret) + dev_err_ratelimited(smmu->dev, "CMD_SYNC timeout at 0x%08x [hwprod 0x%08x, hwcons 0x%08x]\n", + llq.prod, readl_relaxed(cmdq->q.prod_reg), + readl_relaxed(cmdq->q.cons_reg)); - /* - * Try to unlock the cmdq lock. This will fail if we're the last - * reader, in which case we can safely update cmdq->q.llq.cons - */ - if (!arm_smmu_cmdq_shared_tryunlock(cmdq)) { - WRITE_ONCE(cmdq->q.llq.cons, llq.cons); - arm_smmu_cmdq_shared_unlock(cmdq); - } + /* + * Try to unlock the cmdq lock. This will fail if we're the last reader, + * in which case we can safely update cmdq->q.llq.cons + */ + if (!arm_smmu_cmdq_shared_tryunlock(cmdq)) { + WRITE_ONCE(cmdq->q.llq.cons, llq.cons); + arm_smmu_cmdq_shared_unlock(cmdq); } local_irq_restore(flags); @@ -1505,12 +1502,7 @@ static int arm_smmu_cmdq_issue_cmd(struct arm_smmu_device *smmu, return -EINVAL; } - return arm_smmu_cmdq_issue_cmdlist(smmu, cmd, 1, false); -} - -static int arm_smmu_cmdq_issue_sync(struct arm_smmu_device *smmu) -{ - return arm_smmu_cmdq_issue_cmdlist(smmu, NULL, 0, true); + return arm_smmu_cmdq_issue_cmdlist(smmu, cmd, 1); } static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu, @@ -1521,7 +1513,7 @@ static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu, struct arm_smmu_ll_queue *llq = &q->q.llq; if (cmds->num == llq->max_cmd_per_batch) { - arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmds, cmds->num, false); + arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmds, cmds->num); cmds->num = 0; } arm_smmu_cmdq_build_cmd(&cmds->cmds[cmds->num * CMDQ_ENT_DWORDS], cmd); @@ -1531,7 +1523,7 @@ static void arm_smmu_cmdq_batch_add(struct arm_smmu_device *smmu, static int arm_smmu_cmdq_batch_submit(struct arm_smmu_device *smmu, struct arm_smmu_cmdq_batch *cmds) { - return arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmds, cmds->num, true); + return arm_smmu_cmdq_issue_cmdlist(smmu, cmds->cmds, cmds->num); } /* Context descriptor manipulation functions */ @@ -1803,7 +1795,6 @@ static void arm_smmu_sync_ste_for_sid(struct arm_smmu_device *smmu, u32 sid) }; arm_smmu_cmdq_issue_cmd(smmu, &cmd); - arm_smmu_cmdq_issue_sync(smmu); } static void arm_smmu_write_strtab_ent(struct arm_smmu_master *master, u32 sid, @@ -2197,17 +2188,21 @@ arm_smmu_atc_inv_to_cmd(int ssid, unsigned long iova, size_t size, static int arm_smmu_atc_inv_master(struct arm_smmu_master *master) { - int i; + int i, ret = 0; struct arm_smmu_cmdq_ent cmd; arm_smmu_atc_inv_to_cmd(0, 0, 0, &cmd); for (i = 0; i < master->num_sids; i++) { + int rc; + cmd.atc.sid = master->sids[i]; - arm_smmu_cmdq_issue_cmd(master->smmu, &cmd); + rc = arm_smmu_cmdq_issue_cmd(master->smmu, &cmd); + if (rc) + ret = rc; } - return arm_smmu_cmdq_issue_sync(master->smmu); + return ret; } static int arm_smmu_atc_inv_domain(struct arm_smmu_domain *smmu_domain, @@ -2280,7 +2275,6 @@ static void arm_smmu_tlb_inv_context(void *cookie) * careful, 007. */ arm_smmu_cmdq_issue_cmd(smmu, &cmd); - arm_smmu_cmdq_issue_sync(smmu); arm_smmu_atc_inv_domain(smmu_domain, 0, 0, 0); } @@ -3667,7 +3661,6 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass) /* Invalidate any cached configuration */ cmd.opcode = CMDQ_OP_CFGI_ALL; arm_smmu_cmdq_issue_cmd(smmu, &cmd); - arm_smmu_cmdq_issue_sync(smmu); /* Invalidate any stale TLB entries */ if (smmu->features & ARM_SMMU_FEAT_HYP) { @@ -3677,7 +3670,6 @@ static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass) cmd.opcode = CMDQ_OP_TLBI_NSNH_ALL; arm_smmu_cmdq_issue_cmd(smmu, &cmd); - arm_smmu_cmdq_issue_sync(smmu); /* Event queue */ writeq_relaxed(smmu->evtq.q.q_base, smmu->base + ARM_SMMU_EVTQ_BASE); From patchwork Mon Jun 22 17:28:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: John Garry X-Patchwork-Id: 11618537 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 35658618 for ; Mon, 22 Jun 2020 17:35:07 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0DFAF20656 for ; Mon, 22 Jun 2020 17:35:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="dD4wo5Jf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0DFAF20656 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=CRwjd1J+O93GUVE4cRDRXedPr72LAG5qAGeWwFOhKq0=; b=dD4wo5JfXs+FRJOLR+LNTMqw3 tL2//0PPWY6F6n2su6qf1vhzZ+QhAap8r+rGBTICJESPkvrMnKfvGN50V4f0w0WKekvAQ97PmCu8j p0weLpc87uvXW4H1um1ka+/dRoOTm+///UH4XrQI3XJkqDkpG8JXZYMCVF0h62UQhLulMtrrW/Vc9 qalcBEnOgoul4Ulr8PAevAknilySTdMbB8ngl+/m6PDmZFEBZ/4BX1TXkPiUjS30ncF/ExvsKaS+5 ozJ14bPc9j5Xm9mGRTtuVFrBKcaOYCggM8kRcmz8JzqWu7RrMesRUrw5tA40qd4n7GKI+FlqfLkf7 nn0QhvWRQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnQJm-0004cm-7w; Mon, 22 Jun 2020 17:33:14 +0000 Received: from szxga06-in.huawei.com ([45.249.212.32] helo=huawei.com) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1jnQJd-0004Z0-Hk for linux-arm-kernel@lists.infradead.org; Mon, 22 Jun 2020 17:33:09 +0000 Received: from DGGEMS409-HUB.china.huawei.com (unknown [172.30.72.60]) by Forcepoint Email with ESMTP id C293EF80ACF35D700EFE; Tue, 23 Jun 2020 01:32:53 +0800 (CST) Received: from localhost.localdomain (10.69.192.58) by DGGEMS409-HUB.china.huawei.com (10.3.19.209) with Microsoft SMTP Server id 14.3.487.0; Tue, 23 Jun 2020 01:32:43 +0800 From: John Garry To: , Subject: [PATCH 4/4] iommu/arm-smmu-v3: Remove cmpxchg() in arm_smmu_cmdq_issue_cmdlist() Date: Tue, 23 Jun 2020 01:28:40 +0800 Message-ID: <1592846920-45338-5-git-send-email-john.garry@huawei.com> X-Mailer: git-send-email 2.8.1 In-Reply-To: <1592846920-45338-1-git-send-email-john.garry@huawei.com> References: <1592846920-45338-1-git-send-email-john.garry@huawei.com> MIME-Version: 1.0 X-Originating-IP: [10.69.192.58] X-CFilter-Loop: Reflected X-Spam-Note: CRM114 invocation failed X-Spam-Score: -2.3 (--) X-Spam-Report: SpamAssassin version 3.4.4 on merlin.infradead.org summary: Content analysis details: (-2.3 points) pts rule name description ---- ---------------------- -------------------------------------------------- -2.3 RCVD_IN_DNSWL_MED RBL: Sender listed at https://www.dnswl.org/, medium trust [45.249.212.32 listed in list.dnswl.org] 0.0 RCVD_IN_MSPIKE_H4 RBL: Very Good reputation (+4) [45.249.212.32 listed in wl.mailspike.net] -0.0 SPF_PASS SPF: sender matches SPF record -0.0 SPF_HELO_PASS SPF: HELO matches SPF record 0.0 RCVD_IN_MSPIKE_WL Mailspike good senders X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: trivial@kernel.org, maz@kernel.org, joro@8bytes.org, John Garry , linuxarm@huawei.com, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org It has been shown that the cmpxchg() for finding space in the cmdq can be a bottleneck: - for more CPUs contending the cmdq, the cmpxchg() will fail more often - since the software-maintained cons pointer is updated on the same 64b memory region, the chance of cmpxchg() failure increases again The cmpxchg() is removed as part of 2 related changes: - Update prod and cmdq owner in a single atomic add operation. For this, we count the prod and owner in separate regions in prod memory. As with simple binary counting, once the prod+wrap fields overflow, they will zero. They should never overflow into "owner" region, and we zero the non-owner, prod region for each owner. This maintains the prod pointer. As for the "owner", we now count this value, instead of setting a flag. Similar to before, once the owner has finished gathering, it will clear a mask. As such, a CPU declares itself as the "owner" when it reads zero for this region. This zeroing will also clear possible overflow in wrap+prod region, above. The owner is now responsible for all cmdq locking to avoid possible deadlock. The owner will lock the cmdq for all non-owers it has gathered when they have space in the queue and have written their entries. - Check for space in the cmdq after the prod pointer has been assigned. We don't bother checking for space in the cmdq before assigning the prod pointer, as this would be racy. So since the prod pointer is updated unconditionally, it would be common for no space to be available in the cmdq when prod is assigned - that is, according the software-maintained prod and cons pointer. So now it must be ensured that the entries are not yet written and not until there is space. How the prod pointer is maintained also leads to a strange condition where the prod pointer can wrap past the cons pointer. We can detect this condition, and report no space here. However, a prod pointer progressed twice past the cons pointer cannot be detected. But it can be ensured that this that this scenario does not occur, as we limit the amount of commands any CPU can issue at any given time, such that we cannot progress prod pointer further. Signed-off-by: John Garry Reported-by: kernel test robot Signed-off-by: Robin Murphy --- drivers/iommu/arm-smmu-v3.c | 101 ++++++++++++++++++++++-------------- 1 file changed, 61 insertions(+), 40 deletions(-) diff --git a/drivers/iommu/arm-smmu-v3.c b/drivers/iommu/arm-smmu-v3.c index 45a39ccaf455..54d056f10bea 100644 --- a/drivers/iommu/arm-smmu-v3.c +++ b/drivers/iommu/arm-smmu-v3.c @@ -772,10 +772,19 @@ static bool queue_has_space(struct arm_smmu_ll_queue *q, u32 n) prod = Q_IDX(q, q->prod); cons = Q_IDX(q, q->cons); - if (Q_WRP(q, q->prod) == Q_WRP(q, q->cons)) + if (Q_WRP(q, q->prod) == Q_WRP(q, q->cons)) { + /* check if we have wrapped, meaning definitely no space */ + if (cons > prod) + return false; + space = (1 << q->max_n_shift) - (prod - cons); - else + } else { + /* similar check to above */ + if (prod > cons) + return false; + space = cons - prod; + } return space >= n; } @@ -1073,7 +1082,7 @@ static void arm_smmu_cmdq_skip_err(struct arm_smmu_device *smmu) * fails if the caller appears to be the last lock holder (yes, this is * racy). All successful UNLOCK routines have RELEASE semantics. */ -static void arm_smmu_cmdq_shared_lock(struct arm_smmu_cmdq *cmdq) +static void arm_smmu_cmdq_shared_lock(struct arm_smmu_cmdq *cmdq, int count) { int val; @@ -1083,12 +1092,12 @@ static void arm_smmu_cmdq_shared_lock(struct arm_smmu_cmdq *cmdq) * to INT_MIN so these increments won't hurt as the value will remain * negative. */ - if (atomic_fetch_inc_relaxed(&cmdq->lock) >= 0) + if (atomic_fetch_add_relaxed(count, &cmdq->lock) >= 0) return; do { val = atomic_cond_read_relaxed(&cmdq->lock, VAL >= 0); - } while (atomic_cmpxchg_relaxed(&cmdq->lock, val, val + 1) != val); + } while (atomic_cmpxchg_relaxed(&cmdq->lock, val, val + count) != val); } static void arm_smmu_cmdq_shared_unlock(struct arm_smmu_cmdq *cmdq) @@ -1374,8 +1383,10 @@ static void arm_smmu_cmdq_write_entries(struct arm_smmu_cmdq *cmdq, u64 *cmds, * insert their own list of commands then all of the commands from one * CPU will appear before any of the commands from the other CPU. * - * - A CMD_SYNC is always inserted, ensuring that any CPU does not issue - * more than the permitted amount commands at once. + * - A CMD_SYNC is always inserted, which ensures we limit the prod pointer + * for when the cmdq is full, such that we don't wrap more than twice. + * It also makes it easy for the owner to know by how many to increment the + * cmdq lock. */ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, u64 *cmds, int n) @@ -1388,39 +1399,38 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, struct arm_smmu_cmdq *cmdq = &smmu->cmdq; struct arm_smmu_ll_queue llq = { .max_n_shift = cmdq->q.llq.max_n_shift, - }, head = llq; + }, head = llq, space = llq; + u32 owner_val = 1 << cmdq->q.llq.owner_count_shift; + u32 prod_mask = GENMASK(cmdq->q.llq.max_n_shift, 0); + u32 owner_mask = GENMASK(30, cmdq->q.llq.owner_count_shift); int ret = 0; /* 1. Allocate some space in the queue */ local_irq_save(flags); - llq.val = READ_ONCE(cmdq->q.llq.val); - do { - u64 old; - while (!queue_has_space(&llq, n + sync)) { - local_irq_restore(flags); - if (arm_smmu_cmdq_poll_until_not_full(smmu, &llq)) - dev_err_ratelimited(smmu->dev, "CMDQ timeout\n"); - local_irq_save(flags); - } + prod = atomic_fetch_add(n + sync + owner_val, + &cmdq->q.llq.atomic.prod); - head.cons = llq.cons; - head.prod = queue_inc_prod_n(&llq, n + sync) | - CMDQ_PROD_OWNED_FLAG; + owner = !(prod & owner_mask); + llq.prod = prod_mask & prod; + head.prod = queue_inc_prod_n(&llq, n + sync); - old = cmpxchg_relaxed(&cmdq->q.llq.val, llq.val, head.val); - if (old == llq.val) - break; + /* + * Ensure it's safe to write the entries. For this, we need to ensure + * that there is space in the queue from our prod pointer. + */ + space.cons = READ_ONCE(cmdq->q.llq.cons); + space.prod = llq.prod; + while (!queue_has_space(&space, n + sync)) { + if (arm_smmu_cmdq_poll_until_not_full(smmu, &space)) + dev_err_ratelimited(smmu->dev, "CMDQ timeout\n"); - llq.val = old; - } while (1); - owner = !(llq.prod & CMDQ_PROD_OWNED_FLAG); - head.prod &= ~CMDQ_PROD_OWNED_FLAG; - llq.prod &= ~CMDQ_PROD_OWNED_FLAG; + space.prod = llq.prod; + } /* * 2. Write our commands into the queue - * Dependency ordering from the cmpxchg() loop above. + * Dependency ordering from the space-checking while loop, above. */ arm_smmu_cmdq_write_entries(cmdq, cmds, llq.prod, n); @@ -1428,26 +1438,24 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, arm_smmu_cmdq_build_sync_cmd(cmd_sync, smmu, prod); queue_write(Q_ENT(&cmdq->q, prod), cmd_sync, CMDQ_ENT_DWORDS); - /* - * In order to determine completion of our CMD_SYNC, we must ensure - * that the queue can't wrap twice without us noticing. We achieve that - * by taking the cmdq lock as shared before marking our slot as valid. - */ - arm_smmu_cmdq_shared_lock(cmdq); - /* 3. Mark our slots as valid, ensuring commands are visible first */ dma_wmb(); arm_smmu_cmdq_set_valid_map(cmdq, llq.prod, head.prod); /* 4. If we are the owner, take control of the SMMU hardware */ if (owner) { + int owner_count; + u32 prod_tmp; + /* a. Wait for previous owner to finish */ atomic_cond_read_relaxed(&cmdq->owner_prod, VAL == llq.prod); - /* b. Stop gathering work by clearing the owned flag */ - prod = atomic_fetch_andnot_relaxed(CMDQ_PROD_OWNED_FLAG, - &cmdq->q.llq.atomic.prod); - prod &= ~CMDQ_PROD_OWNED_FLAG; + /* b. Stop gathering work by clearing the owned mask */ + prod_tmp = atomic_fetch_andnot_relaxed(~prod_mask, + &cmdq->q.llq.atomic.prod); + prod = prod_tmp & prod_mask; + owner_count = prod_tmp & owner_mask; + owner_count >>= cmdq->q.llq.owner_count_shift; /* * c. Wait for any gathered work to be written to the queue. @@ -1456,6 +1464,19 @@ static int arm_smmu_cmdq_issue_cmdlist(struct arm_smmu_device *smmu, */ arm_smmu_cmdq_poll_valid_map(cmdq, llq.prod, prod); + /* + * In order to determine completion of the CMD_SYNCs, we must + * ensure that the queue can't wrap twice without us noticing. + * We achieve that by taking the cmdq lock as shared before + * progressing the prod pointer. + * The owner does this for all the non-owners it has gathered. + * Otherwise, some non-owner(s) may lock the cmdq, blocking + * cons being updating. This could be when the cmdq has just + * become full. In this case, other sibling non-owners could be + * blocked from progressing, leading to deadlock. + */ + arm_smmu_cmdq_shared_lock(cmdq, owner_count); + /* * d. Advance the hardware prod pointer * Control dependency ordering from the entries becoming valid.