From patchwork Wed Sep 20 08:52:02 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Tomasz Nowicki X-Patchwork-Id: 9961157 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id B23A860234 for ; Wed, 20 Sep 2017 08:55:06 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A089B2903C for ; Wed, 20 Sep 2017 08:55:06 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 915F72903E; Wed, 20 Sep 2017 08:55:06 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAD_ENC_HEADER,BAYES_00, DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id DF6962903C for ; Wed, 20 Sep 2017 08:55:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=8P25ATX+xEQF9xX/ljTX2NayPns/o0Xgak796xueMfU=; b=kU/kgEMBwv/KGJ e3HMYqcT+vk6fDvmrfUx+xxGfAGpfhFsdxHnzwfc0Z5xHxOoNQvBAkegldC4s1I+vqM+uTJqnSKSI NK3WVcNykBTeOUSNLYPxf0J2+tEIkX4+P0z6/8rdJe/kig3nRbr4weJPjgJny0qpotdXkcPPErL3I gzquGM6DpTPNcSUoa6v0xV6fVgayQFCtEq3Zq/oWDxz1MMThByvT16iBs+gF4ytXZFBdP4wUMiwzU gLRf1ZcuThUy8PWWc3EA1ZTZ+zdnNAR4W7+oMPzzNdX/RNsqIWRjEjgnGFzu/Jj7Oxhi2ZY1MdVLc mz9Spn9shSo8g7U2XJ0A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1duame-0001Gy-Ui; Wed, 20 Sep 2017 08:55:04 +0000 Received: from mail-co1nam03on0050.outbound.protection.outlook.com ([104.47.40.50] helo=NAM03-CO1-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1duakg-0008WU-74 for linux-arm-kernel@lists.infradead.org; Wed, 20 Sep 2017 08:53:06 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=+s7OgdJwlNwvfYPkd1phcbCUTfmQnRXdmv1Su7XZSOc=; b=OMhmiD+WdAE/eFFw7T79Amubm8hxXM8DKroF/OH5ALCWzrHmFIxhlFZB/VWbanzA0IoNc5jJuRvlQUTXQwHyPrtjmQdFjezjpUHIuXjN/gtsRKYsRdPlTcqsD144uE2Pfshkk8ZkcAEuLvxO8mOgAk4rguhrczInK0UkaYdfKUY= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Tomasz.Nowicki@cavium.com; Received: from desktop.semihalf.local (31.172.191.173) by SN4PR0701MB3661.namprd07.prod.outlook.com (2603:10b6:803:4d::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.56.11; Wed, 20 Sep 2017 08:52:31 +0000 From: Tomasz Nowicki To: joro@8bytes.org, robin.murphy@arm.com, will.deacon@arm.com Subject: [PATCH V2 1/1] iommu/iova: Make rcache flush optional on IOVA allocation failure Date: Wed, 20 Sep 2017 10:52:02 +0200 Message-Id: <1505897522-9237-2-git-send-email-tomasz.nowicki@caviumnetworks.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1505897522-9237-1-git-send-email-tomasz.nowicki@caviumnetworks.com> References: <1505897522-9237-1-git-send-email-tomasz.nowicki@caviumnetworks.com> MIME-Version: 1.0 X-Originating-IP: [31.172.191.173] X-ClientProxiedBy: VI1PR0101CA0084.eurprd01.prod.exchangelabs.com (2603:10a6:800:1f::52) To SN4PR0701MB3661.namprd07.prod.outlook.com (2603:10b6:803:4d::15) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 5cc4fa76-3ef2-4791-19a6-08d50004f236 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:SN4PR0701MB3661; X-Microsoft-Exchange-Diagnostics: 1; SN4PR0701MB3661; 3:8fZzoZLKRip1wGMefvE6uZct7yYcORRt8cpVcZXVcqTK4e1D3drEyXJQQ6SAM6pUm8obg82qKMQSfpa2l8Y6dK3ZqHMFkDCDdq/MwTd+0meZkM4rgkuWBD4y4pQ5meNG62g/YknixVwobet0QzpLDiX9yNJSb1/Vh58xtCGp7TrjqGmKOKpRsucWm7117tGpllZpsesUYNmfpe0FsxkvcBDBWp7LmRg6DJqbFyszMEYcq54bNhjRDXOOplY+UgIc; 25:uXChoMs0DEjy3foxGZzjF9hOIBSbu8yQn3JDf1Z5shoX9Cy8gaBOAiHk9kPWGLl+idH2I7zfxCWmI8+D5BqpR81gQVNZW4VhH1HedufUj8oVJLfRvav4uvr/v5q32sBdtHTECYgN3HqzPsrHZ52pXWtMB+Cud6X3XdAOyipATAzygnvHufETqpjg0aGkf5/ReX5bb/EY9eko3ryVO4IGNRA5+laWYTzxWkfFvLzRfeIJCck7ZaKluKuLJRsSSkKlpy48GIRwH3IRPPilMicZlVW5Es2M6v3aXXGkn7+LBr5tYzO1lA/ZjVUklPN2r+UnyqbDoDlQ44q1BHTjaGBxeQ==; 31:I1t4V/zzwbVEldyPhEQX7LeGmDsXpmnX5t+SCy92tD/7AZ9IJtJ8A+4Hcn+gHfNygOg5CJT7EpNiD2ja6jk7gX6kWcEZBQ931lP+Ou0pFmjdxk0/0y3JpFqpbijzE54tBovsMB5dlAlxrnMoFHGw7YMa5rXWXkBnZXoQb/EeTJYHh1RsW5rJTG7rbaoshD+KFoqjfA+hQJaZTDnl6Hwt3eTOK5fSUvin7L0/YsJxvvg= X-MS-TrafficTypeDiagnostic: SN4PR0701MB3661: X-Microsoft-Exchange-Diagnostics: 1; SN4PR0701MB3661; 20:4wFodeVWAmp2exA2uXPvB8FShUcJxsiqy+LQFAlkZcHIu3q+So1J3yzNMRXNnlmLZBuEn8ggpqsj+KS5i83OiWA3p/CP5qb0OkeNQLyXpP/hwwiLKiSgkAoto0IUtg6Jrq9g823+AG7QWZPOsfZFEqW3pOT1PL/dLuXgrVZardW1RIz1cwEbRcML9lr22iHhRFFyVf5spOi3Z3C1U5LGDQxWE6jwLFT8M8CsDETjnBj3Owjn1plsGyHkwpK3KLyY2om6SawGMh7USAiRGhYvq4tg5Ww9lX0vuSpCX/2SQ3xjg8BMd3fSTkbOi2pRv4M2ul/zMuTiZhODA9UpIHwZotCxKXnT0qFHaiE3voZQmGMmuvFEQ/7Q5Zr6E3FJmp6xeZZAMN6oPBulCVuoptDor+VDSr4jtsNzue0L8+3WeU5B/QJcAGcQ6q3BykbQ+GZmBvP8FJjt2hCm0wfalWPqVN1istXx7+XgJcyfm0TyUrzsMUxl81VDUK0AFQdce5mM7zNblKN0O61ZFNdY0Gb1HjsXJIzetxl5D92L+maOrYnSvXb9cuo5xYgtR1hQjA+znYGcx7C9jjib9CEtCGFIEi7vpq5Oco1418NU6Wk6uNk=; 4:HGoUL/650xYqUK9K1w7HFcW5qe7CTRS97Mbzt6riCMFzWZn9yT2E63mVOtmCuyNRmFqbLtfY9qLkImCIg3jUriZbJzOe1+yoftlZ2+pCj6PTNfkBvBKSYIDfz5SzzSaCcJ50CtB4sQSamP8RhPPnaoXyXTCi6QgytYknHXPuYP4iE602hFZVr8lQswsFPVzCJoxMlJ/f6Isq1XjWGcbMMNB9q9T81oijP6LOaYKh5sq+t8LSDuAkd8c7fO7+NptSJCho6OKu1powHDCEbiO1v63v19gWgxJLo3Ty4T2Aa0I= X-Exchange-Antispam-Report-Test: UriScan:(180628864354917); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(8121501046)(5005006)(100000703101)(100105400095)(10201501046)(93006095)(3002001)(6041248)(20161123562025)(20161123555025)(20161123564025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123560025)(20161123558100)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:SN4PR0701MB3661; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:SN4PR0701MB3661; X-Forefront-PRVS: 04362AC73B X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6009001)(346002)(376002)(199003)(189002)(97736004)(6116002)(5003940100001)(3846002)(72206003)(8936002)(53936002)(6512007)(2906002)(16526017)(316002)(16586007)(36756003)(48376002)(4326008)(54906003)(47776003)(66066001)(189998001)(68736007)(50466002)(107886003)(25786009)(33646002)(5660300001)(101416001)(81166006)(6486002)(2950100002)(81156014)(6506006)(50226002)(42882006)(575784001)(8676002)(6666003)(50986999)(76176999)(106356001)(105586002)(7736002)(305945005)(478600001); DIR:OUT; SFP:1101; SCL:1; SRVR:SN4PR0701MB3661; H:desktop.semihalf.local; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; SN4PR0701MB3661; 23:Rbv+KK0U9AzVoTJRVSsFAQc6mWVOxi64DjYlE+D?= =?us-ascii?Q?gJyfzkSZBOGDVC3conU+nJgINe0egiV3fFjNu6ejM7+tNNF5Qt6QY5wWwpjr?= =?us-ascii?Q?KG2oL7m3ITUGIDqudk977UH70mDnUfATh7fYHGZSBpUvtxNh+sCV2zq6hxnc?= =?us-ascii?Q?RRZTq1LTDDj6HmSIlxE96cLJ03bU3fhx2CeA5T5ges2O5oP5ONZ7vygqQGzi?= =?us-ascii?Q?t44O22iv+SjnvPMIaDGEGtL1QXUy6tyOsazEKx99IgLOn+sK3zIZdblCjCR5?= =?us-ascii?Q?eFytVwXiwQgX/R5DgQm2erxFIs4VO5XvjgLGpgfJu4UVVWQ0kxXHuIbXpev+?= =?us-ascii?Q?uZROTtKuZXppr5q4g0JNYEVKH6JO2JI1EIS74obuFql3sgVnQTlsSJI9Y3w1?= =?us-ascii?Q?2VH3FFJJ6ZaeyqFxLfOi9ZikMN36Jgw+vdVO5qwiioOYj+IjXC0OwoctFnYW?= =?us-ascii?Q?C9xjQI8MyxsgidbCcjm8ht0Nyu3F5a52iSH/iQ9i9Rl9g8bYT6Da1ueBVZXd?= =?us-ascii?Q?oqmFTtlRBziEJvarfzQPCX0jfg4XW3g3P9RxkhJ5KFLFlxkBpV9q84ZWJwRn?= =?us-ascii?Q?1gws35Hy0I+NrxlAkQAaGcY5ev/dc/ItBucB9DEvUbwfCAfZr8ExUAqjePTW?= =?us-ascii?Q?WFlRyAuEViO6O3xnMRLbLvjWGr1x+64SMcmcYESBtRxEIuvT1BYvKQiKCvG6?= =?us-ascii?Q?d3/sdJ5nUUMhfpIrREOvO7d0ankQ9VDwSw+x0K5V0zkR5dvyQyl8LMBZ5dQo?= =?us-ascii?Q?KIV7h6KkXf3e2S5bs+N9MeoXqtatJx294J1dOn+UxjiCkeavdT/wTBGrZhba?= =?us-ascii?Q?OdVccv1Wz516j+46E4T5j6Qu7bPcVgUwoGLiOHIL31PmTUfPjCUaLBBZs0gY?= =?us-ascii?Q?rXMbiIX7C7OT5akFSTt5HBi2j2x7r9h3eo+pGbQa9NLSpqYKCTQlemhK5E/N?= =?us-ascii?Q?XryQB+6WSYwDeCqGhZs0DPxLLul2ISRFvtSzV6YGvz9V9ZovOyk0RFU25PtV?= =?us-ascii?Q?54AJZTH+v11LMNeb9En5kIJD6Bl6EsVtBPUJGiaHwNc3hnkscA+pFnY0wW2e?= =?us-ascii?Q?rSwwgJug7oKKKWflqgypdN5zUmsKrUjhEWVH7K7sAt/bOghVevfaUwu8WUIQ?= =?us-ascii?Q?R73tnQZFv1P9s/Lej/WQsJgFpCtnVbaAM?= X-Microsoft-Exchange-Diagnostics: 1; SN4PR0701MB3661; 6:Xu1i2b7aRCveN1oLMi8j3lK9/IJBwoHCi2Q9rHBIY77NSkyFiM+ZVcHIsaHK3xlriPAzKrXGY+n3jgAEuVTGMQvftfUO1e6+pq/Z7NA3qJN6WEToR73Ld9Oe0h4PBgtQFD/bZ2bRTpObqqpXN5y00skT62JSFn3XYJQZbyQHD7fqLlR9oFC86DtpVprsogI+Weyz3BMt3qYl2/2S3aeAktYUK0N+aOWhHyRjVXcEApM6hmzQSUMZW48zhxRpRL+YYrxmSjAjts3tGnv6gW6v8ZQ/hxt3XObDXEfql8hi+kLZh5XUSL4MD54PtQMKR3o2hFkznJ7cy2s9oUKWuWNeew==; 5:jRybAP9EqsQ06bGkw3xfM+EAqZV5Puvuc8JkdRJI2Ns9JiYw3eb0QxwjQ82uV58ERdqgdQxO8j0+5/8+DRzm+gaHkE+miyKdGYCNUui9JI4onv5XDpPsu62WpohDeU32MYPmdg8CR+kncdQ8LdAWug==; 24:O+S2+IOJ9EIx20/6I5vLQfvnZlhZM9Qj3JQ3w73yS1vlBqL4ZTgUA4uf6wwbxUzpq8f5rR7hBXL00/RDLiBcNAChUj1KHZxd2FXRhikF94g=; 7:qQSKuMY5DEtG2N6GiTFQWHNrR26DbTC/SkJpwcdK43QYnI7UISxtE2pNTD18osOQv6YnrWXIgmLqTjdy2mhOUWv5PWXjX4OkVmLXI2/IVWgvgfLt23sjB6mrOBaEM/3Mwi71DJhzLcNoyet2sRDWcJ6vEH215XQaFaPOf4dFH/c0g2ho+/dURxxgf0D3M8rdIef/rmTlCxnnuz92gfArjS1+rEY6fZgxvztMLzEWayE= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 20 Sep 2017 08:52:31.3793 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SN4PR0701MB3661 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20170920_015304_137316_EDCB80FD X-CRM114-Status: GOOD ( 14.04 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jayachandran.Nair@cavium.com, lorenzo.pieralisi@arm.com, Tomasz Nowicki , ard.biesheuvel@linaro.org, linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, linux-arm-kernel@lists.infradead.org, Ganapatrao.Kulkarni@cavium.com, nwatters@codeaurora.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Since IOVA allocation failure is not unusual case we need to flush CPUs' rcache in hope we will succeed in next round. However, it is useful to decide whether we need rcache flush step because of two reasons: - Not scalability. On large system with ~100 CPUs iterating and flushing rcache for each CPU becomes serious bottleneck so we may want to defer it. - free_cpu_cached_iovas() does not care about max PFN we are interested in. Thus we may flush our rcaches and still get no new IOVA like in the commonly used scenario: if (dma_limit > DMA_BIT_MASK(32) && dev_is_pci(dev)) iova = alloc_iova_fast(iovad, iova_len, DMA_BIT_MASK(32) >> shift); if (!iova) iova = alloc_iova_fast(iovad, iova_len, dma_limit >> shift); 1. First alloc_iova_fast() call is limited to DMA_BIT_MASK(32) to get PCI devices a SAC address 2. alloc_iova() fails due to full 32-bit space 3. rcaches contain PFNs out of 32-bit space so free_cpu_cached_iovas() throws entries away for nothing and alloc_iova() fails again 4. Next alloc_iova_fast() call cannot take advantage of rcache since we have just defeated caches. In this case we pick the slowest option to proceed. This patch reworks flushed_rcache local flag to be additional function argument instead and control rcache flush step. Also, it updates all users to do the flush as the last chance. Signed-off-by: Tomasz Nowicki Reviewed-by: Robin Murphy Tested-by: Nate Watterson --- drivers/iommu/amd_iommu.c | 5 +++-- drivers/iommu/dma-iommu.c | 6 ++++-- drivers/iommu/intel-iommu.c | 5 +++-- drivers/iommu/iova.c | 11 ++++++----- include/linux/iova.h | 5 +++-- 5 files changed, 19 insertions(+), 13 deletions(-) diff --git a/drivers/iommu/amd_iommu.c b/drivers/iommu/amd_iommu.c index 8d2ec60..ce68986 100644 --- a/drivers/iommu/amd_iommu.c +++ b/drivers/iommu/amd_iommu.c @@ -1604,10 +1604,11 @@ static unsigned long dma_ops_alloc_iova(struct device *dev, if (dma_mask > DMA_BIT_MASK(32)) pfn = alloc_iova_fast(&dma_dom->iovad, pages, - IOVA_PFN(DMA_BIT_MASK(32))); + IOVA_PFN(DMA_BIT_MASK(32)), false); if (!pfn) - pfn = alloc_iova_fast(&dma_dom->iovad, pages, IOVA_PFN(dma_mask)); + pfn = alloc_iova_fast(&dma_dom->iovad, pages, + IOVA_PFN(dma_mask), true); return (pfn << PAGE_SHIFT); } diff --git a/drivers/iommu/dma-iommu.c b/drivers/iommu/dma-iommu.c index 191be9c..25914d3 100644 --- a/drivers/iommu/dma-iommu.c +++ b/drivers/iommu/dma-iommu.c @@ -370,10 +370,12 @@ static dma_addr_t iommu_dma_alloc_iova(struct iommu_domain *domain, /* Try to get PCI devices a SAC address */ if (dma_limit > DMA_BIT_MASK(32) && dev_is_pci(dev)) - iova = alloc_iova_fast(iovad, iova_len, DMA_BIT_MASK(32) >> shift); + iova = alloc_iova_fast(iovad, iova_len, + DMA_BIT_MASK(32) >> shift, false); if (!iova) - iova = alloc_iova_fast(iovad, iova_len, dma_limit >> shift); + iova = alloc_iova_fast(iovad, iova_len, dma_limit >> shift, + true); return (dma_addr_t)iova << shift; } diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c index 05c0c3a..75c8320 100644 --- a/drivers/iommu/intel-iommu.c +++ b/drivers/iommu/intel-iommu.c @@ -3460,11 +3460,12 @@ static unsigned long intel_alloc_iova(struct device *dev, * from higher range */ iova_pfn = alloc_iova_fast(&domain->iovad, nrpages, - IOVA_PFN(DMA_BIT_MASK(32))); + IOVA_PFN(DMA_BIT_MASK(32)), false); if (iova_pfn) return iova_pfn; } - iova_pfn = alloc_iova_fast(&domain->iovad, nrpages, IOVA_PFN(dma_mask)); + iova_pfn = alloc_iova_fast(&domain->iovad, nrpages, + IOVA_PFN(dma_mask), true); if (unlikely(!iova_pfn)) { pr_err("Allocating %ld-page iova for %s failed", nrpages, dev_name(dev)); diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index f88acad..e8c140c 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -353,14 +353,15 @@ EXPORT_SYMBOL_GPL(free_iova); * @iovad: - iova domain in question * @size: - size of page frames to allocate * @limit_pfn: - max limit address + * @flush_rcache: - set to flush rcache on regular allocation failure * This function tries to satisfy an iova allocation from the rcache, - * and falls back to regular allocation on failure. + * and falls back to regular allocation on failure. If regular allocation + * fails too and the flush_rcache flag is set then the rcache will be flushed. */ unsigned long alloc_iova_fast(struct iova_domain *iovad, unsigned long size, - unsigned long limit_pfn) + unsigned long limit_pfn, bool flush_rcache) { - bool flushed_rcache = false; unsigned long iova_pfn; struct iova *new_iova; @@ -373,11 +374,11 @@ alloc_iova_fast(struct iova_domain *iovad, unsigned long size, if (!new_iova) { unsigned int cpu; - if (flushed_rcache) + if (!flush_rcache) return 0; /* Try replenishing IOVAs by flushing rcache. */ - flushed_rcache = true; + flush_rcache = false; for_each_online_cpu(cpu) free_cpu_cached_iovas(cpu, iovad); goto retry; diff --git a/include/linux/iova.h b/include/linux/iova.h index 58c2a36..8fdcb66 100644 --- a/include/linux/iova.h +++ b/include/linux/iova.h @@ -97,7 +97,7 @@ struct iova *alloc_iova(struct iova_domain *iovad, unsigned long size, void free_iova_fast(struct iova_domain *iovad, unsigned long pfn, unsigned long size); unsigned long alloc_iova_fast(struct iova_domain *iovad, unsigned long size, - unsigned long limit_pfn); + unsigned long limit_pfn, bool flush_rcache); struct iova *reserve_iova(struct iova_domain *iovad, unsigned long pfn_lo, unsigned long pfn_hi); void copy_reserved_iova(struct iova_domain *from, struct iova_domain *to); @@ -151,7 +151,8 @@ static inline void free_iova_fast(struct iova_domain *iovad, static inline unsigned long alloc_iova_fast(struct iova_domain *iovad, unsigned long size, - unsigned long limit_pfn) + unsigned long limit_pfn, + bool flush_rcache) { return 0; }