From patchwork Thu Jan 28 11:37:59 2010 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joerg Roedel X-Patchwork-Id: 75578 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter.kernel.org (8.14.3/8.14.3) with ESMTP id o0SBf0Wn008791 for ; Thu, 28 Jan 2010 11:41:01 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1755785Ab0A1Lkv (ORCPT ); Thu, 28 Jan 2010 06:40:51 -0500 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1755710Ab0A1Lku (ORCPT ); Thu, 28 Jan 2010 06:40:50 -0500 Received: from va3ehsobe002.messaging.microsoft.com ([216.32.180.12]:37076 "EHLO VA3EHSOBE002.bigfish.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754786Ab0A1Lie (ORCPT ); Thu, 28 Jan 2010 06:38:34 -0500 Received: from mail137-va3-R.bigfish.com (10.7.14.244) by VA3EHSOBE002.bigfish.com (10.7.40.22) with Microsoft SMTP Server id 8.1.240.5; Thu, 28 Jan 2010 11:38:33 +0000 Received: from mail137-va3 (localhost [127.0.0.1]) by mail137-va3-R.bigfish.com (Postfix) with ESMTP id 62C3F7A0B14; Thu, 28 Jan 2010 11:38:33 +0000 (UTC) X-SpamScore: -4 X-BigFish: VPS-4(zzab9bh936eMzz1202hzzz32i6bh87h62h) X-Spam-TCS-SCL: 1:0 X-FB-DOMAIN-IP-MATCH: fail Received: from mail137-va3 (localhost.localdomain [127.0.0.1]) by mail137-va3 (MessageSwitch) id 1264678711582271_21120; Thu, 28 Jan 2010 11:38:31 +0000 (UTC) Received: from VA3EHSMHS001.bigfish.com (unknown [10.7.14.237]) by mail137-va3.bigfish.com (Postfix) with ESMTP id 7CED3B801C7; Thu, 28 Jan 2010 11:38:31 +0000 (UTC) Received: from ausb3extmailp02.amd.com (163.181.251.22) by VA3EHSMHS001.bigfish.com (10.7.99.11) with Microsoft SMTP Server (TLS) id 14.0.482.39; Thu, 28 Jan 2010 11:38:30 +0000 Received: from ausb3twp01.amd.com (ausb3twp01.amd.com [163.181.250.37]) by ausb3extmailp02.amd.com (Switch-3.2.7/Switch-3.2.7) with ESMTP id o0SBdp6J014268; Thu, 28 Jan 2010 05:39:54 -0600 X-WSS-ID: 0KWYGBZ-01-FXV-02 X-M-MSG: Received: from sausexbh2.amd.com (SAUSEXBH2.amd.com [163.181.22.102]) by ausb3twp01.amd.com (Tumbleweed MailGate 3.7.2) with ESMTP id 2D8361028642; Thu, 28 Jan 2010 05:38:23 -0600 (CST) Received: from sausexmb4.amd.com ([163.181.3.15]) by sausexbh2.amd.com with Microsoft SMTPSVC(6.0.3790.3959); Thu, 28 Jan 2010 05:38:26 -0600 Received: from seurexmb1.amd.com ([165.204.9.130]) by sausexmb4.amd.com with Microsoft SMTPSVC(6.0.3790.3959); Thu, 28 Jan 2010 05:38:25 -0600 Received: from lemmy.osrc.amd.com ([165.204.15.93]) by seurexmb1.amd.com with Microsoft SMTPSVC(6.0.3790.3959); Thu, 28 Jan 2010 12:38:20 +0100 Received: by lemmy.osrc.amd.com (Postfix, from userid 41430) id B880FC9B8D; Thu, 28 Jan 2010 12:38:20 +0100 (CET) From: Joerg Roedel To: Avi Kivity , Marcelo Tosatti , David Woodhouse CC: kvm@vger.kernel.org, iommu@lists.linux-foundation.org, linux-kernel@vger.kernel.org, Joerg Roedel Subject: [PATCH 08/11] x86/amd-iommu: Make iommu_unmap_page and fetch_pte aware of page sizes Date: Thu, 28 Jan 2010 12:37:59 +0100 Message-ID: <1264678682-30655-9-git-send-email-joerg.roedel@amd.com> X-Mailer: git-send-email 1.6.6 In-Reply-To: <1264678682-30655-1-git-send-email-joerg.roedel@amd.com> References: <1264678682-30655-1-git-send-email-joerg.roedel@amd.com> X-OriginalArrivalTime: 28 Jan 2010 11:38:20.0961 (UTC) FILETIME=[64808110:01CAA00E] MIME-Version: 1.0 X-Reverse-DNS: ausb3extmailp02.amd.com Sender: kvm-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.3 (demeter.kernel.org [140.211.167.41]); Thu, 28 Jan 2010 11:41:01 +0000 (UTC) diff --git a/arch/x86/include/asm/amd_iommu_types.h b/arch/x86/include/asm/amd_iommu_types.h index 5e8da56..b150c74 100644 --- a/arch/x86/include/asm/amd_iommu_types.h +++ b/arch/x86/include/asm/amd_iommu_types.h @@ -200,6 +200,12 @@ (((address) | ((pagesize) - 1)) & \ (~(pagesize >> 1)) & PM_ADDR_MASK) +/* + * Takes a PTE value with mode=0x07 and returns the page size it maps + */ +#define PTE_PAGE_SIZE(pte) \ + (1ULL << (1 + ffz(((pte) | 0xfffULL)))) + #define IOMMU_PTE_P (1ULL << 0) #define IOMMU_PTE_TV (1ULL << 1) #define IOMMU_PTE_U (1ULL << 59) diff --git a/arch/x86/kernel/amd_iommu.c b/arch/x86/kernel/amd_iommu.c index 4170031..503d312 100644 --- a/arch/x86/kernel/amd_iommu.c +++ b/arch/x86/kernel/amd_iommu.c @@ -776,28 +776,47 @@ static u64 *alloc_pte(struct protection_domain *domain, * This function checks if there is a PTE for a given dma address. If * there is one, it returns the pointer to it. */ -static u64 *fetch_pte(struct protection_domain *domain, - unsigned long address, int map_size) +static u64 *fetch_pte(struct protection_domain *domain, unsigned long address) { int level; u64 *pte; - level = domain->mode - 1; - pte = &domain->pt_root[PM_LEVEL_INDEX(level, address)]; + if (address > PM_LEVEL_SIZE(domain->mode)) + return NULL; + + level = domain->mode - 1; + pte = &domain->pt_root[PM_LEVEL_INDEX(level, address)]; - while (level > map_size) { + while (level > 0) { + + /* Not Present */ if (!IOMMU_PTE_PRESENT(*pte)) return NULL; + /* Large PTE */ + if (PM_PTE_LEVEL(*pte) == 0x07) { + unsigned long pte_mask, __pte; + + /* + * If we have a series of large PTEs, make + * sure to return a pointer to the first one. + */ + pte_mask = PTE_PAGE_SIZE(*pte); + pte_mask = ~((PAGE_SIZE_PTE_COUNT(pte_mask) << 3) - 1); + __pte = ((unsigned long)pte) & pte_mask; + + return (u64 *)__pte; + } + + /* No level skipping support yet */ + if (PM_PTE_LEVEL(*pte) != level) + return NULL; + level -= 1; + /* Walk to the next level */ pte = IOMMU_PTE_PAGE(*pte); pte = &pte[PM_LEVEL_INDEX(level, address)]; - - if ((PM_PTE_LEVEL(*pte) == 0) && level != map_size) { - pte = NULL; - break; - } } return pte; @@ -850,13 +869,48 @@ static int iommu_map_page(struct protection_domain *dom, return 0; } -static void iommu_unmap_page(struct protection_domain *dom, - unsigned long bus_addr, int map_size) +static unsigned long iommu_unmap_page(struct protection_domain *dom, + unsigned long bus_addr, + unsigned long page_size) { - u64 *pte = fetch_pte(dom, bus_addr, map_size); + unsigned long long unmap_size, unmapped; + u64 *pte; + + BUG_ON(!is_power_of_2(page_size)); + + unmapped = 0; + + while (unmapped < page_size) { + + pte = fetch_pte(dom, bus_addr); + + if (!pte) { + /* + * No PTE for this address + * move forward in 4kb steps + */ + unmap_size = PAGE_SIZE; + } else if (PM_PTE_LEVEL(*pte) == 0) { + /* 4kb PTE found for this address */ + unmap_size = PAGE_SIZE; + *pte = 0ULL; + } else { + int count, i; + + /* Large PTE found which maps this address */ + unmap_size = PTE_PAGE_SIZE(*pte); + count = PAGE_SIZE_PTE_COUNT(unmap_size); + for (i = 0; i < count; i++) + pte[i] = 0ULL; + } + + bus_addr = (bus_addr & ~(unmap_size - 1)) + unmap_size; + unmapped += unmap_size; + } + + BUG_ON(!is_power_of_2(unmapped)); - if (pte) - *pte = 0; + return unmapped; } /* @@ -1054,7 +1108,7 @@ static int alloc_new_range(struct dma_ops_domain *dma_dom, for (i = dma_dom->aperture[index]->offset; i < dma_dom->aperture_size; i += PAGE_SIZE) { - u64 *pte = fetch_pte(&dma_dom->domain, i, PM_MAP_4k); + u64 *pte = fetch_pte(&dma_dom->domain, i); if (!pte || !IOMMU_PTE_PRESENT(*pte)) continue; @@ -2491,7 +2545,7 @@ static void amd_iommu_unmap_range(struct iommu_domain *dom, iova &= PAGE_MASK; for (i = 0; i < npages; ++i) { - iommu_unmap_page(domain, iova, PM_MAP_4k); + iommu_unmap_page(domain, iova, PAGE_SIZE); iova += PAGE_SIZE; } @@ -2506,7 +2560,7 @@ static phys_addr_t amd_iommu_iova_to_phys(struct iommu_domain *dom, phys_addr_t paddr; u64 *pte; - pte = fetch_pte(domain, iova, PM_MAP_4k); + pte = fetch_pte(domain, iova); if (!pte || !IOMMU_PTE_PRESENT(*pte)) return 0;