From patchwork Fri Apr 26 03:43:23 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13644106 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 53929C4345F for ; Fri, 26 Apr 2024 03:43:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 706A76B0099; Thu, 25 Apr 2024 23:43:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 68EC96B009A; Thu, 25 Apr 2024 23:43:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4E04A6B009B; Thu, 25 Apr 2024 23:43:31 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 232886B0099 for ; Thu, 25 Apr 2024 23:43:31 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id D740C801A9 for ; Fri, 26 Apr 2024 03:43:30 +0000 (UTC) X-FDA: 82050288180.15.A10101D Received: from mail-qt1-f180.google.com (mail-qt1-f180.google.com [209.85.160.180]) by imf21.hostedemail.com (Postfix) with ESMTP id 24E551C000B for ; Fri, 26 Apr 2024 03:43:29 +0000 (UTC) Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=bLy4O+ON; spf=pass (imf21.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.180 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714103009; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0qKgoZHcRwjMzBnVukZSI3GExhXHJMz1htG24w59mTg=; b=qby8exZLZPmqWIoON6hjhDm6tkKE8yaVIolu9JWAPF4uK8mbQJJy24Qt94r35tZjlQ7lNq qXZYh5kcWiQUHzbV9AhxzoCmo4FvBOBbK5LxsepAsVDRr7cD28spUYCxSIK76AejgAIieX HFJ3q9YKxAXg1DVsXSALp+2dciBAf1E= ARC-Authentication-Results: i=1; imf21.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=bLy4O+ON; spf=pass (imf21.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.180 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714103009; a=rsa-sha256; cv=none; b=6OU2ivjqmQHminnpjrnBKArxHpYrflqzpNIUXeiWi5l/MOCA00a4e/n4HMR/k+0mjoPOAm yxpV+2qVPz6kQRmwNSUJB2SYYBfh976fdI5AGa+BggQS10+iobJPjgrSabpbR0bIdmQNG1 b+7Mybw2ZyU2e7U3BDsABjToAy0byl4= Received: by mail-qt1-f180.google.com with SMTP id d75a77b69052e-436ffd27871so10818791cf.2 for ; Thu, 25 Apr 2024 20:43:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1714103008; x=1714707808; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:from:to:cc:subject:date:message-id :reply-to; bh=0qKgoZHcRwjMzBnVukZSI3GExhXHJMz1htG24w59mTg=; b=bLy4O+ONaCZE/WM7wS/0bdI7+tLavxac57qgPg8OmeVUqT4Ajv4jbnFD7b3nbCs8XT PSy58vNVw4At+BxfKsbgq8u63npLW13nXAHvy0xvLzJjDOWEn0NIs+I1nJE4KL4XUx1b ORZwHrcqSmtWn3b/8h/pqcHL8j56s3qj8MnEcLjUDn+jggNWyfkA6qoXEeFYaOdbFLxf 0O4o0l1Bpt22tVfP/MeTNxOEYnFJZAMOa5SNMYyECVsmbgQ/JE8tPl3tkXsqUtWBu2hA 4eET368e44tLpPrgopBNHTdKm9kbYz3RdKRpLPV0QBxvq34TTu0jnd3tB/++WPZHuqRd xx6w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714103008; x=1714707808; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0qKgoZHcRwjMzBnVukZSI3GExhXHJMz1htG24w59mTg=; b=DGSt1XmTOA0Zv//I9wr/8H5WzK7nn40wYrIhiQS5Ug8zRbvP7vRhAT4/jxV7+f/NAr loJqK/R/QKWzvVT7M2o952KmjNLzc46u4l1zij2TkTg95bBLNPla5QvAs+7AhsDCVgem gXRq/aUbQWhB+e4FKjXvKsQpWI3LZZuIHb7KdL72ylVfZGZE2q3CCr3CTaYRZ6f2q2K1 k31gNAPgjoeVCWtOu+Dyh7h6CXJyd+3NAQz1HDQNkr495npCTXjZyPcSHcVGfbKgfzPZ Z1cxx6uGkMPOg0O2VAf+2zJbsFyAr27IBPNAkbdhYS6uMy6acJDIyfYLsPsoMbXs54Bw RlcQ== X-Forwarded-Encrypted: i=1; AJvYcCURBWAW4FvSEgnwi9uvfwCMzosg82Ct7uD5xmHRvXbWzhO+bXT8z521TcU5YfQhazkt7X1erXze+QaET8FHxP7hfHg= X-Gm-Message-State: AOJu0Yyk85C2aIy7jB0RSdz3Q143oGYibhfciv3Ff3EabN2bD34t/wDH sxk4OaI9ucrhqPr2CGOtgLkgpx/DB+uwi4A7aS6EoW5N6WRtjT4ax5T1YfiVlf0= X-Google-Smtp-Source: AGHT+IHmUgwf6vDm3AhjFFV4a78DkNgnzXWpFjgQ5FiiDRbS0nCVhPfbCsQnbjR/SyVu2r2tQ6OZiw== X-Received: by 2002:ac8:7e81:0:b0:43a:7bc9:7c2f with SMTP id w1-20020ac87e81000000b0043a7bc97c2fmr325043qtj.2.1714103008254; Thu, 25 Apr 2024 20:43:28 -0700 (PDT) Received: from soleen.c.googlers.com.com (129.177.85.34.bc.googleusercontent.com. [34.85.177.129]) by smtp.gmail.com with ESMTPSA id c6-20020ac80546000000b00436bb57faddsm7540815qth.25.2024.04.25.20.43.27 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Apr 2024 20:43:27 -0700 (PDT) From: Pasha Tatashin To: akpm@linux-foundation.org, linux-mm@kvack.org, pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, rientjes@google.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, iommu@lists.linux.dev Subject: [RFC v2 3/3] iommu/intel: free empty page tables on unmaps Date: Fri, 26 Apr 2024 03:43:23 +0000 Message-ID: <20240426034323.417219-4-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.769.g3c40516874-goog In-Reply-To: <20240426034323.417219-1-pasha.tatashin@soleen.com> References: <20240426034323.417219-1-pasha.tatashin@soleen.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 24E551C000B X-Stat-Signature: xctki885p7rcdtf3ehmqwsbnwekzsnp7 X-Rspam-User: X-Rspamd-Server: rspam07 X-HE-Tag: 1714103009-19102 X-HE-Meta: U2FsdGVkX18xMqhxvF6Eunhy56zYFe7X5qrib7V45HUfFmLecsqSpSIwNNaQg3G46q3qvARN3TUPI9WXYcEd+0h3oqOYkTKcAbQWNtFDtZeh4KbnckXy3hQHFu2Jcs7ogTL22hj++WzPH+2pqPKkR9h/qeiuTTYOZdUQDLFU4iQXX1IExrtCvkV4U9ayzrFpgHh7qSrif853G6NPMNPO8ykwTCvQNllOHAaOxKBNej6PJucQMdZ4OXwiAGO7AzpkeVp3TZ9znAihsyLCNrqNRHcvE+yElu/unRzvjGMY3T2pqwfkAwVyHUOaZruxFEk5sTEj/vgJq+B5F6D1ghn63uV3XAGBKOVewZO+hUhueHF3253vK2wHCLoOqsN63yxpPwVOlfKtttdFwCrTu+tjUFWuRi0KmenfpPoh1J0R7mY5GucEJWdgVoS+6nLY74VGNEH729ZD59SmsWBWv9zgLI4tYYtgDvuunvpgmAbK/VcPE8hDNSadFTYqtdEykiF5kMYgTaPB+ndMF7k/rrc4LMFSBGfCgDo6NFCb8UprD/FC25BUkWZo3jwBpijfjrPWeDMA6+KU83qUihzgsJ8RZBBu5qnrggLv99vg962xp+R3hTdbY40MPlCKmkU1nSjPAiiA5cCVULL7cgWoL/SfSfqZ/Ozh4YS4egh4/HcsTkhEYFYiyj0ZaVcs5vVYb0/dtOyejiuf3IcSKXMt8GGWAU0KInZ8fh5+0X81DvjtL15VsiAgUjRB5g9XI9GFkpu33j30X+txYxke4RboNdsTGdJY51xhxZhq6AEPuuQjLRXUqu9AzH/+ARqmaM2X9vvF827DuBawWE4b7B+lsT9foqvQ+MvfPe7keSxgDav/nsvh5ax9m1Pr7lviWEhJfQ7emYlMIvnvumyhE7LzQ51u0TShodi9AuavEXLOw3lbrHUzsIq62FgaidTqaxLjoOiOvUDHKl+eUcNwACu/ncS XH5kAWzM Dl8+d3UmZoEHr5trl+yHrxxe2muJ1/5toBM/PaZKmbWFa/a3FbqVC8Guo86lKqeJGOi6Sa4/oA6RTCmVlOnoevsTZSgsRlOnGX+3L4ah9BZxmXwzrmKInIAeDgve2jhyqjURQnSid8IGgv1rA7TBLnpIPTHLPomsWkRj4ppqEIJJKaQGWsYFewcshvpg9xMGitAoxzdQDwcMbCu+Lzk9pxGgUad1Z8/Tu5Q6/N3h/DAfhL2hP5o29EUETIShLtAV23ndCvI+3D0xorgc86ogQx89TBFB39kkFGSDNFHjOzaNs8yg6KkMkb0JYJTFN2goR07omvzeRakvFzt1uJhwqdbqZdCJj8eJ8D2l+oARQITprpqMtegMVLcjoQbrRhpjMRHi1nikJ3+Ctifsl0BKgJ6a8wRVB3OCBZzOvgBxd3COJSWctnbFPz545sxKTW+ddPowrUCRi4QhGCYrXHNPBWUEvFyyTTulJBjxpEpYnXqnughI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: When page tables become empty, add them to the freelist so that they can also be freed. This is means that a page tables that are outside of the imediat iova range might be freed as well, therefore, only in the case where such page tables are going to be freed, we take the writer lock. Signed-off-by: Pasha Tatashin --- drivers/iommu/intel/iommu.c | 91 +++++++++++++++++++++++++++++++------ 1 file changed, 77 insertions(+), 14 deletions(-) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index 8c7e596728b5..2dedcd4f6060 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -1044,7 +1044,7 @@ static void dma_pte_list_pagetables(struct dmar_domain *domain, static void dma_pte_clear_level(struct dmar_domain *domain, int level, struct dma_pte *pte, unsigned long pfn, unsigned long start_pfn, unsigned long last_pfn, - struct list_head *freelist) + struct list_head *freelist, int *freed_level) { struct dma_pte *first_pte = NULL, *last_pte = NULL; @@ -1070,11 +1070,47 @@ static void dma_pte_clear_level(struct dmar_domain *domain, int level, first_pte = pte; last_pte = pte; } else if (level > 1) { + struct dma_pte *npte = phys_to_virt(dma_pte_addr(pte)); + struct page *npage = virt_to_page(npte); + /* Recurse down into a level that isn't *entirely* obsolete */ - dma_pte_clear_level(domain, level - 1, - phys_to_virt(dma_pte_addr(pte)), + dma_pte_clear_level(domain, level - 1, npte, level_pfn, start_pfn, last_pfn, - freelist); + freelist, freed_level); + + /* + * Free next level page table if it became empty. + * + * We only holding the reader lock, and it is possible + * that other threads are accessing page table as + * readers as well. We can only free page table that + * is outside of the request IOVA space only if + * we grab the writer lock. Since we need to drop reader + * lock, we are incrementing the mapcount in the npage + * so it (and the current page table) does not + * dissappear due to concurrent unmapping threads. + * + * Store the size maximum size of the freed page table + * into freed_level, so the size of the IOTLB flush + * can be determined. + */ + if (freed_level && !atomic_read(&npage->_mapcount)) { + atomic_inc(&npage->_mapcount); + read_unlock(&domain->pgd_lock); + write_lock(&domain->pgd_lock); + atomic_dec(&npage->_mapcount); + if (!atomic_read(&npage->_mapcount)) { + dma_clear_pte(pte); + if (!first_pte) + first_pte = pte; + last_pte = pte; + page_mapcount_reset(npage); + list_add_tail(&npage->lru, freelist); + *freed_level = level; + } + write_unlock(&domain->pgd_lock); + read_lock(&domain->pgd_lock); + } } next: pfn = level_pfn + level_size(level); @@ -1089,7 +1125,8 @@ static void dma_pte_clear_level(struct dmar_domain *domain, int level, the page tables, and may have cached the intermediate levels. The pages can only be freed after the IOTLB flush has been done. */ static void domain_unmap(struct dmar_domain *domain, unsigned long start_pfn, - unsigned long last_pfn, struct list_head *freelist) + unsigned long last_pfn, struct list_head *freelist, + int *level) { if (WARN_ON(!domain_pfn_supported(domain, last_pfn)) || WARN_ON(start_pfn > last_pfn)) @@ -1098,7 +1135,8 @@ static void domain_unmap(struct dmar_domain *domain, unsigned long start_pfn, read_lock(&domain->pgd_lock); /* we don't need lock here; nobody else touches the iova range */ dma_pte_clear_level(domain, agaw_to_level(domain->agaw), - domain->pgd, 0, start_pfn, last_pfn, freelist); + domain->pgd, 0, start_pfn, last_pfn, freelist, + level); read_unlock(&domain->pgd_lock); /* free pgd */ @@ -1479,11 +1517,11 @@ static void __iommu_flush_iotlb_psi(struct intel_iommu *iommu, u16 did, static void iommu_flush_iotlb_psi(struct intel_iommu *iommu, struct dmar_domain *domain, - unsigned long pfn, unsigned int pages, + unsigned long pfn, unsigned long pages, int ih, int map) { - unsigned int aligned_pages = __roundup_pow_of_two(pages); - unsigned int mask = ilog2(aligned_pages); + unsigned long aligned_pages = __roundup_pow_of_two(pages); + unsigned long mask = ilog2(aligned_pages); uint64_t addr = (uint64_t)pfn << VTD_PAGE_SHIFT; u16 did = domain_id_iommu(domain, iommu); @@ -1837,7 +1875,8 @@ static void domain_exit(struct dmar_domain *domain) if (domain->pgd) { LIST_HEAD(freelist); - domain_unmap(domain, 0, DOMAIN_MAX_PFN(domain->gaw), &freelist); + domain_unmap(domain, 0, DOMAIN_MAX_PFN(domain->gaw), &freelist, + NULL); iommu_put_pages_list(&freelist); } @@ -3419,7 +3458,8 @@ static int intel_iommu_memory_notifier(struct notifier_block *nb, struct intel_iommu *iommu; LIST_HEAD(freelist); - domain_unmap(si_domain, start_vpfn, last_vpfn, &freelist); + domain_unmap(si_domain, start_vpfn, last_vpfn, + &freelist, NULL); rcu_read_lock(); for_each_active_iommu(iommu, drhd) @@ -4080,6 +4120,7 @@ static size_t intel_iommu_unmap(struct iommu_domain *domain, struct iommu_iotlb_gather *gather) { struct dmar_domain *dmar_domain = to_dmar_domain(domain); + bool queued = iommu_iotlb_gather_queued(gather); unsigned long start_pfn, last_pfn; int level = 0; @@ -4099,7 +4140,16 @@ static size_t intel_iommu_unmap(struct iommu_domain *domain, start_pfn = iova >> VTD_PAGE_SHIFT; last_pfn = (iova + size - 1) >> VTD_PAGE_SHIFT; - domain_unmap(dmar_domain, start_pfn, last_pfn, &gather->freelist); + /* + * pass level only if !queued, which means we will do iotlb + * flush callback before freeing pages from freelist. + * + * When level is passed domain_unamp will attempt to add empty + * page tables to freelist, and pass the level number of the highest + * page table that was added to the freelist. + */ + domain_unmap(dmar_domain, start_pfn, last_pfn, &gather->freelist, + queued ? NULL : &level); if (dmar_domain->max_addr == iova + size) dmar_domain->max_addr = iova; @@ -4108,8 +4158,21 @@ static size_t intel_iommu_unmap(struct iommu_domain *domain, * We do not use page-selective IOTLB invalidation in flush queue, * so there is no need to track page and sync iotlb. */ - if (!iommu_iotlb_gather_queued(gather)) - iommu_iotlb_gather_add_page(domain, gather, iova, size); + if (!queued) { + size_t sz = size; + + /* + * Increase iova and sz for flushing if level was returned, + * as it means we also are freeing some page tables. + */ + if (level) { + unsigned long pgsize = level_size(level) << VTD_PAGE_SHIFT; + + iova = ALIGN_DOWN(iova, pgsize); + sz = ALIGN(size, pgsize); + } + iommu_iotlb_gather_add_page(domain, gather, iova, sz); + } return size; }