From patchwork Fri Apr 26 03:43:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13644103 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 17F7AC4345F for ; Fri, 26 Apr 2024 03:43:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9EBC86B0092; Thu, 25 Apr 2024 23:43:29 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 99BB56B0096; Thu, 25 Apr 2024 23:43:29 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83B7A6B0098; Thu, 25 Apr 2024 23:43:29 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 600C96B0092 for ; Thu, 25 Apr 2024 23:43:29 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 11D5340643 for ; Fri, 26 Apr 2024 03:43:29 +0000 (UTC) X-FDA: 82050288138.14.28D2B35 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf16.hostedemail.com (Postfix) with ESMTP id 47DBB180006 for ; Fri, 26 Apr 2024 03:43:27 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=gf22pu3W; spf=pass (imf16.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714103007; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=cjep24xOYUjfKLkGIt5Xg15kGd3msBByaQ3pAIEh0mM=; b=K+JbP4p8tfbUFVnRmoDOfDZR1ayhQuYPV/vM61erDQgHyakbU1DrO1aPzCrAP3nl9FEKtb EWdYYShiNEkmwRlw2q6Hj0VRErzBcp/UZJyM74iQZ+NYWu00RVh+bAnSPm2Zt8Iru1lLFH mybs4rg7kGaXyPjce7AZT4IO4hGz9CM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714103007; a=rsa-sha256; cv=none; b=66pzceelCyi0y4itAM4SpPZFRlz3FnxzfaDYcHfl0JfwajkawUAEYinYtMTPcbL/FxRobi FQYZH/p5ZL9g4crO50zKgYSFyjvf2k5cLqjE+kkhJVtCFrYuBCdU4keOui2KPLbK9f0pC9 KNNGmeK7bQ3qkORgMDGkJ0APeBHpZqE= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=soleen-com.20230601.gappssmtp.com header.s=20230601 header.b=gf22pu3W; spf=pass (imf16.hostedemail.com: domain of pasha.tatashin@soleen.com designates 209.85.160.179 as permitted sender) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=pass (policy=none) header.from=soleen.com Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-43969307359so10806311cf.2 for ; Thu, 25 Apr 2024 20:43:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen-com.20230601.gappssmtp.com; s=20230601; t=1714103006; x=1714707806; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=cjep24xOYUjfKLkGIt5Xg15kGd3msBByaQ3pAIEh0mM=; b=gf22pu3Wmhjwgz4ggluzpkl4IcksgLLbJo7xwtgCI6N+1JYMgK9rFRrO//YSWRCflL jD74kwHhNovcFNMGTwDe4QtT5BwxxWKXdL3hdWx+9VLqKW2DV3YoP3Hk9zrL13GCMc9k ttl3oPHyqVHjaqAJZ4qnpbnV2Or7IxTAACrUL3NeBQ7MO23UcJL6GSlYd2dHEV+Zh72S El5KrondC+KBUIYGMW9HNWXwG/gg8VRjsGpg7SZS8msQIMi0bBPEL29tq/+vL3bY0XRh ifGoq1icrHq+JcwBQGc4lCx4AB+1CsJ77pgpFAFk79jYCNAaUgbLjvGSyVrNFueQdely dtCA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714103006; x=1714707806; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=cjep24xOYUjfKLkGIt5Xg15kGd3msBByaQ3pAIEh0mM=; b=gkAo+cHEx9ghTZtHXsnQF4wtm90EKMQGtwdt6x2E5uFM2Zc3zDTvvEN/q1WQtZ5NgE x7wQseu0US9DSKFKi1Dim3WGfNKswiMMXvp9sA1p8PXgE15l4ta4FrP0wdHMR42RhqBL q5RW2sEVHm+5P5A4dGIzdPZSZjIZpBMTHrqe0SAGZZlm2uhtnrPGlfkHTXIuPz6+cGgz epCzzAZyOa+DczPHzVo9doLP2GCOYKOGaC0gp+p/SRGrONB1RO8r9JusqEYh5g/yHs4A hq9dq+0USO1X1GDlR+aRmPvuVEUoECDuLi4dsOdaPGrlQoIoOfWhbsHm0qoNPtfU0WXx C3eQ== X-Forwarded-Encrypted: i=1; AJvYcCX1vnTdpR0+csU153kLrKwlwlRCVLxV/hwpFqCY03HUdO6kzen0rXtUUh+63EkOv5utUYmNiaPaRNj3UfacJw8aXQ0= X-Gm-Message-State: AOJu0Yyyd3nCG66gjaZSzAVAPxRJYuKUYLk633IARu2oYuJVeoFRZ36V l7++vlvRWF/Bv7KE8IdG/EvBRHFknScOIpz210D6H3WEXFd3Kmh/hSPtUlm7LPGssX6RhY49i/G b X-Google-Smtp-Source: AGHT+IFuoNJDhVkOL87F6xMutiaC8xqNWrWB8I1N4j8YVN/IGWVNHKOf3F9BvXbF9SFqhbkbEzFqrA== X-Received: by 2002:ac8:7dc1:0:b0:437:9877:333 with SMTP id c1-20020ac87dc1000000b0043798770333mr1787129qte.35.1714103006289; Thu, 25 Apr 2024 20:43:26 -0700 (PDT) Received: from soleen.c.googlers.com.com (129.177.85.34.bc.googleusercontent.com. [34.85.177.129]) by smtp.gmail.com with ESMTPSA id c6-20020ac80546000000b00436bb57faddsm7540815qth.25.2024.04.25.20.43.25 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 25 Apr 2024 20:43:26 -0700 (PDT) From: Pasha Tatashin To: akpm@linux-foundation.org, linux-mm@kvack.org, pasha.tatashin@soleen.com, linux-kernel@vger.kernel.org, rientjes@google.com, dwmw2@infradead.org, baolu.lu@linux.intel.com, joro@8bytes.org, will@kernel.org, robin.murphy@arm.com, iommu@lists.linux.dev Subject: [RFC v2 0/3] iommu/intel: Free empty page tables on unmaps Date: Fri, 26 Apr 2024 03:43:20 +0000 Message-ID: <20240426034323.417219-1-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.44.0.769.g3c40516874-goog MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 47DBB180006 X-Stat-Signature: wzpwretasc1mduxrmer4nbgk1r3yi1yk X-HE-Tag: 1714103007-763762 X-HE-Meta: U2FsdGVkX1+UgLRAVVBZq34+Pxo+VUy6CjsRYW7Ouz8nJC3JhvAF1fKf7XuDtqflabOGm9Vm7xtoxO5zrv+fCMxcb8kIki8OcLspv3rD9/EiRJBSVj0qq5bfbXHxcJyrhXTBxIa+iisrNd3T78h4sT1+xOUnUZp+voAclQIS4XobasNkitojCofMm20+WU+GlX51oWhKd3u0XUasFT0VzNpb447zOMGGQo2HiNgDjJy/xGoAdku20US2grGbwq91/x8IkbKalgBnpGeft/8gmUaIe6gi0M+It8WL0n5UKk+P/gWt7GKCdystZi78XFJrcuxiiSeYd/l8Tzjwwq9GBfR90VQNbpfDwMGAKTgRLgl/+icjT4yo2AEsDXQURBq/aL7/NTgfHdFrICVjjc5qUA/irkaY2J3lSAwZEpn6dk8MexSLv8u4WALlQNETKPhgojljbvC2H2f5KKeP5YiJb3v8RqgRCQtr7OxP15X928bIEmSJZ5xQVzVM196R2mPwz2+sQu7SDMqg6ByL1Y4xW7yzb0UW9wC/j+9lnEIdnuND5nHtW/tZArjsUtUnj/FueynKi26wxZppRemYC15Qvco00AAPNuFXD4BiSTyxvpOtq8ODcjlx1ZkmLqV83PO0J+1h2EelEm9o+KGYcTbsS6n+/nmxJgqSCldq5rzrH+/GbS8jTDnEIJROFgBTZfC3bh4i3TAKDWBQ5jFHN04II0j76MdD0xiS9vYnEXwPBzI0hzOxZ2w3azc9vz62GFdej6ti0BwGcS1yKDw9t2NUWLC7LK+BkxLaW5prDp3CM4b2ADbpdBiTL/3rzVs2Yug2IsfLazH581HO81z87qjAVdpyucNjFe6hCbpQOsgXtu97pU4Kja/RZ8Q9ENZ9FSyNzap5fATi4R+AKlyu/tn5xEwK9TZJnglpovgTq/UVjihY3J1AGU0n3nIrHsdl3vtEjin8Bckf1TOv4gldgkS Dre1lgaH FR0HQZrJgk++HpXQAEJ1/HRlQGojNpuA+V1JxeDlxmKwp8fOHBLyj7f9Mt+hUSP4cNJzUimaCPz+V7etj/l6mwY/HNcHYppdBIfkLLlnquEZwU8UOr8FLyKXTFkE8gIo3dO9jdI1X4PtlhATtnuhxzNIB3h4Tz/3rhuF9yGhZwEuXdKHIHkzJNIrbhEMcKafYjvfrVAz5M1do4DOjpI1OaxA+4uhSQp0MSi4JYjao39sWr9bk65B3p12MbUWPlHbK80crH/8NeB6MPbTg1zu7L92+vuCQavX2oyEK5VUVPhdJrdyz3ChbqtES0ACfhGjjlR762lwXio5q/3kImddKWfQc/myjlh8vBqYuYCTmegDsO2NMuCiVtXCkHaCw9bkW/wVEu0VQ1aUrmcEdS4WAawqDe2XbRgWk1SCjZdBrjxHBfv0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Changelog ================================================================ v2: Use mapcount instead of refcount Synchronized with IOMMU Observability changes. ================================================================ This series frees empty page tables on unmaps. It intends to be a low overhead feature. The read-writer lock is used to synchronize page table, but most of time the lock is held is reader. It is held as a writer for short period of time when unmapping a page that is bigger than the current iova request. For all other cases this lock is read-only. page->mapcount is used in order to track number of entries at each page table. Microbenchmark data using iova_stress[1]: Base: $ ./iova_stress -s 16 dma_size: 4K iova space: 16T iommu: ~ 32847M time: 36.074s Fix: $ ./iova_stress -s 16 dma_size: 4K iova space: 16T iommu: ~ 27M time: 38.870s The test maps/unmaps 4K pages and cycles through the IOVA space in a tight loop. Base uses 32G of memory, and test completes in 36.074s Fix uses 0G of memory, and test completes in 38.870s. I believe the proposed fix is a good compromise in terms of complexity/ scalability. A more scalable solution would be to spread read/writer lock per-page table, and user page->private field to store the lock itself. However, since iommu already has some protection: i.e. no-one touches the iova space of the request map/unmap we can avoid the extra complexity and rely on a single per page table RW lock, and be in a reader mode most of the time. [1] https://github.com/soleen/iova_stress Pasha Tatashin (3): iommu/intel: Use page->_mapcount to count number of entries in IOMMU iommu/intel: synchronize page table map and unmap operations iommu/intel: free empty page tables on unmaps drivers/iommu/intel/iommu.c | 154 ++++++++++++++++++++++++++++-------- drivers/iommu/intel/iommu.h | 42 ++++++++-- drivers/iommu/iommu-pages.h | 30 +++++-- 3 files changed, 180 insertions(+), 46 deletions(-)