From patchwork Fri Feb 2 19:28:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13543326 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3CD84C4828F for ; Fri, 2 Feb 2024 19:28:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id BB5396B007D; Fri, 2 Feb 2024 14:28:25 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B3E776B0081; Fri, 2 Feb 2024 14:28:25 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9DF5F6B0082; Fri, 2 Feb 2024 14:28:25 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8B0016B007D for ; Fri, 2 Feb 2024 14:28:25 -0500 (EST) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 535D5C02A4 for ; Fri, 2 Feb 2024 19:28:25 +0000 (UTC) X-FDA: 81747850170.03.85B97C7 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) by imf14.hostedemail.com (Postfix) with ESMTP id 9E438100009 for ; Fri, 2 Feb 2024 19:28:23 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=none ("invalid DKIM record") header.d=soleen.com header.s=google header.b=FehqzAZs; spf=none (imf14.hostedemail.com: domain of pasha.tatashin@soleen.com has no SPF policy when checking 209.85.222.170) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706902103; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=DRMVoXUIDhsqk2wauqeaSYS0KJTLFWCGjlI236Qz/TI=; b=QHWIxoJhBcoRMu0/9B+2wiwOVRxF6VyxjeHE24+3DT5xtTVi3xjeOlb+CeT89OdTY8w4rW /w91sLqCp724L7/pXKMRb+aEi0ajv6f0DsiPiqI8CtV9shhykri7/lcBDbvrDsLEjxbxj7 6WRgLmII2qU10JkUWRHYdBc+hGzkbhw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706902103; a=rsa-sha256; cv=none; b=y+3Bnf67tJBMLG0kpp8M/Eh9cXcSoe78kT1OTzKLo1kMBly6kOcv/I01SeEAGFAhci4N4A BdJu17vc8nid0hT4RHhudTI1etGx18picN2hY6YW2fVm5pSD1mzyWOlA+APb4LEAlskQ/+ f9O+zXPZG4gjb/zEoc3gDwKQ5CG47gw= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=none ("invalid DKIM record") header.d=soleen.com header.s=google header.b=FehqzAZs; spf=none (imf14.hostedemail.com: domain of pasha.tatashin@soleen.com has no SPF policy when checking 209.85.222.170) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-783d4b3a026so151873985a.1 for ; Fri, 02 Feb 2024 11:28:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; t=1706902103; x=1707506903; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=DRMVoXUIDhsqk2wauqeaSYS0KJTLFWCGjlI236Qz/TI=; b=FehqzAZs18SZVMOmKZTuVAh7P7ZLBjrGRktvlb7NNteyqTFnysjqdzHo+szHMBjiQG Nl9KQYU9LPBS0IJHrRxRDubHBipKAQnY2Sjg1EgNBD50nVa3q4Hwj9efWQmk2HJzkV9c j0pSBosT7DIEjgVhXBrPZ+hFPh4TWN8C2KPJnZN7cWhZQZ37L3OZK5pmfyGxKIF2k/d+ Te7MsQg+0Kyc4QWyYaSHcvWyGOGhpSCqrjTsVte3W14Bma6NRovzaWYDj4EJR8koLXId uJx3IY8joEPFtsQqLA5vLxS96+J4d41W8Q6CvVDDPJz7kaCVrVOopXzwA0Vm6gr5wr9N qbsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706902103; x=1707506903; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=DRMVoXUIDhsqk2wauqeaSYS0KJTLFWCGjlI236Qz/TI=; b=cG3qUCl4MBApY/McHF4zHtNAey+fjVgPtEvV2LtfW/3MzsdI/sAOc0BysdAYPv179z uX6+41gDn00HgOf/v3qxrxlUm8o58plJ0Qs9Tv1cWyDQeCDLDuKM+X0LsM1ubOvpXH98 fQ9WURQltDu3mH9CkTUdxTW02E2GaXCjMM6OhyczNlkiVR984f8gHXDudGzmNEB4C+vc vO4S49/Mc2XELsl3zgIBEf+z+vCrbYI/BSMq3RJtFDpS7lVHWctjyamXPeYWon+1HIXr RZCbwvm9BH7HfEcw2ClxtnLQmXMN5WEYAD9dcVeTXm6/aN4Sc9WM1Ri216N1XNXjazb6 GWgA== X-Gm-Message-State: AOJu0YyoyUUac42NziqLTESw9O5KUKwM8I1d0QdgBH9vIoaGfoWQ9Krh Z00guSc1/7CvuvJMBofwGERX69kQC1Elne0l9g4VsiX83ZMNOhLBPEF2a52D54U= X-Google-Smtp-Source: AGHT+IE2j+F9W4BLBuGDUanXeAR8Q/iu6EVu8yI5R6ISNqSNldAm4ZHYxxJTdJtaLcdM+zS8CiYtWg== X-Received: by 2002:a05:620a:3885:b0:783:9a16:307f with SMTP id qp5-20020a05620a388500b007839a16307fmr6694812qkn.18.1706902102702; Fri, 02 Feb 2024 11:28:22 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCUs3qHpOZ2c1uil71kD5W+kCecGQrkbnV/5pNfTp0xVi4h70AJfW4s6lXExDKn+ZZvmTqY5ubjqxI4VzYMk/i0p6bUaKx7KP84/W4cV/5ND5LFV421tdfdVbtHZxKYHoxSEikm3zZBNStesA/22yIK4C6ldTfg8cuPtBgE6U9bI0Y2s854k+somdy0gB6JrbGuBSgfoRUg0ko43jhZbebMVmLMiT/IfJ2BsBYF2RoEMioBoXqGDXh1K0tExwURHX1A5O57CndtgqpSBDh/IuPaU/fG2tfFqCoxMkYB2n5spwg== Received: from soleen.c.googlers.com.com (249.240.85.34.bc.googleusercontent.com. [34.85.240.249]) by smtp.gmail.com with ESMTPSA id u5-20020a05620a022500b007831f8c5aeasm902250qkm.55.2024.02.02.11.28.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 02 Feb 2024 11:28:22 -0800 (PST) From: Pasha Tatashin To: robin.murphy@arm.com, joro@8bytes.org, will@kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-mm@kvack.org, pasha.tatashin@soleen.com, rientjes@google.com, yosryahmed@google.com Subject: [PATCH v2] iommu/iova: use named kmem_cache for iova magazines Date: Fri, 2 Feb 2024 19:28:20 +0000 Message-ID: <20240202192820.536408-1-pasha.tatashin@soleen.com> X-Mailer: git-send-email 2.43.0.594.gd9cf4e227d-goog MIME-Version: 1.0 X-Stat-Signature: eaea97jgptdqchuofohnzwerkt96rxe1 X-Rspamd-Server: rspam10 X-Rspamd-Queue-Id: 9E438100009 X-Rspam-User: X-HE-Tag: 1706902103-707252 X-HE-Meta: U2FsdGVkX19sJKnv2QEx1aqzVaqU9rq5KpD3eDn7tQtDmn0tr4DFqRbnaNB2+ADYalbysFsIuSIc+MUdTi89vq90y3Lvv8q8+LzR+X+RXvJD35eDUO5iDR5JiaD/D/qIbJloV86Yo7OJkvbDYeBMNLX+YWJSNGpOneWbr9JhQeHXnCFTZ5vfR4U0QpBW2dj7G8eNuXhFhNja9el86mLP1QStWEKf3I+t3li6Mw1zRw3Ki7C4orq8g7s3mCLilftMuS3mYFVMk3dNrvNsIpfcE+0txb9Gte2LDnmCdYT/zGrdLtVJqHQOQPpgbrpL3iiaE6HclZvquhQEVOmjx+gtLH3gceiOS7Q534dL0KljtADWOxbqoNvrcBNphapu1RaEylZGP1X/3dFNGaVSalSh7/hhXvc3EgNnFIGY3xu+EoygrB6lI95ut7ouRsux4P7Y4NoXZTETcevM5Ocw7Gv01L5wZvaRQo9w4Y1mmKCvBFESOr956YplHzNCUyE0fC8jZOaQZrU7Lx2N6CWHCI6DbUkS0nNXxRM275sD3WCMkKX4xCXxtdAM8c4NOoRd6KyhFTNudoWZpxuoJUq6RrfEv+ksVwH5kbCnR+xTpawH1ihM5WJ2/ScQab0f6snRIeEDpOxXIif6WB115H3BwAdM8kQYtI/URPK8sxYSXcETFPc1KrJmzWfmIs0OAiw4W7D0OTiVbwqTNcbeKFzzHkKkBiqko/YReCQMVMSvRh41c4XSoCzu71kgHTP2m7VmHlyi/IXN4VPCuJbljR9e/cBqS91PXPhMnVtLmepUl7ZwFpL/p40Jai4EWdsBhrn0XZduQ/ki/Ils4Cz4IlAwPOjIyFpKIjGEUdo5AUc7rO7skxf/4+OiGBulM+PxtKt659smvBoDS2CTJ2ivkqBdf6+kxBKS2QlKWO2SznlWo/Bhy2eZuN54ezQw9MNXzkMR+jxLHAmrTxbZZsO2ZWUCCMp a42AJyML mwSH3BGHXdN+aOEe49InEGzLaK3C2qV7Q1CoFzbfcHmrUqPNRw/YqZ+L0CFNBO32jcR2B3sT+ox5K+faehykL606l2sHGgbs3IREe61x2x5SvqMtk+oKKyr/rXCVDl/oIrXyvkm6srASehudB1qu/8ro5Vv0YdETUuV+2xu0MMTj68y9amCzWhERc+glucMkWqn+uGI2zokiS2o/o5hUdfblAJeMDWuakFK9abRm9LpwKXIczgahnJID2DIH2A9PMqyElIA7cR9WEOv3xx6v5n7WXfZLDp1WEcYKFjKdbiJvd2OrNZ5yrriy7RHHt43JRF0xAR2Zu3wnwsziTBYLdJ2eEGmn9nzKdXGhGS4k9iFrZa8J0zyqA2PzLlw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The magazine buffers can take gigabytes of kmem memory, dominating all other allocations. For observability purpose create named slab cache so the iova magazine memory overhead can be clearly observed. With this change: > slabtop -o | head Active / Total Objects (% used) : 869731 / 952904 (91.3%) Active / Total Slabs (% used) : 103411 / 103974 (99.5%) Active / Total Caches (% used) : 135 / 211 (64.0%) Active / Total Size (% used) : 395389.68K / 411430.20K (96.1%) Minimum / Average / Maximum Object : 0.02K / 0.43K / 8.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 244412 244239 99% 1.00K 61103 4 244412K iommu_iova_magazine 91636 88343 96% 0.03K 739 124 2956K kmalloc-32 75744 74844 98% 0.12K 2367 32 9468K kernfs_node_cache On this machine it is now clear that magazine use 242M of kmem memory. Signed-off-by: Pasha Tatashin Acked-by: David Rientjes --- drivers/iommu/iova.c | 70 ++++++++++++++++++++++++++------------------ 1 file changed, 42 insertions(+), 28 deletions(-) Changelog: v2: - Use iova_cache_get/iova_cache_put to allocate/free "iova_magazine_cache" as suggested by Robin Murphy - Minor fix in the commit log. diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index d30e453d0fb4..88255f9443b5 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -237,6 +237,35 @@ static int __alloc_and_insert_iova_range(struct iova_domain *iovad, return -ENOMEM; } +/* + * Magazine caches for IOVA ranges. For an introduction to magazines, + * see the USENIX 2001 paper "Magazines and Vmem: Extending the Slab + * Allocator to Many CPUs and Arbitrary Resources" by Bonwick and Adams. + * For simplicity, we use a static magazine size and don't implement the + * dynamic size tuning described in the paper. + */ + +/* + * As kmalloc's buffer size is fixed to power of 2, 127 is chosen to + * assure size of 'iova_magazine' to be 1024 bytes, so that no memory + * will be wasted. Since only full magazines are inserted into the depot, + * we don't need to waste PFN capacity on a separate list head either. + */ +#define IOVA_MAG_SIZE 127 + +#define IOVA_DEPOT_DELAY msecs_to_jiffies(100) + +struct iova_magazine { + union { + unsigned long size; + struct iova_magazine *next; + }; + unsigned long pfns[IOVA_MAG_SIZE]; +}; + +static_assert(!(sizeof(struct iova_magazine) & (sizeof(struct iova_magazine) - 1))); + +static struct kmem_cache *iova_magazine_cache; static struct kmem_cache *iova_cache; static unsigned int iova_cache_users; static DEFINE_MUTEX(iova_cache_mutex); @@ -275,6 +304,16 @@ int iova_cache_get(void) pr_err("Couldn't create iova cache\n"); return -ENOMEM; } + + iova_magazine_cache = kmem_cache_create("iommu_iova_magazine", + sizeof(struct iova_magazine), + 0, SLAB_HWCACHE_ALIGN, NULL); + if (!iova_magazine_cache) { + cpuhp_remove_multi_state(CPUHP_IOMMU_IOVA_DEAD); + mutex_unlock(&iova_cache_mutex); + pr_err("Couldn't create iova magazine cache\n"); + return -ENOMEM; + } } iova_cache_users++; @@ -295,6 +334,7 @@ void iova_cache_put(void) if (!iova_cache_users) { cpuhp_remove_multi_state(CPUHP_IOMMU_IOVA_DEAD); kmem_cache_destroy(iova_cache); + kmem_cache_destroy(iova_magazine_cache); } mutex_unlock(&iova_cache_mutex); } @@ -612,32 +652,6 @@ reserve_iova(struct iova_domain *iovad, } EXPORT_SYMBOL_GPL(reserve_iova); -/* - * Magazine caches for IOVA ranges. For an introduction to magazines, - * see the USENIX 2001 paper "Magazines and Vmem: Extending the Slab - * Allocator to Many CPUs and Arbitrary Resources" by Bonwick and Adams. - * For simplicity, we use a static magazine size and don't implement the - * dynamic size tuning described in the paper. - */ - -/* - * As kmalloc's buffer size is fixed to power of 2, 127 is chosen to - * assure size of 'iova_magazine' to be 1024 bytes, so that no memory - * will be wasted. Since only full magazines are inserted into the depot, - * we don't need to waste PFN capacity on a separate list head either. - */ -#define IOVA_MAG_SIZE 127 - -#define IOVA_DEPOT_DELAY msecs_to_jiffies(100) - -struct iova_magazine { - union { - unsigned long size; - struct iova_magazine *next; - }; - unsigned long pfns[IOVA_MAG_SIZE]; -}; -static_assert(!(sizeof(struct iova_magazine) & (sizeof(struct iova_magazine) - 1))); struct iova_cpu_rcache { spinlock_t lock; @@ -658,7 +672,7 @@ static struct iova_magazine *iova_magazine_alloc(gfp_t flags) { struct iova_magazine *mag; - mag = kmalloc(sizeof(*mag), flags); + mag = kmem_cache_alloc(iova_magazine_cache, flags); if (mag) mag->size = 0; @@ -667,7 +681,7 @@ static struct iova_magazine *iova_magazine_alloc(gfp_t flags) static void iova_magazine_free(struct iova_magazine *mag) { - kfree(mag); + kmem_cache_free(iova_magazine_cache, mag); } static void