From patchwork Thu Feb 1 19:30:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Pasha Tatashin X-Patchwork-Id: 13541684 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B0045C48286 for ; Thu, 1 Feb 2024 19:30:20 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 0A0D56B0072; Thu, 1 Feb 2024 14:30:20 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 051536B0074; Thu, 1 Feb 2024 14:30:20 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E5C0F6B0075; Thu, 1 Feb 2024 14:30:19 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id D8D986B0072 for ; Thu, 1 Feb 2024 14:30:19 -0500 (EST) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id AB85C160EF9 for ; Thu, 1 Feb 2024 19:30:19 +0000 (UTC) X-FDA: 81744226158.16.44C8EA1 Received: from mail-vs1-f52.google.com (mail-vs1-f52.google.com [209.85.217.52]) by imf03.hostedemail.com (Postfix) with ESMTP id 05F0A2001B for ; Thu, 1 Feb 2024 19:30:17 +0000 (UTC) Authentication-Results: imf03.hostedemail.com; dkim=none ("invalid DKIM record") header.d=soleen.com header.s=google header.b=eC7onT1p; spf=none (imf03.hostedemail.com: domain of pasha.tatashin@soleen.com has no SPF policy when checking 209.85.217.52) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706815818; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=BSEQBUwLYtDyX8SfM7b6fTPQYYyhCXu8tsAp9rUSiMw=; b=q5wgH6/XusvbEeSiD/OD/JfcgLoAOTj0EHsCK5hwmmhVEiD1sJm/ZZtmdoY6Zg8CpTV3xF iniahRBDcerxEYrDQgQG0r4i5Ozx9QuPUllNqiMyPcbxl5KzJfWfsVaWjvtI+Sm9jobX67 bUK1KRoBuHmwR3kyhtsJUBhlg0Pbc7A= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706815818; a=rsa-sha256; cv=none; b=3VAG75OsQKQz7sOx5XK7EZZkMhvt1/1T8n9ceWqvSZtguVQn4VU5jjSD1QTTg6+u4qeibQ VikHBAPLiTLQ/pS93CpOINitrAB85iwzl4qbJhXxkuPW9wVl9uYg0ST3tlVkg4tqiQierR LGfNrRx1bwqFpzMLm9/XmLECTRiyLj8= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=none ("invalid DKIM record") header.d=soleen.com header.s=google header.b=eC7onT1p; spf=none (imf03.hostedemail.com: domain of pasha.tatashin@soleen.com has no SPF policy when checking 209.85.217.52) smtp.mailfrom=pasha.tatashin@soleen.com; dmarc=none Received: by mail-vs1-f52.google.com with SMTP id ada2fe7eead31-467ed334c40so608532137.0 for ; Thu, 01 Feb 2024 11:30:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=soleen.com; s=google; t=1706815817; x=1707420617; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=BSEQBUwLYtDyX8SfM7b6fTPQYYyhCXu8tsAp9rUSiMw=; b=eC7onT1p3cOcZij9hT49uBJUEfMyWsizFPvEILgCtJjwrXGS76eromc3xwuhVQdsd0 2hs9BlYiDlGIM1U62fwlSAVg7Zs/zaKB6z2jDTOphj8zNJyYN6beUAXkwChRCMgOR3Ro Ai9HTj9PJZ0ei//kHq2mPkqEgdFsjr6AJc7PXGk8sn3PBcSCX5foHkGndA/98FL1d4Iv NAktVXuVJaUtlfsu33Nex8S90+g3xZXdWl2rIb2TfDRJr1tdlhYTwNw4NlmSegNc/mDu 73QzMqV4kDiV1v7BmDrCFmDyIIfgJsiSOwOd5nxZtLzirP8zBwQyz1L5NmibLzhjpZTR /g2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706815817; x=1707420617; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=BSEQBUwLYtDyX8SfM7b6fTPQYYyhCXu8tsAp9rUSiMw=; b=HI1daYkcYcpV6XpC4C+d4bZ79nHwtqtsrKodB28doatWZKd3qr7KejYVuohDpXv2PH OMDlSWqQb4d79XiXnsLhc3/TOwX0Ex0efqP92hIGYLsvPAX9XaSn2tV6R0MIVkyKXVlV H1Xx3Sd0nkf2ty7z4KPB9OmyBPN3R6/7pUPNw6Mhii6GOQHVGEw+GDS9kQ6YAw0IMEhd MECvjZKU6IP7Cm8UYLwbYWuTgEU7Kw0v3gVdz7mhrjEx2xG4yUvUwuOjDoZsIQL3JOVG llXxWS1J9tN6ryrmI8V5ryI7zBulWqVMz//K+cXBDscpL73enQFbSYQpvRSOHg2ZFdTZ wUiw== X-Gm-Message-State: AOJu0YzH5c+R14QzstESzL69eG/GzWdXZChOq0RK17307pJCNzhWdAtg uzPT6NmVTk0xcRhAGSBhUl3ah2SR8i0sMQjQNXmbvEyHshmqnuqG4T4kj27qLH8= X-Google-Smtp-Source: AGHT+IET93Vs0eqCzQUdFrwgDKKIhEbJwk6iAT2cMBeQIiq6TG6SVpcwALrgdlhKSo1JBgLLU5iGkA== X-Received: by 2002:a05:6102:316c:b0:46b:159e:bf04 with SMTP id l12-20020a056102316c00b0046b159ebf04mr6285592vsm.2.1706815816873; Thu, 01 Feb 2024 11:30:16 -0800 (PST) X-Forwarded-Encrypted: i=0; AJvYcCUbTUU/Nlcj6oufqZ9IAZI/9jykB57i+q5vJUcSyMfSXMQRX5nRq/VbOwv5We1qFiKsHSdD7s8/MjqeEFVGLMNwkff7T4Qgj0EeVt9bhyeAoHbv04nlO5LFzIBdsHmNbQf7Vfm2vhlhVWweFrQWRpVA2xQwu/P49MiyEL6Zh1pCk+/vUADBYM2rLeMH8HVCkvoXZJh06/iCgcSd+ue6+o/VG/zOyn/T3mgonWNQ2MjM7t3t27PsCbTZIWLixNH5U5jl/XZH76hxYbE= Received: from soleen.c.googlers.com.com (249.240.85.34.bc.googleusercontent.com. [34.85.240.249]) by smtp.gmail.com with ESMTPSA id r18-20020ad44052000000b0068c50dec857sm58648qvp.128.2024.02.01.11.30.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 01 Feb 2024 11:30:16 -0800 (PST) From: Pasha Tatashin X-Google-Original-From: Pasha Tatashin To: robin.murphy@arm.com, joro@8bytes.org, will@kernel.org, iommu@lists.linux.dev, linux-kernel@vger.kernel.org, linux-mm@kvack.org, pasha.tatashin@soleen.com, rientjes@google.com Subject: [PATCH] iommu/iova: use named kmem_cache for iova magazines Date: Thu, 1 Feb 2024 19:30:14 +0000 Message-ID: <20240201193014.2785570-1-tatashin@google.com> X-Mailer: git-send-email 2.43.0.594.gd9cf4e227d-goog MIME-Version: 1.0 X-Rspamd-Queue-Id: 05F0A2001B X-Rspam-User: X-Stat-Signature: cgoa4bfikj1epgqybwi4t6kqz6juurn5 X-Rspamd-Server: rspam03 X-HE-Tag: 1706815817-955943 X-HE-Meta: U2FsdGVkX1/fsaKxdN5IHMxz2aPR/j0My40T+Jbqfa7fItmKmN6rJUQdQJRJqSP7jvi7uEfylWxTDqE4umP875Et/PVLheSzkoypfKMkM2rTem2xUPWpWwDb23bKlTPLijpxTBIulqCO+XRf3MtzY1AdV5kCkrrzP1GViQBxgtsGMGmMLYgKJ4nFZNUPQUKk8djn7DBtLrQZID3LJrqAG2SN3Uzm9ASjjUo4gM36JCibzHxNsK3Pxe+6r/w0Z62kxBpIy1gLdDLUOi0iHgLqG+rfYe27hoMKMmmkNL7XIm8Hh/FlEVdsD8E2UCW1qRc8lgQzeI8rQiXRbgoGvZh7fCiz4MBW0peggjS99pd/pqSVLq4eZkpZ2lwS34u3wne4rSjolzENhKKeAkUnlXcv5R8Zpc34NcPh2rfvtAccgwHUYgRZZcQBlIe41HEwnf8sAP4flhoZ8tMQ7wlPiN6hWTL23jDQYUaesanBygEUHTOlMtU+qIIOem6AT1P7u5twocpUfG8PPcqgmWwnyowL+z1KEwDbWK4rErUMd8Fm+9WeOGzFQCdHToFxBICohcmOnzPiNPUP4yi0aOmExdPbbjeX56UQKdY2grVvnIJ8trZ4Uxt3Xt4krOTawIWOjFeC1I0XBsM8KcFyTB8Jh3ZSIZENwOJxxXJpXQGHdwq98I+BcsHs72wG4+pwUjIjzuGjy+tF67GyTCVw/f2btPLDFXMmDZtzR5KBoQ9oGO/uuKpz9atCJiN/zHuJn8mJPbEBCMEy1BPifzCfKFTeTmD1553yLY9WHy0r/R048RqTD2vZ/lGxU5EYvsa0swJXsBeuqmuTNRxIBOVHqOng1I9QPVTlPbFbYI+fuQ6AsdjIl5IdRItteCRopuiFB4s6whvYV/ipY2r5Ck35LSPTFgp+2riFP59wpoxCsHLGvWDzc8+khajYi/4MwM+aq6Ilct3lC4x095DkxaYhDZILQbx 1hwBrN+h 8jqIX3glFRVj4WHBZ1lN9XVdd5i2cX85wloiflFBgmxMk/2Q= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Pasha Tatashin The magazine buffers can take gigabytes of kmem memory, dominating all other allocations. For observability prurpose create named slab cache so the iova magazine memory overhead can be clearly observed. With this change: > slabtop -o | head Active / Total Objects (% used) : 869731 / 952904 (91.3%) Active / Total Slabs (% used) : 103411 / 103974 (99.5%) Active / Total Caches (% used) : 135 / 211 (64.0%) Active / Total Size (% used) : 395389.68K / 411430.20K (96.1%) Minimum / Average / Maximum Object : 0.02K / 0.43K / 8.00K OBJS ACTIVE USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME 244412 244239 99% 1.00K 61103 4 244412K iommu_iova_magazine 91636 88343 96% 0.03K 739 124 2956K kmalloc-32 75744 74844 98% 0.12K 2367 32 9468K kernfs_node_cache On this machine it is now clear that magazine use 242M of kmem memory. Signed-off-by: Pasha Tatashin --- drivers/iommu/iova.c | 57 +++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 54 insertions(+), 3 deletions(-) diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index d30e453d0fb4..617bbc2b79f5 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -630,6 +630,10 @@ EXPORT_SYMBOL_GPL(reserve_iova); #define IOVA_DEPOT_DELAY msecs_to_jiffies(100) +static struct kmem_cache *iova_magazine_cache; +static unsigned int iova_magazine_cache_users; +static DEFINE_MUTEX(iova_magazine_cache_mutex); + struct iova_magazine { union { unsigned long size; @@ -654,11 +658,51 @@ struct iova_rcache { struct delayed_work work; }; +static int iova_magazine_cache_init(void) +{ + int ret = 0; + + mutex_lock(&iova_magazine_cache_mutex); + + iova_magazine_cache_users++; + if (iova_magazine_cache_users > 1) + goto out_unlock; + + iova_magazine_cache = kmem_cache_create("iommu_iova_magazine", + sizeof(struct iova_magazine), + 0, SLAB_HWCACHE_ALIGN, NULL); + + if (!iova_magazine_cache) { + pr_err("Couldn't create iova magazine cache\n"); + ret = -ENOMEM; + } + +out_unlock: + mutex_unlock(&iova_magazine_cache_mutex); + + return ret; +} + +static void iova_magazine_cache_fini(void) +{ + mutex_lock(&iova_magazine_cache_mutex); + + if (WARN_ON(!iova_magazine_cache_users)) + goto out_unlock; + + iova_magazine_cache_users--; + if (!iova_magazine_cache_users) + kmem_cache_destroy(iova_magazine_cache); + +out_unlock: + mutex_unlock(&iova_magazine_cache_mutex); +} + static struct iova_magazine *iova_magazine_alloc(gfp_t flags) { struct iova_magazine *mag; - mag = kmalloc(sizeof(*mag), flags); + mag = kmem_cache_alloc(iova_magazine_cache, flags); if (mag) mag->size = 0; @@ -667,7 +711,7 @@ static struct iova_magazine *iova_magazine_alloc(gfp_t flags) static void iova_magazine_free(struct iova_magazine *mag) { - kfree(mag); + kmem_cache_free(iova_magazine_cache, mag); } static void @@ -766,11 +810,17 @@ int iova_domain_init_rcaches(struct iova_domain *iovad) unsigned int cpu; int i, ret; + ret = iova_magazine_cache_init(); + if (ret) + return -ENOMEM; + iovad->rcaches = kcalloc(IOVA_RANGE_CACHE_MAX_SIZE, sizeof(struct iova_rcache), GFP_KERNEL); - if (!iovad->rcaches) + if (!iovad->rcaches) { + iova_magazine_cache_fini(); return -ENOMEM; + } for (i = 0; i < IOVA_RANGE_CACHE_MAX_SIZE; ++i) { struct iova_cpu_rcache *cpu_rcache; @@ -948,6 +998,7 @@ static void free_iova_rcaches(struct iova_domain *iovad) kfree(iovad->rcaches); iovad->rcaches = NULL; + iova_magazine_cache_fini(); } /*