From patchwork Thu Jun 30 07:33:04 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Feng Tang X-Patchwork-Id: 12901293 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BC179C43334 for ; Thu, 30 Jun 2022 07:33:11 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2F7106B0072; Thu, 30 Jun 2022 03:33:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2A6D46B0073; Thu, 30 Jun 2022 03:33:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 16E868E0001; Thu, 30 Jun 2022 03:33:11 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 08B976B0072 for ; Thu, 30 Jun 2022 03:33:11 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id CADA9356FC for ; Thu, 30 Jun 2022 07:33:10 +0000 (UTC) X-FDA: 79634086140.19.A212079 Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by imf03.hostedemail.com (Postfix) with ESMTP id CE76B2002F for ; Thu, 30 Jun 2022 07:33:09 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1656574389; x=1688110389; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=yS4Suwf7GI6LhylNscWfzZ0ukkPLH++SXvgQgtpNGaQ=; b=b1sEyreg3mJ/mH73KBHLz2zAodXDpkuGCfDkRWO9SxjPm1kMOzJ/1ues tkZxjyJFH9QFdruzy9KAp0c+zSvkgir9WFwvNYzRZ3hjyRVlQCQsm8exg pu0mRLKhjgPH+JsibXqZ2NrfWlsvmiIOLKIR/u7DHVkLjk3MSQF9TAhxZ M/1hHCI5MK9XmfIACSastc48F24qHr46pYdLrCWKv3B/bhCT1aR5ETPU9 MIOp0iBLGvB3l4X7BQFHqvuCS8FfR57z3McQMYgWp5/mvHr+Dx0IKsc/p 9TOiFxWYRGeNkXgSpMej/I3xs92KVeNIM4zSgSQXgAqYl/zAbPGNd7QNU g==; X-IronPort-AV: E=McAfee;i="6400,9594,10393"; a="279817398" X-IronPort-AV: E=Sophos;i="5.92,233,1650956400"; d="scan'208";a="279817398" Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga102.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 30 Jun 2022 00:33:08 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.92,233,1650956400"; d="scan'208";a="680866208" Received: from shbuild999.sh.intel.com ([10.239.146.138]) by FMSMGA003.fm.intel.com with ESMTP; 30 Jun 2022 00:33:05 -0700 From: Feng Tang To: Joerg Roedel , Will Deacon , Robin Murphy , iommu@lists.linux-foundation.org, iommu@lists.linux.dev, Andrew Morton , Christoph Lameter , Vlastimil Babka Cc: linux-mm@kvack.org, Paul Menzel , linux-kernel@vger.kernel.org, Feng Tang Subject: [PATCH] iommu/iova: change IOVA_MAG_SIZE to 127 to save memory Date: Thu, 30 Jun 2022 15:33:04 +0800 Message-Id: <20220630073304.26945-1-feng.tang@intel.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1656574390; a=rsa-sha256; cv=none; b=xpGIEiIp68UTyyc1oKE61nCgcAiy0e8XuLm3+qSzN+vCBjBkNsWzs6hxzjsG4kH3HZu7Zj L380KYFbvwjPlDaZkVTzmHn21L9i1ZxXyr7xXlKaZH7/yF1K/umQW20wQjvTZHF0e6C8Ap p+pxbPOVd2BXrg9cekvh68fFKC5IGW4= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=b1sEyreg; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf03.hostedemail.com: domain of feng.tang@intel.com has no SPF policy when checking 192.55.52.93) smtp.mailfrom=feng.tang@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1656574390; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=7DziQ76URbDQnzh3vqtJ4ZNfvyGa+B8hK4DwpSzpwok=; b=IToKcC4s2skHEwdsya+yM6rKqdMKwXl9jXs437uizwQhdpSpUMa0FipNCW+jCgz2vPQlfg WfvQD8aCAwEjyPlcssXFT/f1vb9zxGtqesdwHlGorFgxsJC0So5B+4Wi+VmbZZOjIvmQpA Q8k6uH7ilXegvlEkTNfqB+QbnwxnJaI= X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: CE76B2002F Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=b1sEyreg; dmarc=pass (policy=none) header.from=intel.com; spf=none (imf03.hostedemail.com: domain of feng.tang@intel.com has no SPF policy when checking 192.55.52.93) smtp.mailfrom=feng.tang@intel.com X-Rspam-User: X-Stat-Signature: huh333wx8yqznxpcf3k7k3xzhmrprxbo X-HE-Tag: 1656574389-327262 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: kmalloc will round up the request size to power of 2, and current iova_magazine's size is 1032 (1024+8) bytes, so each instance allocated will get 2048 bytes from kmalloc, causing around 1KB waste. And in some exstreme case, the memory wasted can trigger OOM as reported in 2019 on a crash kernel with 256 MB memory [1]. [ 4.319253] iommu: Adding device 0000:06:00.2 to group 5 [ 4.325869] iommu: Adding device 0000:20:01.0 to group 15 [ 4.332648] iommu: Adding device 0000:20:02.0 to group 16 [ 4.338946] swapper/0 invoked oom-killer: gfp_mask=0x6040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null), order=0, oom_score_adj=0 [ 4.350251] swapper/0 cpuset=/ mems_allowed=0 [ 4.354618] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.19.57.mx64.282 #1 [ 4.355612] Hardware name: Dell Inc. PowerEdge R7425/08V001, BIOS 1.9.3 06/25/2019 [ 4.355612] Call Trace: [ 4.355612] dump_stack+0x46/0x5b [ 4.355612] dump_header+0x6b/0x289 [ 4.355612] out_of_memory+0x470/0x4c0 [ 4.355612] __alloc_pages_nodemask+0x970/0x1030 [ 4.355612] cache_grow_begin+0x7d/0x520 [ 4.355612] fallback_alloc+0x148/0x200 [ 4.355612] kmem_cache_alloc_trace+0xac/0x1f0 [ 4.355612] init_iova_domain+0x112/0x170 [ 4.355612] amd_iommu_domain_alloc+0x138/0x1a0 [ 4.355612] iommu_group_get_for_dev+0xc4/0x1a0 [ 4.355612] amd_iommu_add_device+0x13a/0x610 [ 4.355612] add_iommu_group+0x20/0x30 [ 4.355612] bus_for_each_dev+0x76/0xc0 [ 4.355612] bus_set_iommu+0xb6/0xf0 [ 4.355612] amd_iommu_init_api+0x112/0x132 [ 4.355612] state_next+0xfb1/0x1165 [ 4.355612] amd_iommu_init+0x1f/0x67 [ 4.355612] pci_iommu_init+0x16/0x3f ... [ 4.670295] Unreclaimable slab info: ... [ 4.857565] kmalloc-2048 59164KB 59164KB Change IOVA_MAG_SIZE from 128 to 127 to make size of 'iova_magazine' 1024 bytes so that no memory will be wasted. [1]. https://lkml.org/lkml/2019/8/12/266 Signed-off-by: Feng Tang Acked-by: Robin Murphy --- drivers/iommu/iova.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/drivers/iommu/iova.c b/drivers/iommu/iova.c index db77aa675145b..27634ddd9b904 100644 --- a/drivers/iommu/iova.c +++ b/drivers/iommu/iova.c @@ -614,7 +614,12 @@ EXPORT_SYMBOL_GPL(reserve_iova); * dynamic size tuning described in the paper. */ -#define IOVA_MAG_SIZE 128 +/* + * As kmalloc's buffer size is fixed to power of 2, 127 is chosen to + * assure size of 'iova_magzine' to be 1024 bytes, so that no memory + * will be wasted. + */ +#define IOVA_MAG_SIZE 127 #define MAX_GLOBAL_MAGS 32 /* magazines per bin */ struct iova_magazine {