From patchwork Mon Jun 12 08:55:35 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jay Patel X-Patchwork-Id: 13275842 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8E0AC7EE23 for ; Mon, 12 Jun 2023 08:56:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D6B906B0072; Mon, 12 Jun 2023 04:56:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CF4B58E0003; Mon, 12 Jun 2023 04:56:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BBBCE8E0002; Mon, 12 Jun 2023 04:56:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id A97516B0072 for ; Mon, 12 Jun 2023 04:56:14 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 5B6A012018E for ; Mon, 12 Jun 2023 08:56:14 +0000 (UTC) X-FDA: 80893489068.16.8AD8712 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf08.hostedemail.com (Postfix) with ESMTP id 85565160011 for ; Mon, 12 Jun 2023 08:56:11 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=HOpInjiN; spf=pass (imf08.hostedemail.com: domain of jaypatel@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=jaypatel@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1686560172; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=eHogwO6tCTp8LeACnGXsOgFPXnIIA6u2bH/CPmU7R4A=; b=Qi44sujtyaER5KqrkJNg3RU8nesqCXGqWPiyuUBRetpP+v0ai+7rwvMk1pVNbJzN23Obo8 ZBJ2Gyoch12x82ogqq7nFWDFZenXLO0kj70IkuL6aygR0itDQgXGfjomKfDfXwOFJAiQTs 7z/TBO/t7TbqC2bvX8ZJsxsiirCZpbM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1686560172; a=rsa-sha256; cv=none; b=y/VoqfuffWpRkTSeugruOLGgrc60LG67cfMhMAwJR6ScXDvOb59JXGVg/+8jbmZJVgAn41 tluEyseSX9R6JA86z6J4c9176UafftWueDrwEZarLUsL12aEDQkjm61waa/4U/3Qxn7Ov5 gAsxhpNeRbtiNok2lUfTiebDX+TyDI0= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=ibm.com header.s=pp1 header.b=HOpInjiN; spf=pass (imf08.hostedemail.com: domain of jaypatel@linux.ibm.com designates 148.163.156.1 as permitted sender) smtp.mailfrom=jaypatel@linux.ibm.com; dmarc=pass (policy=none) header.from=ibm.com Received: from pps.filterd (m0353727.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 35C8l330015366; Mon, 12 Jun 2023 08:56:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id : mime-version : content-transfer-encoding; s=pp1; bh=eHogwO6tCTp8LeACnGXsOgFPXnIIA6u2bH/CPmU7R4A=; b=HOpInjiNa/Utsd8C1e38a+KqFhzZWN00rQl5tpK+/DR6Qks9A6DdJWmFvSa/BR9m4h59 9AMfau4r/2fQ1u2XTbnScSB18rYB943hCt8YdCtKzISksu/O3O/+IxUlnAQTZsWjebeL EXIVBJH5kqvgHvK1B1fCm83ocAa+ImX9lssd84SerMHOoB/zAhNf8yrPpHQTJpSRlU4G 7dBHtbdUAYqhjXLyRSmfrMnnmGXq7U6EQ43YMj2tjtEbg3glBmCv0Js72MJ08GKDmz6g 3mxdiGdB0aCHgHetBmdAKbfjj+KDzg8v9gPbPmZrIa8fD+mzZ3Abq6R5g570ZNbI9zQp Bw== Received: from pps.reinject (localhost [127.0.0.1]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r605eg7ns-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 12 Jun 2023 08:56:00 +0000 Received: from m0353727.ppops.net (m0353727.ppops.net [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 35C8m4VE017485; Mon, 12 Jun 2023 08:56:00 GMT Received: from ppma04dal.us.ibm.com (7a.29.35a9.ip4.static.sl-reverse.com [169.53.41.122]) by mx0a-001b2d01.pphosted.com (PPS) with ESMTPS id 3r605eg7m4-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 12 Jun 2023 08:56:00 +0000 Received: from pps.filterd (ppma04dal.us.ibm.com [127.0.0.1]) by ppma04dal.us.ibm.com (8.17.1.19/8.17.1.19) with ESMTP id 35C7Tgk7011854; Mon, 12 Jun 2023 08:55:57 GMT Received: from smtprelay03.wdc07v.mail.ibm.com ([9.208.129.113]) by ppma04dal.us.ibm.com (PPS) with ESMTPS id 3r4gt5fktu-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 12 Jun 2023 08:55:57 +0000 Received: from smtpav03.dal12v.mail.ibm.com (smtpav03.dal12v.mail.ibm.com [10.241.53.102]) by smtprelay03.wdc07v.mail.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 35C8tt2g3736300 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Mon, 12 Jun 2023 08:55:55 GMT Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 8510358056; Mon, 12 Jun 2023 08:55:55 +0000 (GMT) Received: from smtpav03.dal12v.mail.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 9329D5803F; Mon, 12 Jun 2023 08:55:51 +0000 (GMT) Received: from patel.in.ibm.com (unknown [9.109.195.231]) by smtpav03.dal12v.mail.ibm.com (Postfix) with ESMTP; Mon, 12 Jun 2023 08:55:51 +0000 (GMT) From: Jay Patel To: linux-mm@kvack.org Cc: cl@linux.com, penberg@kernel.org, rientjes@google.com, iamjoonsoo.kim@lge.com, akpm@linux-foundation.org, vbabka@suse.cz, aneesh.kumar@linux.ibm.com, tsahu@linux.ibm.com, piyushs@linux.ibm.com, jaypatel@linux.ibm.com Subject: [RFC PATCH ] mm/slub: Reducing slub memory wastage Date: Mon, 12 Jun 2023 14:25:35 +0530 Message-Id: <20230612085535.275206-1-jaypatel@linux.ibm.com> X-Mailer: git-send-email 2.31.1 MIME-Version: 1.0 X-TM-AS-GCONF: 00 X-Proofpoint-ORIG-GUID: Vs91-CM-fy8I0CzmFAb1oNUPX8TTKSo7 X-Proofpoint-GUID: w_CIJI2gsGcpd2VklkZFpqjxdpM7nf6c X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.254,Aquarius:18.0.957,Hydra:6.0.573,FMLib:17.11.176.26 definitions=2023-06-12_05,2023-06-09_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 impostorscore=0 lowpriorityscore=0 suspectscore=0 priorityscore=1501 phishscore=0 bulkscore=0 adultscore=0 clxscore=1011 spamscore=0 mlxscore=0 malwarescore=0 mlxlogscore=892 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2305260000 definitions=main-2306120073 X-Rspamd-Queue-Id: 85565160011 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: os3p3ux9s6msdsfe6ewdq7k9zccqkbo5 X-HE-Tag: 1686560171-805090 X-HE-Meta: U2FsdGVkX1+nQjdK8IrXAtBkeKa/kDzCiNZw/K0U2/W28r68DgxPNNg5wzf7nR13/CSbfJqjdn5GZ3sJzlMzXvkdXwPReYCwN9BKwnnZNcGHaYlTdeCLBSnVffZ/Y5bfWxM5XJlxDJ65lj4iWecfCYQqUstpFCBZo35sj+UIgD+QRI46EPhJd3lMlLSqG9ZfvbTlkjPcJJ2U60zcs+6aflLwiScYDXvnyXpRffZrYnKdx69KBrjm7K+2bLZuwm2QBoDRpvrtBDpoK0OQuJpgUYOrHbXt3fmiQfO0CaecIGinrrWOgBv+k5P4ax6jCh4uM0OqbZiNQ4jcF+Odrn9AxeIsQGCpam3sXSeky1ZP40epLdsklSTj8EOmk0mjCkLwmqvIiEz96eHaZdm/g2ULlFzQ6Cuv3jFH9Q+adggx2uFaHZDixt0ecFwdRj+et68Gx/J7nFCjgKV0O/QlquKZI+KbhCdmNk5iTJT89l54BwSlPCQp5HGB78AABESbUHMHDu/veXS3BP2CERR2JbPFMxFaif+rdxrM1SAqkBgI3Gg27wsvsKkzeCpduKC3bAT9wWshC1YaRPgQFxZxVEESDIT6mlkn/4Cg5SEWslvfQh8NdZuLq8gdewrTta6VnJaUUldUqR7RC2OFLgxPa8C4lv0SuhKzc/r7SIfWJjbnCwR8sfQb6Q6FQrtnuxtrm9bBfbvtZddWG/aOInYYDpMjN0TMj+1xqKVneZxgodZPDBzsHt7oHsSCfFIyyW+xheIsGHSvPeBIqqHHu/mWkA7RiGkOl8xzozkir7IE4wiTsyQzOKnqpski3Zdsj1Nc0+JhgV1kaX8HuOWVz0+vFBnUkvixiY5wRxIHngN5q96j9vRNC42tCzXuU7qa+2D0cvcuec+ym7068l5sKzUG0w1XJlaCdDTfvnuVVwyomC8eGJgHazOhW6imx2vh+d3QBbh0M/JwuughgPpf6CJ+A/o KTz1hvLu fU2uAIITUgWby+1ti3jjFHPtt7JhwwUIqeJSV/huMQVUFjhXLBZcLn6k71zS8+CEEh+COv23Ibq8jtrGU/OQ+uPl7zTTRiDAFUHEjKcO8976Si9PEXVS7q55IBqe4l8OODEAwraowJkI4D48LyYgWbiHvusZGhCgcPPHALm13MX81zxjcUm/dCQrp+SWWCSn/m1bw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: In the current implementation of the slub memory allocator, the slab order selection process follows these criteria: 1) Determine the minimum order required to serve the minimum number of objects (min_objects). This calculation is based on the formula (order = min_objects * object_size / PAGE_SIZE). 2) If the minimum order is greater than the maximum allowed order (slub_max_order), set slub_max_order as the order for this slab. 3) If the minimum order is less than the slub_max_order, iterate through a loop from minimum order to slub_max_order and check if the condition (rem <= slab_size / fract_leftover) holds true. Here, slab_size is calculated as (PAGE_SIZE << order), rem is (slab_size % object_size), and fract_leftover can have values of 16, 8, or 4. If the condition is true, select that order for the slab. However, in point 3, when calculating the fraction left over, it can result in a large range of values (like 1 Kb to 256 bytes on 4K page size & 4 Kb to 16 Kb on 64K page size with order 0 and goes on increasing with higher order) when compared to the remainder (rem). This can lead to the selection of an order that results in more memory wastage. To mitigate such wastage, we have modified point 3 as follows: instead of selecting the first order that satisfies the condition (rem <= slab_size / fract_leftover), we iterate through the loop from min_order to slub_max_order and choose the order that minimizes memory wastage for the slab. Let's consider an example using mm_struct on 160 CPUs so min_objects is 32, and let's assume a page size of 64K. The size of mm_struct is 1536 bytes, which means a single page can serve 42 objects, exceeding the min_objects requirement. With the current logic, order 0 is selected for this slab since the remainder (rem) is 1 Kb (64 Kb % 1536 bytes), which is less than 4 Kb (slab_size is 64 Kb/fraction_size is 16). However, this results in wasting 1 Kb of memory for each mm_struct slab. But with this patch, order 1 (2 pages) is chosen, leading to a wastage of 512 bytes of memory for each mm_struct slab. Consequently, reducing memory wastage for this slab, increases the numbers of objects per slab. I conducted tests on systems with 160 CPUs and 16 CPUs, using 4K and 64K page sizes. Through these tests, it was observed that the patch successfully reduces the wastage of slab memory without any noticeable performance degradation in the hackbench test report. However, it should be noted that the patch also increases the total number of objects, leading to an overall increase in total slab memory usage. Test results are as follows: 1) On 160 CPUs with 4K Page size +----------------+----------------+----------------+ | Total wastage in slub memory | +----------------+----------------+----------------+ | | After Boot | After Hackbench| | Normal | 1819 Kb | 3056 Kb | | With Patch | 1288 Kb | 2217 Kb | | Wastage reduce | ~29% | ~27% | +----------------+----------------+----------------+ +-----------------+----------------+----------------+ | Total slub memory | +-----------------+----------------+----------------+ | | After Boot | After Hackbench| | Normal | 469336 | 725960 | | With Patch | 488032 | 726416 | | Memory increase | ~4% | ~0.06% | +-----------------+----------------+----------------+ hackbench-process-sockets +-------+-----+----------+----------+-----------+ | | Normal |With Patch| | +-------+-----+----------+----------+-----------+ | Amean | 1 | 1.2887 | 1.2143 | ( 5.77%) | | Amean | 4 | 1.5633 | 1.5993 | ( -2.30%) | | Amean | 7 | 2.3993 | 2.3813 | ( 0.75%) | | Amean | 12 | 3.9543 | 3.9637 | ( -0.24%) | | Amean | 21 | 6.9723 | 6.9290 | ( 0.62%) | | Amean | 30 | 10.1407 | 10.1067 | ( 0.34%) | | Amean | 48 | 16.6730 | 16.6697 | ( 0.02%) | | Amean | 79 | 28.6743 | 28.8970 | ( -0.78%) | | Amean | 110 | 39.0990 | 39.1857 | ( -0.22%) | | Amean | 141 | 51.2667 | 51.2003 | ( 0.13%) | | Amean | 172 | 62.0797 | 62.3190 | ( -0.39%) | | Amean | 203 | 73.5273 | 74.3567 | ( -1.13%) | | Amean | 234 | 84.7130 | 85.7940 | ( -1.28%) | | Amean | 265 | 97.0863 | 96.5810 | ( 0.52%) | | Amean | 296 | 108.4597 | 108.2987 | ( 0.15%) | +-------+-----+----------+----------+-----------+ 2) On 160 CPUs with 64K Page size +-----------------+----------------+----------------+ | Total wastage in slub memory | +-----------------+----------------+----------------+ | | After Boot |After Hackbench | | Normal | 729 Kb | 1597 Kb | | With Patch | 512 Kb | 1066 Kb | | Wastage reduce | ~30% | ~33% | +-----------------+----------------+----------------+ +-----------------+----------------+----------------+ | Total slub memory | +-----------------+----------------+----------------+ | | After Boot | After Hackbench| | Normal | 1612608 | 2667200 | | With Patch | 2147456 | 3500096 | | Memory increase | ~33% | ~31% | +-----------------+----------------+----------------+ hackbench-process-sockets +-------+-----+----------+----------+-----------+ | Amean | 1 | 1.2667 | 1.2053 | ( 4.84%) | | Amean | 4 | 1.5997 | 1.6453 | ( -2.85%) | | Amean | 7 | 2.3797 | 2.4017 | ( -0.92%) | | Amean | 12 | 3.9763 | 3.9987 | ( -0.56%) | | Amean | 21 | 6.9760 | 6.9917 | ( -0.22%) | | Amean | 30 | 10.2150 | 10.2093 | ( 0.06%) | | Amean | 48 | 16.8080 | 16.7707 | ( 0.22%) | | Amean | 79 | 28.2237 | 28.1583 | ( 0.23%) | | Amean | 110 | 39.7710 | 39.8420 | ( -0.18%) | | Amean | 141 | 51.3563 | 51.9233 | ( -1.10%) | | Amean | 172 | 63.4027 | 63.7463 | ( -0.54%) | | Amean | 203 | 74.4970 | 74.9327 | ( -0.58%) | | Amean | 234 | 86.1483 | 85.9420 | ( 0.24%) | | Amean | 265 | 97.5137 | 97.6100 | ( -0.10%) | | Amean | 296 | 109.2327 | 110.2417 | ( -0.92%) | +-------+-----+----------+----------+-----------+ 3) On 16 CPUs with 4K Page size +-----------------+----------------+------------------+ | Total wastage in slub memory | +-----------------+----------------+------------------+ | | After Boot | After Hackbench | | Normal | 666 Kb | 902 Kb | | With Patch | 533 Kb | 694 Kb | | Wastage reduce | ~20% | ~23% | +-----------------+----------------+------------------+ +-----------------+----------------+----------------+ | Total slub memory | +-----------------+----------------+----------------+ | | After Boot | After Hackbench| | Normal | 82360 | 122532 | | With Patch | 87372 | 129180 | | Memory increase | ~6% | ~5% | +-----------------+----------------+----------------+ hackbench-process-sockets +-------+----+---------+---------+-----------+ | Amean | 1 | 1.4983 | 1.4867 | ( 0.78%) | | Amean | 4 | 5.6613 | 5.6793 | ( -0.32%) | | Amean | 7 | 9.9813 | 9.9873 | ( -0.06%) | | Amean | 12 | 17.6963 | 17.8527 | ( -0.88%) | | Amean | 21 | 31.2017 | 31.2060 | ( -0.01%) | | Amean | 30 | 44.0297 | 44.1750 | ( -0.33%) | | Amean | 48 | 70.2073 | 69.6210 | ( 0.84%) | | Amean | 64 | 92.3257 | 93.7410 | ( -1.53%) | +-------+----+---------+---------+-----------+ 4) On 16 CPUs with 64K Page size +----------------+----------------+----------------+ | Total wastage in slub memory | +----------------+----------------+----------------+ | | After Boot | After Hackbench| | Normal | 239 Kb | 484 Kb | | With Patch | 135 Kb | 234 Kb | | Wastage reduce | ~43% | ~51% | +----------------+----------------+----------------+ +-----------------+----------------+----------------+ | Total slub memory | +-----------------+----------------+----------------+ | | After Boot | After Hackbench| | Normal | 227136 | 328110 | | With Patch | 284352 | 451391 | | Memory increase | ~25% | ~37% | +-----------------+----------------+----------------+ hackbench-process-sockets +-------+----+---------+---------+-----------+ | Amean | 1 | 1.3597 | 1.3583 | ( 0.10%) | | Amean | 4 | 5.2633 | 5.2503 | ( 0.25%) | | Amean | 7 | 9.2700 | 9.1710 | ( 1.07%) | | Amean | 12 | 16.3730 | 16.3103 | ( 0.38%) | | Amean | 21 | 28.7140 | 28.7510 | ( -0.13%) | | Amean | 30 | 40.3987 | 40.4940 | ( -0.24%) | | Amean | 48 | 63.8477 | 63.9457 | ( -0.15%) | | Amean | 64 | 86.4917 | 85.3810 | ( 1.28%) | +-------+----+---------+---------+-----------+ Signed-off-by: Jay Patel --- mm/slub.c | 43 +++++++++++++++++-------------------------- 1 file changed, 17 insertions(+), 26 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index c87628cd8a9a..e0b465173ed3 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -4087,11 +4087,10 @@ static unsigned int slub_min_objects; * the smallest order which will fit the object. */ static inline unsigned int calc_slab_order(unsigned int size, - unsigned int min_objects, unsigned int max_order, - unsigned int fract_leftover) + unsigned int min_objects, unsigned int max_order) { unsigned int min_order = slub_min_order; - unsigned int order; + unsigned int order, min_wastage = size, min_wastage_order = slub_max_order+1; if (order_objects(min_order, size) > MAX_OBJS_PER_PAGE) return get_order(size * MAX_OBJS_PER_PAGE) - 1; @@ -4104,11 +4103,17 @@ static inline unsigned int calc_slab_order(unsigned int size, rem = slab_size % size; - if (rem <= slab_size / fract_leftover) - break; + if (rem < min_wastage) { + min_wastage = rem; + min_wastage_order = order; + } } - return order; + if (min_wastage_order <= slub_max_order) + return min_wastage_order; + else + return order; + } static inline int calculate_order(unsigned int size) @@ -4145,32 +4150,18 @@ static inline int calculate_order(unsigned int size) max_objects = order_objects(slub_max_order, size); min_objects = min(min_objects, max_objects); - while (min_objects > 1) { - unsigned int fraction; - - fraction = 16; - while (fraction >= 4) { - order = calc_slab_order(size, min_objects, - slub_max_order, fraction); - if (order <= slub_max_order) - return order; - fraction /= 2; - } + while (min_objects >= 1) { + order = calc_slab_order(size, min_objects, + slub_max_order); + if (order <= slub_max_order) + return order; min_objects--; } - /* - * We were unable to place multiple objects in a slab. Now - * lets see if we can place a single object there. - */ - order = calc_slab_order(size, 1, slub_max_order, 1); - if (order <= slub_max_order) - return order; - /* * Doh this slab cannot be placed using slub_max_order. */ - order = calc_slab_order(size, 1, MAX_ORDER, 1); + order = calc_slab_order(size, 1, MAX_ORDER); if (order <= MAX_ORDER) return order; return -ENOSYS;