From patchwork Mon Jun 19 14:35:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexey Romanov X-Patchwork-Id: 13284591 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 979D7EB64DA for ; Mon, 19 Jun 2023 14:35:26 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52DDA8D0003; Mon, 19 Jun 2023 10:35:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DDE58D0001; Mon, 19 Jun 2023 10:35:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A7328D0003; Mon, 19 Jun 2023 10:35:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 2B8CF8D0001 for ; Mon, 19 Jun 2023 10:35:25 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D24FDC067C for ; Mon, 19 Jun 2023 14:35:24 +0000 (UTC) X-FDA: 80919745368.15.3C31996 Received: from mx.sberdevices.ru (mx.sberdevices.ru [45.89.227.171]) by imf05.hostedemail.com (Postfix) with ESMTP id 56BBD100015 for ; Mon, 19 Jun 2023 14:35:22 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=sberdevices.ru header.s=mail header.b=Kx0Y2IWD; dmarc=pass (policy=quarantine) header.from=sberdevices.ru; spf=pass (imf05.hostedemail.com: domain of AVRomanov@sberdevices.ru designates 45.89.227.171 as permitted sender) smtp.mailfrom=AVRomanov@sberdevices.ru ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687185322; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=yyJFbEiRHCma+uvofoZbZvm1fPuV2gLaYZpyW0jnz8Q=; b=pA9PGbFVTMRPWveXvaVTqdo7TdyDKJxPwU85wQUQtrjMu2tQMLMSKfZixN1/Uiyycevxc5 Od+7OLpDLHJDN/sXKJa7+mVMmJt7216TYewHnfkwtJjlGqCIjHrU1yxwlO1vrQgg8nBA5Z PsLEyHOYaTAILeQu5VtdoiALbEDsFqE= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=sberdevices.ru header.s=mail header.b=Kx0Y2IWD; dmarc=pass (policy=quarantine) header.from=sberdevices.ru; spf=pass (imf05.hostedemail.com: domain of AVRomanov@sberdevices.ru designates 45.89.227.171 as permitted sender) smtp.mailfrom=AVRomanov@sberdevices.ru ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687185322; a=rsa-sha256; cv=none; b=8XamvFeitlammuYRLD1AquWBRXEXa71kgb2PUPFX+bPuE/IXWv97MSUuC5K3f63qhaS0xo czN2tgVBRre3mjkdEAY2FOwKAiRyBDRHmWIMH3CxeNPpICr6CgOJWFWYhY15xDNWTxUcdq AjvjiqjFYgrWgENlFv3MyRHmNZAn5zM= Received: from s-lin-edge02.sberdevices.ru (localhost [127.0.0.1]) by mx.sberdevices.ru (Postfix) with ESMTP id 3F04F5FD2A; Mon, 19 Jun 2023 17:35:20 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sberdevices.ru; s=mail; t=1687185320; bh=yyJFbEiRHCma+uvofoZbZvm1fPuV2gLaYZpyW0jnz8Q=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type; b=Kx0Y2IWD//KCWUqHs31kllelCYkzawNTLoFolOJ7WtBIBaE8DqUSOB/P56564TZCF 8wHNaQcpS0AVi1fqBHPqXLECjUBexiVLjDkZ5UdwkYp1XGCgYp/0B0c3mrALy9vibg rFrlXFmHizU4e85qqg4LuyznALDrWT5bILn/SP5GOjJlfezNjEJqwTzTgl6KrtexYm tK0rqzSGYboERdOMW0HqAtqyogAl/3fYtBrstj2z++mPdW9xVfZjCwFl5ywnksS9Jc z0AJeymCexXYTe04JqBI0f9SoFpOE726nKIfOwYtI5nXkrleuAxli5DTU//5vUt5S+ wOPWWE+e+QW+A== Received: from p-i-exch-sc-m01.sberdevices.ru (p-i-exch-sc-m01.sberdevices.ru [172.16.192.107]) by mx.sberdevices.ru (Postfix) with ESMTP; Mon, 19 Jun 2023 17:35:20 +0300 (MSK) From: Alexey Romanov To: , , CC: , , , Alexey Romanov Subject: [PATCH v1 1/2] zsmalloc: add allocated objects counter for subpage Date: Mon, 19 Jun 2023 17:35:05 +0300 Message-ID: <20230619143506.45253-2-avromanov@sberdevices.ru> X-Mailer: git-send-email 2.33.0 In-Reply-To: <20230619143506.45253-1-avromanov@sberdevices.ru> References: <20230619143506.45253-1-avromanov@sberdevices.ru> MIME-Version: 1.0 X-Originating-IP: [172.16.1.6] X-ClientProxiedBy: S-MS-EXCH02.sberdevices.ru (172.16.1.5) To p-i-exch-sc-m01.sberdevices.ru (172.16.192.107) X-KSMG-Rule-ID: 4 X-KSMG-Message-Action: clean X-KSMG-AntiSpam-Status: not scanned, disabled by settings X-KSMG-AntiSpam-Interceptor-Info: not scanned X-KSMG-AntiPhishing: not scanned, disabled by settings X-KSMG-AntiVirus: Kaspersky Secure Mail Gateway, version 1.1.2.30, bases: 2023/06/19 10:56:00 #21523989 X-KSMG-AntiVirus-Status: Clean, skipped X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: 56BBD100015 X-Stat-Signature: jpj8jy8enymqoe1yjgiqwjsnhfc7u7ti X-HE-Tag: 1687185321-265179 X-HE-Meta: U2FsdGVkX1/t0fFGQ4mh9ecNHLBn8uGe4hrjv3lmbXJuHqdq0dUBUPq4bRGoN0xKWUWTH9TYByc9tgujH9hIpmePKhjviQVyIN31T5GdrYGzUer408bQo9C3lYwMWVAzhDRZ0mhsU/cOwYg2yRZqKMhS3n3rogMnsBAtU+dCdL6DjjkO+R4E+D/YJJdGY/sjysMRQK09TpuRK7ppH8aYyH1RYGgdmtF5omheTEntVJjDSwCwuEGbdqolH4Smfi5K7bZAb7T2sW6fsfiOGqYSENDQDItTQdcA4xs9EGmBvhOakZLSV6kjULliBPLPfwDSmT2lI5tJk0X+61SSlWMPnmCwJND2+MwszlA2cmUt26zoZMrrkJD8UHzQZW7W21ToyORskWRbi6grWQwnLcs169MFdmJ3gCpNxgczK+BUz2iY1IRtuNObhmYcs2e2hctX7AVRs5XPow+Jtb1N7OxFW78twu3DTb4RmItnvLV4oTy30pemgez5I3yjDnV8Fa2CKcHbNWxmFZvXCTRSjjqYZ9BP9gRLAl+9dmTMVBRUG5wWo/4xpn/2IZmaAVlHSE8D15V4HE4MTaxaX6dGgvjxqXAZSMfyzoRs8ScJgGf9BmQRRG7+OsvAt3c37UxPfLFof7EslOPZuGBjYXyjdFwYFQyBvPlRuUXvt7+ofRQ3ckg0byLq8rlEtvh/LYw/zYMbMQCoFNlwhtwCtvJKNdaSenqdhcMVhaveQQIxhloWo9GLCNqe3J2VwT5V3R6YRsuqXZneaWowdO01JuKSwhvNmfEpZn7QjTNC7ZJnQ397C6zNXv/VvicligwClTq4yyefWzZG0kcRbLFFzcPVglLCCblwGWC6qQfYWV84AhF0GC/adGmMehEGZ5Uz6a1Yafk+47MTFWZ+Vc8PGJKnvtREZ8KybOYtT9aiipqgFmOBYDb2dDO9I5dNcv4FRCNPRrAY9bx7CdwlSujVCClryDz KQBD0b81 829G8cl3RNmCkkSTCwroYGUfbxIp48X37vALpwt822fRqux14c9bgZKcmSmsFHeEvOo9TtDhs0cy2a72MXlTnepAvH053Kd3U4rC8C/2c5tw0D8RZyQhpghLdTzykRGvRRFFB+s5zBCU61n46LqWz9PGllK7EYlrewdZpZ+wyeyIjqhdxkfpmfmsIYat6F7p8EJ2/pubIm6v1NPEcHEXGKE58eWh+os9x+sRR8BQniigQK3ahWI5MH6ZBAOJEXhDuxILaftVgjf5kXP0qjZP73f4KZt0PwqMNJHuVzD1kT+7QQsYZuiThi8oqlLP2ilLyoCmrsZZrGCjftjAVE877FZLSXR4rf0F7DFBI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: We use a variable of type unsigned int to store the offset of the first object at the subpage In turn, the offset cannot exceed the size of PAGE_SIZE, which is usually 4096. Thus, 12 bits are enough to store the offset. We can use the remaining bytes to store, for example, the count of allocated objects on a subpage. If the page size is 4Kb, then no more than 128 (4096 / 32) objects can be allocated on the subpage, which means that one byte is enough to store the counter of allocated objects. This patch adds support for counting the number of allocated objects on a subpage in the first byte of the page_type field. The offset of the first object is now stored in the remaining bytes of this field. The sum of allocated counter for all subpages is zspage->inuse. We only count objects that have been tagged (I'm talking about OBJ_ALLOCATED_TAG) on a subpage. So, for example, in the situation: subpage 1 subpage 2 [obj1_s - obj1_e, obj2_s - ] -> [obj2_e, obj3_s - obj3_e, free space] Allocated counter for subpage 1 will be 2, and 1 for subpage 2. Signed-off-by: Alexey Romanov --- mm/zsmalloc.c | 38 +++++++++++++++++++++++++++++++++++--- 1 file changed, 35 insertions(+), 3 deletions(-) diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index c0d433541636..dd6e2c3429e0 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -20,7 +20,10 @@ * page->index: links together all component pages of a zspage * For the huge page, this is always 0, so we use this field * to store handle. - * page->page_type: first object offset in a subpage of zspage + * page->page_type: + * First byte: count of allocated objects (OBJ_ALLOCATED_TAG) + * in a subpage of zspage. + * Other bytes: first object offset in a subpage of zspage. * * Usage of struct page flags: * PG_private: identifies the first component page @@ -126,6 +129,9 @@ #define OBJ_INDEX_BITS (BITS_PER_LONG - _PFN_BITS - OBJ_TAG_BITS) #define OBJ_INDEX_MASK ((_AC(1, UL) << OBJ_INDEX_BITS) - 1) +#define OBJ_ALLOCATED_BITS (sizeof(u8) * BITS_PER_BYTE) +#define OBJ_ALLOCATED_MASK ((1UL << OBJ_ALLOCATED_BITS) - 1) + #define HUGE_BITS 1 #define FULLNESS_BITS 4 #define CLASS_BITS 8 @@ -520,14 +526,37 @@ static inline struct page *get_first_page(struct zspage *zspage) return first_page; } +static inline u8 get_obj_allocated(struct page *page) +{ + return page->page_type & OBJ_ALLOCATED_MASK; +} + +static inline void set_obj_allocated(struct page *page, u8 value) +{ + page->page_type = (page->page_type & ~OBJ_ALLOCATED_MASK) | value; +} + +static inline void mod_obj_allocated(struct page *page, s8 value) +{ + u8 inuse = get_obj_allocated(page); + /* + * Overflow is not possible: + * 1. Maximum number of objects allocated on a subpage is 128. + * 2. We use this function only with value = 1 or -1. + */ + inuse += value; + set_obj_allocated(page, inuse); +} + static inline unsigned int get_first_obj_offset(struct page *page) { - return page->page_type; + return page->page_type >> OBJ_ALLOCATED_BITS; } static inline void set_first_obj_offset(struct page *page, unsigned int offset) { - page->page_type = offset; + page->page_type = (page->page_type & OBJ_ALLOCATED_MASK) | + (offset << OBJ_ALLOCATED_BITS); } static inline unsigned int get_freeobj(struct zspage *zspage) @@ -1126,6 +1155,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, } inc_zone_page_state(page, NR_ZSPAGES); + set_obj_allocated(page, 0); pages[i] = page; } @@ -1456,6 +1486,7 @@ static unsigned long obj_malloc(struct zs_pool *pool, kunmap_atomic(vaddr); mod_zspage_inuse(zspage, 1); + mod_obj_allocated(m_page, 1); obj = location_to_obj(m_page, obj); @@ -1576,6 +1607,7 @@ static void obj_free(int class_size, unsigned long obj, unsigned long *handle) kunmap_atomic(vaddr); mod_zspage_inuse(zspage, -1); + mod_obj_allocated(f_page, -1); } void zs_free(struct zs_pool *pool, unsigned long handle)