From patchwork Tue Oct 25 20:52:46 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13019897 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 63FE0C38A2D for ; Tue, 25 Oct 2022 20:54:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=osD7OgH672Ig2zBGyjrCkUTdY2rsT25jraPKZCKuL8M=; b=5DcyRy2YjarDWk WUBDOGJYgL0Hgl4GIfsq6TJ9oxlwJNMW8HeUljVJDzRqGexkkwm0blTgA+4Yp2UmAnl3wYi73/5kX t4AsKUNnhyyZVjioX9qXGn4nT3SLoHUoDs3JlPdioj7gVFmw8NBDKn0U7qh6Nk736AGJgo6J3S0IK PhJnV5z6gncyHKz+BKYoaAtwKp6okq4sP/rRKRJMgymMupuMn+QF6Agfp6y4J6BpAX+bcHj6Es2b6 D+KrLNpDVueU8/XERC+wk9oJIFUJClPv3y8LgKVfCuhgrz6gf9AmLCx0lr7zLb6ry6hjZnmBYp7Ni a3qIn/T0w6bPDHSD9irw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1onQuy-0078J0-4v; Tue, 25 Oct 2022 20:53:00 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1onQuu-0078HE-Dj for linux-arm-kernel@lists.infradead.org; Tue, 25 Oct 2022 20:52:58 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id B26CB61B7F; Tue, 25 Oct 2022 20:52:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A0860C4347C; Tue, 25 Oct 2022 20:52:52 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann Cc: Will Deacon , Marc Zyngier , Greg Kroah-Hartman , Andrew Morton , Herbert Xu , Ard Biesheuvel , Christoph Hellwig , Isaac Manjarres , Saravana Kannan , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 1/2] mm: slab: Introduce __GFP_PACKED for smaller kmalloc() alignments Date: Tue, 25 Oct 2022 21:52:46 +0100 Message-Id: <20221025205247.3264568-2-catalin.marinas@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221025205247.3264568-1-catalin.marinas@arm.com> References: <20221025205247.3264568-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221025_135256_564517_5C179910 X-CRM114-Status: GOOD ( 21.63 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org By default kmalloc() returns objects aligned to ARCH_KMALLOC_MINALIGN. This can be somewhat large on architectures defining ARCH_DMA_MINALIGN (e.g. 128 on arm64) and significant memory is wasted through small kmalloc() allocations. Reduce the minimum alignment for kmalloc() to the default KMALLOC_MIN_SIZE (8 for slub, 32 for slab) but align the requested size to the bigger ARCH_KMALLOC_MINALIGN unless a newly added __GFP_PACKED flag is passed. With this gfp flag, the alignment is reduced to KMALLOC_PACKED_ALIGN, at least sizeof(unsigned long long). There's no slob support. Signed-off-by: Catalin Marinas --- include/linux/gfp_types.h | 10 ++++++++-- include/linux/slab.h | 22 ++++++++++++++++++---- mm/slab_common.c | 3 ++- 3 files changed, 28 insertions(+), 7 deletions(-) diff --git a/include/linux/gfp_types.h b/include/linux/gfp_types.h index d88c46ca82e1..305cb8cb6f8b 100644 --- a/include/linux/gfp_types.h +++ b/include/linux/gfp_types.h @@ -55,8 +55,9 @@ typedef unsigned int __bitwise gfp_t; #define ___GFP_SKIP_KASAN_UNPOISON 0 #define ___GFP_SKIP_KASAN_POISON 0 #endif +#define ___GFP_PACKED 0x8000000u #ifdef CONFIG_LOCKDEP -#define ___GFP_NOLOCKDEP 0x8000000u +#define ___GFP_NOLOCKDEP 0x10000000u #else #define ___GFP_NOLOCKDEP 0 #endif @@ -243,6 +244,10 @@ typedef unsigned int __bitwise gfp_t; * * %__GFP_SKIP_KASAN_POISON makes KASAN skip poisoning on page deallocation. * Typically, used for userspace pages. Only effective in HW_TAGS mode. + * + * %__GFP_PACKED returns a pointer aligned to the possibly smaller + * KMALLOC_PACKED_ALIGN rather than ARCH_KMALLOC_MINALIGN. Useful for small + * object allocation on architectures that define large ARCH_DMA_MINALIGN. */ #define __GFP_NOWARN ((__force gfp_t)___GFP_NOWARN) #define __GFP_COMP ((__force gfp_t)___GFP_COMP) @@ -251,12 +256,13 @@ typedef unsigned int __bitwise gfp_t; #define __GFP_SKIP_ZERO ((__force gfp_t)___GFP_SKIP_ZERO) #define __GFP_SKIP_KASAN_UNPOISON ((__force gfp_t)___GFP_SKIP_KASAN_UNPOISON) #define __GFP_SKIP_KASAN_POISON ((__force gfp_t)___GFP_SKIP_KASAN_POISON) +#define __GFP_PACKED ((__force gfp_t)___GFP_PACKED) /* Disable lockdep for GFP context tracking */ #define __GFP_NOLOCKDEP ((__force gfp_t)___GFP_NOLOCKDEP) /* Room for N __GFP_FOO bits */ -#define __GFP_BITS_SHIFT (27 + IS_ENABLED(CONFIG_LOCKDEP)) +#define __GFP_BITS_SHIFT (28 + IS_ENABLED(CONFIG_LOCKDEP)) #define __GFP_BITS_MASK ((__force gfp_t)((1 << __GFP_BITS_SHIFT) - 1)) /** diff --git a/include/linux/slab.h b/include/linux/slab.h index 90877fcde70b..0f59585b5fbf 100644 --- a/include/linux/slab.h +++ b/include/linux/slab.h @@ -223,8 +223,6 @@ void kmem_dump_obj(void *object); */ #if defined(ARCH_DMA_MINALIGN) && ARCH_DMA_MINALIGN > 8 #define ARCH_KMALLOC_MINALIGN ARCH_DMA_MINALIGN -#define KMALLOC_MIN_SIZE ARCH_DMA_MINALIGN -#define KMALLOC_SHIFT_LOW ilog2(ARCH_DMA_MINALIGN) #else #define ARCH_KMALLOC_MINALIGN __alignof__(unsigned long long) #endif @@ -310,6 +308,11 @@ static inline unsigned int arch_slab_minalign(void) #define KMALLOC_MIN_SIZE (1 << KMALLOC_SHIFT_LOW) #endif +/* + * This alignment should be at least sizeof(unsigned long long). + */ +#define KMALLOC_PACKED_ALIGN (KMALLOC_MIN_SIZE) + /* * This restriction comes from byte sized index implementation. * Page size is normally 2^12 bytes and, in this case, if we want to use @@ -382,6 +385,17 @@ static __always_inline enum kmalloc_cache_type kmalloc_type(gfp_t flags) return KMALLOC_CGROUP; } +/* + * Align the size to ARCH_KMALLOC_MINALIGN unless __GFP_PACKED is passed. + */ +static __always_inline size_t kmalloc_size_align(size_t size, gfp_t flags) +{ + if (ARCH_KMALLOC_MINALIGN > KMALLOC_PACKED_ALIGN && + !(flags & __GFP_PACKED)) + size = ALIGN(size, ARCH_KMALLOC_MINALIGN); + return size; +} + /* * Figure out which kmalloc slab an allocation of a certain size * belongs to. @@ -568,7 +582,7 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) if (size > KMALLOC_MAX_CACHE_SIZE) return kmalloc_large(size, flags); #ifndef CONFIG_SLOB - index = kmalloc_index(size); + index = kmalloc_index(kmalloc_size_align(size, flags)); if (!index) return ZERO_SIZE_PTR; @@ -590,7 +604,7 @@ static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t fla if (size > KMALLOC_MAX_CACHE_SIZE) return kmalloc_large_node(size, flags, node); - index = kmalloc_index(size); + index = kmalloc_index(kmalloc_size_align(size, flags)); if (!index) return ZERO_SIZE_PTR; diff --git a/mm/slab_common.c b/mm/slab_common.c index 33b1886b06eb..0e4ea396cd4f 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -627,7 +627,7 @@ void __init create_boot_cache(struct kmem_cache *s, const char *name, unsigned int useroffset, unsigned int usersize) { int err; - unsigned int align = ARCH_KMALLOC_MINALIGN; + unsigned int align = KMALLOC_PACKED_ALIGN; s->name = name; s->size = s->object_size = size; @@ -720,6 +720,7 @@ struct kmem_cache *kmalloc_slab(size_t size, gfp_t flags) { unsigned int index; + size = kmalloc_size_align(size, flags); if (size <= 192) { if (!size) return ZERO_SIZE_PTR; From patchwork Tue Oct 25 20:52:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13019899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4D358C38A2D for ; Tue, 25 Oct 2022 20:54:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kxDVZMQJDT/XiyFdXGeDOrixsv9+52iIfQlGUq8Xlkc=; b=f+edoJHVytOcEo TEQBHf2YnO78rSJvPqtAE/jbZtttD0wlND019WEd+nq5x81v+G8TxDlJBMvGrZOHxVyId0FZbKlwh n33g4rg4+HzEdma4QSY9Zrnyd+4wRzZgQHzlou5zjQGpJKIcp8oyiqkdb2+CViJeB9z1A5Ha5XRM3 SvJ3nRwg2X2cQmgDP5HIZCu+ugqOTT7E6QMVVOo4QFCcJWJm7DtNmlocAEypl3ZRHGpjRfLX+HI3D ZOHW7Ggkk163rokHJTic7PKmSA2HN4i33fFdfmFikBvzsnOAH5V3OLYFHPmyvOMKhbbJ6ksw+MC8a qcknR2o3saZODzWxnwyQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1onQvG-0078OP-DD; Tue, 25 Oct 2022 20:53:18 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1onQuy-0078JC-QH for linux-arm-kernel@lists.infradead.org; Tue, 25 Oct 2022 20:53:04 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 56784B81BE1; Tue, 25 Oct 2022 20:52:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 87836C43470; Tue, 25 Oct 2022 20:52:55 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann Cc: Will Deacon , Marc Zyngier , Greg Kroah-Hartman , Andrew Morton , Herbert Xu , Ard Biesheuvel , Christoph Hellwig , Isaac Manjarres , Saravana Kannan , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 2/2] treewide: Add the __GFP_PACKED flag to several non-DMA kmalloc() allocations Date: Tue, 25 Oct 2022 21:52:47 +0100 Message-Id: <20221025205247.3264568-3-catalin.marinas@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221025205247.3264568-1-catalin.marinas@arm.com> References: <20221025205247.3264568-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221025_135301_223907_8D256362 X-CRM114-Status: GOOD ( 26.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Using the ftrace kmalloc histogram shows several hot spots for small memory allocations that would benefit from the smaller KMALLOC_PACKED_ALIGN alignment. To set up the ftrace events for small kmalloc() allocations (only showing those not already having the __GFP_PACKED flag): echo 'hist:key=call_site.sym-offset:vals=bytes_req,bytes_alloc:sort=bytes_alloc.descending if bytes_req<=96 && !(gfp_flags&0x8000000)' \ > /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger To enable tracing of boot-time allocations, use the following bootconfig: ftrace.event.kmem.kmalloc.hist { keys = call_site.sym-offset values = bytes_req, bytes_alloc sort = bytes_alloc.descending filter = "bytes_req <= 96 && !(gfp_flags & 0x8000000)" } To view the allocation hot spots: head /sys/kernel/debug/tracing/events/kmem/kmalloc/hist Signed-off-by: Catalin Marinas --- drivers/usb/core/message.c | 3 ++- fs/binfmt_elf.c | 6 ++++-- fs/dcache.c | 3 ++- fs/ext4/dir.c | 4 ++-- fs/ext4/extents.c | 4 ++-- fs/file.c | 2 +- fs/kernfs/file.c | 8 ++++---- fs/nfs/dir.c | 7 ++++--- fs/nfs/inode.c | 2 +- fs/nfs/nfs4state.c | 2 +- fs/nfs/write.c | 3 ++- fs/notify/inotify/inotify_fsnotify.c | 3 ++- fs/proc/self.c | 2 +- fs/seq_file.c | 5 +++-- include/acpi/platform/aclinuxex.h | 6 ++++-- kernel/trace/ftrace.c | 2 +- kernel/trace/tracing_map.c | 2 +- lib/kasprintf.c | 2 +- lib/kobject.c | 4 ++-- lib/mpi/mpiutil.c | 2 +- mm/list_lru.c | 6 ++++-- mm/memcontrol.c | 4 ++-- mm/util.c | 6 +++--- mm/vmalloc.c | 6 ++++-- net/sunrpc/auth_unix.c | 2 +- net/sunrpc/xdr.c | 2 +- security/apparmor/lsm.c | 2 +- security/security.c | 4 ++-- security/tomoyo/realpath.c | 2 +- 29 files changed, 60 insertions(+), 46 deletions(-) diff --git a/drivers/usb/core/message.c b/drivers/usb/core/message.c index 4d59d927ae3e..bff8901dc426 100644 --- a/drivers/usb/core/message.c +++ b/drivers/usb/core/message.c @@ -525,7 +525,8 @@ int usb_sg_init(struct usb_sg_request *io, struct usb_device *dev, } /* initialize all the urbs we'll use */ - io->urbs = kmalloc_array(io->entries, sizeof(*io->urbs), mem_flags); + io->urbs = kmalloc_array(io->entries, sizeof(*io->urbs), + mem_flags | __GFP_PACKED); if (!io->urbs) goto nomem; diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index 63c7ebb0da89..e5ad1a3244fb 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -883,7 +883,8 @@ static int load_elf_binary(struct linux_binprm *bprm) goto out_free_ph; retval = -ENOMEM; - elf_interpreter = kmalloc(elf_ppnt->p_filesz, GFP_KERNEL); + elf_interpreter = kmalloc(elf_ppnt->p_filesz, + GFP_KERNEL | __GFP_PACKED); if (!elf_interpreter) goto out_free_ph; @@ -908,7 +909,8 @@ static int load_elf_binary(struct linux_binprm *bprm) */ would_dump(bprm, interpreter); - interp_elf_ex = kmalloc(sizeof(*interp_elf_ex), GFP_KERNEL); + interp_elf_ex = kmalloc(sizeof(*interp_elf_ex), + GFP_KERNEL | __GFP_PACKED); if (!interp_elf_ex) { retval = -ENOMEM; goto out_free_ph; diff --git a/fs/dcache.c b/fs/dcache.c index 52e6d5fdab6b..9c66679365ca 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -1785,7 +1785,8 @@ static struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) size_t size = offsetof(struct external_name, name[1]); struct external_name *p = kmalloc(size + name->len, GFP_KERNEL_ACCOUNT | - __GFP_RECLAIMABLE); + __GFP_RECLAIMABLE | + __GFP_PACKED); if (!p) { kmem_cache_free(dentry_cache, dentry); return NULL; diff --git a/fs/ext4/dir.c b/fs/ext4/dir.c index 3985f8c33f95..fc8415ff8880 100644 --- a/fs/ext4/dir.c +++ b/fs/ext4/dir.c @@ -435,7 +435,7 @@ static struct dir_private_info *ext4_htree_create_dir_info(struct file *filp, { struct dir_private_info *p; - p = kzalloc(sizeof(*p), GFP_KERNEL); + p = kzalloc(sizeof(*p), GFP_KERNEL | __GFP_PACKED); if (!p) return NULL; p->curr_hash = pos2maj_hash(filp, pos); @@ -471,7 +471,7 @@ int ext4_htree_store_dirent(struct file *dir_file, __u32 hash, /* Create and allocate the fname structure */ len = sizeof(struct fname) + ent_name->len + 1; - new_fn = kzalloc(len, GFP_KERNEL); + new_fn = kzalloc(len, GFP_KERNEL | __GFP_PACKED); if (!new_fn) return -ENOMEM; new_fn->hash = hash; diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c index f1956288307f..d8ffb65ac318 100644 --- a/fs/ext4/extents.c +++ b/fs/ext4/extents.c @@ -912,7 +912,7 @@ ext4_find_extent(struct inode *inode, ext4_lblk_t block, if (!path) { /* account possible depth increase */ path = kcalloc(depth + 2, sizeof(struct ext4_ext_path), - gfp_flags); + gfp_flags | __GFP_PACKED); if (unlikely(!path)) return ERR_PTR(-ENOMEM); path[0].p_maxdepth = depth + 1; @@ -2937,7 +2937,7 @@ int ext4_ext_remove_space(struct inode *inode, ext4_lblk_t start, le16_to_cpu(path[k].p_hdr->eh_entries)+1; } else { path = kcalloc(depth + 1, sizeof(struct ext4_ext_path), - GFP_NOFS | __GFP_NOFAIL); + GFP_NOFS | __GFP_NOFAIL | __GFP_PACKED); if (path == NULL) { ext4_journal_stop(handle); return -ENOMEM; diff --git a/fs/file.c b/fs/file.c index 5f9c802a5d8d..3f732736eaf3 100644 --- a/fs/file.c +++ b/fs/file.c @@ -129,7 +129,7 @@ static struct fdtable * alloc_fdtable(unsigned int nr) if (unlikely(nr > sysctl_nr_open)) nr = ((sysctl_nr_open - 1) | (BITS_PER_LONG - 1)) + 1; - fdt = kmalloc(sizeof(struct fdtable), GFP_KERNEL_ACCOUNT); + fdt = kmalloc(sizeof(struct fdtable), GFP_KERNEL_ACCOUNT | __GFP_PACKED); if (!fdt) goto out; fdt->max_fds = nr; diff --git a/fs/kernfs/file.c b/fs/kernfs/file.c index 9ab6c92e02da..882980965c55 100644 --- a/fs/kernfs/file.c +++ b/fs/kernfs/file.c @@ -304,7 +304,7 @@ static ssize_t kernfs_fop_write_iter(struct kiocb *iocb, struct iov_iter *iter) if (buf) mutex_lock(&of->prealloc_mutex); else - buf = kmalloc(len + 1, GFP_KERNEL); + buf = kmalloc(len + 1, GFP_KERNEL | __GFP_PACKED); if (!buf) return -ENOMEM; @@ -565,7 +565,7 @@ static int kernfs_get_open_node(struct kernfs_node *kn, if (!on) { /* not there, initialize a new one */ - on = kzalloc(sizeof(*on), GFP_KERNEL); + on = kzalloc(sizeof(*on), GFP_KERNEL | __GFP_PACKED); if (!on) { mutex_unlock(mutex); return -ENOMEM; @@ -663,7 +663,7 @@ static int kernfs_fop_open(struct inode *inode, struct file *file) /* allocate a kernfs_open_file for the file */ error = -ENOMEM; - of = kzalloc(sizeof(struct kernfs_open_file), GFP_KERNEL); + of = kzalloc(sizeof(struct kernfs_open_file), GFP_KERNEL | __GFP_PACKED); if (!of) goto err_out; @@ -706,7 +706,7 @@ static int kernfs_fop_open(struct inode *inode, struct file *file) goto err_free; if (ops->prealloc) { int len = of->atomic_write_len ?: PAGE_SIZE; - of->prealloc_buf = kmalloc(len + 1, GFP_KERNEL); + of->prealloc_buf = kmalloc(len + 1, GFP_KERNEL | __GFP_PACKED); error = -ENOMEM; if (!of->prealloc_buf) goto err_free; diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c index 58036f657126..592ed327b1df 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -78,7 +78,7 @@ alloc_nfs_open_dir_context(struct inode *dir) struct nfs_inode *nfsi = NFS_I(dir); struct nfs_open_dir_context *ctx; - ctx = kzalloc(sizeof(*ctx), GFP_KERNEL_ACCOUNT); + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL_ACCOUNT | __GFP_PACKED); if (ctx != NULL) { ctx->attr_gencount = nfsi->attr_gencount; ctx->dtsize = NFS_INIT_DTSIZE; @@ -1230,7 +1230,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) nfs_revalidate_mapping(inode, file->f_mapping); res = -ENOMEM; - desc = kzalloc(sizeof(*desc), GFP_KERNEL); + desc = kzalloc(sizeof(*desc), GFP_KERNEL | __GFP_PACKED); if (!desc) goto out; desc->file = file; @@ -3074,7 +3074,8 @@ static void nfs_access_add_rbtree(struct inode *inode, void nfs_access_add_cache(struct inode *inode, struct nfs_access_entry *set, const struct cred *cred) { - struct nfs_access_entry *cache = kmalloc(sizeof(*cache), GFP_KERNEL); + struct nfs_access_entry *cache = kmalloc(sizeof(*cache), + GFP_KERNEL | __GFP_PACKED); if (cache == NULL) return; RB_CLEAR_NODE(&cache->rb_node); diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c index 6b2cfa59a1a2..993e2e56dccb 100644 --- a/fs/nfs/inode.c +++ b/fs/nfs/inode.c @@ -948,7 +948,7 @@ struct nfs_lock_context *nfs_get_lock_context(struct nfs_open_context *ctx) res = __nfs_find_lock_context(ctx); rcu_read_unlock(); if (res == NULL) { - new = kmalloc(sizeof(*new), GFP_KERNEL_ACCOUNT); + new = kmalloc(sizeof(*new), GFP_KERNEL_ACCOUNT | __GFP_PACKED); if (new == NULL) return ERR_PTR(-ENOMEM); nfs_init_lock_context(new); diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c index c3503fb26fa2..4bba9b8f40e8 100644 --- a/fs/nfs/nfs4state.c +++ b/fs/nfs/nfs4state.c @@ -1074,7 +1074,7 @@ struct nfs_seqid *nfs_alloc_seqid(struct nfs_seqid_counter *counter, gfp_t gfp_m { struct nfs_seqid *new; - new = kmalloc(sizeof(*new), gfp_mask); + new = kmalloc(sizeof(*new), gfp_mask | __GFP_PACKED); if (new == NULL) return ERR_PTR(-ENOMEM); new->sequence = counter; diff --git a/fs/nfs/write.c b/fs/nfs/write.c index f41d24b54fd1..d6711c600a9d 100644 --- a/fs/nfs/write.c +++ b/fs/nfs/write.c @@ -114,7 +114,8 @@ static void nfs_writehdr_free(struct nfs_pgio_header *hdr) static struct nfs_io_completion *nfs_io_completion_alloc(gfp_t gfp_flags) { - return kmalloc(sizeof(struct nfs_io_completion), gfp_flags); + return kmalloc(sizeof(struct nfs_io_completion), gfp_flags | + __GFP_PACKED); } static void nfs_io_completion_init(struct nfs_io_completion *ioc, diff --git a/fs/notify/inotify/inotify_fsnotify.c b/fs/notify/inotify/inotify_fsnotify.c index 49cfe2ae6d23..78b5e41a247e 100644 --- a/fs/notify/inotify/inotify_fsnotify.c +++ b/fs/notify/inotify/inotify_fsnotify.c @@ -86,7 +86,8 @@ int inotify_handle_inode_event(struct fsnotify_mark *inode_mark, u32 mask, * security repercussion. */ old_memcg = set_active_memcg(group->memcg); - event = kmalloc(alloc_len, GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL); + event = kmalloc(alloc_len, GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL | + __GFP_PACKED); set_active_memcg(old_memcg); if (unlikely(!event)) { diff --git a/fs/proc/self.c b/fs/proc/self.c index 72cd69bcaf4a..e97ff8416856 100644 --- a/fs/proc/self.c +++ b/fs/proc/self.c @@ -19,7 +19,7 @@ static const char *proc_self_get_link(struct dentry *dentry, if (!tgid) return ERR_PTR(-ENOENT); /* max length of unsigned int in decimal + NULL term */ - name = kmalloc(10 + 1, dentry ? GFP_KERNEL : GFP_ATOMIC); + name = kmalloc(10 + 1, (dentry ? GFP_KERNEL : GFP_ATOMIC) | __GFP_PACKED); if (unlikely(!name)) return dentry ? ERR_PTR(-ENOMEM) : ERR_PTR(-ECHILD); sprintf(name, "%u", tgid); diff --git a/fs/seq_file.c b/fs/seq_file.c index 9456a2032224..326afd30169d 100644 --- a/fs/seq_file.c +++ b/fs/seq_file.c @@ -572,7 +572,8 @@ static void single_stop(struct seq_file *p, void *v) int single_open(struct file *file, int (*show)(struct seq_file *, void *), void *data) { - struct seq_operations *op = kmalloc(sizeof(*op), GFP_KERNEL_ACCOUNT); + struct seq_operations *op = kmalloc(sizeof(*op), GFP_KERNEL_ACCOUNT | + __GFP_PACKED); int res = -ENOMEM; if (op) { @@ -634,7 +635,7 @@ void *__seq_open_private(struct file *f, const struct seq_operations *ops, void *private; struct seq_file *seq; - private = kzalloc(psize, GFP_KERNEL_ACCOUNT); + private = kzalloc(psize, GFP_KERNEL_ACCOUNT | __GFP_PACKED); if (private == NULL) goto out; diff --git a/include/acpi/platform/aclinuxex.h b/include/acpi/platform/aclinuxex.h index 28c72744decf..298ccbc0b949 100644 --- a/include/acpi/platform/aclinuxex.h +++ b/include/acpi/platform/aclinuxex.h @@ -49,12 +49,14 @@ acpi_status acpi_os_terminate(void); */ static inline void *acpi_os_allocate(acpi_size size) { - return kmalloc(size, irqs_disabled()? GFP_ATOMIC : GFP_KERNEL); + return kmalloc(size, (irqs_disabled()? GFP_ATOMIC : GFP_KERNEL) | + __GFP_PACKED); } static inline void *acpi_os_allocate_zeroed(acpi_size size) { - return kzalloc(size, irqs_disabled()? GFP_ATOMIC : GFP_KERNEL); + return kzalloc(size, (irqs_disabled()? GFP_ATOMIC : GFP_KERNEL) | + __GFP_PACKED); } static inline void acpi_os_free(void *memory) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index fbf2543111c0..f7fa75587ed6 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -7284,7 +7284,7 @@ static void add_to_clear_hash_list(struct list_head *clear_list, { struct ftrace_init_func *func; - func = kmalloc(sizeof(*func), GFP_KERNEL); + func = kmalloc(sizeof(*func), GFP_KERNEL | __GFP_PACKED); if (!func) { MEM_FAIL(1, "alloc failure, ftrace filter could be stale\n"); return; diff --git a/kernel/trace/tracing_map.c b/kernel/trace/tracing_map.c index c774e560f2f9..ab1dc9352122 100644 --- a/kernel/trace/tracing_map.c +++ b/kernel/trace/tracing_map.c @@ -948,7 +948,7 @@ create_sort_entry(void *key, struct tracing_map_elt *elt) { struct tracing_map_sort_entry *sort_entry; - sort_entry = kzalloc(sizeof(*sort_entry), GFP_KERNEL); + sort_entry = kzalloc(sizeof(*sort_entry), GFP_KERNEL | __GFP_PACKED); if (!sort_entry) return NULL; diff --git a/lib/kasprintf.c b/lib/kasprintf.c index cd2f5974ed98..9409012c9a21 100644 --- a/lib/kasprintf.c +++ b/lib/kasprintf.c @@ -22,7 +22,7 @@ char *kvasprintf(gfp_t gfp, const char *fmt, va_list ap) first = vsnprintf(NULL, 0, fmt, aq); va_end(aq); - p = kmalloc_track_caller(first+1, gfp); + p = kmalloc_track_caller(first+1, gfp | __GFP_PACKED); if (!p) return NULL; diff --git a/lib/kobject.c b/lib/kobject.c index a0b2dbfcfa23..2c4acb36925d 100644 --- a/lib/kobject.c +++ b/lib/kobject.c @@ -144,7 +144,7 @@ char *kobject_get_path(struct kobject *kobj, gfp_t gfp_mask) len = get_kobj_path_length(kobj); if (len == 0) return NULL; - path = kzalloc(len, gfp_mask); + path = kzalloc(len, gfp_mask | __GFP_PACKED); if (!path) return NULL; fill_kobj_path(kobj, path, len); @@ -749,7 +749,7 @@ static struct kobject *kobject_create(void) { struct kobject *kobj; - kobj = kzalloc(sizeof(*kobj), GFP_KERNEL); + kobj = kzalloc(sizeof(*kobj), GFP_KERNEL | __GFP_PACKED); if (!kobj) return NULL; diff --git a/lib/mpi/mpiutil.c b/lib/mpi/mpiutil.c index aa8c46544af8..441bb69b588e 100644 --- a/lib/mpi/mpiutil.c +++ b/lib/mpi/mpiutil.c @@ -88,7 +88,7 @@ MPI mpi_alloc(unsigned nlimbs) { MPI a; - a = kmalloc(sizeof *a, GFP_KERNEL); + a = kmalloc(sizeof *a, GFP_KERNEL | __GFP_PACKED); if (!a) return a; diff --git a/mm/list_lru.c b/mm/list_lru.c index a05e5bef3b40..754a7598206b 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -340,7 +340,8 @@ static struct list_lru_memcg *memcg_init_list_lru_one(gfp_t gfp) int nid; struct list_lru_memcg *mlru; - mlru = kmalloc(struct_size(mlru, node, nr_node_ids), gfp); + mlru = kmalloc(struct_size(mlru, node, nr_node_ids), + gfp | __GFP_PACKED); if (!mlru) return NULL; @@ -484,7 +485,8 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, return 0; gfp &= GFP_RECLAIM_MASK; - table = kmalloc_array(memcg->css.cgroup->level, sizeof(*table), gfp); + table = kmalloc_array(memcg->css.cgroup->level, sizeof(*table), + gfp | __GFP_PACKED); if (!table) return -ENOMEM; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2d8549ae1b30..626001f081da 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2882,8 +2882,8 @@ int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, void *vec; gfp &= ~OBJCGS_CLEAR_MASK; - vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, - slab_nid(slab)); + vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), + gfp | __GFP_PACKED, slab_nid(slab)); if (!vec) return -ENOMEM; diff --git a/mm/util.c b/mm/util.c index 12984e76767e..11c4bbf3849b 100644 --- a/mm/util.c +++ b/mm/util.c @@ -58,7 +58,7 @@ char *kstrdup(const char *s, gfp_t gfp) return NULL; len = strlen(s) + 1; - buf = kmalloc_track_caller(len, gfp); + buf = kmalloc_track_caller(len, gfp | __GFP_PACKED); if (buf) memcpy(buf, s, len); return buf; @@ -149,7 +149,7 @@ char *kmemdup_nul(const char *s, size_t len, gfp_t gfp) if (!s) return NULL; - buf = kmalloc_track_caller(len + 1, gfp); + buf = kmalloc_track_caller(len + 1, gfp | __GFP_PACKED); if (buf) { memcpy(buf, s, len); buf[len] = '\0'; @@ -558,7 +558,7 @@ EXPORT_SYMBOL(vm_mmap); */ void *kvmalloc_node(size_t size, gfp_t flags, int node) { - gfp_t kmalloc_flags = flags; + gfp_t kmalloc_flags = flags | __GFP_PACKED; void *ret; /* diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ccaa461998f3..46d9fa305ec8 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2491,7 +2491,8 @@ static struct vm_struct *__get_vm_area_node(unsigned long size, align = 1ul << clamp_t(int, get_count_order_long(size), PAGE_SHIFT, IOREMAP_MAX_ORDER); - area = kzalloc_node(sizeof(*area), gfp_mask & GFP_RECLAIM_MASK, node); + area = kzalloc_node(sizeof(*area), (gfp_mask & GFP_RECLAIM_MASK) | + __GFP_PACKED, node); if (unlikely(!area)) return NULL; @@ -3026,7 +3027,8 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, area->pages = __vmalloc_node(array_size, 1, nested_gfp, node, area->caller); } else { - area->pages = kmalloc_node(array_size, nested_gfp, node); + area->pages = kmalloc_node(array_size, nested_gfp | + __GFP_PACKED, node); } if (!area->pages) { diff --git a/net/sunrpc/auth_unix.c b/net/sunrpc/auth_unix.c index 1e091d3fa607..4129180dc4c8 100644 --- a/net/sunrpc/auth_unix.c +++ b/net/sunrpc/auth_unix.c @@ -45,7 +45,7 @@ static struct rpc_cred *unx_lookup_cred(struct rpc_auth *auth, { struct rpc_cred *ret; - ret = kmalloc(sizeof(*ret), rpc_task_gfp_mask()); + ret = kmalloc(sizeof(*ret), rpc_task_gfp_mask() | __GFP_PACKED); if (!ret) { if (!(flags & RPCAUTH_LOOKUP_ASYNC)) return ERR_PTR(-ENOMEM); diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index 336a7c7833e4..e3019e0c8109 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -146,7 +146,7 @@ xdr_alloc_bvec(struct xdr_buf *buf, gfp_t gfp) size_t i, n = xdr_buf_pagecount(buf); if (n != 0 && buf->bvec == NULL) { - buf->bvec = kmalloc_array(n, sizeof(buf->bvec[0]), gfp); + buf->bvec = kmalloc_array(n, sizeof(buf->bvec[0]), gfp | __GFP_PACKED); if (!buf->bvec) return -ENOMEM; for (i = 0; i < n; i++) { diff --git a/security/apparmor/lsm.c b/security/apparmor/lsm.c index f56070270c69..e23d510d57ed 100644 --- a/security/apparmor/lsm.c +++ b/security/apparmor/lsm.c @@ -809,7 +809,7 @@ static int apparmor_sk_alloc_security(struct sock *sk, int family, gfp_t flags) { struct aa_sk_ctx *ctx; - ctx = kzalloc(sizeof(*ctx), flags); + ctx = kzalloc(sizeof(*ctx), flags | __GFP_PACKED); if (!ctx) return -ENOMEM; diff --git a/security/security.c b/security/security.c index 79d82cb6e469..b410c284e872 100644 --- a/security/security.c +++ b/security/security.c @@ -537,7 +537,7 @@ static int lsm_cred_alloc(struct cred *cred, gfp_t gfp) return 0; } - cred->security = kzalloc(blob_sizes.lbs_cred, gfp); + cred->security = kzalloc(blob_sizes.lbs_cred, gfp | __GFP_PACKED); if (cred->security == NULL) return -ENOMEM; return 0; @@ -614,7 +614,7 @@ static int lsm_task_alloc(struct task_struct *task) return 0; } - task->security = kzalloc(blob_sizes.lbs_task, GFP_KERNEL); + task->security = kzalloc(blob_sizes.lbs_task, GFP_KERNEL | __GFP_PACKED); if (task->security == NULL) return -ENOMEM; return 0; diff --git a/security/tomoyo/realpath.c b/security/tomoyo/realpath.c index 1c483ee7f93d..8004d8e9ad66 100644 --- a/security/tomoyo/realpath.c +++ b/security/tomoyo/realpath.c @@ -42,7 +42,7 @@ char *tomoyo_encode2(const char *str, int str_len) } len++; /* Reserve space for appending "/". */ - cp = kzalloc(len + 10, GFP_NOFS); + cp = kzalloc(len + 10, GFP_NOFS | __GFP_PACKED); if (!cp) return NULL; cp0 = cp;