From patchwork Tue Oct 25 20:52:47 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Catalin Marinas X-Patchwork-Id: 13019899 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4D358C38A2D for ; Tue, 25 Oct 2022 20:54:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=kxDVZMQJDT/XiyFdXGeDOrixsv9+52iIfQlGUq8Xlkc=; b=f+edoJHVytOcEo TEQBHf2YnO78rSJvPqtAE/jbZtttD0wlND019WEd+nq5x81v+G8TxDlJBMvGrZOHxVyId0FZbKlwh n33g4rg4+HzEdma4QSY9Zrnyd+4wRzZgQHzlou5zjQGpJKIcp8oyiqkdb2+CViJeB9z1A5Ha5XRM3 SvJ3nRwg2X2cQmgDP5HIZCu+ugqOTT7E6QMVVOo4QFCcJWJm7DtNmlocAEypl3ZRHGpjRfLX+HI3D ZOHW7Ggkk163rokHJTic7PKmSA2HN4i33fFdfmFikBvzsnOAH5V3OLYFHPmyvOMKhbbJ6ksw+MC8a qcknR2o3saZODzWxnwyQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1onQvG-0078OP-DD; Tue, 25 Oct 2022 20:53:18 +0000 Received: from ams.source.kernel.org ([2604:1380:4601:e00::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1onQuy-0078JC-QH for linux-arm-kernel@lists.infradead.org; Tue, 25 Oct 2022 20:53:04 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 56784B81BE1; Tue, 25 Oct 2022 20:52:59 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 87836C43470; Tue, 25 Oct 2022 20:52:55 +0000 (UTC) From: Catalin Marinas To: Linus Torvalds , Arnd Bergmann Cc: Will Deacon , Marc Zyngier , Greg Kroah-Hartman , Andrew Morton , Herbert Xu , Ard Biesheuvel , Christoph Hellwig , Isaac Manjarres , Saravana Kannan , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Subject: [PATCH v2 2/2] treewide: Add the __GFP_PACKED flag to several non-DMA kmalloc() allocations Date: Tue, 25 Oct 2022 21:52:47 +0100 Message-Id: <20221025205247.3264568-3-catalin.marinas@arm.com> X-Mailer: git-send-email 2.30.2 In-Reply-To: <20221025205247.3264568-1-catalin.marinas@arm.com> References: <20221025205247.3264568-1-catalin.marinas@arm.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221025_135301_223907_8D256362 X-CRM114-Status: GOOD ( 26.95 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Using the ftrace kmalloc histogram shows several hot spots for small memory allocations that would benefit from the smaller KMALLOC_PACKED_ALIGN alignment. To set up the ftrace events for small kmalloc() allocations (only showing those not already having the __GFP_PACKED flag): echo 'hist:key=call_site.sym-offset:vals=bytes_req,bytes_alloc:sort=bytes_alloc.descending if bytes_req<=96 && !(gfp_flags&0x8000000)' \ > /sys/kernel/debug/tracing/events/kmem/kmalloc/trigger To enable tracing of boot-time allocations, use the following bootconfig: ftrace.event.kmem.kmalloc.hist { keys = call_site.sym-offset values = bytes_req, bytes_alloc sort = bytes_alloc.descending filter = "bytes_req <= 96 && !(gfp_flags & 0x8000000)" } To view the allocation hot spots: head /sys/kernel/debug/tracing/events/kmem/kmalloc/hist Signed-off-by: Catalin Marinas --- drivers/usb/core/message.c | 3 ++- fs/binfmt_elf.c | 6 ++++-- fs/dcache.c | 3 ++- fs/ext4/dir.c | 4 ++-- fs/ext4/extents.c | 4 ++-- fs/file.c | 2 +- fs/kernfs/file.c | 8 ++++---- fs/nfs/dir.c | 7 ++++--- fs/nfs/inode.c | 2 +- fs/nfs/nfs4state.c | 2 +- fs/nfs/write.c | 3 ++- fs/notify/inotify/inotify_fsnotify.c | 3 ++- fs/proc/self.c | 2 +- fs/seq_file.c | 5 +++-- include/acpi/platform/aclinuxex.h | 6 ++++-- kernel/trace/ftrace.c | 2 +- kernel/trace/tracing_map.c | 2 +- lib/kasprintf.c | 2 +- lib/kobject.c | 4 ++-- lib/mpi/mpiutil.c | 2 +- mm/list_lru.c | 6 ++++-- mm/memcontrol.c | 4 ++-- mm/util.c | 6 +++--- mm/vmalloc.c | 6 ++++-- net/sunrpc/auth_unix.c | 2 +- net/sunrpc/xdr.c | 2 +- security/apparmor/lsm.c | 2 +- security/security.c | 4 ++-- security/tomoyo/realpath.c | 2 +- 29 files changed, 60 insertions(+), 46 deletions(-) diff --git a/drivers/usb/core/message.c b/drivers/usb/core/message.c index 4d59d927ae3e..bff8901dc426 100644 --- a/drivers/usb/core/message.c +++ b/drivers/usb/core/message.c @@ -525,7 +525,8 @@ int usb_sg_init(struct usb_sg_request *io, struct usb_device *dev, } /* initialize all the urbs we'll use */ - io->urbs = kmalloc_array(io->entries, sizeof(*io->urbs), mem_flags); + io->urbs = kmalloc_array(io->entries, sizeof(*io->urbs), + mem_flags | __GFP_PACKED); if (!io->urbs) goto nomem; diff --git a/fs/binfmt_elf.c b/fs/binfmt_elf.c index 63c7ebb0da89..e5ad1a3244fb 100644 --- a/fs/binfmt_elf.c +++ b/fs/binfmt_elf.c @@ -883,7 +883,8 @@ static int load_elf_binary(struct linux_binprm *bprm) goto out_free_ph; retval = -ENOMEM; - elf_interpreter = kmalloc(elf_ppnt->p_filesz, GFP_KERNEL); + elf_interpreter = kmalloc(elf_ppnt->p_filesz, + GFP_KERNEL | __GFP_PACKED); if (!elf_interpreter) goto out_free_ph; @@ -908,7 +909,8 @@ static int load_elf_binary(struct linux_binprm *bprm) */ would_dump(bprm, interpreter); - interp_elf_ex = kmalloc(sizeof(*interp_elf_ex), GFP_KERNEL); + interp_elf_ex = kmalloc(sizeof(*interp_elf_ex), + GFP_KERNEL | __GFP_PACKED); if (!interp_elf_ex) { retval = -ENOMEM; goto out_free_ph; diff --git a/fs/dcache.c b/fs/dcache.c index 52e6d5fdab6b..9c66679365ca 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -1785,7 +1785,8 @@ static struct dentry *__d_alloc(struct super_block *sb, const struct qstr *name) size_t size = offsetof(struct external_name, name[1]); struct external_name *p = kmalloc(size + name->len, GFP_KERNEL_ACCOUNT | - __GFP_RECLAIMABLE); + __GFP_RECLAIMABLE | + __GFP_PACKED); if (!p) { kmem_cache_free(dentry_cache, dentry); return NULL; diff --git a/fs/ext4/dir.c b/fs/ext4/dir.c index 3985f8c33f95..fc8415ff8880 100644 --- a/fs/ext4/dir.c +++ b/fs/ext4/dir.c @@ -435,7 +435,7 @@ static struct dir_private_info *ext4_htree_create_dir_info(struct file *filp, { struct dir_private_info *p; - p = kzalloc(sizeof(*p), GFP_KERNEL); + p = kzalloc(sizeof(*p), GFP_KERNEL | __GFP_PACKED); if (!p) return NULL; p->curr_hash = pos2maj_hash(filp, pos); @@ -471,7 +471,7 @@ int ext4_htree_store_dirent(struct file *dir_file, __u32 hash, /* Create and allocate the fname structure */ len = sizeof(struct fname) + ent_name->len + 1; - new_fn = kzalloc(len, GFP_KERNEL); + new_fn = kzalloc(len, GFP_KERNEL | __GFP_PACKED); if (!new_fn) return -ENOMEM; new_fn->hash = hash; diff --git a/fs/ext4/extents.c b/fs/ext4/extents.c index f1956288307f..d8ffb65ac318 100644 --- a/fs/ext4/extents.c +++ b/fs/ext4/extents.c @@ -912,7 +912,7 @@ ext4_find_extent(struct inode *inode, ext4_lblk_t block, if (!path) { /* account possible depth increase */ path = kcalloc(depth + 2, sizeof(struct ext4_ext_path), - gfp_flags); + gfp_flags | __GFP_PACKED); if (unlikely(!path)) return ERR_PTR(-ENOMEM); path[0].p_maxdepth = depth + 1; @@ -2937,7 +2937,7 @@ int ext4_ext_remove_space(struct inode *inode, ext4_lblk_t start, le16_to_cpu(path[k].p_hdr->eh_entries)+1; } else { path = kcalloc(depth + 1, sizeof(struct ext4_ext_path), - GFP_NOFS | __GFP_NOFAIL); + GFP_NOFS | __GFP_NOFAIL | __GFP_PACKED); if (path == NULL) { ext4_journal_stop(handle); return -ENOMEM; diff --git a/fs/file.c b/fs/file.c index 5f9c802a5d8d..3f732736eaf3 100644 --- a/fs/file.c +++ b/fs/file.c @@ -129,7 +129,7 @@ static struct fdtable * alloc_fdtable(unsigned int nr) if (unlikely(nr > sysctl_nr_open)) nr = ((sysctl_nr_open - 1) | (BITS_PER_LONG - 1)) + 1; - fdt = kmalloc(sizeof(struct fdtable), GFP_KERNEL_ACCOUNT); + fdt = kmalloc(sizeof(struct fdtable), GFP_KERNEL_ACCOUNT | __GFP_PACKED); if (!fdt) goto out; fdt->max_fds = nr; diff --git a/fs/kernfs/file.c b/fs/kernfs/file.c index 9ab6c92e02da..882980965c55 100644 --- a/fs/kernfs/file.c +++ b/fs/kernfs/file.c @@ -304,7 +304,7 @@ static ssize_t kernfs_fop_write_iter(struct kiocb *iocb, struct iov_iter *iter) if (buf) mutex_lock(&of->prealloc_mutex); else - buf = kmalloc(len + 1, GFP_KERNEL); + buf = kmalloc(len + 1, GFP_KERNEL | __GFP_PACKED); if (!buf) return -ENOMEM; @@ -565,7 +565,7 @@ static int kernfs_get_open_node(struct kernfs_node *kn, if (!on) { /* not there, initialize a new one */ - on = kzalloc(sizeof(*on), GFP_KERNEL); + on = kzalloc(sizeof(*on), GFP_KERNEL | __GFP_PACKED); if (!on) { mutex_unlock(mutex); return -ENOMEM; @@ -663,7 +663,7 @@ static int kernfs_fop_open(struct inode *inode, struct file *file) /* allocate a kernfs_open_file for the file */ error = -ENOMEM; - of = kzalloc(sizeof(struct kernfs_open_file), GFP_KERNEL); + of = kzalloc(sizeof(struct kernfs_open_file), GFP_KERNEL | __GFP_PACKED); if (!of) goto err_out; @@ -706,7 +706,7 @@ static int kernfs_fop_open(struct inode *inode, struct file *file) goto err_free; if (ops->prealloc) { int len = of->atomic_write_len ?: PAGE_SIZE; - of->prealloc_buf = kmalloc(len + 1, GFP_KERNEL); + of->prealloc_buf = kmalloc(len + 1, GFP_KERNEL | __GFP_PACKED); error = -ENOMEM; if (!of->prealloc_buf) goto err_free; diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c index 58036f657126..592ed327b1df 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -78,7 +78,7 @@ alloc_nfs_open_dir_context(struct inode *dir) struct nfs_inode *nfsi = NFS_I(dir); struct nfs_open_dir_context *ctx; - ctx = kzalloc(sizeof(*ctx), GFP_KERNEL_ACCOUNT); + ctx = kzalloc(sizeof(*ctx), GFP_KERNEL_ACCOUNT | __GFP_PACKED); if (ctx != NULL) { ctx->attr_gencount = nfsi->attr_gencount; ctx->dtsize = NFS_INIT_DTSIZE; @@ -1230,7 +1230,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) nfs_revalidate_mapping(inode, file->f_mapping); res = -ENOMEM; - desc = kzalloc(sizeof(*desc), GFP_KERNEL); + desc = kzalloc(sizeof(*desc), GFP_KERNEL | __GFP_PACKED); if (!desc) goto out; desc->file = file; @@ -3074,7 +3074,8 @@ static void nfs_access_add_rbtree(struct inode *inode, void nfs_access_add_cache(struct inode *inode, struct nfs_access_entry *set, const struct cred *cred) { - struct nfs_access_entry *cache = kmalloc(sizeof(*cache), GFP_KERNEL); + struct nfs_access_entry *cache = kmalloc(sizeof(*cache), + GFP_KERNEL | __GFP_PACKED); if (cache == NULL) return; RB_CLEAR_NODE(&cache->rb_node); diff --git a/fs/nfs/inode.c b/fs/nfs/inode.c index 6b2cfa59a1a2..993e2e56dccb 100644 --- a/fs/nfs/inode.c +++ b/fs/nfs/inode.c @@ -948,7 +948,7 @@ struct nfs_lock_context *nfs_get_lock_context(struct nfs_open_context *ctx) res = __nfs_find_lock_context(ctx); rcu_read_unlock(); if (res == NULL) { - new = kmalloc(sizeof(*new), GFP_KERNEL_ACCOUNT); + new = kmalloc(sizeof(*new), GFP_KERNEL_ACCOUNT | __GFP_PACKED); if (new == NULL) return ERR_PTR(-ENOMEM); nfs_init_lock_context(new); diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c index c3503fb26fa2..4bba9b8f40e8 100644 --- a/fs/nfs/nfs4state.c +++ b/fs/nfs/nfs4state.c @@ -1074,7 +1074,7 @@ struct nfs_seqid *nfs_alloc_seqid(struct nfs_seqid_counter *counter, gfp_t gfp_m { struct nfs_seqid *new; - new = kmalloc(sizeof(*new), gfp_mask); + new = kmalloc(sizeof(*new), gfp_mask | __GFP_PACKED); if (new == NULL) return ERR_PTR(-ENOMEM); new->sequence = counter; diff --git a/fs/nfs/write.c b/fs/nfs/write.c index f41d24b54fd1..d6711c600a9d 100644 --- a/fs/nfs/write.c +++ b/fs/nfs/write.c @@ -114,7 +114,8 @@ static void nfs_writehdr_free(struct nfs_pgio_header *hdr) static struct nfs_io_completion *nfs_io_completion_alloc(gfp_t gfp_flags) { - return kmalloc(sizeof(struct nfs_io_completion), gfp_flags); + return kmalloc(sizeof(struct nfs_io_completion), gfp_flags | + __GFP_PACKED); } static void nfs_io_completion_init(struct nfs_io_completion *ioc, diff --git a/fs/notify/inotify/inotify_fsnotify.c b/fs/notify/inotify/inotify_fsnotify.c index 49cfe2ae6d23..78b5e41a247e 100644 --- a/fs/notify/inotify/inotify_fsnotify.c +++ b/fs/notify/inotify/inotify_fsnotify.c @@ -86,7 +86,8 @@ int inotify_handle_inode_event(struct fsnotify_mark *inode_mark, u32 mask, * security repercussion. */ old_memcg = set_active_memcg(group->memcg); - event = kmalloc(alloc_len, GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL); + event = kmalloc(alloc_len, GFP_KERNEL_ACCOUNT | __GFP_RETRY_MAYFAIL | + __GFP_PACKED); set_active_memcg(old_memcg); if (unlikely(!event)) { diff --git a/fs/proc/self.c b/fs/proc/self.c index 72cd69bcaf4a..e97ff8416856 100644 --- a/fs/proc/self.c +++ b/fs/proc/self.c @@ -19,7 +19,7 @@ static const char *proc_self_get_link(struct dentry *dentry, if (!tgid) return ERR_PTR(-ENOENT); /* max length of unsigned int in decimal + NULL term */ - name = kmalloc(10 + 1, dentry ? GFP_KERNEL : GFP_ATOMIC); + name = kmalloc(10 + 1, (dentry ? GFP_KERNEL : GFP_ATOMIC) | __GFP_PACKED); if (unlikely(!name)) return dentry ? ERR_PTR(-ENOMEM) : ERR_PTR(-ECHILD); sprintf(name, "%u", tgid); diff --git a/fs/seq_file.c b/fs/seq_file.c index 9456a2032224..326afd30169d 100644 --- a/fs/seq_file.c +++ b/fs/seq_file.c @@ -572,7 +572,8 @@ static void single_stop(struct seq_file *p, void *v) int single_open(struct file *file, int (*show)(struct seq_file *, void *), void *data) { - struct seq_operations *op = kmalloc(sizeof(*op), GFP_KERNEL_ACCOUNT); + struct seq_operations *op = kmalloc(sizeof(*op), GFP_KERNEL_ACCOUNT | + __GFP_PACKED); int res = -ENOMEM; if (op) { @@ -634,7 +635,7 @@ void *__seq_open_private(struct file *f, const struct seq_operations *ops, void *private; struct seq_file *seq; - private = kzalloc(psize, GFP_KERNEL_ACCOUNT); + private = kzalloc(psize, GFP_KERNEL_ACCOUNT | __GFP_PACKED); if (private == NULL) goto out; diff --git a/include/acpi/platform/aclinuxex.h b/include/acpi/platform/aclinuxex.h index 28c72744decf..298ccbc0b949 100644 --- a/include/acpi/platform/aclinuxex.h +++ b/include/acpi/platform/aclinuxex.h @@ -49,12 +49,14 @@ acpi_status acpi_os_terminate(void); */ static inline void *acpi_os_allocate(acpi_size size) { - return kmalloc(size, irqs_disabled()? GFP_ATOMIC : GFP_KERNEL); + return kmalloc(size, (irqs_disabled()? GFP_ATOMIC : GFP_KERNEL) | + __GFP_PACKED); } static inline void *acpi_os_allocate_zeroed(acpi_size size) { - return kzalloc(size, irqs_disabled()? GFP_ATOMIC : GFP_KERNEL); + return kzalloc(size, (irqs_disabled()? GFP_ATOMIC : GFP_KERNEL) | + __GFP_PACKED); } static inline void acpi_os_free(void *memory) diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index fbf2543111c0..f7fa75587ed6 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -7284,7 +7284,7 @@ static void add_to_clear_hash_list(struct list_head *clear_list, { struct ftrace_init_func *func; - func = kmalloc(sizeof(*func), GFP_KERNEL); + func = kmalloc(sizeof(*func), GFP_KERNEL | __GFP_PACKED); if (!func) { MEM_FAIL(1, "alloc failure, ftrace filter could be stale\n"); return; diff --git a/kernel/trace/tracing_map.c b/kernel/trace/tracing_map.c index c774e560f2f9..ab1dc9352122 100644 --- a/kernel/trace/tracing_map.c +++ b/kernel/trace/tracing_map.c @@ -948,7 +948,7 @@ create_sort_entry(void *key, struct tracing_map_elt *elt) { struct tracing_map_sort_entry *sort_entry; - sort_entry = kzalloc(sizeof(*sort_entry), GFP_KERNEL); + sort_entry = kzalloc(sizeof(*sort_entry), GFP_KERNEL | __GFP_PACKED); if (!sort_entry) return NULL; diff --git a/lib/kasprintf.c b/lib/kasprintf.c index cd2f5974ed98..9409012c9a21 100644 --- a/lib/kasprintf.c +++ b/lib/kasprintf.c @@ -22,7 +22,7 @@ char *kvasprintf(gfp_t gfp, const char *fmt, va_list ap) first = vsnprintf(NULL, 0, fmt, aq); va_end(aq); - p = kmalloc_track_caller(first+1, gfp); + p = kmalloc_track_caller(first+1, gfp | __GFP_PACKED); if (!p) return NULL; diff --git a/lib/kobject.c b/lib/kobject.c index a0b2dbfcfa23..2c4acb36925d 100644 --- a/lib/kobject.c +++ b/lib/kobject.c @@ -144,7 +144,7 @@ char *kobject_get_path(struct kobject *kobj, gfp_t gfp_mask) len = get_kobj_path_length(kobj); if (len == 0) return NULL; - path = kzalloc(len, gfp_mask); + path = kzalloc(len, gfp_mask | __GFP_PACKED); if (!path) return NULL; fill_kobj_path(kobj, path, len); @@ -749,7 +749,7 @@ static struct kobject *kobject_create(void) { struct kobject *kobj; - kobj = kzalloc(sizeof(*kobj), GFP_KERNEL); + kobj = kzalloc(sizeof(*kobj), GFP_KERNEL | __GFP_PACKED); if (!kobj) return NULL; diff --git a/lib/mpi/mpiutil.c b/lib/mpi/mpiutil.c index aa8c46544af8..441bb69b588e 100644 --- a/lib/mpi/mpiutil.c +++ b/lib/mpi/mpiutil.c @@ -88,7 +88,7 @@ MPI mpi_alloc(unsigned nlimbs) { MPI a; - a = kmalloc(sizeof *a, GFP_KERNEL); + a = kmalloc(sizeof *a, GFP_KERNEL | __GFP_PACKED); if (!a) return a; diff --git a/mm/list_lru.c b/mm/list_lru.c index a05e5bef3b40..754a7598206b 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -340,7 +340,8 @@ static struct list_lru_memcg *memcg_init_list_lru_one(gfp_t gfp) int nid; struct list_lru_memcg *mlru; - mlru = kmalloc(struct_size(mlru, node, nr_node_ids), gfp); + mlru = kmalloc(struct_size(mlru, node, nr_node_ids), + gfp | __GFP_PACKED); if (!mlru) return NULL; @@ -484,7 +485,8 @@ int memcg_list_lru_alloc(struct mem_cgroup *memcg, struct list_lru *lru, return 0; gfp &= GFP_RECLAIM_MASK; - table = kmalloc_array(memcg->css.cgroup->level, sizeof(*table), gfp); + table = kmalloc_array(memcg->css.cgroup->level, sizeof(*table), + gfp | __GFP_PACKED); if (!table) return -ENOMEM; diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 2d8549ae1b30..626001f081da 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -2882,8 +2882,8 @@ int memcg_alloc_slab_cgroups(struct slab *slab, struct kmem_cache *s, void *vec; gfp &= ~OBJCGS_CLEAR_MASK; - vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), gfp, - slab_nid(slab)); + vec = kcalloc_node(objects, sizeof(struct obj_cgroup *), + gfp | __GFP_PACKED, slab_nid(slab)); if (!vec) return -ENOMEM; diff --git a/mm/util.c b/mm/util.c index 12984e76767e..11c4bbf3849b 100644 --- a/mm/util.c +++ b/mm/util.c @@ -58,7 +58,7 @@ char *kstrdup(const char *s, gfp_t gfp) return NULL; len = strlen(s) + 1; - buf = kmalloc_track_caller(len, gfp); + buf = kmalloc_track_caller(len, gfp | __GFP_PACKED); if (buf) memcpy(buf, s, len); return buf; @@ -149,7 +149,7 @@ char *kmemdup_nul(const char *s, size_t len, gfp_t gfp) if (!s) return NULL; - buf = kmalloc_track_caller(len + 1, gfp); + buf = kmalloc_track_caller(len + 1, gfp | __GFP_PACKED); if (buf) { memcpy(buf, s, len); buf[len] = '\0'; @@ -558,7 +558,7 @@ EXPORT_SYMBOL(vm_mmap); */ void *kvmalloc_node(size_t size, gfp_t flags, int node) { - gfp_t kmalloc_flags = flags; + gfp_t kmalloc_flags = flags | __GFP_PACKED; void *ret; /* diff --git a/mm/vmalloc.c b/mm/vmalloc.c index ccaa461998f3..46d9fa305ec8 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -2491,7 +2491,8 @@ static struct vm_struct *__get_vm_area_node(unsigned long size, align = 1ul << clamp_t(int, get_count_order_long(size), PAGE_SHIFT, IOREMAP_MAX_ORDER); - area = kzalloc_node(sizeof(*area), gfp_mask & GFP_RECLAIM_MASK, node); + area = kzalloc_node(sizeof(*area), (gfp_mask & GFP_RECLAIM_MASK) | + __GFP_PACKED, node); if (unlikely(!area)) return NULL; @@ -3026,7 +3027,8 @@ static void *__vmalloc_area_node(struct vm_struct *area, gfp_t gfp_mask, area->pages = __vmalloc_node(array_size, 1, nested_gfp, node, area->caller); } else { - area->pages = kmalloc_node(array_size, nested_gfp, node); + area->pages = kmalloc_node(array_size, nested_gfp | + __GFP_PACKED, node); } if (!area->pages) { diff --git a/net/sunrpc/auth_unix.c b/net/sunrpc/auth_unix.c index 1e091d3fa607..4129180dc4c8 100644 --- a/net/sunrpc/auth_unix.c +++ b/net/sunrpc/auth_unix.c @@ -45,7 +45,7 @@ static struct rpc_cred *unx_lookup_cred(struct rpc_auth *auth, { struct rpc_cred *ret; - ret = kmalloc(sizeof(*ret), rpc_task_gfp_mask()); + ret = kmalloc(sizeof(*ret), rpc_task_gfp_mask() | __GFP_PACKED); if (!ret) { if (!(flags & RPCAUTH_LOOKUP_ASYNC)) return ERR_PTR(-ENOMEM); diff --git a/net/sunrpc/xdr.c b/net/sunrpc/xdr.c index 336a7c7833e4..e3019e0c8109 100644 --- a/net/sunrpc/xdr.c +++ b/net/sunrpc/xdr.c @@ -146,7 +146,7 @@ xdr_alloc_bvec(struct xdr_buf *buf, gfp_t gfp) size_t i, n = xdr_buf_pagecount(buf); if (n != 0 && buf->bvec == NULL) { - buf->bvec = kmalloc_array(n, sizeof(buf->bvec[0]), gfp); + buf->bvec = kmalloc_array(n, sizeof(buf->bvec[0]), gfp | __GFP_PACKED); if (!buf->bvec) return -ENOMEM; for (i = 0; i < n; i++) { diff --git a/security/apparmor/lsm.c b/security/apparmor/lsm.c index f56070270c69..e23d510d57ed 100644 --- a/security/apparmor/lsm.c +++ b/security/apparmor/lsm.c @@ -809,7 +809,7 @@ static int apparmor_sk_alloc_security(struct sock *sk, int family, gfp_t flags) { struct aa_sk_ctx *ctx; - ctx = kzalloc(sizeof(*ctx), flags); + ctx = kzalloc(sizeof(*ctx), flags | __GFP_PACKED); if (!ctx) return -ENOMEM; diff --git a/security/security.c b/security/security.c index 79d82cb6e469..b410c284e872 100644 --- a/security/security.c +++ b/security/security.c @@ -537,7 +537,7 @@ static int lsm_cred_alloc(struct cred *cred, gfp_t gfp) return 0; } - cred->security = kzalloc(blob_sizes.lbs_cred, gfp); + cred->security = kzalloc(blob_sizes.lbs_cred, gfp | __GFP_PACKED); if (cred->security == NULL) return -ENOMEM; return 0; @@ -614,7 +614,7 @@ static int lsm_task_alloc(struct task_struct *task) return 0; } - task->security = kzalloc(blob_sizes.lbs_task, GFP_KERNEL); + task->security = kzalloc(blob_sizes.lbs_task, GFP_KERNEL | __GFP_PACKED); if (task->security == NULL) return -ENOMEM; return 0; diff --git a/security/tomoyo/realpath.c b/security/tomoyo/realpath.c index 1c483ee7f93d..8004d8e9ad66 100644 --- a/security/tomoyo/realpath.c +++ b/security/tomoyo/realpath.c @@ -42,7 +42,7 @@ char *tomoyo_encode2(const char *str, int str_len) } len++; /* Reserve space for appending "/". */ - cp = kzalloc(len + 10, GFP_NOFS); + cp = kzalloc(len + 10, GFP_NOFS | __GFP_PACKED); if (!cp) return NULL; cp0 = cp;