From patchwork Tue Sep 15 13:20:40 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Marco Elver X-Patchwork-Id: 11776663 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 75BA259D for ; Tue, 15 Sep 2020 13:42:35 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 34D2720936 for ; Tue, 15 Sep 2020 13:42:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="NcDSAHII"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="tVCZzDNH"; dkim=fail reason="signature verification failed" (2048-bit key) header.d=google.com header.i=@google.com header.b="NpaUC7cf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 34D2720936 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:Cc:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:To:From:Subject:References:Mime-Version:Message-Id: In-Reply-To:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=tQ/IHjhc3tSQmv/aoCq4PgbXHwJyWryt/smLC4zT0Gw=; b=NcDSAHIIQLWVK0Ufp2zgtyYxO OO13zIyR/BpiKbWmb0MJaXC+jtMBQyTTDvEjxdJCCg4ewTYAr5jncdvxXKdJChric25o41CNWzptv qtADv4MVSHIyA0cIIc942KRQuinRP3GE2aFlGva5nRsznxTQD8ZvqIhDdkHAf9E09t/HPVQXxtmk9 FOZMKeBSEQdaEBpwH9mUSSIjdfvtBYvw+f5pil/aZW1Xa1tjL6Jthr8knbeQaHhVjpabXmpLV80Ed d5mZV0jgaJDzh/kVaWDGe/astb6hwilAqqpqgE6xLMveizHhWPSyRMLSfZTwlop2Yw9bvMHtmCAat Sc505fkLw==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1kIBDt-00065Z-O0; Tue, 15 Sep 2020 13:42:17 +0000 Received: from casper.infradead.org ([2001:8b0:10b:1236::1]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kIBDF-0005ok-18 for linux-arm-kernel@merlin.infradead.org; Tue, 15 Sep 2020 13:41:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Type:Cc:To:From:Subject: References:Mime-Version:Message-Id:In-Reply-To:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=xuqVHa2SvjiSIGMIomiVLdoP1dWqddUAsUtVImDYYgU=; b=tVCZzDNHSTMr518gifnj6qEZRx R7/k34Zx8qwAG8qV8hD16d/Z+1DZfIv9cfCaFz+EkJOogPoJ0ZAyQSXqbaU/q/B7m7CfpZ0j4q2kh P8gsB6NMPfQ8oC91b/Ttl1e/JxDvLWMg85ecPrH7CABpdUBr8t4wP//ECQBPpP820qLgkedce0INz dP6sIBNbj4yye9NetdLuTV6gMXEzK6WNMZyEZyl0jfgIR875ZTakIarnprsqFq0lFyRYDFMXxm1+1 H7u5jii+z4nASjJ/kPb85OoAuZFg5HJTqGErAXD6uvppU+E9dNgkhb1wfdjHRSTqXL0NRin/QBZd2 fPmUE7Ew==; Received: from mail-wm1-x349.google.com ([2a00:1450:4864:20::349]) by casper.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1kIAtW-0001ht-1K for linux-arm-kernel@lists.infradead.org; Tue, 15 Sep 2020 13:21:17 +0000 Received: by mail-wm1-x349.google.com with SMTP id l15so887271wmh.9 for ; Tue, 15 Sep 2020 06:21:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=sender:date:in-reply-to:message-id:mime-version:references:subject :from:to:cc; bh=xuqVHa2SvjiSIGMIomiVLdoP1dWqddUAsUtVImDYYgU=; b=NpaUC7cfpzPMpRx17nryihoxa1AlcB17W3fVb3fXaxB9vMemW2FSTgKHFb7nk9XNnF gjMN6Vyjs/nv83d2V9vMJ2aydL+Ay6YM4ALes4xgUTpRXOf3mBGn8x3ldQp6QWVE02dX LElEwjmkfQTIgWbW9qqfSZIVjrv6jia0LTP5U2mY4tNDAlPIbIlSENZqo53MmZ8/Rn6S 70PKKnm6cH1CgAX/68BRgcqTfiyjhbo5Ih+pFiGGicAD7gjzLlyKqhJL0WtrOG9BcjzO lWY1MiWhTwjuzBAePYDWffsTNaUobcscX9PRDhZB7RmnLDgtnRbMMASpQ9tTuS2TukNr 6JCw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=xuqVHa2SvjiSIGMIomiVLdoP1dWqddUAsUtVImDYYgU=; b=KuSh5Je68FyvVDrTM85UYJXqgfhr9JncjRQjtcD+zMe9R/LxP1IarqO+ZFP8eUBUcL zaCKh/CyNy1M01fWmGRmhbmbPSgCJDeHwaBImp+euOD8gVSo/udPTl6vRA/76I4zaaKb gRvckYZcL4hKrfdmH53NxCZyIE+KD4bEtmVo1v8K0h+UrWovUzvtkF9n8fVpStHZiEf0 MFWNfxmMo8B689BipWskIDBNWZrwZeHKnrMa2w6qrmnO4gOVeWP+ITVXsGljvb7Ug8Uf ERvYb2STX3G9hKFjaQ6qSVxDeH6NfVfLlIFJVT23fZnEbJcJUw4kk6+rB8OwPTMiDDJZ hnjA== X-Gm-Message-State: AOAM5300641fxLliN0czJhfjZdSXNM47laM9OclETnhL5oFS1LGO5iWa RMDIQVgdTvUD4hCNXp+64j3/8mapZQ== X-Google-Smtp-Source: ABdhPJw9K95P957+uQw+uYA/RC6Zpb6e6m9pUCjJM+e6tY7J5w0Vg4MK4a/lIOMfiC2EWBE2575wJpxckA== X-Received: from elver.muc.corp.google.com ([2a00:79e0:15:13:f693:9fff:fef4:2449]) (user=elver job=sendgmr) by 2002:a7b:c958:: with SMTP id i24mr4957022wml.50.1600176068261; Tue, 15 Sep 2020 06:21:08 -0700 (PDT) Date: Tue, 15 Sep 2020 15:20:40 +0200 In-Reply-To: <20200915132046.3332537-1-elver@google.com> Message-Id: <20200915132046.3332537-5-elver@google.com> Mime-Version: 1.0 References: <20200915132046.3332537-1-elver@google.com> X-Mailer: git-send-email 2.28.0.618.gf4bc123cb7-goog Subject: [PATCH v2 04/10] mm, kfence: insert KFENCE hooks for SLAB From: Marco Elver To: elver@google.com, akpm@linux-foundation.org, glider@google.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200915_142114_214048_F931E19B X-CRM114-Status: GOOD ( 21.78 ) X-Spam-Score: -9.6 (---------) X-Spam-Report: SpamAssassin version 3.4.4 on casper.infradead.org summary: Content analysis details: (-9.6 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [2a00:1450:4864:20:0:0:0:349 listed in] [list.dnswl.org] -0.0 SPF_PASS SPF: sender matches SPF record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -7.5 USER_IN_DEF_DKIM_WL From: address is in the default DKIM white-list -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.0 DKIMWL_WL_MED DKIMwl.org - Medium sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: mark.rutland@arm.com, linux-doc@vger.kernel.org, peterz@infradead.org, catalin.marinas@arm.com, dave.hansen@linux.intel.com, linux-mm@kvack.org, edumazet@google.com, hpa@zytor.com, cl@linux.com, will@kernel.org, corbet@lwn.net, x86@kernel.org, kasan-dev@googlegroups.com, mingo@redhat.com, vbabka@suse.cz, rientjes@google.com, aryabinin@virtuozzo.com, keescook@chromium.org, paulmck@kernel.org, jannh@google.com, andreyknvl@google.com, bp@alien8.de, luto@kernel.org, Jonathan.Cameron@huawei.com, tglx@linutronix.de, dvyukov@google.com, linux-arm-kernel@lists.infradead.org, gregkh@linuxfoundation.org, linux-kernel@vger.kernel.org, penberg@kernel.org, cai@lca.pw, iamjoonsoo.kim@lge.com Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org From: Alexander Potapenko Inserts KFENCE hooks into the SLAB allocator. We note the addition of the 'orig_size' argument to slab_alloc*() functions, to be able to pass the originally requested size to KFENCE. When KFENCE is disabled, there is no additional overhead, since these functions are __always_inline. Co-developed-by: Marco Elver Signed-off-by: Marco Elver Signed-off-by: Alexander Potapenko --- mm/slab.c | 46 ++++++++++++++++++++++++++++++++++------------ mm/slab_common.c | 6 +++++- 2 files changed, 39 insertions(+), 13 deletions(-) diff --git a/mm/slab.c b/mm/slab.c index 3160dff6fd76..30aba06ae02b 100644 --- a/mm/slab.c +++ b/mm/slab.c @@ -100,6 +100,7 @@ #include #include #include +#include #include #include #include @@ -3206,7 +3207,7 @@ static void *____cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, } static __always_inline void * -slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, +slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, size_t orig_size, unsigned long caller) { unsigned long save_flags; @@ -3219,6 +3220,10 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, if (unlikely(!cachep)) return NULL; + ptr = kfence_alloc(cachep, orig_size, flags); + if (unlikely(ptr)) + goto out_hooks; + cache_alloc_debugcheck_before(cachep, flags); local_irq_save(save_flags); @@ -3251,6 +3256,7 @@ slab_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid, if (unlikely(slab_want_init_on_alloc(flags, cachep)) && ptr) memset(ptr, 0, cachep->object_size); +out_hooks: slab_post_alloc_hook(cachep, objcg, flags, 1, &ptr); return ptr; } @@ -3288,7 +3294,7 @@ __do_cache_alloc(struct kmem_cache *cachep, gfp_t flags) #endif /* CONFIG_NUMA */ static __always_inline void * -slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) +slab_alloc(struct kmem_cache *cachep, gfp_t flags, size_t orig_size, unsigned long caller) { unsigned long save_flags; void *objp; @@ -3299,6 +3305,10 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) if (unlikely(!cachep)) return NULL; + objp = kfence_alloc(cachep, orig_size, flags); + if (unlikely(objp)) + goto leave; + cache_alloc_debugcheck_before(cachep, flags); local_irq_save(save_flags); objp = __do_cache_alloc(cachep, flags); @@ -3309,6 +3319,7 @@ slab_alloc(struct kmem_cache *cachep, gfp_t flags, unsigned long caller) if (unlikely(slab_want_init_on_alloc(flags, cachep)) && objp) memset(objp, 0, cachep->object_size); +leave: slab_post_alloc_hook(cachep, objcg, flags, 1, &objp); return objp; } @@ -3414,6 +3425,11 @@ static void cache_flusharray(struct kmem_cache *cachep, struct array_cache *ac) static __always_inline void __cache_free(struct kmem_cache *cachep, void *objp, unsigned long caller) { + if (kfence_free(objp)) { + kmemleak_free_recursive(objp, cachep->flags); + return; + } + /* Put the object into the quarantine, don't touch it for now. */ if (kasan_slab_free(cachep, objp, _RET_IP_)) return; @@ -3479,7 +3495,7 @@ void ___cache_free(struct kmem_cache *cachep, void *objp, */ void *kmem_cache_alloc(struct kmem_cache *cachep, gfp_t flags) { - void *ret = slab_alloc(cachep, flags, _RET_IP_); + void *ret = slab_alloc(cachep, flags, cachep->object_size, _RET_IP_); trace_kmem_cache_alloc(_RET_IP_, ret, cachep->object_size, cachep->size, flags); @@ -3512,7 +3528,7 @@ int kmem_cache_alloc_bulk(struct kmem_cache *s, gfp_t flags, size_t size, local_irq_disable(); for (i = 0; i < size; i++) { - void *objp = __do_cache_alloc(s, flags); + void *objp = kfence_alloc(s, s->object_size, flags) ?: __do_cache_alloc(s, flags); if (unlikely(!objp)) goto error; @@ -3545,7 +3561,7 @@ kmem_cache_alloc_trace(struct kmem_cache *cachep, gfp_t flags, size_t size) { void *ret; - ret = slab_alloc(cachep, flags, _RET_IP_); + ret = slab_alloc(cachep, flags, size, _RET_IP_); ret = kasan_kmalloc(cachep, ret, size, flags); trace_kmalloc(_RET_IP_, ret, @@ -3571,7 +3587,7 @@ EXPORT_SYMBOL(kmem_cache_alloc_trace); */ void *kmem_cache_alloc_node(struct kmem_cache *cachep, gfp_t flags, int nodeid) { - void *ret = slab_alloc_node(cachep, flags, nodeid, _RET_IP_); + void *ret = slab_alloc_node(cachep, flags, nodeid, cachep->object_size, _RET_IP_); trace_kmem_cache_alloc_node(_RET_IP_, ret, cachep->object_size, cachep->size, @@ -3589,7 +3605,7 @@ void *kmem_cache_alloc_node_trace(struct kmem_cache *cachep, { void *ret; - ret = slab_alloc_node(cachep, flags, nodeid, _RET_IP_); + ret = slab_alloc_node(cachep, flags, nodeid, size, _RET_IP_); ret = kasan_kmalloc(cachep, ret, size, flags); trace_kmalloc_node(_RET_IP_, ret, @@ -3650,7 +3666,7 @@ static __always_inline void *__do_kmalloc(size_t size, gfp_t flags, cachep = kmalloc_slab(size, flags); if (unlikely(ZERO_OR_NULL_PTR(cachep))) return cachep; - ret = slab_alloc(cachep, flags, caller); + ret = slab_alloc(cachep, flags, size, caller); ret = kasan_kmalloc(cachep, ret, size, flags); trace_kmalloc(caller, ret, @@ -4138,18 +4154,24 @@ void __check_heap_object(const void *ptr, unsigned long n, struct page *page, bool to_user) { struct kmem_cache *cachep; - unsigned int objnr; + unsigned int objnr = 0; unsigned long offset; + bool is_kfence = is_kfence_address(ptr); ptr = kasan_reset_tag(ptr); /* Find and validate object. */ cachep = page->slab_cache; - objnr = obj_to_index(cachep, page, (void *)ptr); - BUG_ON(objnr >= cachep->num); + if (!is_kfence) { + objnr = obj_to_index(cachep, page, (void *)ptr); + BUG_ON(objnr >= cachep->num); + } /* Find offset within object. */ - offset = ptr - index_to_obj(cachep, page, objnr) - obj_offset(cachep); + if (is_kfence_address(ptr)) + offset = ptr - kfence_object_start(ptr); + else + offset = ptr - index_to_obj(cachep, page, objnr) - obj_offset(cachep); /* Allow address range falling entirely within usercopy region. */ if (offset >= cachep->useroffset && diff --git a/mm/slab_common.c b/mm/slab_common.c index f9ccd5dc13f3..6e35e273681a 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -12,6 +12,7 @@ #include #include #include +#include #include #include #include @@ -448,6 +449,9 @@ static int shutdown_cache(struct kmem_cache *s) /* free asan quarantined objects */ kasan_cache_shutdown(s); + if (!kfence_shutdown_cache(s)) + return -EBUSY; + if (__kmem_cache_shutdown(s) != 0) return -EBUSY; @@ -1171,7 +1175,7 @@ size_t ksize(const void *objp) if (unlikely(ZERO_OR_NULL_PTR(objp)) || !__kasan_check_read(objp, 1)) return 0; - size = __ksize(objp); + size = kfence_ksize(objp) ?: __ksize(objp); /* * We assume that ksize callers could use whole allocated area, * so we need to unpoison this area.