From patchwork Thu Jan 18 12:41:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: lizhe.67@bytedance.com X-Patchwork-Id: 13522813 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8DFADC4707B for ; Thu, 18 Jan 2024 12:41:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2106E6B009E; Thu, 18 Jan 2024 07:41:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1C0C96B009F; Thu, 18 Jan 2024 07:41:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 03B4D6B00A0; Thu, 18 Jan 2024 07:41:32 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id E1C936B009E for ; Thu, 18 Jan 2024 07:41:32 -0500 (EST) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id AE5CE80C7D for ; Thu, 18 Jan 2024 12:41:32 +0000 (UTC) X-FDA: 81692392824.19.532337F Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf01.hostedemail.com (Postfix) with ESMTP id 0BE8140011 for ; Thu, 18 Jan 2024 12:41:30 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=KIPAOalC; spf=pass (imf01.hostedemail.com: domain of lizhe.67@bytedance.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=lizhe.67@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1705581691; a=rsa-sha256; cv=none; b=LxJ0+BgKS3V3hjOEZkBNRyxeE16grAw8X5gLRIsa2sDsbDyHZOAwQ1IgwizJpquNfPPzA5 roNHEwCcwpkxKYpvF3fCStfvdnmBboJu1YHHO7esCF5zN0Az4Vf8JSd5parV4PMzqWwTpb WY1t7xNxpOtT9PVQuxir7UeOvppsdfE= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=KIPAOalC; spf=pass (imf01.hostedemail.com: domain of lizhe.67@bytedance.com designates 209.85.210.176 as permitted sender) smtp.mailfrom=lizhe.67@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1705581691; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7X0c+MMnT5ixfOkPAVKJUdxf0xJI7ufL6KL0JiMq+9I=; b=Hu/Dxd4gqgzNrqmxcLnfO/mswzMjCcTHcW/d66V/v5P9UArXMr2S9FVRvzQ9/5IlhBphuL 0XSW6hzvE1JTkpDQwDyluUnH+fxEj+2wh9icMSpHMH9+Kjvhu5HKpXWT7ah7fyEDgqdd/n +6W57PowHfo8Dnztz+ycxNJw3B6hmhc= Received: by mail-pf1-f176.google.com with SMTP id d2e1a72fcca58-6d9af1f52bcso6450426b3a.3 for ; Thu, 18 Jan 2024 04:41:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1705581690; x=1706186490; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7X0c+MMnT5ixfOkPAVKJUdxf0xJI7ufL6KL0JiMq+9I=; b=KIPAOalCRHPUTWlmhkWVjzvUNAbeG5edrg+Jsa1/ZjZQDmZ2rZAXqX7sX+bdewspn0 2wH3NMculNSLb22V9lArOHX0utRM+FhwuDt/dyUXulGThRIpyOIVCP5MXIAAjN8xV0NW sIsHJVxtL8kRowH0ykpBSblIzMSWJvAnwKIWqFphDat4Zgov2fH4X3ef69g2K+fb9H6t 8QpR0ULwoN3UFR9WPbfoYFr7/uyzGxGijVme5XwEkGtzSBzoOV4GtAfJQo/tbFoURvtd 15YiMKpFLrYfjt+Plzgh1v6c1/H2QU3nMiN8Fys80Qt3K/KpBnt1ZVmJQV1KoF2ipFhi s3Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1705581690; x=1706186490; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7X0c+MMnT5ixfOkPAVKJUdxf0xJI7ufL6KL0JiMq+9I=; b=eZgnMD1xHrSK1aKVLNpH8bxwkGXLoYEaChjDMKb/qRbjLl79mbhverziqEN8xUOp7L 67Ir4eqhDW+tUvJx8iCYQFnVNecxqhHEzyXwmHTWk/Ca2oDXg7DgMhwIRmcxQsPi46l0 JiTGaAgrxr1tGGtDf9GAdjTd2l0ndahdriN93HKy2st2Z9yanBO3pyfvqKNa4aBuLDpl kFXIAuN8T3eqT1rOODXsdOZCZP+X+3jrmB3PhqYneAuk3BHLL8c9S5hQQuHHpZi4XtSv jM024ibkKWFwf0oyZcjJFnepIO4EziQ/uWHa1l3g/fxltbu/sRh7hvkw+N99WIzZMNfs R86w== X-Gm-Message-State: AOJu0Yza0AlVcZavMgo4/C1dnYYu0mnBr7xeEikD4Vc2Q4Hj+LWFimM8 6d6lQUbLyzhBaxsbFdJO5bqEuEgRxMre8amIboe6IWit2kDn8RKkMb4iugZXDmU= X-Google-Smtp-Source: AGHT+IGBdBisqTsaUdtEa3qeUxnfNMREnCCIw38YZPXu7WLO84oXd/0ARs5+MTwtlPtZtHQY9MAdGQ== X-Received: by 2002:aa7:998a:0:b0:6db:dae:c5aa with SMTP id k10-20020aa7998a000000b006db0daec5aamr709361pfh.63.1705581689774; Thu, 18 Jan 2024 04:41:29 -0800 (PST) Received: from GQ6QX3JCW2.bytedance.net ([203.208.189.13]) by smtp.gmail.com with ESMTPSA id y17-20020a056a00191100b006d977f70cd5sm3199744pfi.23.2024.01.18.04.41.25 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Thu, 18 Jan 2024 04:41:29 -0800 (PST) From: lizhe.67@bytedance.com To: ryabinin.a.a@gmail.com, glider@google.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, akpm@linux-foundation.org Cc: kasan-dev@googlegroups.com, linux-mm@kvack.org, lizefan.x@bytedance.com, lizhe.67@bytedance.com Subject: [RFC 1/2] kasan: introduce mem track feature base on kasan Date: Thu, 18 Jan 2024 20:41:08 +0800 Message-ID: <20240118124109.37324-2-lizhe.67@bytedance.com> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240118124109.37324-1-lizhe.67@bytedance.com> References: <20240118124109.37324-1-lizhe.67@bytedance.com> MIME-Version: 1.0 X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: 0BE8140011 X-Stat-Signature: 4pqu38xhdtqp7mmzuej7n37ey843rsur X-Rspam-User: X-HE-Tag: 1705581690-629985 X-HE-Meta: U2FsdGVkX1+ZX7skLrZrP7oF+Vz8KZeR7IK5fsFizPiVea2vTy8WHBscdCg+XiDmPX3CoOvnLRUi64m/L4RLzVhcOwF8NISQsu2nj2QMrVjNW7/+0sKXSZ4IGJqE+y2Gs/woEjdYPl7NfFqjqE2yepdT2pDuOltwDyNWrm+Fou0IUy90+rQ6ZVuqwlAnqTi/481LInhByJXJOSm9V1sLlD+uxUKvoJDwdA6zF/cDkBaJ7D+sF8BEhlnXklxj97qjLkAZvV247rmcQC4WNH7ZGOayPzgg886LXd+CdB7J8CqoyhjL7/Wx5FT2ds7uc+tupa7HDUJjxx0nqdLmNWdWKBay2/Naqp8liDiHUNPCL0q2Ce4Fecs4ft5SDAIpRY6Ft+q5grGDPyDNb7SWg8JSeFb6p0pA66+4QgCse84iYPb1sOTlLvMtVhksUhJQmEm4AkcmR/S6ZhMeIQ7JWAvW1NsC2CLraWr9uEOkdwR84XXn7exDTZ8ZnooPWoVVdueOaNu5d5QdUu4z2AO59j/E8SlDm71jhpM1JM9Ri69whGyX7MXpUs3Gf0pYjT3UkmPci4NRjcjVChtaQhPsbPMCgelzX6HVSvURcnbvk5Cir9nez5VHuGT3Kxm80zwOrkycoc5L31qzy+RBSS97GKE5euOf9IJ+0ePKKAl5ezDGD55zycZ5bkerDRsrXypBKGYd+hj5BdiMLNRra6mixq/cezHamz552H5QkTygvWZ3JrL7q5dEGRreYWgAnG6rl0o9Pl48nlCYiEhmofF7bLZSgK77ozVPSvnYnBYqvucXqhv+l8f5x+1wI0b4ikLqbcVlWzJuirRuz7QgAwe1lMq+w1A+BoNLCTsv4WBjA5Ai9J1vdiQRNcQasIFAP9nyx/hczurQEA6/CbDqjWlZ1NUUOJutf+BjMu4ZMwsYn6jRTwAmdHd+oKOqEysnscEX6ETsJO7LS5iebIqRWoZ/EW2 PDysPsBx wMQD2rKyLVzqd4q9yxWDgPGfSl0AAF1hzyoFt0NDqMAj7KM3ttD7e70+7/vNq60yWbJaWa8QT4HNQXWTIF9vN9tvw9oX94QWFdb6bfwSszxW9dC77RgMjUcBh3K0iPPMaMYvMBSrNZ6L8jfSpJy3E7iYAN0ySfRUXZU7L04NOiiUsVJxnWhz6UTyrBZAaVbPWK0CX/ZbeJVK7WLQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: From: Li Zhe This is a feature based on KASAN_GENERIC. The current implementation of kasan can help us locate memory's problems such as out-of-bounds, use-after-free, etc. But it cannot identify memory tramples on allocated memory by software. This type of problem may appear in our daily development. Generally, the phenomenon is rather strange and problem is difficult to locate. With this tool, we can easily locate memory corruption on allocated memory. In the current kernel implementation, we use bits 0-2 of each shadow memory byte to store how many bytes in the 8 byte memory corresponding to the shadow memory byte can be accessed. In addition, for inaccessible memory, the highest bit of its shadow mem is 1. Therefore, we can use the free bits 3-6 of shadow mem to record the track information corresponding to 8-byte of memory, that is, one bit records track information of 2 bytes. If the track bit of the shadow mem corresponding to a certain memory is 1, it means that the corresponding 2-byte memory is tracked. Of course, if we configure a byte to be tracked, when we access its paired byte, the track check will also be successfully triggered, which will cause us some interference. But for this type of false positives, we can easily identify them by checking kasan logs. And I think this shortcoming should not overshadow the convenience that this feature brings to our debugging. Signed-off-by: Li Zhe --- lib/Kconfig.kasan | 9 ++ mm/kasan/generic.c | 276 +++++++++++++++++++++++++++++++++++--- mm/kasan/report_generic.c | 6 + 3 files changed, 275 insertions(+), 16 deletions(-) diff --git a/lib/Kconfig.kasan b/lib/Kconfig.kasan index e6eda054ab27..d96e28757fb7 100644 --- a/lib/Kconfig.kasan +++ b/lib/Kconfig.kasan @@ -183,6 +183,15 @@ config KASAN_VMALLOC With Hardware Tag-Based KASAN, only non-executable VM_ALLOC mappings are checked. There is no additional memory usage. +config KASAN_MEM_TRACK + bool "Capture allocated memory corruption based on KASAN" + depends on KASAN_GENERIC && KASAN_OUTLINE + help + Enable memory tracking bases on kasan. This is a tools to capture + memory corruption on allocated memory. + + If unsure, say N. + config KASAN_KUNIT_TEST tristate "KUnit-compatible tests of KASAN bug detection capabilities" if !KUNIT_ALL_TESTS depends on KASAN && KUNIT && TRACEPOINTS diff --git a/mm/kasan/generic.c b/mm/kasan/generic.c index 24c13dfb1e94..a204ddcbaa3f 100644 --- a/mm/kasan/generic.c +++ b/mm/kasan/generic.c @@ -42,9 +42,94 @@ * depending on memory access size X. */ -static __always_inline bool memory_is_poisoned_1(const void *addr) +#ifdef CONFIG_KASAN_MEM_TRACK +#define KASAN_SHADOW_VALUE_MASK_ONE_BYTE 0x07 +#define KASAN_TRACK_VALUE_MASK_ONE_BYTE 0x78 +#define KASAN_SHADOW_VALUE_MASK_TWO_BYTE 0x0707 +#define KASAN_SHADOW_VALUE_MASK_EIGHT_BYTE 0x0707070707070707 +#define KASAN_TRACK_VALUE_MASK_EIGHT_BYTE 0x7878787878787878 +#define KASAN_TRACK_VALUE_OFFSET 3 +static __always_inline bool is_poison_value_1_byte(s8 shadow_value) +{ + if (shadow_value & 0x80) + return true; + return false; +} + +static __always_inline bool is_poison_value_8_byte(u64 shadow_value) +{ + if (shadow_value & 0x8080808080808080) + return true; + return false; +} + +static __always_inline s8 to_shadow_value_1_byte(s8 shadow_value) +{ + if (is_poison_value_1_byte(shadow_value)) + return shadow_value; + return shadow_value & KASAN_SHADOW_VALUE_MASK_ONE_BYTE; +} + +static __always_inline s8 to_track_value_1_byte(s8 shadow_value) +{ + if (is_poison_value_1_byte(shadow_value)) + return shadow_value; + return (shadow_value & KASAN_TRACK_VALUE_MASK_ONE_BYTE) >> + KASAN_TRACK_VALUE_OFFSET; +} + +static __always_inline u64 to_shadow_value_8_byte(u64 shadow_value) +{ + if (is_poison_value_8_byte(shadow_value)) + return shadow_value; + return shadow_value & KASAN_SHADOW_VALUE_MASK_EIGHT_BYTE; +} + +static __always_inline u64 to_track_value_8_byte(u64 shadow_value) +{ + if (is_poison_value_8_byte(shadow_value)) + return shadow_value; + return shadow_value & KASAN_TRACK_VALUE_MASK_EIGHT_BYTE; +} + +static __always_inline s8 get_shadow_value_1_byte(const void *addr) { s8 shadow_value = *(s8 *)kasan_mem_to_shadow(addr); + return to_shadow_value_1_byte(shadow_value); +} + +static __always_inline u16 get_shadow_value_2_byte(const void *addr) +{ + u16 shadow_value = *(u16 *)kasan_mem_to_shadow(addr); + + return shadow_value & KASAN_SHADOW_VALUE_MASK_TWO_BYTE; +} +#else +static __always_inline s8 to_shadow_value_1_byte(s8 shadow_value) +{ + return shadow_value; +} +static __always_inline u64 to_shadow_value_8_byte(u64 shadow_value) +{ + return shadow_value; +} +static __always_inline s8 get_shadow_value_1_byte(const void *addr) +{ + return *(s8 *)kasan_mem_to_shadow(addr); +} +static __always_inline u16 get_shadow_value_2_byte(const void *addr) +{ + return *(u16 *)kasan_mem_to_shadow(addr); +} +static __always_inline bool memory_is_tracked(const void *addr, size_t size) +{ + return 0; +} +#endif + +static __always_inline bool memory_is_poisoned_1(const void *addr) +{ + s8 shadow_value = get_shadow_value_1_byte(addr); if (unlikely(shadow_value)) { s8 last_accessible_byte = (unsigned long)addr & KASAN_GRANULE_MASK; @@ -57,34 +142,30 @@ static __always_inline bool memory_is_poisoned_1(const void *addr) static __always_inline bool memory_is_poisoned_2_4_8(const void *addr, unsigned long size) { - u8 *shadow_addr = (u8 *)kasan_mem_to_shadow(addr); - /* * Access crosses 8(shadow size)-byte boundary. Such access maps * into 2 shadow bytes, so we need to check them both. */ if (unlikely((((unsigned long)addr + size - 1) & KASAN_GRANULE_MASK) < size - 1)) - return *shadow_addr || memory_is_poisoned_1(addr + size - 1); + return get_shadow_value_1_byte(addr) || memory_is_poisoned_1(addr + size - 1); return memory_is_poisoned_1(addr + size - 1); } static __always_inline bool memory_is_poisoned_16(const void *addr) { - u16 *shadow_addr = (u16 *)kasan_mem_to_shadow(addr); - /* Unaligned 16-bytes access maps into 3 shadow bytes. */ if (unlikely(!IS_ALIGNED((unsigned long)addr, KASAN_GRANULE_SIZE))) - return *shadow_addr || memory_is_poisoned_1(addr + 15); + return get_shadow_value_2_byte(addr) || memory_is_poisoned_1(addr + 15); - return *shadow_addr; + return get_shadow_value_2_byte(addr); } -static __always_inline unsigned long bytes_is_nonzero(const u8 *start, +static __always_inline unsigned long bytes_is_nonzero(const s8 *start, size_t size) { while (size) { - if (unlikely(*start)) + if (unlikely(to_shadow_value_1_byte(*start))) return (unsigned long)start; start++; size--; @@ -93,7 +174,7 @@ static __always_inline unsigned long bytes_is_nonzero(const u8 *start, return 0; } -static __always_inline unsigned long memory_is_nonzero(const void *start, +static __always_inline unsigned long shadow_val_is_nonzero(const void *start, const void *end) { unsigned int words; @@ -113,7 +194,7 @@ static __always_inline unsigned long memory_is_nonzero(const void *start, words = (end - start) / 8; while (words) { - if (unlikely(*(u64 *)start)) + if (unlikely(to_shadow_value_8_byte(*(u64 *)start))) return bytes_is_nonzero(start, 8); start += 8; words--; @@ -126,7 +207,7 @@ static __always_inline bool memory_is_poisoned_n(const void *addr, size_t size) { unsigned long ret; - ret = memory_is_nonzero(kasan_mem_to_shadow(addr), + ret = shadow_val_is_nonzero(kasan_mem_to_shadow(addr), kasan_mem_to_shadow(addr + size - 1) + 1); if (unlikely(ret)) { @@ -135,7 +216,7 @@ static __always_inline bool memory_is_poisoned_n(const void *addr, size_t size) s8 last_accessible_byte = (unsigned long)last_byte & KASAN_GRANULE_MASK; if (unlikely(ret != (unsigned long)last_shadow || - last_accessible_byte >= *last_shadow)) + last_accessible_byte >= to_shadow_value_1_byte(*last_shadow))) return true; } return false; @@ -161,6 +242,168 @@ static __always_inline bool memory_is_poisoned(const void *addr, size_t size) return memory_is_poisoned_n(addr, size); } +#ifdef CONFIG_KASAN_MEM_TRACK +static __always_inline s8 get_track_value(const void *addr) +{ + s8 shadow_value = *(s8 *)kasan_mem_to_shadow(addr); + + /* In the early stages of system startup, when Kasan is not fully ready, + * some illegal values may be obtained. Ignore it. + */ + if (unlikely(shadow_value & 0x80)) + return 0; + return (shadow_value >> KASAN_TRACK_VALUE_OFFSET); +} + +/* ================================== size : 1 2 3 4 5 6 7 8 */ +static const s8 kasan_track_mask_odd_array[] = {0x01, 0x03, 0x03, 0x07, 0x07, 0x0f, 0x0f}; +static const s8 kasan_track_mask_even_array[] = {-1, 0x01, -1, 0x03, -1, 0x07, -1, 0x0f}; +static s8 kasan_track_mask_odd(size_t size) +{ + return kasan_track_mask_odd_array[size - 1]; +} + +static s8 kasan_track_mask_even(size_t size) +{ + return kasan_track_mask_even_array[size - 1]; +} + +/* check with addr do not cross 8(shadow size)-byte boundary */ +static __always_inline bool _memory_is_tracked(const void *addr, size_t size) +{ + s8 mask; + u8 offset = (unsigned long)addr & KASAN_GRANULE_MASK; + + if ((unsigned long)addr & 0x01) + mask = kasan_track_mask_odd(size); + else + mask = kasan_track_mask_even(size); + + return unlikely(get_track_value(addr) & (mask << (offset >> 1))); +} + +static __always_inline bool memory_is_tracked_1(const void *addr) +{ + u8 last_accessible_byte = (unsigned long)addr & KASAN_GRANULE_MASK; + + return unlikely(get_track_value(addr) & (0x01 << (last_accessible_byte >> 1))); +} + +static __always_inline bool memory_is_tracked_2_4_8(const void *addr, size_t size) +{ + /* + * Access crosses 8(shadow size)-byte boundary. Such access maps + * into 2 shadow bytes, so we need to check them both. + */ + if (unlikely((((unsigned long)addr + size - 1) & KASAN_GRANULE_MASK) < size - 1)) { + u8 part = (unsigned long)addr & KASAN_GRANULE_MASK; + + part = 8 - part; + return ((unlikely(get_track_value(addr)) && _memory_is_tracked(addr, part)) || + _memory_is_tracked(addr + part, size - part)); + } + + return _memory_is_tracked(addr, size); +} + +static __always_inline bool memory_is_tracked_16(const void *addr) +{ + /* Unaligned 16-bytes access maps into 3 shadow bytes. */ + if (unlikely(!IS_ALIGNED((unsigned long)addr, KASAN_GRANULE_SIZE))) { + u8 part = (unsigned long)addr & KASAN_GRANULE_MASK; + + part = 8 - part; + return ((unlikely(get_track_value(addr)) && _memory_is_tracked(addr, part)) || + _memory_is_tracked(addr + part, 8) || + _memory_is_tracked(addr + part + 8, 8 - part)); + } + + return unlikely(get_track_value(addr) || get_track_value(addr + 8)); +} + +static __always_inline unsigned long track_bytes_is_nonzero(const s8 *start, + size_t size) +{ + while (size) { + if (unlikely(to_track_value_1_byte(*start))) + return (unsigned long)start; + start++; + size--; + } + + return 0; +} + +static __always_inline unsigned long track_val_is_nonzero(const void *start, + const void *end) +{ + unsigned int words; + unsigned long ret; + unsigned int prefix = (unsigned long)start % 8; + + if (end - start <= 16) + return track_bytes_is_nonzero(start, end - start); + + if (prefix) { + prefix = 8 - prefix; + ret = track_bytes_is_nonzero(start, prefix); + if (unlikely(ret)) + return ret; + start += prefix; + } + + words = (end - start) / 8; + while (words) { + if (unlikely(to_track_value_8_byte(*(u64 *)start))) + return track_bytes_is_nonzero(start, 8); + start += 8; + words--; + } + + return track_bytes_is_nonzero(start, (end - start) % 8); +} + +static __always_inline bool memory_is_tracked_n(const void *addr, size_t size) +{ + unsigned long ret; + + ret = track_val_is_nonzero(kasan_mem_to_shadow(addr), + kasan_mem_to_shadow(addr + size - 1) + 1); + + if (unlikely(ret)) { + const void *last_byte = addr + size - 1; + s8 *last_shadow = (s8 *)kasan_mem_to_shadow(last_byte); + + if (unlikely(ret != (unsigned long)last_shadow || + _memory_is_tracked( + (void *)((unsigned long)last_byte & ~KASAN_GRANULE_MASK), + ((unsigned long)last_byte & KASAN_GRANULE_MASK) + 1))) + return true; + } + return false; +} + +static __always_inline bool memory_is_tracked(const void *addr, size_t size) +{ + if (__builtin_constant_p(size)) { + switch (size) { + case 1: + return memory_is_tracked_1(addr); + case 2: + case 4: + case 8: + return memory_is_tracked_2_4_8(addr, size); + case 16: + return memory_is_tracked_16(addr); + default: + BUILD_BUG(); + } + } + + return memory_is_tracked_n(addr, size); +} +#endif + static __always_inline bool check_region_inline(const void *addr, size_t size, bool write, unsigned long ret_ip) @@ -177,7 +420,8 @@ static __always_inline bool check_region_inline(const void *addr, if (unlikely(!addr_has_metadata(addr))) return !kasan_report(addr, size, write, ret_ip); - if (likely(!memory_is_poisoned(addr, size))) + if ((likely(!memory_is_poisoned(addr, size))) && + (!write || likely(!memory_is_tracked(addr, size)))) return true; return !kasan_report(addr, size, write, ret_ip); @@ -196,7 +440,7 @@ bool kasan_byte_accessible(const void *addr) if (!kasan_arch_is_ready()) return true; - shadow_byte = READ_ONCE(*(s8 *)kasan_mem_to_shadow(addr)); + shadow_byte = (s8)to_shadow_value_1_byte(READ_ONCE(*(s8 *)kasan_mem_to_shadow(addr))); return shadow_byte >= 0 && shadow_byte < KASAN_GRANULE_SIZE; } diff --git a/mm/kasan/report_generic.c b/mm/kasan/report_generic.c index f5b8e37b3805..e264c5f3c3e6 100644 --- a/mm/kasan/report_generic.c +++ b/mm/kasan/report_generic.c @@ -120,6 +120,12 @@ static const char *get_shadow_bug_type(struct kasan_report_info *info) case KASAN_VMALLOC_INVALID: bug_type = "vmalloc-out-of-bounds"; break; +#ifdef CONFIG_KASAN_MEM_TRACK + default: + if (!((*shadow_addr) & 0x80)) + bug_type = "memory-track"; + break; +#endif } return bug_type;