From patchwork Tue Dec 12 10:28:55 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 13488892 X-Patchwork-Delegate: paul@paul-moore.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="JKjv6fYA" Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 6AC32F7 for ; Tue, 12 Dec 2023 02:29:19 -0800 (PST) Received: from pps.filterd (m0246632.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3BC7hh1r004073; Tue, 12 Dec 2023 10:29:01 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-11-20; bh=Mc6avwsJwUdwv06HUKlEkdrvcvLbKRHmxiNe/zxUa8U=; b=JKjv6fYAO+s8NAodJKgJRzfQqdEvEDRNDq86Pq/GA1NNdf0WuY5nkLfHpDPGFzltDup7 T2JsnlX01sW7s9w8GJ9TvycoB9F+2uAgrEK5m2x7NmOw7UpwKKV6MNkLuTEh6TJqQiir 3yokGbSrGInxLcEbbVFTbzgggSvpybWSqjtDw/rvltegv4v71visugv5FagX39QSREUC RgQ7L+tG77fKOvQvjkXEzD+C7Lpau04pu6mhixP5Q4ppdTSigB0xxiUDv0O8gA63w3/s E0mvOz9w6j13obhOlO9pTk7wTV8qlsIaLnifPdtwRHyXGTlW9H4CNF4MPZvd/wmdtPvB zg== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3uvfuu5939-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 12 Dec 2023 10:29:01 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3BC95OqQ003177; Tue, 12 Dec 2023 10:29:00 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3uvep6cmrx-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 12 Dec 2023 10:29:00 +0000 Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3BCASxHQ033207; Tue, 12 Dec 2023 10:29:00 GMT Received: from lab61.no.oracle.com (lab61.no.oracle.com [10.172.144.82]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3uvep6cmqk-2; Tue, 12 Dec 2023 10:29:00 +0000 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: Paul Moore , Eric Paris , audit@vger.kernel.org Cc: Ankur Arora Subject: [PATCH 1/2] audit: Vary struct audit_entry alignment Date: Tue, 12 Dec 2023 11:28:55 +0100 Message-Id: <20231212102857.803984-2-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231212102857.803984-1-haakon.bugge@oracle.com> References: <20231212102857.803984-1-haakon.bugge@oracle.com> Precedence: bulk X-Mailing-List: audit@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-12_04,2023-12-12_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 spamscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312120083 X-Proofpoint-GUID: yFg3laE4T9lPTOKeSu4z9Z74IHVuU50V X-Proofpoint-ORIG-GUID: yFg3laE4T9lPTOKeSu4z9Z74IHVuU50V We allocate struct audit_entry using kzalloc() which aligns the structure at its natural boundary and so uses the kmalloc-512 SLAB. That means that the lower order 9 bits are equal for these allocations. Which on architectures with limited L1D cache-sets width would cause conflict misses. With STIG compliant audit rules, up-to 26 L1D misses per syscall have been observed. This, of course, varies from boot to boot. Fix this by using a kmem_cache aligned at cacheline width, which greatly increases the entropy in the lower order VA bits. With this fix, we observe a maximum of 0.8 misses per syscall using the same set of audit rules. Testing: run "perf stat -d -r 5 ./syscall_lat", where syscall_lat is a simple getpid() loop, running 100 million rounds. This test is run 10 times, with a reboot in between. After each reboot, we wait until the last minute load is less than 1.0. We boot the kernel, v6.6-rc4, with "mitigations=off", in order to amplify the changes in the audit system. Base vs. base + this commit gives: ns per call: min avg max pstdev - 205 210 227 6.402000 + 203 203 209 0.954149 L1d misses per syscall: min avg max pstdev - 3.147 12.017 26.045 6.568284 + 0.012 0.103 0.817 0.238352 ipc: min avg max pstdev - 2.090 2.259 2.300 0.060075 + 2.320 2.329 2.330 0.003000 The above metrics are all improved, and the L1D misses per syscall has a significant reduction. Co-developed-by: Ankur Arora Signed-off-by: Ankur Arora Signed-off-by: Håkon Bugge --- kernel/auditfilter.c | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/kernel/auditfilter.c b/kernel/auditfilter.c index 8317a37dea0bb..b7597a63a3950 100644 --- a/kernel/auditfilter.c +++ b/kernel/auditfilter.c @@ -80,6 +80,7 @@ static void audit_free_lsm_field(struct audit_field *f) } } +static struct kmem_cache *entry_cache; static inline void audit_free_rule(struct audit_entry *e) { int i; @@ -93,7 +94,7 @@ static inline void audit_free_rule(struct audit_entry *e) audit_free_lsm_field(&erule->fields[i]); kfree(erule->fields); kfree(erule->filterkey); - kfree(e); + kmem_cache_free(entry_cache, e); } void audit_free_rule_rcu(struct rcu_head *head) @@ -108,13 +109,20 @@ static inline struct audit_entry *audit_init_entry(u32 field_count) struct audit_entry *entry; struct audit_field *fields; - entry = kzalloc(sizeof(*entry), GFP_KERNEL); + if (!entry_cache) { + entry_cache = kmem_cache_create("audit_entry", sizeof(*entry), 0, + SLAB_HWCACHE_ALIGN, NULL); + if (!entry_cache) + return NULL; + } + + entry = kmem_cache_zalloc(entry_cache, GFP_KERNEL); if (unlikely(!entry)) return NULL; fields = kcalloc(field_count, sizeof(*fields), GFP_KERNEL); if (unlikely(!fields)) { - kfree(entry); + kmem_cache_free(entry_cache, entry); return NULL; } entry->rule.fields = fields; From patchwork Tue Dec 12 10:28:57 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Haakon Bugge X-Patchwork-Id: 13488890 X-Patchwork-Delegate: paul@paul-moore.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=oracle.com header.i=@oracle.com header.b="kxVw7zDg" Received: from mx0b-00069f02.pphosted.com (mx0b-00069f02.pphosted.com [205.220.177.32]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 296E5AB for ; Tue, 12 Dec 2023 02:29:06 -0800 (PST) Received: from pps.filterd (m0333520.ppops.net [127.0.0.1]) by mx0b-00069f02.pphosted.com (8.17.1.19/8.17.1.19) with ESMTP id 3BC7hll0028870; Tue, 12 Dec 2023 10:29:03 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : mime-version : content-type : content-transfer-encoding; s=corp-2023-11-20; bh=tJc/FOodJSHcQG0F9yHAqD4iooQS++do2a0zd/mCcHQ=; b=kxVw7zDghT/+bZ+HLAPPA9j0iO6O7UOZ3+tratE1gO2WsnGL/P4v/h6q+66/WUJEE7m7 ac5TVlFFAyxt0tVoVHLl7cVR64Y5gbZdgxjefc3n6JEzH9LCpDBMb9ZcfLKiPmSijl1e 446mqnyqaX6U5fC5jq9NDETWqJx/CLuokPXvpturYCrjWyX6IISZ8TD/VvCcmm/+IHL2 pCmDi+S5xSzqYKg78StCue/qxRLeftFjIkQwmBK7eHd8lFA/T4Vi1dUQy4gxGcSbfgr7 bOVeAs3kGOdcOLrf3A/fdFw/Hf1tcC74Q3U8AdV/N4kXPQ9uat06GRIxZoBKEsC0X5UL Jw== Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.appoci.oracle.com [147.154.18.20]) by mx0b-00069f02.pphosted.com (PPS) with ESMTPS id 3uvg9d5auq-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 12 Dec 2023 10:29:03 +0000 Received: from pps.filterd (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (8.17.1.19/8.17.1.19) with ESMTP id 3BC9IEgN003222; Tue, 12 Dec 2023 10:29:03 GMT Received: from pps.reinject (localhost [127.0.0.1]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTPS id 3uvep6cmt7-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 12 Dec 2023 10:29:03 +0000 Received: from iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com [127.0.0.1]) by pps.reinject (8.17.1.5/8.17.1.5) with ESMTP id 3BCASxHU033207; Tue, 12 Dec 2023 10:29:02 GMT Received: from lab61.no.oracle.com (lab61.no.oracle.com [10.172.144.82]) by iadpaimrmta02.imrmtpd1.prodappiadaev1.oraclevcn.com (PPS) with ESMTP id 3uvep6cmqk-4; Tue, 12 Dec 2023 10:29:02 +0000 From: =?utf-8?q?H=C3=A5kon_Bugge?= To: Paul Moore , Eric Paris , audit@vger.kernel.org Cc: Ankur Arora Subject: [PATCH 2/2] audit: Apply special optimizations Date: Tue, 12 Dec 2023 11:28:57 +0100 Message-Id: <20231212102857.803984-4-haakon.bugge@oracle.com> X-Mailer: git-send-email 2.39.3 In-Reply-To: <20231212102857.803984-1-haakon.bugge@oracle.com> References: <20231212102857.803984-1-haakon.bugge@oracle.com> Precedence: bulk X-Mailing-List: audit@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=baseguard engine=ICAP:2.0.272,Aquarius:18.0.997,Hydra:6.0.619,FMLib:17.11.176.26 definitions=2023-12-12_04,2023-12-12_01,2023-05-22_02 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 phishscore=0 bulkscore=0 spamscore=0 malwarescore=0 mlxscore=0 mlxlogscore=999 adultscore=0 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2311290000 definitions=main-2312120083 X-Proofpoint-GUID: MBAKxHxuO-Rdt-Yz2-oa3drQzBdL3U9i X-Proofpoint-ORIG-GUID: MBAKxHxuO-Rdt-Yz2-oa3drQzBdL3U9i For the most time-consuming function, when running a syscall benchmark with STIG compliant audit rules: Overhead Command Shared Object Symbol ......... ............ ................. ........................ 27.62% syscall_lat [kernel.kallsyms] [k] __audit_filter_op we apply special optimizations, which speeds up the syscall performance by around 17% on an Intel Cascade Lake system. We run "perf stat -d -r 5 ./syscall_lat", where syscall_lat is a C application that measures average syscall latency from getpid() running 100 million rounds. Between each perf run, we reboot the system and waits until the last minute load is less than 1.0. We boot the kernel, v6.6-rc4, with "mitigations=off", in order to amplify the changes in the audit system. Let the base kernel be v6.6-rc4 with booted with "audit=1" and "mitigations=off" and with the commit "audit: Vary struct audit_entry alignment" on an Intel Cascade Lake system. The following three metrics are reported, nanoseconds per syscall, L1D misses per syscall, and finally Intructions Per Cycle, ipc. Base vs. base + this commit gives: ns per call: min avg max pstdev - 203 203 209 0.954149 + 173 173 178 0.884534 L1d misses per syscall: min avg max pstdev - 0.012 0.103 0.817 0.238352 + 0.010 0.209 1.235 0.399416 ipc: min avg max pstdev - 2.320 2.329 2.330 0.003000 + 2.430 2.436 2.440 0.004899 Signed-off-by: Håkon Bugge --- kernel/auditsc.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/kernel/auditsc.c b/kernel/auditsc.c index 6f0d6fb6523fa..84d0dfe75a4ac 100644 --- a/kernel/auditsc.c +++ b/kernel/auditsc.c @@ -822,6 +822,7 @@ static int audit_in_mask(const struct audit_krule *rule, unsigned long val) * parameter can be NULL, but all others must be specified. * Returns 1/true if the filter finds a match, 0/false if none are found. */ +#pragma GCC optimize("unswitch-loops", "align-loops=16", "align-jumps=16") static int __audit_filter_op(struct task_struct *tsk, struct audit_context *ctx, struct list_head *list, @@ -841,6 +842,7 @@ static int __audit_filter_op(struct task_struct *tsk, } return 0; } +#pragma GCC reset_options /** * audit_filter_uring - apply filters to an io_uring operation