From patchwork Wed Mar 25 16:12:27 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alexander Potapenko X-Patchwork-Id: 11458263 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9FF0A913 for ; Wed, 25 Mar 2020 16:13:58 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6095120774 for ; Wed, 25 Mar 2020 16:13:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="Eoavv/bO" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6095120774 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 497466B007B; Wed, 25 Mar 2020 12:13:49 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 46DBE6B007D; Wed, 25 Mar 2020 12:13:49 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3AC686B007E; Wed, 25 Mar 2020 12:13:49 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0112.hostedemail.com [216.40.44.112]) by kanga.kvack.org (Postfix) with ESMTP id 202936B007B for ; Wed, 25 Mar 2020 12:13:49 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id EC5642BFC1 for ; Wed, 25 Mar 2020 16:13:48 +0000 (UTC) X-FDA: 76634380536.03.fall62_78935664e6f3b X-Spam-Summary: 2,0,0,2230771300a11d4f,d41d8cd98f00b204,3oon7xgykccklqnijwlttlqj.htrqnszc-rrpafhp.twl@flex--glider.bounces.google.com,,RULES_HIT:41:152:334:355:379:541:800:960:973:988:989:1260:1277:1313:1314:1345:1359:1431:1437:1516:1518:1535:1544:1593:1594:1711:1730:1747:1777:1792:2194:2197:2199:2200:2393:2559:2562:2904:3138:3139:3140:3141:3142:3152:3355:3653:3865:3867:3868:3870:3871:3872:3874:4119:4250:4605:5007:6119:6261:6653:6742:6743:7875:7974:8660:8784:9969:10004:11026:11473:11658:11914:12043:12048:12296:12297:12438:12555:12895:13148:13230:13846:14096:14097:14181:14394:14659:14721:21080:21221:21365:21433:21444:21451:21627:21740:21795:21990:30051:30054:30064:30069,0,RBL:209.85.221.74:@flex--glider.bounces.google.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:2,LUA_SUMMARY:none X-HE-Tag: fall62_78935664e6f3b X-Filterd-Recvd-Size: 8043 Received: from mail-wr1-f74.google.com (mail-wr1-f74.google.com [209.85.221.74]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Wed, 25 Mar 2020 16:13:48 +0000 (UTC) Received: by mail-wr1-f74.google.com with SMTP id f8so1378652wrp.1 for ; Wed, 25 Mar 2020 09:13:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=1/Qp93MRgGz2zAATzzImSzxxf/Q7LJs0a04QBt6vCuA=; b=Eoavv/bOMMPQ7/F30RbMbnA358EezMcW1w3DjDNCuPE+YiGYk2V3km0+nvUeQFpXU4 nKul8ASmwCshOHdhEhFeDBFvVIbpKAAzG/+qZkJzeW7usvF/LJeEQCb1EbWWI+JeZiWU lq5/6RJrp08FITrwGFvgk0YIoQyNJPsMD65zQdSg/lso8/mJZmACNjDtbgVNTPwQAsiz 0kHDP1Rt22sNYqhMEvxoh2DRlz+ijV0ih97ykN2WptDXTYsHeTnaGoUZTfa/sRvlijcF +w8O9OG1WO4Owp3fJyY4tuoYXBc6IVzQyKgDJtTjQW8oc3wdh7ciYB0EwGx1vZmWUHwU gFkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=1/Qp93MRgGz2zAATzzImSzxxf/Q7LJs0a04QBt6vCuA=; b=AbBJ1hFP/5bwUsZ2srlK7H62mFgjXb7QE1G/t6xWQ2dN/zIe6mVdU8ndI3VrhUF9jp ptwi+5JG4FCHs6kQxg1gcRmz/IWtC6Vw2t2c4AmyXA/cD6xJiTqlkSn7HCKX8cBRsraB IyGEzoKFZLZeXgF8F+yrdTfBl2Kflhj18xkIUlwCORd2D/GodE5WVX6hKySVblfRWoMQ bTLZEbhTrAsvJI0xE9D+F2eaTZyXxEZLfoWhtdxZs79SP1t7MroFkNnkjrryXhvAi6AD huAQmgVY2YXmWbyqGZdmmY3W+NjAPWR7y+u6LjruocdbOzKYNSKLAyUozNP0IVcI3TGz rUJw== X-Gm-Message-State: ANhLgQ0Zg0t2JlTaj4hvrXSLwww3O1GcshoLb+u5d/D9UJ9IQ3/gA8yE nOLQGj1TXS2AKGnZKVvWQjNIh12MAhQ= X-Google-Smtp-Source: ADFU+vuTgQHXpTp20DET2xBebeWvzPSM6yB5Roam+3SechTtO6BrGbXsbeY7g6xqqFCF6DzGWkeJuA0oBx0= X-Received: by 2002:adf:a21a:: with SMTP id p26mr4218863wra.102.1585152826967; Wed, 25 Mar 2020 09:13:46 -0700 (PDT) Date: Wed, 25 Mar 2020 17:12:27 +0100 In-Reply-To: <20200325161249.55095-1-glider@google.com> Message-Id: <20200325161249.55095-17-glider@google.com> Mime-Version: 1.0 References: <20200325161249.55095-1-glider@google.com> X-Mailer: git-send-email 2.25.1.696.g5e7596f4ac-goog Subject: [PATCH v5 16/38] kmsan: x86/asm: softirq: add KMSAN IRQ entry hooks From: glider@google.com To: Jens Axboe , Andy Lutomirski , Vegard Nossum , Dmitry Vyukov , Marco Elver , Andrey Konovalov , Christoph Hellwig , linux-mm@kvack.org Cc: glider@google.com, viro@zeniv.linux.org.uk, adilger.kernel@dilger.ca, akpm@linux-foundation.org, aryabinin@virtuozzo.com, ard.biesheuvel@linaro.org, arnd@arndb.de, hch@infradead.org, darrick.wong@oracle.com, davem@davemloft.net, dmitry.torokhov@gmail.com, ebiggers@google.com, edumazet@google.com, ericvh@gmail.com, gregkh@linuxfoundation.org, harry.wentland@amd.com, herbert@gondor.apana.org.au, iii@linux.ibm.com, mingo@elte.hu, jasowang@redhat.com, m.szyprowski@samsung.com, mark.rutland@arm.com, martin.petersen@oracle.com, schwidefsky@de.ibm.com, willy@infradead.org, mst@redhat.com, mhocko@suse.com, monstr@monstr.eu, pmladek@suse.com, cai@lca.pw, rdunlap@infradead.org, robin.murphy@arm.com, sergey.senozhatsky@gmail.com, rostedt@goodmis.org, tiwai@suse.com, tytso@mit.edu, tglx@linutronix.de, gor@linux.ibm.com, wsa@the-dreams.de X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Add assembly helpers to entry_64.S that invoke hooks from kmsan_entry.c and notify KMSAN about interrupts. Also call these hooks from kernel/softirq.c This is needed to switch between several KMSAN contexts holding function parameter metadata. Signed-off-by: Alexander Potapenko To: Alexander Potapenko Cc: Jens Axboe Cc: Andy Lutomirski Cc: Vegard Nossum Cc: Dmitry Vyukov Cc: Marco Elver Cc: Andrey Konovalov Cc: Christoph Hellwig Cc: linux-mm@kvack.org Acked-by: Andrey Konovalov --- v4: - moved softirq changes to this patch Change-Id: I3037d51672fe69d09e588b27adb2d9fdc6ad3a7d --- arch/x86/entry/entry_64.S | 16 ++++++++++++++++ kernel/softirq.c | 5 +++++ 2 files changed, 21 insertions(+) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 0e9504fabe526..03f5a32b0af4d 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -35,6 +35,7 @@ #include #include #include +#include #include #include #include @@ -575,6 +576,7 @@ SYM_CODE_START(interrupt_entry) 1: ENTER_IRQ_STACK old_rsp=%rdi save_ret=1 + KMSAN_INTERRUPT_ENTER /* We entered an interrupt context - irqs are off: */ TRACE_IRQS_OFF @@ -604,12 +606,14 @@ SYM_CODE_START_LOCAL(common_interrupt) addq $-0x80, (%rsp) /* Adjust vector to [-256, -1] range */ call interrupt_entry UNWIND_HINT_REGS indirect=1 + KMSAN_UNPOISON_PT_REGS call do_IRQ /* rdi points to pt_regs */ /* 0(%rsp): old RSP */ ret_from_intr: DISABLE_INTERRUPTS(CLBR_ANY) TRACE_IRQS_OFF + KMSAN_INTERRUPT_EXIT LEAVE_IRQ_STACK testb $3, CS(%rsp) @@ -801,6 +805,7 @@ SYM_CODE_START(\sym) .Lcommon_\sym: call interrupt_entry UNWIND_HINT_REGS indirect=1 + KMSAN_UNPOISON_PT_REGS call \do_sym /* rdi points to pt_regs */ jmp ret_from_intr SYM_CODE_END(\sym) @@ -908,15 +913,18 @@ apicinterrupt IRQ_WORK_VECTOR irq_work_interrupt smp_irq_work_interrupt .if \shift_ist != -1 subq $\ist_offset, CPU_TSS_IST(\shift_ist) + KMSAN_IST_ENTER(\shift_ist) .endif .if \read_cr2 movq %r12, %rdx /* Move CR2 into 3rd argument */ .endif + KMSAN_UNPOISON_PT_REGS call \do_sym .if \shift_ist != -1 + KMSAN_IST_EXIT(\shift_ist) addq $\ist_offset, CPU_TSS_IST(\shift_ist) .endif @@ -1079,7 +1087,9 @@ SYM_FUNC_START(do_softirq_own_stack) pushq %rbp mov %rsp, %rbp ENTER_IRQ_STACK regs=0 old_rsp=%r11 + KMSAN_SOFTIRQ_ENTER call __do_softirq + KMSAN_SOFTIRQ_EXIT LEAVE_IRQ_STACK regs=0 leaveq ret @@ -1466,9 +1476,12 @@ SYM_CODE_START(nmi) * done with the NMI stack. */ + KMSAN_NMI_ENTER movq %rsp, %rdi movq $-1, %rsi + KMSAN_UNPOISON_PT_REGS call do_nmi + KMSAN_NMI_EXIT /* * Return back to user mode. We must *not* do the normal exit @@ -1678,10 +1691,13 @@ end_repeat_nmi: call paranoid_entry UNWIND_HINT_REGS + KMSAN_NMI_ENTER /* paranoidentry do_nmi, 0; without TRACE_IRQS_OFF */ movq %rsp, %rdi movq $-1, %rsi + KMSAN_UNPOISON_PT_REGS call do_nmi + KMSAN_NMI_EXIT /* Always restore stashed CR3 value (see paranoid_entry) */ RESTORE_CR3 scratch_reg=%r15 save_reg=%r14 diff --git a/kernel/softirq.c b/kernel/softirq.c index 0427a86743a46..98c5f4062cbfe 100644 --- a/kernel/softirq.c +++ b/kernel/softirq.c @@ -11,6 +11,7 @@ #include #include +#include #include #include #include @@ -370,7 +371,9 @@ static inline void invoke_softirq(void) * it is the irq stack, because it should be near empty * at this stage. */ + kmsan_context_enter(); __do_softirq(); + kmsan_context_exit(); #else /* * Otherwise, irq_exit() is called on the task stack that can @@ -600,7 +603,9 @@ static void run_ksoftirqd(unsigned int cpu) * We can safely run softirq on inline stack, as we are not deep * in the task stack here. */ + kmsan_context_enter(); __do_softirq(); + kmsan_context_exit(); local_irq_enable(); cond_resched(); return;