From patchwork Mon Feb 24 20:30:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Michel Lespinasse X-Patchwork-Id: 11401517 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0C55C14BC for ; Mon, 24 Feb 2020 20:31:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id C100E20675 for ; Mon, 24 Feb 2020 20:31:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="anyjOpnq" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org C100E20675 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B61EE6B009F; Mon, 24 Feb 2020 15:31:43 -0500 (EST) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id B12E56B00A1; Mon, 24 Feb 2020 15:31:43 -0500 (EST) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 98F436B00A2; Mon, 24 Feb 2020 15:31:43 -0500 (EST) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0059.hostedemail.com [216.40.44.59]) by kanga.kvack.org (Postfix) with ESMTP id 7C6E06B009F for ; Mon, 24 Feb 2020 15:31:43 -0500 (EST) Received: from smtpin28.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 33229181AC9CB for ; Mon, 24 Feb 2020 20:31:43 +0000 (UTC) X-FDA: 76526166486.28.blood45_2c0baa40c0a48 X-Spam-Summary: 2,0,0,392f84b4231d4031,d41d8cd98f00b204,3rtjuxgykcfwq4fe8haiiaf8.6igfchor-ggep46e.ila@flex--walken.bounces.google.com,,RULES_HIT:41:152:355:379:541:800:960:973:988:989:1260:1277:1313:1314:1345:1359:1437:1516:1518:1535:1543:1593:1594:1605:1711:1730:1747:1777:1792:2194:2199:2393:2559:2562:2693:2901:2915:3138:3139:3140:3141:3142:3152:3865:3866:3867:3868:3870:3871:3872:3874:4117:4321:5007:6119:6120:6261:6653:7875:7903:8660:9969:10004:10400:11026:11658:11914:12043:12048:12291:12296:12297:12438:12555:12683:12895:12986:13141:13148:13161:13229:13230:14096:14097:14181:14659:14721:21080:21324:21444:21451:21627:21740:21795:21987:21990:30003:30051:30054:30069:30070:30079,0,RBL:209.85.216.73:@flex--walken.bounces.google.com:.lbl8.mailshell.net-66.100.201.100 62.18.0.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: blood45_2c0baa40c0a48 X-Filterd-Recvd-Size: 6681 Received: from mail-pj1-f73.google.com (mail-pj1-f73.google.com [209.85.216.73]) by imf07.hostedemail.com (Postfix) with ESMTP for ; Mon, 24 Feb 2020 20:31:42 +0000 (UTC) Received: by mail-pj1-f73.google.com with SMTP id d7so411396pjx.6 for ; Mon, 24 Feb 2020 12:31:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=fF9G1om+3zD9UiRxzXGgjpBttylL5tHFqZGsQikK5a8=; b=anyjOpnqEIA6JB1h4gyDURU3XcceQXgd85VoQys20OoNSPiV/9UXyPV8zOWsVAO2GS 9KdjP/lk2LcvElu/pX9CriaeCyaUHO00fYTPPqBR4dN1BvI0PcV1JGzT4Nz7KqT0KQtQ NSdnaz6HUbcg8ERiOKKqjrhHRNq9cgO59ISk83GJqELVMhEF6wnhM07ggz/yVveeSEx0 pPPmI62yOoWQHDzEa2DeS+O5rp6ZlgPdoqtZbXFJ2w5aIycZnoOFj1QPQpfXEIC50EPS M7JMyKPWdfx7vAQYmmUKEAC4A8GAeWgj7caegiwmDgexEE7N7Ap/2EiYVgielakQ8VXg C8Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=fF9G1om+3zD9UiRxzXGgjpBttylL5tHFqZGsQikK5a8=; b=oHSFq7oER9t6JuDNUbfKyQbFtnk1pr3+99/7ObaHgQZyipVzWokf0NoMfuh1e28zbs AILAlo2rQD+bHKmLkZY/og1ZiHiGrOUdQtuA74rukMVjTsS3ttcwjN/mEl+2YVkp91j6 oYubEZF4rxdeXP+Vw0Gg4X8TxJQdkiq7yxhGyDnrGUORIsi0HCwX9xxCrssfYj6sgCOT jZ2Vgda2nlrslBBAvmOgxuArSmZhO7UQhThWKOJGSWFOuuTL/in/okeXh5liiEO/DALN J+UMk3xcPHmN1tgfAMOhJ28Lw8tquiPWnmHDxLkTANKIvVK0BZIYlcGmFLHJ4bmhZPaJ J8ng== X-Gm-Message-State: APjAAAVzJbNF5Hk3iNgNBzUx1DPYrnCSCFsyPSA5KrDfRHwQopZ8thd2 o7znCvMfsAilY5P7FVjGB5+gxuV9C+c= X-Google-Smtp-Source: APXvYqxWJLA2+YbvGOULRF/z+UOc+lTeiHoJRxsCPIzB6u9S5yojVHZhb7PNVChBpF5h7KvFQN5wnJiemZ4= X-Received: by 2002:a63:da45:: with SMTP id l5mr12775702pgj.273.1582576301384; Mon, 24 Feb 2020 12:31:41 -0800 (PST) Date: Mon, 24 Feb 2020 12:30:50 -0800 In-Reply-To: <20200224203057.162467-1-walken@google.com> Message-Id: <20200224203057.162467-18-walken@google.com> Mime-Version: 1.0 References: <20200224203057.162467-1-walken@google.com> X-Mailer: git-send-email 2.25.0.265.gbab2e86ba0-goog Subject: [RFC PATCH 17/24] x86 fault handler: implement range locking From: Michel Lespinasse To: Peter Zijlstra , Andrew Morton , Laurent Dufour , Vlastimil Babka , Matthew Wilcox , "Liam R . Howlett" , Jerome Glisse , Davidlohr Bueso , David Rientjes Cc: linux-mm , Michel Lespinasse X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Change the x86 fault handler to implement range locking. Initially we try to lock a pmd sized range around the faulting address, which is appropriate for anon vmas. After finding the correct vma for the faulting address, we verify that it is anonymous and fall back to a coarse grained lock if necessary. If a fine grained lock is workable, we copy the vma of record into a pseudo-vma and release the mm_vma_lock before handling the fault. Signed-off-by: Michel Lespinasse --- arch/x86/mm/fault.c | 40 ++++++++++++++++++++++++++++++++-------- 1 file changed, 32 insertions(+), 8 deletions(-) diff --git arch/x86/mm/fault.c arch/x86/mm/fault.c index 52333272e14e..1e37284d373c 100644 --- arch/x86/mm/fault.c +++ arch/x86/mm/fault.c @@ -941,6 +941,7 @@ bad_area(struct pt_regs *regs, unsigned long error_code, unsigned long address, struct vm_area_struct *vma, struct mm_lock_range *range) { + struct mm_struct *mm; u32 pkey = 0; int si_code = SEGV_MAPERR; @@ -984,7 +985,10 @@ bad_area(struct pt_regs *regs, unsigned long error_code, * Something tried to access memory that isn't in our memory map.. * Fix it, but check if it's kernel or user first.. */ - mm_read_range_unlock(current->mm, range); + mm = current->mm; + if (!mm_range_is_coarse(range)) + mm_vma_unlock(mm); + mm_read_range_unlock(mm, range); __bad_area_nosemaphore(regs, error_code, address, pkey, si_code); } @@ -1278,7 +1282,7 @@ void do_user_addr_fault(struct pt_regs *regs, unsigned long hw_error_code, unsigned long address) { - struct mm_lock_range *range; + struct mm_lock_range pmd_range, *range; struct vm_area_struct pvma, *vma; struct task_struct *tsk; struct mm_struct *mm; @@ -1363,7 +1367,10 @@ void do_user_addr_fault(struct pt_regs *regs, } #endif - range = mm_coarse_lock_range(); + mm_init_lock_range(&pmd_range, + address & PMD_MASK, + (address & PMD_MASK) + PMD_SIZE); + range = &pmd_range; /* * Kernel-mode access to the user address space should only occur @@ -1397,6 +1404,8 @@ void do_user_addr_fault(struct pt_regs *regs, might_sleep(); } + if (!mm_range_is_coarse(range)) + mm_vma_lock(mm); vma = find_vma(mm, address); if (unlikely(!vma)) { bad_area(regs, hw_error_code, address, NULL, range); @@ -1408,6 +1417,10 @@ void do_user_addr_fault(struct pt_regs *regs, bad_area(regs, hw_error_code, address, NULL, range); return; } + /* + * Note that if range is fine grained, we can still safely call + * expand_stack as we are protected by the mm_vma_lock(). + */ if (unlikely(expand_stack(vma, address))) { bad_area(regs, hw_error_code, address, NULL, range); return; @@ -1423,23 +1436,34 @@ void do_user_addr_fault(struct pt_regs *regs, return; } - if (vma_is_anonymous(vma)) { + if (!mm_range_is_coarse(range)) { /* * Allocate anon_vma if needed. * This needs to operate on the vma of record. */ fault = prepare_mm_fault(vma, flags); - if (fault) - goto got_fault; /* * Copy vma attributes into a pseudo-vma. - * This will be required when using fine grained locks. + * The vma of record is only valid until mm_vma_unlock(). */ pvma = *vma; vma = &pvma; - } + mm_vma_unlock(mm); + if (fault) + goto got_fault; + + /* + * Fall back to locking the entire MM + * when operating on file vma. + */ + if (!vma_is_anonymous(vma)) { + mm_read_range_unlock(mm, range); + range = mm_coarse_lock_range(); + goto retry; + } + } /* * If for any reason at all we couldn't handle the fault, * make sure we exit gracefully rather than endlessly redo