From patchwork Fri Aug 28 14:03:12 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Gerald Schaefer X-Patchwork-Id: 11742909 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CA16E739 for ; Fri, 28 Aug 2020 14:04:07 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 872F52086A for ; Fri, 28 Aug 2020 14:04:07 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ibm.com header.i=@ibm.com header.b="InVcVV9y" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 872F52086A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.ibm.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id A198C8D0001; Fri, 28 Aug 2020 10:04:03 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 906B28D0002; Fri, 28 Aug 2020 10:04:03 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6DEBD6B0005; Fri, 28 Aug 2020 10:04:03 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0140.hostedemail.com [216.40.44.140]) by kanga.kvack.org (Postfix) with ESMTP id 4404D6B0006 for ; Fri, 28 Aug 2020 10:04:03 -0400 (EDT) Received: from smtpin16.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id BC72452B3 for ; Fri, 28 Aug 2020 14:04:02 +0000 (UTC) X-FDA: 77200146324.16.table58_101861527076 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin16.hostedemail.com (Postfix) with ESMTP id 6AF5F100E5A50 for ; Fri, 28 Aug 2020 14:03:59 +0000 (UTC) X-Spam-Summary: 1,0,0,7af6f80720f71255,d41d8cd98f00b204,gerald.schaefer@linux.ibm.com,,RULES_HIT:41:355:379:541:967:973:988:989:1260:1261:1345:1437:1535:1544:1605:1711:1730:1747:1777:1792:1801:2194:2198:2199:2200:2393:2525:2559:2565:2570:2682:2685:2693:2703:2859:2933:2937:2939:2942:2945:2947:2951:2954:3022:3865:3866:3867:3868:3870:3871:3872:3874:3934:3936:3938:3941:3944:3947:3950:3953:3956:3959:4118:4321:4361:4470:4605:5007:6119:6120:6261:6630:7903:8985:9025:9121:10004:11658:13146:13161:13229:13230,0,RBL:148.163.156.1:@linux.ibm.com:.lbl8.mailshell.net-64.100.201.201 62.2.0.100;04ygrt9yaapqopf9tadcumx8cwcd3ocizx7gccjypx31bad6zy61hcoz5e9bmsq.9nh3hd9bkcgb18rifofan4ykqictmz5uj1fc6he5ituiu713nbfh9k7ztset8j5.s-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:neutral,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: table58_101861527076 X-Filterd-Recvd-Size: 7901 Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by imf18.hostedemail.com (Postfix) with ESMTP for ; Fri, 28 Aug 2020 14:03:44 +0000 (UTC) Received: from pps.filterd (m0098396.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 07SE3ZrU029648; Fri, 28 Aug 2020 10:03:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ibm.com; h=from : to : cc : subject : date : message-id; s=pp1; bh=ERoOd0Q/7NlQ0VKtxMcS5uxz9ImhNBo0Vidm9kzUOYo=; b=InVcVV9yMS6plnwQJg8J6G7DpMxcIkh84j2ZVOt1YvVl0c7xcoGNXFSz3TuvYqeunij0 vKY0noKsXZpf10GS5rCeaka48hCedfqt7WuLyhMTOGEubPF98du7Mt680ZS0lmUadf4f QK65zhg8B+Z/eSmtfavk3huobMUdMi1xM6YS9FUPog2VSjXp9offRbJkFIFaM82ZnNIi S92QxFzsD5Azp0pdh00M5Okt6W7c/PoqH11P6vYGfEZCvAW4vOciIAluaND89GVY1/Vh rmMJL35pZ64ltJL/ndvHLhly9mNnCi7Z5MV7MArK8xK8294edCI2eqMrgX9jenHLE2PW JA== Received: from ppma02fra.de.ibm.com (47.49.7a9f.ip4.static.sl-reverse.com [159.122.73.71]) by mx0a-001b2d01.pphosted.com with ESMTP id 3371cabu5u-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Aug 2020 10:03:42 -0400 Received: from pps.filterd (ppma02fra.de.ibm.com [127.0.0.1]) by ppma02fra.de.ibm.com (8.16.0.42/8.16.0.42) with SMTP id 07SE2YTY003363; Fri, 28 Aug 2020 14:03:40 GMT Received: from b06avi18626390.portsmouth.uk.ibm.com (b06avi18626390.portsmouth.uk.ibm.com [9.149.26.192]) by ppma02fra.de.ibm.com with ESMTP id 336fvdrkfw-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Fri, 28 Aug 2020 14:03:40 +0000 Received: from d06av24.portsmouth.uk.ibm.com (mk.ibm.com [9.149.105.60]) by b06avi18626390.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id 07SE26SV49742244 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Fri, 28 Aug 2020 14:02:06 GMT Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 768C742047; Fri, 28 Aug 2020 14:03:37 +0000 (GMT) Received: from d06av24.portsmouth.uk.ibm.com (unknown [127.0.0.1]) by IMSVA (Postfix) with ESMTP id 12BCA42045; Fri, 28 Aug 2020 14:03:37 +0000 (GMT) Received: from tuxmaker.boeblingen.de.ibm.com (unknown [9.152.85.9]) by d06av24.portsmouth.uk.ibm.com (Postfix) with ESMTP; Fri, 28 Aug 2020 14:03:37 +0000 (GMT) From: Gerald Schaefer To: Linus Torvalds , Andrew Morton Cc: linux-mm , LKML , Vasily Gorbik , Alexander Gordeev , linux-s390@vger.kernel.org, Heiko Carstens , Claudio Imbrenda , Christian Borntraeger Subject: [RFC PATCH 0/2] mm/gup: fix gup_fast with dynamic page table folding Date: Fri, 28 Aug 2020 16:03:12 +0200 Message-Id: <20200828140314.8556-1-gerald.schaefer@linux.ibm.com> X-Mailer: git-send-email 2.17.1 X-TM-AS-GCONF: 00 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.235,18.0.687 definitions=2020-08-28_08:2020-08-28,2020-08-28 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 clxscore=1011 suspectscore=0 spamscore=0 bulkscore=0 mlxscore=0 phishscore=0 malwarescore=0 mlxlogscore=999 priorityscore=1501 adultscore=0 impostorscore=0 lowpriorityscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2006250000 definitions=main-2008280108 X-Rspamd-Queue-Id: 6AF5F100E5A50 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, Commit 1a42010cdc26 ("s390/mm: convert to the generic get_user_pages_fast code") introduced a subtle but severe bug on s390 with gup_fast, due to dynamic page table folding. The question "What would it require for the generic code to work for s390" has already been discussed here https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 and ended with a promising approach here https://lkml.kernel.org/r/20190419153307.4f2911b5@mschwideX1 which in the end unfortunately didn't quite work completely. We tried to mimic static level folding by changing pgd_offset to always dynamically calculate top level page table offset, and do nothing in folded pXd_offset. What has been overlooked is that PxD_SIZE/MASK and thus pXd_addr_end do not reflect this dynamic behaviour, and still act like static 5-level page tables. Here is an example of what happens with gup_fast on s390, for a task with 3-levels paging, crossing a 2 GB pud boundary: // addr = 0x1007ffff000, end = 0x10080001000 static int gup_pud_range(p4d_t p4d, unsigned long addr, unsigned long end, unsigned int flags, struct page **pages, int *nr) { unsigned long next; pud_t *pudp; // pud_offset returns &p4d itself (a pointer to a value on stack) pudp = pud_offset(&p4d, addr); do { // on second iteratation reading "random" stack value pud_t pud = READ_ONCE(*pudp); // next = 0x10080000000 next = pud_addr_end(addr, end); ... } while (pudp++, addr = next, addr != end); // pudp++ iterating over stack return 1; } pud_addr_end = 0x10080000000 is correct, but the previous pgd/p4d_addr_end should also have returned that limit, instead of the 5-level static pgd/p4d limits with PUD_SIZE/MASK != PGDIR_SIZE/MASK. Then the "end" parameter for gup_pud_range would also have been 0x10080000000, and we would not iterate further in gup_pud_range, but rather go back and (correctly) do it in gup_pgd_range. So, for the second iteration in gup_pud_range, we will increase pudp, which pointed to a stack value and not the real pud table. This new pudp will then point to whatever lies behind the p4d stack value. In general, this happens to be the previously read pgd, but it probably could also be something different, depending on compiler decisions. Most unfortunately, if it happens to be the pgd value, which is the same as the p4d / pud due to folding, it is a valid and present entry. So after the increment, we would still point to the same pud entry. The addr however has been increased in the second iteration, so that we now have different pmd/pte_index values, which will result in very wrong behaviour for the remaining gup_pmd/pte_range calls. We will effectively operate on an address minus 2 GB, due to missing pudp increase. In the "good case", if nothing is mapped there, we will fall back to the slow gup path. But if something is mapped there, and valid for gup_fast, we will end up (silently) getting references on the wrong pages and also add the wrong pages to the **pages result array. This can cause data corruption. We came up with two possible fix-ups, both will introduce some gup-specific helper functions, which will have no effect on other archs than s390. Patch 1 is the solution that has already been discussed in https://lkml.kernel.org/r/20190418100218.0a4afd51@mschwideX1 It will additionally pass along pXd pointers in gup_pXd_range, and also introduce pXd_offset_orig for s390, which takes an additional pXd entry value parameter. This allows correctly returning / incrementing pointers while still preseving the READ_ONCE logic for gup_fast. Patch 2 would instead introduce new gup_pXd_addr_end helpers, which take an additional pXd entry value parameter, that can be used on s390 to determine the correct page table level and return corresponding end / boundary. With that, the pointer iteration will always happen in gup_pgd_range. Comments / preferences are welcome. As a last resort, we might also revert back to the s390-specific gup_fast code, if none of the options are agreeable. Regards, Gerald