From patchwork Wed Feb 3 22:00:23 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Joao Martins X-Patchwork-Id: 12065533 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-18.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,MSGID_FROM_MTA_HEADER,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7D8EFC433DB for ; Wed, 3 Feb 2021 22:01:14 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id EE9DF64E4E for ; Wed, 3 Feb 2021 22:01:13 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org EE9DF64E4E Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=oracle.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0F9206B0078; Wed, 3 Feb 2021 17:01:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0AAAA8D0001; Wed, 3 Feb 2021 17:01:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D7A306B007D; Wed, 3 Feb 2021 17:01:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id C069A6B0078 for ; Wed, 3 Feb 2021 17:01:11 -0500 (EST) Received: from smtpin18.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id 85F0A181AEF1F for ; Wed, 3 Feb 2021 22:01:11 +0000 (UTC) X-FDA: 77778327942.18.farm91_5500410275d6 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin18.hostedemail.com (Postfix) with ESMTP id EBEAE100ED9C4 for ; Wed, 3 Feb 2021 22:01:10 +0000 (UTC) X-HE-Tag: farm91_5500410275d6 X-Filterd-Recvd-Size: 12882 Received: from aserp2130.oracle.com (aserp2130.oracle.com [141.146.126.79]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Wed, 3 Feb 2021 22:01:09 +0000 (UTC) Received: from pps.filterd (aserp2130.oracle.com [127.0.0.1]) by aserp2130.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 113LxkTv136679; Wed, 3 Feb 2021 22:01:08 GMT DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.com; h=from : to : cc : subject : date : message-id : in-reply-to : references : content-type : mime-version; s=corp-2020-01-29; bh=vzB97g62e/yEZSWcUFQa5IU9s8oLQsgB8kq9NlEZFvQ=; b=HTwTbtXS1k1/mfdr473Is2ZLsSbpw/LW0mEedRvVryuhTCCFGziEH4Qp3NwdnFYo51Jf ymu554pdXMK7n6hQ9ILnWxhdd83/+554E5Cutztx8bpP2SA7VG8zMrzR7bNLxEk8wxt/ r+eyAIVLYnLsUywv9N8LbSNnCRPQ0AlQ1myPmPdjI8n9XRE6gUVT18YtLn6VScoClIjN 8k8p7mkO4NVtGMwcw6pwRG6MkNuYpYlnd0XQCtdrcQ6duqRkpRsjxUJdzY6LOTN8nqGM QQoELNJRbJqM9zLmZpAYVe4h/HYiaGpUVlFbnN0IHG9Px7AM1yuzT2sMnbAJZurx3M1s LA== Received: from aserp3020.oracle.com (aserp3020.oracle.com [141.146.126.70]) by aserp2130.oracle.com with ESMTP id 36cvyb2mgj-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Feb 2021 22:01:07 +0000 Received: from pps.filterd (aserp3020.oracle.com [127.0.0.1]) by aserp3020.oracle.com (8.16.0.42/8.16.0.42) with SMTP id 113M14Fa042767; Wed, 3 Feb 2021 22:01:07 GMT Received: from nam10-bn7-obe.outbound.protection.outlook.com (mail-bn7nam10lp2108.outbound.protection.outlook.com [104.47.70.108]) by aserp3020.oracle.com with ESMTP id 36dhc1t3at-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Wed, 03 Feb 2021 22:01:07 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KiDOZUqstscJyHD8aol7jbL4c+vLI7Bm8FVkbxjzHPCC74frNMgj8QPVvs4zyoAvl10SK04/yLeZmb/wwewYSMqgl9RDSOMsc/Eo6dO1ZqQZgPK4AGQc23SJ2Yo2bPcJyMVQt+iIR4nHCVh5BQq3sWFO7o/enpqs6hT4RLtHOSziUz6m7HmksVskkv424HnwgkUQRy3kBZ9QJlG9NEVRiF2dQ/BbZfDxa7MDer5IRwfMClX0ANq8z+Z+F3tLAUeN+KNQKv1afNHHhVKSCPFyOTRFhtgqJtLWyC+yuwUJV4sxUXcs/g2E2mcpztQ5XnrW7hFIchEnIfcjTPQuOqvIgw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vzB97g62e/yEZSWcUFQa5IU9s8oLQsgB8kq9NlEZFvQ=; b=m/2srNaU2hZZ3wbaUTVkhsW36vFNnCOBk3l+MTxPZazGu1rMayNKDuznz70O8v/iIQ3Ua87iCtgfqK2Qb6bTtDlXd+bHEZqhJjZOHDTnn0YEiVPEcUwTjwRoFYQNVRU0ZDOYNFtljxhVtEnvuGCibW1Y7JiFcFL+IcONXGZT1V5tk3MWucExXKRS0MR5i6BX1nBFyQjmEi/bkvzMnnVpDBVrtjteaDJTRifiGwpd/6qM6l/iSCb9a9aujDkMJiDKrP652rZ0g7DRk05sDodJzYca/IVqVuF2XLemKbR+XeUmfsSGFrxoABDXzMGL0pdyTlH+RgB6igmA6Qai1lkBVw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=oracle.com; dmarc=pass action=none header.from=oracle.com; dkim=pass header.d=oracle.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=oracle.onmicrosoft.com; s=selector2-oracle-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=vzB97g62e/yEZSWcUFQa5IU9s8oLQsgB8kq9NlEZFvQ=; b=ghh+uq6rACMp7VOHVnoQfbDndrUkS1D5VTO/X5Pxx7S8AHkjIEVuoHLr0h/heq16TbDNzJ1ne1yCB1M5yRr5rvucN1dfvPR7nWm+k8hABHvJZSu13xkjyACKoFLwJu8IXchaA81ZHwjllS3sKXEvqEb3KhVLMs1j/i77jV2o82M= Authentication-Results: kvack.org; dkim=none (message not signed) header.d=none;kvack.org; dmarc=none action=none header.from=oracle.com; Received: from BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) by BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.3805.19; Wed, 3 Feb 2021 22:01:03 +0000 Received: from BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571]) by BYAPR10MB3077.namprd10.prod.outlook.com ([fe80::74a8:8649:e20b:d571%7]) with mapi id 15.20.3805.024; Wed, 3 Feb 2021 22:01:03 +0000 From: Joao Martins To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, linux-rdma@vger.kernel.org, Andrew Morton , Jason Gunthorpe , Doug Ledford , John Hubbard , Matthew Wilcox , Joao Martins Subject: [PATCH 2/4] mm/gup: decrement head page once for group of subpages Date: Wed, 3 Feb 2021 22:00:23 +0000 Message-Id: <20210203220025.8568-3-joao.m.martins@oracle.com> X-Mailer: git-send-email 2.11.0 In-Reply-To: <20210203220025.8568-1-joao.m.martins@oracle.com> References: <20210203220025.8568-1-joao.m.martins@oracle.com> X-Originating-IP: [94.61.1.144] X-ClientProxiedBy: LO4P123CA0266.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:194::19) To BYAPR10MB3077.namprd10.prod.outlook.com (2603:10b6:a03:8c::12) MIME-Version: 1.0 X-MS-Exchange-MessageSentRepresentingType: 1 Received: from paddy.uk.oracle.com (94.61.1.144) by LO4P123CA0266.GBRP123.PROD.OUTLOOK.COM (2603:10a6:600:194::19) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256) id 15.20.3825.17 via Frontend Transport; Wed, 3 Feb 2021 22:01:00 +0000 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 526eac20-7b3b-42a2-49fb-08d8c88f32a1 X-MS-TrafficTypeDiagnostic: BYAPR10MB3077: X-MS-Exchange-Transport-Forked: True X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:4502; X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: eX2iuzmXIXVRUJFbsdpMidDqiatd4hW3Bfahrnni+d1zWcDfKJnjUf+Ix4XW9lIoYl5EIKSr6GA0IPSr3ixEsasf1k4oHeGIs94Lj5EiGrwQsiu9d+MTVF+uzdwEUtuGWtdBDefa8zWPGHIBVYRQrwm4ADI4rWfaU6QKLXFoTYE8rp6Tw4tEjS73syrJ2MjAnWtwgNqOAXWjMAHts3YA3PNAf5JGLVcvI6BA9NO1YKajSEIc57EEWnW4OYO7wk1RbyT92bERl1DkikagbxoTCn7nzWawxJw6vSD0LyBIyeS7hI4Pnf9o9zn6G64vYRuH1ogsPN7g49b/h4nBRUf8hh08cGZ1ZnyJ7JhMGQPe6WvVcuYgctrsnfnBTz3nGW8vAkxdpwDTUv/ex7D1UEF32Uw3J2JxTQgTPfpaS6Z2M/VO8Ixd5dI1ps+1doT9KbL4H+LJ0Nj7E3130AeZJ9ceo4Zwh59BLKnLtmIL1OWKA82XP3yaMAYX0YMaqgIKrsR1vyOjMePWtA8OQO85NivwFw== X-Forefront-Antispam-Report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:BYAPR10MB3077.namprd10.prod.outlook.com;PTR:;CAT:NONE;SFS:(39860400002)(346002)(366004)(376002)(396003)(136003)(6486002)(83380400001)(86362001)(52116002)(5660300002)(1076003)(478600001)(8676002)(107886003)(4326008)(26005)(36756003)(2906002)(103116003)(16526019)(66946007)(316002)(54906003)(186003)(66476007)(8936002)(66556008)(6916009)(7696005)(956004)(6666004)(2616005);DIR:OUT;SFP:1101; X-MS-Exchange-AntiSpam-MessageData: q3pdFOFTrzCakjbsQBZV6BEaENo+bkB8eob3CLGSk2Ay+ONqhVYfVkWG5ffR0KNsDK78LP4V9ibSLvJku3ZWW6K69/hd3zAJZDAJUQ9lanTfSW7Ez0JX0BnA3h9/clBQQ4JleDtfPiHDUpGJy1ciJHkEmUgRW6U9AxpTqbaQlH4nLXSGN6+4kYEAFHKWuE7YTSYhK+PQI6lkRmkMtpSbgbDrxe0ylqHPZJaEiyt3iSSNCuOhvouogQp78nOhPRpi0pfDWLlvvlneBeK/phnkeFVXf0eRAjZodHp1DY+T3RdB0atpBvRHzvNlDT8kFwhc+wsGOqd6aP3vdak+cyGyWoVD1U9DhDxL0ARRd07ECXr4Le4q7J0oPY50SC0Gt4ZgfAf07dhwPmzTVEUaMzFty1E4/5fQ0QTZnxnQl0reCWym9/6w2tYVR6Fkz2FRvGUtg0nY4rBZsxHOckpki9rWK9v0wn4sulksLDJcL6Fueo4IDwLoIpm4bMyjxxV4P635yFUWzuZLzEfQPqtIXA4XmMvhKpXGEvoRODxsjESzwLtvJGv8LIAU4SVl/ZN1C1iq3zM+beitEO7lomNrW1HuApMa2sTyQJAQmvG5aCg7ro8yzgmmB6ZkXJAmwfR1lyMnK7pU4oJc8FDvtt3cGowKGlrnX5E/3P+vOH3y/acP0m7uIuRhyN1ACMwOEjtsE+PinjZZc75v/LzVJcpnjYeRqlQBwuubN5B6tSUg99IBHCBsNzKfT3e6wZHb3ALOtTiE25A0408keFZ8x35ThNLR23bar0HzmFZXimS426Ju0gygDv0zazwLhVGd0/dZ3XocrwPSE9EDRwbAvNi19xJmo7Md0pk80BKZlvoVZqbIGObEtnCzOl3V1heAvOLMTFTxwOj4WHS4kpWW6brCZjBOg/LAqRpLRN30vB7jkVKfSz5TBiPleEt4AZ9f55NsKlLyK0cQWHGYaDc7S4T7RWQb4P4jY660OTPhxm8m6Qo2TFQ+WBHj/rqkhBwSjGD2xSdzEG8NYvH75YmlA77uci7uOoGJrP6f6WvwKqbLxqhqL9fR/5dHcvkcCdS/WCRQkkGl X-OriginatorOrg: oracle.com X-MS-Exchange-CrossTenant-Network-Message-Id: 526eac20-7b3b-42a2-49fb-08d8c88f32a1 X-MS-Exchange-CrossTenant-AuthSource: BYAPR10MB3077.namprd10.prod.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-OriginalArrivalTime: 03 Feb 2021 22:01:03.0243 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 4e2c6054-71cb-48f1-bd6c-3a9705aca71b X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: wBbsQCCQ8vfDktq1Myp/0uCAH4ke0nnFw8qGouDpOzsmVFfgjgU+1wC9Fb9VEGDGsQpB+xBK827ORt2aJoWaCEvhXZJdvoUFueq3wmaFy4U= X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR10MB3077 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 bulkscore=0 adultscore=0 suspectscore=0 spamscore=0 mlxscore=0 malwarescore=0 mlxlogscore=999 phishscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102030133 X-Proofpoint-Virus-Version: vendor=nai engine=6200 definitions=9884 signatures=668683 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 clxscore=1015 impostorscore=0 mlxscore=0 spamscore=0 bulkscore=0 priorityscore=1501 adultscore=0 lowpriorityscore=0 malwarescore=0 phishscore=0 mlxlogscore=999 suspectscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2009150000 definitions=main-2102030133 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Rather than decrementing the head page refcount one by one, we walk the page array and checking which belong to the same compound_head. Later on we decrement the calculated amount of references in a single write to the head page. To that end switch to for_each_compound_head() does most of the work. set_page_dirty() needs no adjustment as it's a nop for non-dirty head pages and it doesn't operate on tail pages. This considerably improves unpinning of pages with THP and hugetlbfs: - THP gup_test -t -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK (put values): ~87.6k us -> ~23.2k us - 16G with 1G huge page size gup_test -f /mnt/huge/file -m 16384 -r 10 [-L|-a] -S -n 512 -w PIN_LONGTERM_BENCHMARK: (put values): ~87.6k us -> ~27.5k us Signed-off-by: Joao Martins Reviewed-by: John Hubbard --- mm/gup.c | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-) diff --git a/mm/gup.c b/mm/gup.c index 4f88dcef39f2..971a24b4b73f 100644 --- a/mm/gup.c +++ b/mm/gup.c @@ -270,20 +270,15 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, bool make_dirty) { unsigned long index; - - /* - * TODO: this can be optimized for huge pages: if a series of pages is - * physically contiguous and part of the same compound page, then a - * single operation to the head page should suffice. - */ + struct page *head; + unsigned int ntails; if (!make_dirty) { unpin_user_pages(pages, npages); return; } - for (index = 0; index < npages; index++) { - struct page *page = compound_head(pages[index]); + for_each_compound_head(index, pages, npages, head, ntails) { /* * Checking PageDirty at this point may race with * clear_page_dirty_for_io(), but that's OK. Two key @@ -304,9 +299,9 @@ void unpin_user_pages_dirty_lock(struct page **pages, unsigned long npages, * written back, so it gets written back again in the * next writeback cycle. This is harmless. */ - if (!PageDirty(page)) - set_page_dirty_lock(page); - unpin_user_page(page); + if (!PageDirty(head)) + set_page_dirty_lock(head); + put_compound_head(head, ntails, FOLL_PIN); } } EXPORT_SYMBOL(unpin_user_pages_dirty_lock); @@ -323,6 +318,8 @@ EXPORT_SYMBOL(unpin_user_pages_dirty_lock); void unpin_user_pages(struct page **pages, unsigned long npages) { unsigned long index; + struct page *head; + unsigned int ntails; /* * If this WARN_ON() fires, then the system *might* be leaking pages (by @@ -331,13 +328,9 @@ void unpin_user_pages(struct page **pages, unsigned long npages) */ if (WARN_ON(IS_ERR_VALUE(npages))) return; - /* - * TODO: this can be optimized for huge pages: if a series of pages is - * physically contiguous and part of the same compound page, then a - * single operation to the head page should suffice. - */ - for (index = 0; index < npages; index++) - unpin_user_page(pages[index]); + + for_each_compound_head(index, pages, npages, head, ntails) + put_compound_head(head, ntails, FOLL_PIN); } EXPORT_SYMBOL(unpin_user_pages);