From patchwork Mon Jun 11 14:06:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew Wilcox X-Patchwork-Id: 10457995 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 157FC6020F for ; Mon, 11 Jun 2018 14:11:02 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id ED6F42867F for ; Mon, 11 Jun 2018 14:11:01 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EA1F8287A0; Mon, 11 Jun 2018 14:11:01 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.8 required=2.0 tests=BAYES_00,DKIM_SIGNED, MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE, T_DKIM_INVALID autolearn=unavailable version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 65F752867F for ; Mon, 11 Jun 2018 14:10:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 836B76B0296; Mon, 11 Jun 2018 10:07:08 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 7959D6B0298; Mon, 11 Jun 2018 10:07:08 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 60FDD6B0297; Mon, 11 Jun 2018 10:07:08 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-pl0-f72.google.com (mail-pl0-f72.google.com [209.85.160.72]) by kanga.kvack.org (Postfix) with ESMTP id 1BCF36B0295 for ; Mon, 11 Jun 2018 10:07:08 -0400 (EDT) Received: by mail-pl0-f72.google.com with SMTP id a5-v6so12106703plp.8 for ; Mon, 11 Jun 2018 07:07:08 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:dkim-signature:from:to:cc:subject:date :message-id:in-reply-to:references; bh=l1OnaQk+EM5IeSocf5IaBTCNqRdXO66+RLL6O/1Wzas=; b=MwrmEUxWvPWneHsdfrQ4pBR+90pPRpUB1z9J1q9Icg5lJUMdEdr/r+WJXw8EzYO9ht pLtGZbs5T5HonoVKC3R47Gfw1axTiVwZLbJwT08240wt4ewfFpUU3wfc42YsHNUmEbmO 7oe/HYEKAQxhfBvm4JZzT4GSowzq8O5GizwRgQw7+x0qyHPBrBEyZcUhIfV3EW7BMcuB 4Zf4TF/dmf4os4MB6Yp6FatUDPtCKOHCSBwG84xdscFoLwitUaB81hxjN6RlwEqzmoWw PDHpsd95wKNL8H9FUcDLHtImzj0C4CWH4RIIXNgdwDewuxu27cmuyjyiOsiDzil4Dmhy iwhw== X-Gm-Message-State: APt69E17EHMCXx66GG6gCgSPr2XwLqKMPKnxwzBP9JZrd8ZCJZCAzU/2 HHhl3+4KbZgzvQNoir0bTn0sLHrsUad+MX0wwMZlcnlJwX2Ut5UzWNYnCxsB3O8U5tGDmhRDkJb qNMZoaVdYTx5lmri4bsJIJ+p4TUzS8za6domn9QznxCNUmPG37JNnU9faIBlmHSCCoA== X-Received: by 2002:a62:3bd2:: with SMTP id w79-v6mr17842457pfj.129.1528726027769; Mon, 11 Jun 2018 07:07:07 -0700 (PDT) X-Google-Smtp-Source: ADUXVKK2vWE9Yrc81+95M2jBO4a1CzuysHMwvFqt23ziSpPfpmgzEd1AXk5HqK0kBz77/rCUsV19 X-Received: by 2002:a62:3bd2:: with SMTP id w79-v6mr17842381pfj.129.1528726026736; Mon, 11 Jun 2018 07:07:06 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1528726026; cv=none; d=google.com; s=arc-20160816; b=PHE06luMAQ6/+U8ctqrnaZ9G5FTT0ktJGrg7n+dubKpXpOHQd69hLzqXLwU0k3w2XZ EDfbO7CA51IRFkHCoHjHYGUT2fC0Jfen77CJJ3T/gRGWoaoW+tWR6q7LZv3yJnv3vNy3 3+GhZzQ0zQxsPkKGRcrkaarpeClOvtppKSQ14bwiGPBhsWoZSjHORzeYY/LTunscETPU basU34vofyRcf0PJgz70FuQtaKbPSwHmuffi3A0ohjltEaUBdAksUDbLfja8cGYx3vGV 4K02c4g4C9u5d0l1tPNnj9mFlgm2LP8cG1xUzMKx2fqsRBmNcVvVSSOf3RXcuOKHqTIh 4MwA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=references:in-reply-to:message-id:date:subject:cc:to:from :dkim-signature:arc-authentication-results; bh=l1OnaQk+EM5IeSocf5IaBTCNqRdXO66+RLL6O/1Wzas=; b=wX0ecl2ejZBnKOMYzklAQSkWDTwLAw4offyUs5uX++k8C64j/kVBIdlg3AlVT65h8S TZ0A0tU0A3Cn0nJB9XPnUQHXX69FyDfEGjeeSU06GbhXTihmbBfPjdKyzSoZ637q8ERo 9oI9wzQZvJjz/IsR7jb43J2GhVol8n49T/j/eAenvFbzkSEIdxBvuxnAdRj8dzpm9bDX wseU05iFamkxIBVDesEw6Fe0gZes1YqIHieA4mI/HAKXU9neb0wVElqal8shZK6N/YrT AqCwYcuXGxUH7SwiL+X6Cuyusrx01Mv7rrboey0Xvz82teNsTgIUxBJdgMXcaP3WYibj HY8w== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@infradead.org header.s=bombadil.20170209 header.b=IdTHfLk1; spf=pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) smtp.mailfrom=willy@infradead.org Received: from bombadil.infradead.org (bombadil.infradead.org. [2607:7c80:54:e::133]) by mx.google.com with ESMTPS id 31-v6si16291328plk.191.2018.06.11.07.07.06 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 11 Jun 2018 07:07:06 -0700 (PDT) Received-SPF: pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) client-ip=2607:7c80:54:e::133; Authentication-Results: mx.google.com; dkim=pass header.i=@infradead.org header.s=bombadil.20170209 header.b=IdTHfLk1; spf=pass (google.com: best guess record for domain of willy@infradead.org designates 2607:7c80:54:e::133 as permitted sender) smtp.mailfrom=willy@infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=References:In-Reply-To:Message-Id: Date:Subject:Cc:To:From:Sender:Reply-To:MIME-Version:Content-Type: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=l1OnaQk+EM5IeSocf5IaBTCNqRdXO66+RLL6O/1Wzas=; b=IdTHfLk1l2F2LMWp3NbAsxsI1 KcAQDoxS3ZRNT9BIkMk2OxZqwA5Yva4tvrNZYZAb7p5YrDxB8GMjVm90Q6mQn3Y73uMcq2rI6+gYq PHc9v1HEZ4a8V3HpR+zGmGXLXxQcMmO7hCTptlrWmBIJXPG8jry0iloT180yf+/9EI9fCv53ExYLt LO+l7ixu/qJXOBdJGGCwEKv0lMyOuKFkjqKxLWZg/rdVsJICkMRz96vnlVgIFgLT2x8TpBN1hMU16 UAvYhdSvMX4rLiBCEHuIe/71gAcek3VEV6MEoE1cS+6Fq6sUPd/1sLw6e8CvFuvn5RLfwE3NCAhsF ol9WflbfA==; Received: from willy by bombadil.infradead.org with local (Exim 4.90_1 #2 (Red Hat Linux)) id 1fSNTN-0004vN-OS; Mon, 11 Jun 2018 14:07:05 +0000 From: Matthew Wilcox To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org, linux-kernel@vger.kernel.org Cc: Matthew Wilcox , Jan Kara , Jeff Layton , Lukas Czerner , Ross Zwisler , Christoph Hellwig , Goldwyn Rodrigues , Nicholas Piggin , Ryusuke Konishi , linux-nilfs@vger.kernel.org, Jaegeuk Kim , Chao Yu , linux-f2fs-devel@lists.sourceforge.net Subject: [PATCH v13 53/72] memfd: Convert memfd_wait_for_pins to XArray Date: Mon, 11 Jun 2018 07:06:20 -0700 Message-Id: <20180611140639.17215-54-willy@infradead.org> X-Mailer: git-send-email 2.14.3 In-Reply-To: <20180611140639.17215-1-willy@infradead.org> References: <20180611140639.17215-1-willy@infradead.org> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Matthew Wilcox Simplify the locking by taking the spinlock while we walk the tree on the assumption that many acquires and releases of the lock will be worse than holding the lock while we process an entire batch of pages. Signed-off-by: Matthew Wilcox Reviewed-by: Mike Kravetz --- mm/memfd.c | 61 ++++++++++++++++++++++-------------------------------- 1 file changed, 25 insertions(+), 36 deletions(-) diff --git a/mm/memfd.c b/mm/memfd.c index 27069518e3c5..e7d6be725b7a 100644 --- a/mm/memfd.c +++ b/mm/memfd.c @@ -21,7 +21,7 @@ #include /* - * We need a tag: a new tag would expand every radix_tree_node by 8 bytes, + * We need a tag: a new tag would expand every xa_node by 8 bytes, * so reuse a tag which we firmly believe is never set or cleared on tmpfs * or hugetlbfs because they are memory only filesystems. */ @@ -72,9 +72,7 @@ static void memfd_tag_pins(struct address_space *mapping) */ static int memfd_wait_for_pins(struct address_space *mapping) { - struct radix_tree_iter iter; - void __rcu **slot; - pgoff_t start; + XA_STATE(xas, &mapping->i_pages, 0); struct page *page; int error, scan; @@ -82,7 +80,9 @@ static int memfd_wait_for_pins(struct address_space *mapping) error = 0; for (scan = 0; scan <= LAST_SCAN; scan++) { - if (!radix_tree_tagged(&mapping->i_pages, MEMFD_TAG_PINNED)) + unsigned int tagged = 0; + + if (!xas_tagged(&xas, MEMFD_TAG_PINNED)) break; if (!scan) @@ -90,45 +90,34 @@ static int memfd_wait_for_pins(struct address_space *mapping) else if (schedule_timeout_killable((HZ << scan) / 200)) scan = LAST_SCAN; - start = 0; - rcu_read_lock(); - radix_tree_for_each_tagged(slot, &mapping->i_pages, &iter, - start, MEMFD_TAG_PINNED) { - - page = radix_tree_deref_slot(slot); - if (radix_tree_exception(page)) { - if (radix_tree_deref_retry(page)) { - slot = radix_tree_iter_retry(&iter); - continue; - } - - page = NULL; - } - - if (page && - page_count(page) - page_mapcount(page) != 1) { - if (scan < LAST_SCAN) - goto continue_resched; - + xas_set(&xas, 0); + xas_lock_irq(&xas); + xas_for_each_tagged(&xas, page, ULONG_MAX, MEMFD_TAG_PINNED) { + bool clear = true; + if (xa_is_value(page)) + continue; + if (page_count(page) - page_mapcount(page) != 1) { /* * On the last scan, we clean up all those tags * we inserted; but make a note that we still * found pages pinned. */ - error = -EBUSY; + if (scan == LAST_SCAN) + error = -EBUSY; + else + clear = false; } + if (clear) + xas_clear_tag(&xas, MEMFD_TAG_PINNED); + if (++tagged % XA_CHECK_SCHED) + continue; - xa_lock_irq(&mapping->i_pages); - radix_tree_tag_clear(&mapping->i_pages, - iter.index, MEMFD_TAG_PINNED); - xa_unlock_irq(&mapping->i_pages); -continue_resched: - if (need_resched()) { - slot = radix_tree_iter_resume(slot, &iter); - cond_resched_rcu(); - } + xas_pause(&xas); + xas_unlock_irq(&xas); + cond_resched(); + xas_lock_irq(&xas); } - rcu_read_unlock(); + xas_unlock_irq(&xas); } return error;