From patchwork Wed Aug 9 06:11:05 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13347446 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FB3FC001B0 for ; Wed, 9 Aug 2023 06:13:16 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10F9E6B0078; Wed, 9 Aug 2023 02:13:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0BEF86B007B; Wed, 9 Aug 2023 02:13:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC93B8D0001; Wed, 9 Aug 2023 02:13:15 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id D924B6B0078 for ; Wed, 9 Aug 2023 02:13:15 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id B09FD1C8E6F for ; Wed, 9 Aug 2023 06:13:15 +0000 (UTC) X-FDA: 81103548750.09.5BC34E9 Received: from mgamail.intel.com (mgamail.intel.com [192.55.52.43]) by imf19.hostedemail.com (Postfix) with ESMTP id 8527D1A000D for ; Wed, 9 Aug 2023 06:13:13 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=OQ2R2dLD; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf19.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1691561593; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nTat4hjNwabT9RgLwLQU/73bsULgdhGM/jU6a6BKHXE=; b=i6+CQkUQmn6tSdqWrAqzo+Mt8giyMmxq+eYTSnxNvyqVjHkh5KkHkXlUyav00xHjB6fPUH FmAhVwb41ugj50yATfpGidPSzRJPb4cXuCibK6WpoxR4DSXiiec0ew/5udIaNQrXw0gmNo 70+P346eJyIpRa4fJCH6DIbcQrMs5I8= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=OQ2R2dLD; dmarc=pass (policy=none) header.from=intel.com; spf=pass (imf19.hostedemail.com: domain of fengwei.yin@intel.com designates 192.55.52.43 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1691561593; a=rsa-sha256; cv=none; b=KbyUeSC4j3fS3aMnD9sz92Ch36+V9CrvOJ8iZn10R+y2O0MT23+b1cfzF7kdxF9hyxkrgM BmWojTJgJMQKj6R8BamGzH1w55FB5Y/6uGLCppbVAJTcdOHAm39y7YN7j48aUcqIFzEwji IG4pvD0cENe+QvnYpxlB8IKHKhtW/Yo= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1691561593; x=1723097593; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=LGJZorJ3E6t3J456VFnC+I2HIJq1tAE/jnA+v0QWpWc=; b=OQ2R2dLDymLFdKjHa3rBoYlT5aTlSA2ntQJ633rtJXZ5YMXY+qwCIumb jVp3mQylWFeeS7bTqILjaVp0rvUiF0g6rHlN0ok6LeV2F1tm6FWpOR0D5 DXAbsMa8ricGwVyPJDjiwIT1/hiEAAl5pe4r5A8k/Q4OBJKiSX4LsTata VJXPaUhrKj9MjSCxau5krcMNRKtU65R+TGGuVKW5dDV3t7Y/QJcy+7Aff bV5lxU+04nso2guKD0fThgCBwAn4r8+m63nE3Ugumuv30LuFh2eYwF4ec m3qpV1+7QaItzTkfWkg/NWKxgulx3VS4v3oNt/VllO4tSUcO0/MbqApLX g==; X-IronPort-AV: E=McAfee;i="6600,9927,10795"; a="457410063" X-IronPort-AV: E=Sophos;i="6.01,158,1684825200"; d="scan'208";a="457410063" Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga105.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 08 Aug 2023 23:13:12 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10795"; a="731680634" X-IronPort-AV: E=Sophos;i="6.01,158,1684825200"; d="scan'208";a="731680634" Received: from fyin-dev.sh.intel.com ([10.239.159.32]) by orsmga002.jf.intel.com with ESMTP; 08 Aug 2023 23:13:08 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, yuzhao@google.com, willy@infradead.org, hughd@google.com, yosryahmed@google.com, ryan.roberts@arm.com, david@redhat.com, shy828301@gmail.com Cc: fengwei.yin@intel.com Subject: [PATCH v2 3/3] mm: mlock: update mlock_pte_range to handle large folio Date: Wed, 9 Aug 2023 14:11:05 +0800 Message-Id: <20230809061105.3369958-4-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230809061105.3369958-1-fengwei.yin@intel.com> References: <20230809061105.3369958-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 8527D1A000D X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: h6era7x3qi8ocuxfe1tp8y1u5hb1qdki X-HE-Tag: 1691561593-859328 X-HE-Meta: U2FsdGVkX1+VUReA7JkuCWhiIIMVRXrYEnX0xCALADTrymT2X/u8EJaLHdXxEfRJoE0lHPCU5CmUSQ0DSKl/ro9dLRqSFKQ4+xxikQcX/rTN2B0uq+5XrQLk7fajbZ4pNDnPTZd8lQJPov7melbWAqCvvm3irIEfPChj8d3uAqmyPoB28xU2VllECsg+bc6SeRmyhC91YUEiLcJySLcxQXKZOY7weVncgt46Gt+6r45F5mK+MPZrWeKifGY+U2DBYrd23CQgJo2ECUEFPiqsUALYSNzFrjeeY8XkABbB4kgOnoZzrDG8OS+NIU3I2E2ysJ04lntrlrRX0bILay9C36FX9e5LUR4qa0MqyVF9xIqAU8wkkcLEgQ314nu3gS5l9Z0DUSa3Wtio05Hxt/1ByZ6epFJirHHruXWKXvflCU7b2S+UoBPyKNGNDwRrALllyyn5ibdMxXAt0Xq7ldI5nh5rT1F0pJznvxBfIcbcKLtKT/++Ur5t4ZC5LD+hqWW2yl8NjZkvmqC9WaCJanDlqtE6PDuPcGJKNFBkrZe/sSZe5fZtEGw2punLf0DLq9XDQMjimWfoJ3zg7HV8BJgQkYtwDuxbHTSvXt7zM1g6ZKUBB0LCmlXXyr6cgNQjo9FLIz+o2Ux1Po2W9A2WU3UXHa+9KWJJxVJ/n87rFqR5GeawimUf+xVCA36OXDWxwNlNyHmuqtOy4KiaeDBmeUVXekRpZmGWgwav2QDjEP/4hWDSy5ZUzwuZI1BLdC+0GDkViN6uYW9bUepK8w8EIYTeFa8cjqZHtErlqzI+aYFRUOuZ95mWjHtX8i6K5UBR0XZOH7a7e0T0Ju2NJkgL+BrLj9JVED+hJouika2U+ATAIL69ssWV4ohM+n015M2JeNmlGKZZNOKMxp4OaX84L3EsKwm6QnTANaQa1ZkfrXkp1EqaJurMr+k3as5O+CUtZ/92ZepkY2x6tLM36+JpdXT Z9by32br T2UCGn0tCQV8RJD/UVIKpJsX7VLjk+0z6sE3+Y74ByPcgV8hf/B9kYlfrSoxOk4LMC7ljHYPDejWntCWqj4mOj9xJp7fjNy+vmNGKHhRjg1g0bbWqwB9HsbkLFhgseHBQdDQG/2vfSxTAW5SR/3yHVrpVFdD28EwGSxnqJypJ6qvwKyKH3lE0qxiR631x+HCQcaaHkvTUvJlyZtI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Current kernel only lock base size folio during mlock syscall. Add large folio support with following rules: - Only mlock large folio when it's in VM_LOCKED VMA range and fully mapped to page table. fully mapped folio is required as if folio is not fully mapped to a VM_LOCKED VMA, if system is in memory pressure, page reclaim is allowed to pick up this folio, split it and reclaim the pages which are not in VM_LOCKED VMA. - munlock will apply to the large folio which is in VMA range or cross the VMA boundary. This is required to handle the case that the large folio is mlocked, later the VMA is split in the middle of large folio. Signed-off-by: Yin Fengwei --- mm/mlock.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 64 insertions(+), 2 deletions(-) diff --git a/mm/mlock.c b/mm/mlock.c index 06bdfab83b58..1da1996745e7 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -305,6 +305,58 @@ void munlock_folio(struct folio *folio) local_unlock(&mlock_fbatch.lock); } +static inline unsigned int folio_mlock_step(struct folio *folio, + pte_t *pte, unsigned long addr, unsigned long end) +{ + unsigned int count, i, nr = folio_nr_pages(folio); + unsigned long pfn = folio_pfn(folio); + pte_t ptent = ptep_get(pte); + + if (!folio_test_large(folio)) + return 1; + + count = pfn + nr - pte_pfn(ptent); + count = min_t(unsigned int, count, (end - addr) >> PAGE_SHIFT); + + for (i = 0; i < count; i++, pte++) { + pte_t entry = ptep_get(pte); + + if (!pte_present(entry)) + break; + if (pte_pfn(entry) - pfn >= nr) + break; + } + + return i; +} + +static inline bool allow_mlock_munlock(struct folio *folio, + struct vm_area_struct *vma, unsigned long start, + unsigned long end, unsigned int step) +{ + /* + * For unlock, allow munlock large folio which is partially + * mapped to VMA. As it's possible that large folio is + * mlocked and VMA is split later. + * + * During memory pressure, such kind of large folio can + * be split. And the pages are not in VM_LOCKed VMA + * can be reclaimed. + */ + if (!(vma->vm_flags & VM_LOCKED)) + return true; + + /* folio not in range [start, end), skip mlock */ + if (!folio_in_range(folio, vma, start, end)) + return false; + + /* folio is not fully mapped, skip mlock */ + if (step != folio_nr_pages(folio)) + return false; + + return true; +} + static int mlock_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -314,6 +366,8 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, pte_t *start_pte, *pte; pte_t ptent; struct folio *folio; + unsigned int step = 1; + unsigned long start = addr; ptl = pmd_trans_huge_lock(pmd, vma); if (ptl) { @@ -334,6 +388,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, walk->action = ACTION_AGAIN; return 0; } + for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { ptent = ptep_get(pte); if (!pte_present(ptent)) @@ -341,12 +396,19 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, folio = vm_normal_folio(vma, addr, ptent); if (!folio || folio_is_zone_device(folio)) continue; - if (folio_test_large(folio)) - continue; + + step = folio_mlock_step(folio, pte, addr, end); + if (!allow_mlock_munlock(folio, vma, start, end, step)) + goto next_entry; + if (vma->vm_flags & VM_LOCKED) mlock_folio(folio); else munlock_folio(folio); + +next_entry: + pte += step - 1; + addr += (step - 1) << PAGE_SHIFT; } pte_unmap(start_pte); out: