From patchwork Mon Sep 18 07:33:18 2023 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yin Fengwei X-Patchwork-Id: 13388986 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE279C46CA1 for ; Mon, 18 Sep 2023 07:34:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5B9536B029D; Mon, 18 Sep 2023 03:34:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 569DC6B029F; Mon, 18 Sep 2023 03:34:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 431746B02A0; Mon, 18 Sep 2023 03:34:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 330626B029D for ; Mon, 18 Sep 2023 03:34:09 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0FC2F1C9B73 for ; Mon, 18 Sep 2023 07:34:09 +0000 (UTC) X-FDA: 81248904618.17.0E7A25D Received: from mgamail.intel.com (mgamail.intel.com [134.134.136.126]) by imf02.hostedemail.com (Postfix) with ESMTP id 0849B8000B for ; Mon, 18 Sep 2023 07:34:06 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=no2FPlDO; spf=pass (imf02.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1695022447; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=omkQZJY6GX4oLfOXjnuAwvtA5BrOk+Wt2K7yMw3C+oU=; b=WNPOc+og6/cUQ7GLWjRPRw1IQicUiFz4Z+gF/aIFLeX7vSFspGZi9b73Y0rXF/dwOgepnE xgj4VPY/uZu7YGeDPqIUYpB6kzPoT6Y6bQBdI8o3HXSEKA0M8JNEpitk29H9xTaa7DRStJ 7mZ+WpNeMmc+PuMVXZm3m4Z8VrwPbJQ= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1695022447; a=rsa-sha256; cv=none; b=hwNtuxp43ucep8d/6CkhojqvTbWa5GTYLEE3TD8q79asGcTzbHVluMj/htTl2xmzzD/his D9ztNHpvPMUo9yUi1txV/kiSrsZB1/rK1tv7PAQw0JoCgzQ8IQx1dL3I3xeUpza+bPgpPD 4LLFza6sxulTBE8Kb+0vtth5UbpKG4Q= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=no2FPlDO; spf=pass (imf02.hostedemail.com: domain of fengwei.yin@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=fengwei.yin@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1695022447; x=1726558447; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=Q8dtjLXEWzMrNVlVSld+SWqxhUjc/Zr37nPzybsHJrg=; b=no2FPlDO/am7ummY826gAFNYZhiInjA7VoZiXMLxb5vBZ8jO3M7GVx98 Dgd9JR54QZq4wGj/QO5+1Rp1zxtz8hY4xKHuDehFvCJThPClDksyjS488 hPy6Sd+zgwmpdzkTfWyIB4BkrhAtB4d9aThwMTZgqrlovl2NvvpbUwXY6 ejTL3ASybQ4K2hGJHR2HY54xchnzCrPzuIxODUV6ATFihYyqljZqXEzuJ oH40RKBvZ5DS71JE+g1uiIcsq/buRpVhpifLRUdZIEY3OaLGox6libBw0 8Vx9ETI8k+0EaeF9ro+7MmmIKvSKzoH6NmwOn7MT/FoLn2sBZsGYUPYRu g==; X-IronPort-AV: E=McAfee;i="6600,9927,10836"; a="364630978" X-IronPort-AV: E=Sophos;i="6.02,156,1688454000"; d="scan'208";a="364630978" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 18 Sep 2023 00:34:05 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10836"; a="815915477" X-IronPort-AV: E=Sophos;i="6.02,156,1688454000"; d="scan'208";a="815915477" Received: from fyin-dev.sh.intel.com ([10.239.159.24]) by fmsmga004.fm.intel.com with ESMTP; 18 Sep 2023 00:34:02 -0700 From: Yin Fengwei To: linux-mm@kvack.org, linux-kernel@vger.kernel.org, akpm@linux-foundation.org, yuzhao@google.com, willy@infradead.org, hughd@google.com, yosryahmed@google.com, ryan.roberts@arm.com, david@redhat.com, shy828301@gmail.com Cc: fengwei.yin@intel.com Subject: [PATCH v3 3/3] mm: mlock: update mlock_pte_range to handle large folio Date: Mon, 18 Sep 2023 15:33:18 +0800 Message-Id: <20230918073318.1181104-4-fengwei.yin@intel.com> X-Mailer: git-send-email 2.39.2 In-Reply-To: <20230918073318.1181104-1-fengwei.yin@intel.com> References: <20230918073318.1181104-1-fengwei.yin@intel.com> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0849B8000B X-Rspam-User: X-Stat-Signature: b3fdfbh1hq5cjc68qgmyu3ur3opaxzzw X-Rspamd-Server: rspam03 X-HE-Tag: 1695022446-224853 X-HE-Meta: U2FsdGVkX1/3PfuHikwLTGuJxvWQ1YtSjvfx1fXZ58w4Wa1glNG+5PKdyYEbzy6SSvp92LAxXydOzBQc0bdRRj16ahqhSiphkoA2WoMHLtuOsAWwoafPZOIlsvsq9gAbQFix80kggHQjCJtMOOoXMmgrMO7laX+E2Zui/UmMElfasQuyAVqykJzH5xo8v4FvLyTVUFDEYtgapqpMoU0Si/shPPDxdGvpxlBvey/bTBkSVB16C/yQv9HEPVX+aZm0paZ2ngIVfsYGzIy03gHy3lg/c899T1L4JxQQ5V0iinC4re0Q+0Tp8go0yInH/kplFIwgvGwV1yvuPvEWXAlvSlie68t4oJ648zF2+gVfXtLT/omt34OzbXss/3M+hMtKXdn5b1L261JS+MlyURsH2bzpVQrPr+6nrL5alwxpGlEfy2o55eh2vgzH0uLlJi89MA3D122h+zBxEmdEbyPh9oiAMLG4FrzE6IJtfD7qXM66p768xBgP3cBd0nD8dM6MPzikO+4Iw9t0Pti0dDGK/NNz86/ay3S0YG827gl5DAM+2vVIB2mz2sfT9D5i039t+9WD9lbmSvJTGPjGshLp5oNH/L9C6sbcXnFAU4YqegwDsZdDLEf5HqqbMaWVq7WfWR+whaOtYbWWo2OLu5EuWO1tcfprIPnczaqJyWTlchcYMHSwZ/O1FqNbT8Uul2wyq48yuW9y3YqIaCfWXkprW4tQEKmECtUNf8+22AV9TzP8TmXFjBOcXxffKgis0Umt2L4ke8fJimO8tPIgYCYhcbsndhUOeWksi43zvFD/6s2+06uDrUUAZZLf60EwHRPSP9UomslJHdh0puHuFEk+AciyowgcJ6/3CE+5zz4bcsgPAwyo/uBiRe6TQHIhlaO5xFGRrD0l3MolnTxYuWmSzLTCqTWUBNtbIQ6DX0MJXVgOoTmxtxG/UTe4zT8afdWZwXranL6H11iR3k8v9D/ k3GL5sGc REYYutT7h8Nc0T4iqaTBlkyAPJ0bJIEeRqc6hOa4ZLPG/shUnZhuniZ+d6U9DBymovdJqHn162iitZtg5XDCam0RxwgzC4bmdXfqOaTGPGLwx/dPLMIorBWQdu+iqJsRkYlQcoWGfJeM2sY/qTp3KTWaMaCkoCzz1vRbwqq7nZ/2PbXwSyHj0+XddYVG71tHxoYn4R5iQUxUtXa0= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Current kernel only lock base size folio during mlock syscall. Add large folio support with following rules: - Only mlock large folio when it's in VM_LOCKED VMA range and fully mapped to page table. fully mapped folio is required as if folio is not fully mapped to a VM_LOCKED VMA, if system is in memory pressure, page reclaim is allowed to pick up this folio, split it and reclaim the pages which are not in VM_LOCKED VMA. - munlock will apply to the large folio which is in VMA range or cross the VMA boundary. This is required to handle the case that the large folio is mlocked, later the VMA is split in the middle of large folio. Signed-off-by: Yin Fengwei --- mm/mlock.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 64 insertions(+), 2 deletions(-) diff --git a/mm/mlock.c b/mm/mlock.c index 06bdfab83b58..42b6865f8f82 100644 --- a/mm/mlock.c +++ b/mm/mlock.c @@ -305,6 +305,58 @@ void munlock_folio(struct folio *folio) local_unlock(&mlock_fbatch.lock); } +static inline unsigned int folio_mlock_step(struct folio *folio, + pte_t *pte, unsigned long addr, unsigned long end) +{ + unsigned int count, i, nr = folio_nr_pages(folio); + unsigned long pfn = folio_pfn(folio); + pte_t ptent = ptep_get(pte); + + if (!folio_test_large(folio)) + return 1; + + count = pfn + nr - pte_pfn(ptent); + count = min_t(unsigned int, count, (end - addr) >> PAGE_SHIFT); + + for (i = 0; i < count; i++, pte++) { + pte_t entry = ptep_get(pte); + + if (!pte_present(entry)) + break; + if (pte_pfn(entry) - pfn >= nr) + break; + } + + return i; +} + +static inline bool allow_mlock_munlock(struct folio *folio, + struct vm_area_struct *vma, unsigned long start, + unsigned long end, unsigned int step) +{ + /* + * For unlock, allow munlock large folio which is partially + * mapped to VMA. As it's possible that large folio is + * mlocked and VMA is split later. + * + * During memory pressure, such kind of large folio can + * be split. And the pages are not in VM_LOCKed VMA + * can be reclaimed. + */ + if (!(vma->vm_flags & VM_LOCKED)) + return true; + + /* folio not in range [start, end), skip mlock */ + if (!folio_within_range(folio, vma, start, end)) + return false; + + /* folio is not fully mapped, skip mlock */ + if (step != folio_nr_pages(folio)) + return false; + + return true; +} + static int mlock_pte_range(pmd_t *pmd, unsigned long addr, unsigned long end, struct mm_walk *walk) @@ -314,6 +366,8 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, pte_t *start_pte, *pte; pte_t ptent; struct folio *folio; + unsigned int step = 1; + unsigned long start = addr; ptl = pmd_trans_huge_lock(pmd, vma); if (ptl) { @@ -334,6 +388,7 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, walk->action = ACTION_AGAIN; return 0; } + for (pte = start_pte; addr != end; pte++, addr += PAGE_SIZE) { ptent = ptep_get(pte); if (!pte_present(ptent)) @@ -341,12 +396,19 @@ static int mlock_pte_range(pmd_t *pmd, unsigned long addr, folio = vm_normal_folio(vma, addr, ptent); if (!folio || folio_is_zone_device(folio)) continue; - if (folio_test_large(folio)) - continue; + + step = folio_mlock_step(folio, pte, addr, end); + if (!allow_mlock_munlock(folio, vma, start, end, step)) + goto next_entry; + if (vma->vm_flags & VM_LOCKED) mlock_folio(folio); else munlock_folio(folio); + +next_entry: + pte += step - 1; + addr += (step - 1) << PAGE_SHIFT; } pte_unmap(start_pte); out: