From patchwork Fri Feb 2 02:20:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ming Lei X-Patchwork-Id: 13541994 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7BE82C48286 for ; Fri, 2 Feb 2024 02:21:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 99EB56B006E; Thu, 1 Feb 2024 21:21:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 926B36B0071; Thu, 1 Feb 2024 21:21:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C7816B0072; Thu, 1 Feb 2024 21:21:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 66D316B006E for ; Thu, 1 Feb 2024 21:21:06 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id F149480633 for ; Fri, 2 Feb 2024 02:21:05 +0000 (UTC) X-FDA: 81745261290.09.E2EFC66 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf22.hostedemail.com (Postfix) with ESMTP id 56AC9C0014 for ; Fri, 2 Feb 2024 02:21:03 +0000 (UTC) Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Y0snY1EF; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of ming.lei@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=ming.lei@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706840463; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references:dkim-signature; bh=/AFNq394fKdD6rutt1qh+bLin+C/DGw45kLcKx0YqTM=; b=8KUnMxhsgTfOacRXkB17oWcnvXFeproY8mUQSjohCotqHTNfmJ/UTY3XO2nXK5CMkOOFNE 5brYUGNzkh0uecsm1sPJLvsxeqtlkkupDI1HrOehgJKPjou8JELrbXadSWq/mV9OcjhT20 CkmwxAgFE/uwoY5YoyzQ2jr8o9eI9+M= ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=Y0snY1EF; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf22.hostedemail.com: domain of ming.lei@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=ming.lei@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706840463; a=rsa-sha256; cv=none; b=Hz1fZL6rBxQcFNaQPvSLYLDQsXDtI1sJ6clkKC4p0+gNndfH2BjlGnv9o9btDFt5c2hK9z qDLgF+5uveVM2jr66V+takqPe5lY85RTK083xQN722kNn0XC4orQbMhmhaDECGUuA+FNJJ N+ZpyYZDjBfWZsfGk/pNE9uAL12/ea4= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1706840462; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding; bh=/AFNq394fKdD6rutt1qh+bLin+C/DGw45kLcKx0YqTM=; b=Y0snY1EFrVIIZtWusiS6DVRH33NwRD2bCxPn3CaL1q1oHyQjqAHr1s12c3CmpX0ONhww+M f+Y00dUhTs2l+u5zm9h10xyG/jKhbTAvqE85kpsX5J2bBw+VqGNKDxDcF/rQYQSyjsQeOR mOoC0/oThI2etkHZJUg+wY2O3KezYFk= Received: from mimecast-mx02.redhat.com (mx-ext.redhat.com [66.187.233.73]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-124-aBo1uFCGPYOOyb7-_Gf_gw-1; Thu, 01 Feb 2024 21:20:58 -0500 X-MC-Unique: aBo1uFCGPYOOyb7-_Gf_gw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mimecast-mx02.redhat.com (Postfix) with ESMTPS id 975C729AC00A; Fri, 2 Feb 2024 02:20:57 +0000 (UTC) Received: from localhost (unknown [10.72.116.16]) by smtp.corp.redhat.com (Postfix) with ESMTP id ABCD01121306; Fri, 2 Feb 2024 02:20:56 +0000 (UTC) From: Ming Lei To: Andrew Morton , linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: linux-kernel@vger.kernel.org, Ming Lei , David Hildenbrand , Matthew Wilcox , Alexander Viro , Christian Brauner , Don Dutile , Rafael Aquini , Dave Chinner , Mike Snitzer Subject: [PATCH] mm/madvise: set ra_pages as device max request size during ADV_POPULATE_READ Date: Fri, 2 Feb 2024 10:20:29 +0800 Message-ID: <20240202022029.1903629-1-ming.lei@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.4.1 on 10.11.54.3 X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 56AC9C0014 X-Stat-Signature: kecghn9xkxpjd8qro6gfcfs3p53gu9je X-Rspam-User: X-HE-Tag: 1706840463-426387 X-HE-Meta: U2FsdGVkX185797m12KrscbMnlwUYiX+zlsp00nx7KDrJzWzkwt3mgEQGSpdZxyFgZ0FjEbf6fv2+1POOVlXEB4/aOHHHLRmyq7pikjsd5QWqYiQn7I38kfipo5nrqKjx/m2N8vR+Kz71FprLg6a6BPtfAxaFy7WVCcqFHSZBo6YR3nn7sjDs/G8cGRpGHvkGGDcTfZiRB/cjURXPjd8GffQDw/WAmRx+SMvSSchzxHm+JDntJaXBZagsdlZ43sBS7bDuXz3f54iFnrZtonz4nWyg4U88AogKZ3sRT1xjRmlAYepUVvbbP/5Qp63lAkQkaUVqosaFWw2xvpDH8GWklsd1ysKqAzLLBbIgt0iQ1iECp372zlj1bhxsKPIbMwYPVXBjLybCv7+VaqjsRu7qOGTGf60lDaLf5rYHrUu24PaBxu44Ct6YwBiT8poEMe6w2p2/TATbQ6G8/YTdvKkmKWd3pyjUWTby8i0FI8OsfJfy0g4F39DHbFKgCb0QT4uHQUfzbTTqXJEKXPO68skapYnSkAx1RSXpeux4YzkMnhB0BAQ5nyDAgTF9/0bD/W51MW9Lzy+0PE03MbH3qope7EohRQPq6nNcxrDk15+qtR05HswkDUlsEQXBF+fwzJkWM4o8zvmjK1HygCbznvUHeCkAGBIS8TUmkZJNfQtTjbqgAgamGebrjUply2eW45FnRv2813r1xt33ungvbHjZpPtllZsJTfKf17zwlUW6b29NdNzWxYKkg41CUJiifnUHox83Xbjblw9cfSx9Jwfjvva8W2ykBdB/sLZTCPSQA7aG399rVqMzNWFRN8XSMpDiuUQUxfujf+5Rr/O659y4VPiCLDB/8iJMiGnFIbvujv2j7bOA88U7Z3BXYy3xgO0fkJTDUCgjOzW/5Po1XFlfbX6+2V6m8vwQiLNMaVVS20GjPoQ5pFbbrfQmgfjFZlmVsuLw9cEdQZv/mlxMjb /dFd8cMQ 86Q0M3R5PlIooyyhKI0oyh64Bwbb9vcGPtuNLfzHvmJV6FeYqJPF7SYZMCsZFiParkVHXu8R//IsFwkEeajzsrkFYhReC6kdbfRMeIaJy2Cdgfh4VHKOz1eF7PxNoSRzlh7AID/2MjZkF+OaUjtxadXGNmIMyTUqLxqxqbFJbePDx9b1rjfbpNgn+JAPsvYj15zNB71r2eILFZfkyRF/DmMUP8cdaMfX0/qoI8GO1usOW5gHQl8l23AqsA8aFnxcJQYAA+GymwFGKgIFWiMrBIXhX94NYrAYh76b74XEqPKMNJB2vpZpnKzAtFb05c1k8dC32KZ7eKNnFKXVcIiipSKXxrOfTwP981ereldoEYB4S4pvE8xadKbwbCuvx5YH1yyMujz+lO/3e2xVVOJDIlqrew8wtnjrW2z8LWpBSq6TGUiQ40v8xik+nKhRf/pWTltD/E7BymqnOr7h3SpiBHXlAJld4XsTMHwhG8Xy/nEj6kEYiL3s1OVwzE01s33jFCJpicL0iIesXgiI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: madvise(MADV_POPULATE_READ) tries to populate all page tables in the specific range, so it is usually sequential IO if VMA is backed by file. Set ra_pages as device max request size for the involved readahead in the ADV_POPULATE_READ, this way reduces latency of madvise(MADV_POPULATE_READ) to 1/10 when running madvise(MADV_POPULATE_READ) over one 1GB file with usual(default) 128KB of read_ahead_kb. Cc: David Hildenbrand Cc: Matthew Wilcox Cc: Alexander Viro Cc: Christian Brauner Cc: Don Dutile Cc: Rafael Aquini Cc: Dave Chinner Cc: Mike Snitzer Cc: Andrew Morton Signed-off-by: Ming Lei --- mm/madvise.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 51 insertions(+), 1 deletion(-) diff --git a/mm/madvise.c b/mm/madvise.c index 912155a94ed5..db5452c8abdd 100644 --- a/mm/madvise.c +++ b/mm/madvise.c @@ -900,6 +900,37 @@ static long madvise_dontneed_free(struct vm_area_struct *vma, return -EINVAL; } +static void madvise_restore_ra_win(struct file **file, unsigned int ra_pages) +{ + if (*file) { + struct file *f = *file; + + f->f_ra.ra_pages = ra_pages; + fput(f); + *file = NULL; + } +} + +static struct file *madvise_override_ra_win(struct file *f, + unsigned long start, unsigned long end, + unsigned int *old_ra_pages) +{ + unsigned int io_pages; + + if (!f || !f->f_mapping || !f->f_mapping->host) + return NULL; + + io_pages = inode_to_bdi(f->f_mapping->host)->io_pages; + if (((end - start) >> PAGE_SHIFT) < io_pages) + return NULL; + + f = get_file(f); + *old_ra_pages = f->f_ra.ra_pages; + f->f_ra.ra_pages = io_pages; + + return f; +} + static long madvise_populate(struct vm_area_struct *vma, struct vm_area_struct **prev, unsigned long start, unsigned long end, @@ -908,9 +939,21 @@ static long madvise_populate(struct vm_area_struct *vma, const bool write = behavior == MADV_POPULATE_WRITE; struct mm_struct *mm = vma->vm_mm; unsigned long tmp_end; + unsigned int ra_pages; + struct file *file; int locked = 1; long pages; + /* + * In case of file backing mapping, increase readahead window + * for reducing the whole populate latency, and restore it + * after the populate is done + */ + if (behavior == MADV_POPULATE_READ) + file = madvise_override_ra_win(vma->vm_file, start, end, + &ra_pages); + else + file = NULL; *prev = vma; while (start < end) { @@ -920,8 +963,10 @@ static long madvise_populate(struct vm_area_struct *vma, */ if (!vma || start >= vma->vm_end) { vma = vma_lookup(mm, start); - if (!vma) + if (!vma) { + madvise_restore_ra_win(&file, ra_pages); return -ENOMEM; + } } tmp_end = min_t(unsigned long, end, vma->vm_end); @@ -935,6 +980,9 @@ static long madvise_populate(struct vm_area_struct *vma, vma = NULL; } if (pages < 0) { + /* restore ra pages back in case of any failure */ + madvise_restore_ra_win(&file, ra_pages); + switch (pages) { case -EINTR: return -EINTR; @@ -954,6 +1002,8 @@ static long madvise_populate(struct vm_area_struct *vma, } start += pages * PAGE_SIZE; } + + madvise_restore_ra_win(&file, ra_pages); return 0; }