From patchwork Tue Dec 3 15:31:37 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892618 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 242DFE69EA6 for ; Tue, 3 Dec 2024 15:32:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD38C6B0085; Tue, 3 Dec 2024 10:32:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A82C36B0088; Tue, 3 Dec 2024 10:32:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8FC8D6B0089; Tue, 3 Dec 2024 10:32:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 72A4D6B0085 for ; Tue, 3 Dec 2024 10:32:50 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 3150A80A60 for ; Tue, 3 Dec 2024 15:32:50 +0000 (UTC) X-FDA: 82854039954.07.B7DBB5A Received: from mail-oi1-f175.google.com (mail-oi1-f175.google.com [209.85.167.175]) by imf06.hostedemail.com (Postfix) with ESMTP id E85ED180002 for ; Tue, 3 Dec 2024 15:32:38 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=l6LJQLx5; spf=pass (imf06.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.175 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239958; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ReI+Zr1x7wQ/SlfmJGhzAq3WrMPRqrYdy9B8AN1mepA=; b=QiSGLpUONlKa2GT8sce9i2dUSAQTy3lJInOhZJluLRLh1FCmGYdheVyxY0a91Tdu/s2Nqb L8CraG9zuvuXciIMF/897Zg+VFkZZdSntRFwhVLYN5v1UDp5+WAnwGWs5wv7omIB5k7VtS wGktfiMWsGkvm32Mm6L3HC7+gkO6deI= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=l6LJQLx5; spf=pass (imf06.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.175 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239958; a=rsa-sha256; cv=none; b=eRCurWrQ9ycF03NwVTin3oxYATfZpTIAHwNI9+kFnY7MaD6PEoqfgrtR25V46b925FrfAN LAv7TllbSHCACQ6TMhk7/H2wfwfAkzEEb8hXfGAg0Y8e8MSU2fVMOPg+CuI5Zeuct16lI/ Dg9Bf39C5DQYINcBbXvJSfCFQUnhsT4= Received: by mail-oi1-f175.google.com with SMTP id 5614622812f47-3ea47651a10so2502008b6e.0 for ; Tue, 03 Dec 2024 07:32:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239967; x=1733844767; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ReI+Zr1x7wQ/SlfmJGhzAq3WrMPRqrYdy9B8AN1mepA=; b=l6LJQLx5x0hWs03XVt1uRT9ROb10+MLH0v5g5qJjuBSUvpkI5H8sFyvImA/sq2yO6i Nigp6oEsixUr6N+ZswEMOzHNO7qxrxHh73Z+VIhxcpZBlkCxP0kQxsqa6hb2z79v9PRW tkAQ0CdfentDGoZ9e0nc9us1iNxuBCCOQLZAfmj8d6MILxzDpCdINZc2n/ahCV2MMr/d rrw4MKJGZ2mR24Y5IT49XAxoGcVY8LkeQy81rTNJFzR8QpMzTF/tjn1BjpdSbfFL0WV9 IJWpcfYipThaQ0XYa4JnWiilrztpFEb+B+M37hB2sEIu9vb4r1EvppBGi6WN+LvGIFba 8SEg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239967; x=1733844767; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ReI+Zr1x7wQ/SlfmJGhzAq3WrMPRqrYdy9B8AN1mepA=; b=dqEUAmSy9Kmw1HE7S7mtTANDz+57GJvKwWV+0cjukBSbvv6qR4nu/mc0PI+EMk2dBW HPE6PiRNRhlUB7jsRPgvq5H9EsdbjagguZxUY6tXcXoRBWEf3pazAucOF0ccGu404z4J nLUqlYD5tGraPHEQ8eyXFLpElaxjm7BtUToLVJTVnN/MX/tq2ZlATASJBVgVGLY+8k42 o+hxrVo8IZ4WvFK2NSCQq7bMsPtdtl1V0JWq+AfX1dVnx7D9rfWEcS3eT0pMGRRmnwOI 74Hg03HcQPlG8LpKVxyfYCIJAWEbefHrHum8cZtOtvCd6VMqlWzBCTOlpZSH52msQ8Jf Rphw== X-Gm-Message-State: AOJu0YwfCEv2eL7zOOo64LYN8spHIUGiLoX6CIO1R5hDhGWwcsOKobFJ nSMLxDg9EB9xaPa7sabRXJR+M+jMdVSAg2BEtqzatlmyoSjinSw2nXb6VBKUFYyKAmkK+jvSKZL r X-Gm-Gg: ASbGncvk7JuQ3im4UWbu7VA7FXXhXIlPiyV6ya1OZ3UCI6VYB6QzphD+bQDJ1oto4SV mE5a2TYLVYK7pd/+V6BBw6t00ghw+Ua7jMC1RG191so4RbAE5ByhyUhqFdz/hx5DfaCk4mHWP/f BV/auf1DxMPhuJ/rIv4ZXzbe0I4tyj02JEnTcPZdl06dN+tumxv914gdJZwje984wMH4AQc6b1y dOf7QRz08hk6TRFzPo6NTWK8R3x4+Douq3EFvAXXIvy8BB1r01i+uqkAmc= X-Google-Smtp-Source: AGHT+IEjGwc7GFJjqyvYWDgJBqgGGBcMfQCEEJJRf9YUZZ1W6BgQFHNlpJWElLeXDmEpw+9XPHnUvQ== X-Received: by 2002:a05:6808:1406:b0:3e6:1ea5:6b30 with SMTP id 5614622812f47-3eae4f9162amr3792647b6e.24.1733239966815; Tue, 03 Dec 2024 07:32:46 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:46 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 01/12] mm/filemap: change filemap_create_folio() to take a struct kiocb Date: Tue, 3 Dec 2024 08:31:37 -0700 Message-ID: <20241203153232.92224-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspamd-Queue-Id: E85ED180002 X-Rspamd-Server: rspam12 X-Stat-Signature: jtwd6df8homtkrg5de8i5djb1kewecc6 X-Rspam-User: X-HE-Tag: 1733239958-421543 X-HE-Meta: U2FsdGVkX1/NGC0ApLaZIR9o4kPcnopkw3reHoDwY482G4Mf849vC9UrlO3/D1Wil6LqjirXWnjxVvIcubKn7jYroaRqPznMoNnmKexzX9TUvHuePuhjXAuMwpAzEERS59RobwBiB8U2QgfYctOiAGm1HUkXiXzTkTtSzAJYktfSfLKftQs+o3uwMaX75YHRh/Yef6X/kbc6LD7NVjcKqYh7eEHyazQUkJ14tkLIyHQ3rer64vsVmIMFa5LF4Rp1Ofx4Kc8xukyltsFpUmlY/Mpzv8RPjNEmqI7lp26xmepgoJ+Q0Qz+NigQ7OKqKWfSRYli9QfFil/LKF5FrwgdLPK+jlXTXM77o6AzSIlKarho9ftSmEuXGcff3P3UNoS6QKtvBQfN1Q5SaYkKmNW3SFivEDD06WDyFQjM2yrkbc6pPHxZffYLwNB79L1W+sXJ4yX5pjsXHucEJRHBomxbmdoK/bAh2rsx1+XuV39r8fadJ20Z6ShDJAO1LHprEJcZWSX+Ka7ya/xS8lkqz00AZRI5uBvr5my8ycHMSI4TTcRVz7fCaZXeA22bgqocuMUDQxyI+Voh0pVwkQpA8A4gEOOZzlOfLn+/P77TDvQQHoooEzMa7JMB6fX6Q68nlXcAbNTBgxLqQhpxPppYSHA05T4R9ksYI5c8wItyoRYI58Jr4o5sbvETwIWqTXJWYf0f5rNd0RIdM+agCj0FgHo69OaPcJNiLfVqDhP2UtGbkRsD+tYKwW6h5xm6+MR+kKkLeosSVcTRfg0OkXL/zn11Zj+Nw//dJWp41bEdE9WbqsAMTeiTLdNO9d7q5o3ciT38T5qDXWDvTl0X7b1WfQfXS/l0nnfA64V7n9xr28rUZJpGnfuoZgQvLjUojLY+fNf1MWeEgIXVrY+1TxycZI83RrxElQ8sco3TX2EOiL21sPpfUYMWmQ4T4YddXMuI6pm4DrBXscc7m/WDZWh1b/t O6jndVHx 3rHuUqbC1bnGiktt0gyWciIqH2W/7/vljsKoD5RH8SP5Op5kz/8o8f5/+qnnzW3C+PdALemvwfzwiGD7Sr3zs4LVlf4BSYbOoKnXC3cpjm+sGwLdIjk4yWoWgMwfoW1UHrp7OxLXLxgts2bSGXMs1yJ53x8j291Ib14anOXxJaCfXT/ChKr4o2xnu+OIMmgAIofCuTPKUgk9usvrsOuTB7QtEdpv0oV6Af2emOH15BBuHnR3qGCpe14NJZ2XBGhl1GOQP7GT0Xm4uBYFOQ9/zgBPd2cMkCMaTjfi8QwCvqp/bzogLVNCcp9dnHglBscLuK4zWUDZi7mdHQSyITons6EVDjHVSukKmMlYZ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Rather than pass in both the file and position directly from the kiocb, just take a struct kiocb instead. While doing so, move the ki_flags checking into filemap_create_folio() as well. In preparation for actually needing the kiocb in the function. No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/filemap.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 7c76a123ba18..898e992039e8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2459,15 +2459,17 @@ static int filemap_update_page(struct kiocb *iocb, return error; } -static int filemap_create_folio(struct file *file, - struct address_space *mapping, loff_t pos, - struct folio_batch *fbatch) +static int filemap_create_folio(struct kiocb *iocb, + struct address_space *mapping, struct folio_batch *fbatch) { struct folio *folio; int error; unsigned int min_order = mapping_min_folio_order(mapping); pgoff_t index; + if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) + return -EAGAIN; + folio = filemap_alloc_folio(mapping_gfp_mask(mapping), min_order); if (!folio) return -ENOMEM; @@ -2486,7 +2488,7 @@ static int filemap_create_folio(struct file *file, * well to keep locking rules simple. */ filemap_invalidate_lock_shared(mapping); - index = (pos >> (PAGE_SHIFT + min_order)) << min_order; + index = (iocb->ki_pos >> (PAGE_SHIFT + min_order)) << min_order; error = filemap_add_folio(mapping, folio, index, mapping_gfp_constraint(mapping, GFP_KERNEL)); if (error == -EEXIST) @@ -2494,7 +2496,8 @@ static int filemap_create_folio(struct file *file, if (error) goto error; - error = filemap_read_folio(file, mapping->a_ops->read_folio, folio); + error = filemap_read_folio(iocb->ki_filp, mapping->a_ops->read_folio, + folio); if (error) goto error; @@ -2550,9 +2553,7 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, filemap_get_read_batch(mapping, index, last_index - 1, fbatch); } if (!folio_batch_count(fbatch)) { - if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) - return -EAGAIN; - err = filemap_create_folio(filp, mapping, iocb->ki_pos, fbatch); + err = filemap_create_folio(iocb, mapping, fbatch); if (err == AOP_TRUNCATED_PAGE) goto retry; return err; From patchwork Tue Dec 3 15:31:38 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892619 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9BC01E64ABB for ; Tue, 3 Dec 2024 15:32:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 55A3E6B0088; Tue, 3 Dec 2024 10:32:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4423E6B0089; Tue, 3 Dec 2024 10:32:52 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 296D66B008C; Tue, 3 Dec 2024 10:32:52 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 063BC6B0088 for ; Tue, 3 Dec 2024 10:32:52 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id A759EC0A01 for ; Tue, 3 Dec 2024 15:32:51 +0000 (UTC) X-FDA: 82854039744.09.5801185 Received: from mail-oi1-f181.google.com (mail-oi1-f181.google.com [209.85.167.181]) by imf01.hostedemail.com (Postfix) with ESMTP id A6C134000C for ; Tue, 3 Dec 2024 15:32:39 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=iGaGUsAi; dmarc=none; spf=pass (imf01.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.181 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239963; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5mQz2vdNaN9LdJ9L4qyJBm7ibQRSfX5SzH0nqDyaIZA=; b=aizupOTEF7MXDv27zlO5cMuL6iX6dGKOEoJq5tHNQKndm9DgqkMsZ2g93TbXouSVRntcUW Lu2jlRKlZiiwZmqbufG0K+hDSTWFVfRHFTGtgFJA8SdznRF63hUOtp3oLT3hL6w31zBbGT gzSQ+LqO7GYKtb+tFgzUeZwn8hc2rm0= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239963; a=rsa-sha256; cv=none; b=yRpZ2GX5dvwgAAT65Dllqq9V3BWXPSSL40nNJqQcSTUglm7Ozx/Nnvgmneh/A4ZsTrryb6 W6ZlmqMjSlUvFxwtzX4/1UUrH2FJJnqVDCzaz5jjGSwQoWf0i3mmtrWhi+J5ZkEwLuSA/7 hxdhBeqCExzo5Om1tDg8fsDXrXdD57Y= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=iGaGUsAi; dmarc=none; spf=pass (imf01.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.181 as permitted sender) smtp.mailfrom=axboe@kernel.dk Received: by mail-oi1-f181.google.com with SMTP id 5614622812f47-3ea552bb91aso2362576b6e.1 for ; Tue, 03 Dec 2024 07:32:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239968; x=1733844768; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=5mQz2vdNaN9LdJ9L4qyJBm7ibQRSfX5SzH0nqDyaIZA=; b=iGaGUsAiPGqpwrCDD4+5nkz4d7R49osJBn+6IF39ckasruGOFWBwnNv89bB74Tbf63 brdlx7q9U0fC6kDQWQUZqffDRt7Wyfbrcz0FiQSkk02nnGT+F0+EvVo8DsOb+vLrjNTB Hz/DM0WrsrBPZ+EdjO/igKxlXlncIZ5YQf1IoBe3bJq8PELwet/rvLrnjeTQoyeKEIf2 ls5YtSmZbed6zv2K1bvggFqMzEWg9ylSAmLBYAllyiL4yRx+wKJBMyqruDBmALsA/Vpb +kQ/eCK+zTagtLaV+LnvJrmRLCIAOeQwm72Ul9EeYN2QYGddR8ej/z8y9rKM+lywjdaO kiCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239968; x=1733844768; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=5mQz2vdNaN9LdJ9L4qyJBm7ibQRSfX5SzH0nqDyaIZA=; b=RC+e2JFObEKXH/FWXHjz5cDiGKhPHMiHlyAN/RZ0JoFANNPyAVZurrCY52vYtrdU8e v6vI7hyEyxty0EK47F1V+EsWgTVVb9DXb8DD7sSJEqArbSFoCIurZOfxkwbODrehCe0K yEgcLcb5mTbLGvoKtjgxtBdco5dzCcT//oLzUP+XjIeIX9QMMd7l5eEGQOL+A/0zDWvv 6Twttb+5G9EN8DomIw4uq2qqfcdNnQf9qTjIO1w2iQsG7v2lVORSZjGNVuG85ezzt0FI u4tdz6JfKidNDIj6OwiRZdBDW/SAaWwKxd0/IRu3i9ooktc7kvvno8ma6UgM2QS6D84v cFnA== X-Gm-Message-State: AOJu0YzmFBBWZDlZL+LcLjz/jal8GskHE0zUrJj9Kvgxz1utFOEwuVhh 45p3rhx6y+R9Sp0UPuVcrWTNtqqo9A5JtgMwLzbNirqESfrwUwfxaeQrb/L2YlNa7Jumocmghzr G X-Gm-Gg: ASbGnctxdKtUJKifeVBhbVuWFgIL2SzcvphNP0o5BgliOIndyoGvSuYC7txak4T3AYn u9kPh2gp4rPUYyc1P0a9dvNb6XwVh+8EV6vtubUAbxsVQ57O4gGoUmyUY7lGqb9gfz+wGzwhYOR KWZ5YZnyX47dprnmGiWnIjn1BEej8evAJr71iN3fVptoywEiSImZg8sYHOOFToaZyF3glA4ObN5 sErr8utg+MslnBUIjxWqyqQonp2+lQqs9gmYGSEx4KtK0jv9l6s7nIidtc= X-Google-Smtp-Source: AGHT+IFPC3QIjfGmFueYpArbFjHsCE/nJs1cTRfjgQkZc7hdX+t3EH5eMgGO4h5vvPNqW5LyAgQfEA== X-Received: by 2002:a05:6808:1454:b0:3ea:45d1:de13 with SMTP id 5614622812f47-3eae499b2b5mr1606169b6e.11.1733239968341; Tue, 03 Dec 2024 07:32:48 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:47 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 02/12] mm/readahead: add folio allocation helper Date: Tue, 3 Dec 2024 08:31:38 -0700 Message-ID: <20241203153232.92224-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Stat-Signature: afis7rwwauezhkzo8m98axwtyjigzyuq X-Rspamd-Queue-Id: A6C134000C X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1733239959-814584 X-HE-Meta: U2FsdGVkX18zgXsAAy42QRGJfVmQ/+b4sStUI4wLdzxwjIPJeeDm3SC/2wYhfKst1dfqW4LVql7iVLA85teqVW/GGT/ZJDS6eUdag1koxiQzIinYE3sKIFdiaJuLLw2S54LIf/Tmk3rAbBZ67Jf44v0Jqzp4AwyhCUnTCBAmi84mHnzJw/Xek7pACuF5mzmH8Y72iA4WKcl/xILnQhrkfp+FXBlCw5/v591Z0fSyEZfi/CWjAQtFFxQwGivsr36MciTmiM9edQTLrg4FfccoZUwltv1mH79dza4U1VQR3JAAYPyHeCxkGLG9Ip80PrzCDYr8U5m6Tz0iBmtGHjKpz/assMQxLczvoAb/La8GTsaee1PCZCh3Y+ch4hkHm9P+bXkvxT65TV++X5W8ani0YRbj+oMgncNNOk1F/aBPMa5ZtnV9/UayGAIXCXYd+J62QtsU/VFyHStgDUx2ZUsPsghNeAdAq9EqM8ynPi41IcdXFnKEwU2jWxeeMl2n120tSLzcY6FtXLuk+7OJtqjgnWU554TMX6aWzsB6bfXZtR1N5V1SvGcU9m9SISL0i/GG1/aP/R6Dzp77OSBePcYrxdDXS04n8JqzVAoPk4GmmSdLuMQsvSjVfBd0LBF+uJVQnsbW4k9EsDuJ/mNVzeEWGjcbXT5IFjsrHX5+xIP7LSbwjJIZlhFsCyxqht5Mmdh2XD3IaRpmKXwr3rj3JSQQtT3Nx4pRZQl/2Iwfv3esYd0ih1On/A0yI9tAkxKF119NAAV6DCuGNctjOhmpzpvWX7oa6yK68G9bPmIRNXDfVUAWJSbwWDDtkUvtIkRqw3/j7nHJNU63QHLkPkSC0yz3etglTpqlM58SSPquiQuNNxh9fJphRbd2KrMdiXvHTbhGymiZIpqqmbgq+zO2SLLYDFtLdgoKUX69/5gKkPbvcxkX0cnVMAAcxGD9Lo+Uhyp7Dy16+h2c2Giyat3AFl6 OVyLAePA 7+Ir3Y02CyaY4lMS3SkXTH15K9Vrkm8AYfBhpvZUmL/YimWkppOFGudDGaaqfU1J9c8+i7wJ/zDr2AjDjRIiYKgVzBuoC/bBpNsQgGc6CON8QWLu1Jz6yBV0SCeVE28wLJQcUfi7Fmahsu39eklUdR8yXqnbTFjom+iIMOe+3Kgv7eeLTtW62x5Zu6bitE5UFk/sp2X/8K5Uvma7nlv6UlhhxbNh8pbA9jZrc3GoVmg7Bq0oDKZxBPDoQPbaUPBE0S4/J33DbkQr0IM+QKTz3wBNXpvQBstSFHs21Ya7g1qQotv4= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Just a wrapper around filemap_alloc_folio() for now, but add it in preparation for modifying the folio based on the 'ractl' being passed in. No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/readahead.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/readahead.c b/mm/readahead.c index 8f1cf599b572..8424bf1b67e2 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -188,6 +188,12 @@ static void read_pages(struct readahead_control *rac) BUG_ON(readahead_count(rac)); } +static struct folio *ractl_alloc_folio(struct readahead_control *ractl, + gfp_t gfp_mask, unsigned int order) +{ + return filemap_alloc_folio(gfp_mask, order); +} + /** * page_cache_ra_unbounded - Start unchecked readahead. * @ractl: Readahead control. @@ -265,8 +271,8 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, continue; } - folio = filemap_alloc_folio(gfp_mask, - mapping_min_folio_order(mapping)); + folio = ractl_alloc_folio(ractl, gfp_mask, + mapping_min_folio_order(mapping)); if (!folio) break; @@ -436,7 +442,7 @@ static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index, pgoff_t mark, unsigned int order, gfp_t gfp) { int err; - struct folio *folio = filemap_alloc_folio(gfp, order); + struct folio *folio = ractl_alloc_folio(ractl, gfp, order); if (!folio) return -ENOMEM; @@ -751,7 +757,7 @@ void readahead_expand(struct readahead_control *ractl, if (folio && !xa_is_value(folio)) return; /* Folio apparently present */ - folio = filemap_alloc_folio(gfp_mask, min_order); + folio = ractl_alloc_folio(ractl, gfp_mask, min_order); if (!folio) return; @@ -780,7 +786,7 @@ void readahead_expand(struct readahead_control *ractl, if (folio && !xa_is_value(folio)) return; /* Folio apparently present */ - folio = filemap_alloc_folio(gfp_mask, min_order); + folio = ractl_alloc_folio(ractl, gfp_mask, min_order); if (!folio) return; From patchwork Tue Dec 3 15:31:39 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892620 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3A3ABE69EAF for ; Tue, 3 Dec 2024 15:32:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8FF7F6B0089; Tue, 3 Dec 2024 10:32:53 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8B19B6B008C; Tue, 3 Dec 2024 10:32:53 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7509D6B0092; Tue, 3 Dec 2024 10:32:53 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 52DA96B0089 for ; Tue, 3 Dec 2024 10:32:53 -0500 (EST) Received: from smtpin04.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 08090140AA7 for ; Tue, 3 Dec 2024 15:32:53 +0000 (UTC) X-FDA: 82854040332.04.74919BB Received: from mail-oi1-f177.google.com (mail-oi1-f177.google.com [209.85.167.177]) by imf05.hostedemail.com (Postfix) with ESMTP id C9F8510000B for ; Tue, 3 Dec 2024 15:32:25 +0000 (UTC) Authentication-Results: imf05.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=VaZvsrj7; spf=pass (imf05.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.177 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239958; a=rsa-sha256; cv=none; b=WqvxA5DBEd5MjZmYbknL8AjoS2Qj+zVJFcFesFJJsjxJG60dBgScpw1l4wqPLJDsN+yZqY Lj8DyKAqOPCMJYxakOdJAu6IsiA01vl/+jP5qKrl7emwigAFEJ2WMghkHb45qoTWIG+496 JKqzVzUCvPlRuUDDvhNFU9dQgc8qKqg= ARC-Authentication-Results: i=1; imf05.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=VaZvsrj7; spf=pass (imf05.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.177 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239958; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=dSQf4jIfLBAkAFBsDD5CVUxWLID5QVKIZqR96ciKoMc=; b=xOT2QBYtCW5a5ybr369S/UCiZWxTZZ+l/xqSt3dHlFdkWiJUQLZKDfrNSZGTyMPr0LCVrj eZQDnqnACWMTtkrwV5hjP5+xVq4f3NhcRbhn+mGaiPWvdB7kZEF/bGLq2l2nFwPE83JeHC CeR1yYcxM7WmdHu4D74Y+OjX2v63Wo0= Received: by mail-oi1-f177.google.com with SMTP id 5614622812f47-3ea55e95d0fso2782724b6e.2 for ; Tue, 03 Dec 2024 07:32:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239970; x=1733844770; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=dSQf4jIfLBAkAFBsDD5CVUxWLID5QVKIZqR96ciKoMc=; b=VaZvsrj7JQ1edIoZQHj9jOxy6WehrD48e5e5R392z4G6GdO59t5mjZl4VrnZpITase mv8/cbDleOokcf6GxZWxQB9u6BSZA4ZwagEZKnt8YON2u+ev7hGfajlXCrxzNxQtzHWm p63UvwnV4w/twYYYs7E5q5dUpgHFvRfNG7cA417wA9Hv/pVfvGynDLmH9apVTRsTsSJZ nvBa5I8cg/lMADkdAcXFF39lQMC3kRXEbmni352T5y3/A7OV7g4bPnU40gYug5fKBWzH sgoplhXniX3J2iLh/9ppma20285N48bOoFz5lw7GsqYAR7X6CKPuZmsvaXIzki8UWzmV HRXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239970; x=1733844770; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=dSQf4jIfLBAkAFBsDD5CVUxWLID5QVKIZqR96ciKoMc=; b=kqeRWZF/PQh97sQmLddVJjhIUNAgxUl31Qbb8S25K3RmMGVoG9t/4h86MDYw95S1Qj AxWQfImRSdjjqGJe6rOtXmcG7RlaL5bR+aatLujvUFMfJskc8dWBrN9ZGgzQp+Zv58zj l4BLr3YLA/8maQ0qNf8/S87E9/4J1wkyOFHuu0H6M9vWTq0tNht81Xp00ja2yGr4vzrQ QHTpcoNgKzyDfMxUukKWrCDrayClTXihlJFX2Hp0kEEUP9mFk7xEbyuBOxiGUX+dsju8 Y4WnZXYWxukf6GEvqEg6oqM9MP3mfsVtK005s0K6+maiggW+NJ5c+DQq3yLegX2Q1NGw pLAQ== X-Gm-Message-State: AOJu0YwIPfpNlHE0RT22IO7KY0rrzYTPXkGtxtUYHJbdXikexZePNEet SgeAbQORuPwYtrvBxLE7SDgkb65WAV5gktFH0DbaX9rxLB4aR+/AASTnRrk9FIbLESSR9AELonV U X-Gm-Gg: ASbGnctHGXAXhbCvCWD62zVpEh+7Y8XJSp56QG2KkgtWaKvDdo0Ng6jE+/j+9dDP+Lt dK5974CPfS8il7oaqvS4U3yaPBGJtr/0oOmpZ8C5M1XBmOJgJZ3M23y4um/V82pZ/gjbsLcZou8 p0IxZOmlmhSfd8l+jRxn29DxP8mmle2F7j4wUM8KUj60m+LZ+oAx1viQr9OyiuxStqoUDdGiM2R /a85iHGwI7fJuulkYH6e2Iool4sBgsamwxDdr0GGpU3ynwdE8W+nXHorao= X-Google-Smtp-Source: AGHT+IF59+hbl1oxKnagTF4XdkSvu7+brogdJGlmHp3dV6C6q4drpkxZSCaanT7IxLge7RXlKXRl5g== X-Received: by 2002:a05:6808:1795:b0:3e7:9e07:432b with SMTP id 5614622812f47-3eae505e041mr2729493b6e.29.1733239969773; Tue, 03 Dec 2024 07:32:49 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:49 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 03/12] mm: add PG_uncached page flag Date: Tue, 3 Dec 2024 08:31:39 -0700 Message-ID: <20241203153232.92224-5-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspamd-Queue-Id: C9F8510000B X-Stat-Signature: bi3xhmd8te1u5czu67449kmroc3ab14f X-Rspam-User: X-Rspamd-Server: rspam09 X-HE-Tag: 1733239945-845931 X-HE-Meta: U2FsdGVkX18K3l8eNBLrha3XEfHd0/2flTU6GJeaCBcbrBbY5RDo8tvT1bPvypeCrrZB6ZBRY+q4Zg1VxK+2rG/xJIlMG5nH1RS36ip//IdwmED2oXAYt7xhBjDpuuoNnGEXjdw0k9syRaWtcNpXq42XCsP0hjhTCCUhNxEVM3kxV5ZaFAY5kHzNKof11Qsx5TsKbnrAhrVK11u+2SdITF9mXdwviZmNsKAVoeJcP73k/yoc3LW0xAzganlzH/Cri7fisMWB+ud8mtVWnryfPRCcTGJRd2+/akE0raLQfjjWC3Uphn8wVRYZYIafIsghet2D4YCeBe0ul9gKRJWpyna/eW0FwExtnPUuu1XFGNHnOZqpJwclrS0WLEkGaqCnc/EFMUa70wCTVyy/ZUm/DPUbVxGqhexpZHB6s4xtR0leVGAZuZi14jWHqKh7SY1UpO9zU9uF3niYkTfCnx8slO/C6w4Crud/gWShe5X1wjh9Il9PVvhbT+7rfqipZMhzTZBVAePZuJENK8WJj35eVky23rx1g8PO47vTqxQ8mAesJneeccwMKAnXylnJx9LOjQjlL4gMtcsFtH7LghlaBw1vpK/kP2rjhNRREo5qRrnvak7DMZHTPLuuJmFHhLFZeVM/UzluLGe3tYQqQKMYf4TTtDvAmEfa3EFbDmuLBtoX8vnz30P5U2EDM6/Kp+HLxc7HLZC16+R/qlcSory4c0CMZ8hXmiX335013f2QtLHqcZUB8DyorsBZidhMZd1+0UVBou88n2QybQ+TvsBaPXZHLbz5e8J5b+EwchfIOP4AASLlLqpiRW5YI2KDqagGiBgbPr+4QpHJ+aXsQUIIYS7TIype2hXExJn6qpKqvsRltbJjEfOI5vNmroYEj60AQs912pxlrp9UKNU4ei3Cz42FdZ/hs2ZhhUw3W+97STXDolDBc35Jvdc1pHcJbc0Cit0/5a3hM5hmVz5UZps 5BWvW1ar /BXR/epqgqVE/GfQeXENsZ236hwaxWJyyuuWlu1YEM0RkZXcRdWVv3kwzmixp0waoOzFaItHbmeGwFkL6B1P+pWPht7vUtotpvzPIzBE7950uq3NOg2/LXRM2MVesokEji+53x3bPYiV6uOC3H118enHvsH6ehfh05T6TYL6kkXSWOE7PrIRH5Bf0N3qY+FcJ3O1ZJPRFCPQGRztExY37RmWFYHKNNW7s9rPr42ARNq2JZP4KUbMvQfwaqOJxJKMF1yulVdQ3zYb42BzP5x2M8QnH3RTR29TEhn6+Ugz8riWe2foa6ShBtAaXQLLYZiIrzJFNRMzlahBbICy5D6acMoy6kUYNN6JAT+OX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a page flag that file IO can use to indicate that the IO being done is uncached, as in it should not persist in the page cache after the IO has been completed. Signed-off-by: Jens Axboe --- include/linux/page-flags.h | 5 +++++ include/trace/events/mmflags.h | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index 2220bfec278e..14346fa2470f 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -110,6 +110,7 @@ enum pageflags { PG_reclaim, /* To be reclaimed asap */ PG_swapbacked, /* Page is backed by RAM/swap */ PG_unevictable, /* Page is "unevictable" */ + PG_uncached, /* uncached read/write IO */ #ifdef CONFIG_MMU PG_mlocked, /* Page is vma mlocked */ #endif @@ -562,6 +563,10 @@ PAGEFLAG(Reclaim, reclaim, PF_NO_TAIL) FOLIO_FLAG(readahead, FOLIO_HEAD_PAGE) FOLIO_TEST_CLEAR_FLAG(readahead, FOLIO_HEAD_PAGE) +FOLIO_FLAG(uncached, FOLIO_HEAD_PAGE) + FOLIO_TEST_CLEAR_FLAG(uncached, FOLIO_HEAD_PAGE) + __FOLIO_SET_FLAG(uncached, FOLIO_HEAD_PAGE) + #ifdef CONFIG_HIGHMEM /* * Must use a macro here due to header dependency issues. page_zone() is not diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index bb8a59c6caa2..b60057284102 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -116,7 +116,8 @@ DEF_PAGEFLAG_NAME(head), \ DEF_PAGEFLAG_NAME(reclaim), \ DEF_PAGEFLAG_NAME(swapbacked), \ - DEF_PAGEFLAG_NAME(unevictable) \ + DEF_PAGEFLAG_NAME(unevictable), \ + DEF_PAGEFLAG_NAME(uncached) \ IF_HAVE_PG_MLOCK(mlocked) \ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ From patchwork Tue Dec 3 15:31:40 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892621 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id F16F9E64ABB for ; Tue, 3 Dec 2024 15:32:58 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 380EB6B0092; Tue, 3 Dec 2024 10:32:55 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 331866B0093; Tue, 3 Dec 2024 10:32:55 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 15B126B0095; Tue, 3 Dec 2024 10:32:55 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id E742E6B0092 for ; Tue, 3 Dec 2024 10:32:54 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 77F9E1A09C4 for ; Tue, 3 Dec 2024 15:32:54 +0000 (UTC) X-FDA: 82854039702.17.99C93CD Received: from mail-oi1-f180.google.com (mail-oi1-f180.google.com [209.85.167.180]) by imf24.hostedemail.com (Postfix) with ESMTP id 0EBD1180018 for ; Tue, 3 Dec 2024 15:32:50 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=VJM6RuND; dmarc=none; spf=pass (imf24.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.180 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239960; a=rsa-sha256; cv=none; b=GC4KFlrtXzFmozVfNkA1Fy8XdRKTG0B5XNpBtTK4QTFF9DN4OHB2b2u2hZ7btbPcmZ0VUb w6y8SUC0ijxBBXBL0Ou3aT83XWx3pd6vTwuhehMgFjXIdNAgr/mushfPHYLj8xWVz9CDb0 AF+hIlShwb1DBTv1yGcINc2b1KpzkWI= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=VJM6RuND; dmarc=none; spf=pass (imf24.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.180 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239960; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=RwN531wgOLFM8EGeUWgrWwcMRYwBcN09BWbzS6SO3q8=; b=0o9Th8dHBJvYEVnhL2Aa93IN3d1bTtHViZSoa55TTa7AeObveZYA8/xWvRIwq9ZNDFAA9d HUTuzA6bgcXeLtwAlVFiWUZOa6/ATYCTkOy7uwTXtOVdKsU/c7gaDCiPP4lwwNgawYP4XG lcSk48K9GaAq58xtk2MDCxfCxFGJjUg= Received: by mail-oi1-f180.google.com with SMTP id 5614622812f47-3ea405aa7b3so2867587b6e.3 for ; Tue, 03 Dec 2024 07:32:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239971; x=1733844771; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=RwN531wgOLFM8EGeUWgrWwcMRYwBcN09BWbzS6SO3q8=; b=VJM6RuNDShDKwq6UoGr+kXupDV6UL46sqnAVMq7EX82n6WtKT+BWf6JY6x7spOPzXQ wo8C5+jUnAA4m00376F9Ft4JuPg2G8DSWqq1a/0e7OQZYPjB+AHDAZ+FFzLeTd2evXnV oA4FqFpFsxUX4BJD0EL9iYvhWBtDelmk1dHAm0cEiXX4ZZcaIwSs1vCIiVX+onhWIIYh 7Ey8x2JOrMjBAekl60GBk8fn1NdsxkS+ZaoGWO3Tv0HEKxlp41jxvhk93Ew68QnFFq5S K0w5D156ibn9Jk5kcaEfJfDjLZ7lKt+eVF0VcQQG6vEVFBP0XXaoxNVC7nVjksAuff3o w6uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239971; x=1733844771; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=RwN531wgOLFM8EGeUWgrWwcMRYwBcN09BWbzS6SO3q8=; b=oHUN/DX734/dHaoKwgKo4u1Ly89/r1qgQE+vJIC5xqfNCtHc5u4sRNQHFu6d2I0kZQ FI4cxLC6xQ0ekaCMqkVcMlfdtL1N5V0YkZaDwwemwA1sO5pdkTn2z2+dMVx05IVf+aA6 OUf99zIq6b/4tpb0kIDJ89DM8+mt3fZjeIPh0YxHg6m6QQCLOdOr7pvlvC6WNmxDPfvk mvy4G4MUKfx9MT9inJK844D1vMGp5O6o0mGJWtndoX0azrwDJAdeT/EmDoHJiZvbWNrV fS2RBS6AJaPOMq6QQ96CjzprHBPm6OaLr5pb893E9+AoSZmtSHP5XDtXKDYi5ya65JkL zleQ== X-Gm-Message-State: AOJu0Yxy92muY27WNWgXw9aBNeGTRERUL/pyEzbh3u4c1HmIMCxJlFO/ ROTiV+YCKLkMXeJISYvoEEOc3LuISUU/hb69m6hiAZx+ev1f2K6JI5i5FwObruSOIW7S4vF4pXr n X-Gm-Gg: ASbGncsIvY0RkjbX5eNBo06YvPiowsxJs21ORsyRZ3ZDjCMSYjrrM+Fg1jL/EBslhJf L1gQXfe/NaTqDlqw6k4pmYj2qonR/zW3TfQ49RT2uCpymxnNVC0qpMLEjuvcI7itiYiFi5NsKDR XbLviDWy6g2NG0SfO/nSsyhPrQ0jOW/Y9nrowWpjQc0ZbO+25Itm3UbaEiHsy8ZNzVyZxStmpyR G1d8IPG3AWLb1tbtAm36N8XSJxgVqX09C6JaMd7YvG+ZZS9OQwj4uJrUlk= X-Google-Smtp-Source: AGHT+IF9UmPrglgGWAW3JFFN5BU2DOVhhB7roaz8QIV4CUJFyvDEFcQOUcjiZ4xJCcSYYgdgH9P94g== X-Received: by 2002:a05:6808:1405:b0:3e0:c13:9837 with SMTP id 5614622812f47-3eaf041efccmr453615b6e.37.1733239971292; Tue, 03 Dec 2024 07:32:51 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:50 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 04/12] mm/readahead: add readahead_control->uncached member Date: Tue, 3 Dec 2024 08:31:40 -0700 Message-ID: <20241203153232.92224-6-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0EBD1180018 X-Stat-Signature: r8qetffbf5wfzcu5c74sme84e6ozps4o X-Rspam-User: X-HE-Tag: 1733239970-257604 X-HE-Meta: U2FsdGVkX18oRzFOujQBbOtMtM9r5wTx50EWbdpjbXwCeid1AZt5sXIumL6T9foGc4pIyHfcK9GtyDd1Uy8YWWM/lvFKLAOKF9yz8GAT13EVcd2knEqM491nMSDcKGo10AS6g/4brQinZuTEFWnbw3WkD9cTblLFWv743tAM8POZWJo7Loaqq7sRat/rTf+KxojLw1ycM/xcWYtFAmxACC7bEQS7lsD0XAAe8D++idsi7mCtvr6F0lbcNp2b2LMXMEuw2Pa7LAkoNwmNVWHkHVLX0fAf4dvlFsirnKhH1jzhp1p3GXciKNmmLA/I802PwlpmAsxEx2ouMoCSOtct0VDU5NfVvR4z5Bxh+car57FQcjXDDRqB/MbXXxXEDeSsz4CivBTNgRmsRgxa9EwqSgD6Lsbqz3plF9K6QwO5zUNrvazvN81DYc4vgwq8nG4gGwW1cLrQECZu9hEBeGNpXsJiZiqveOOorME2MPfeXosICRqrEEr3g8xzv8mxsEQM3JJK32+ZWxso71SkWLU4+4RYs6/liMXHRY0kr5dkxLXJH0djgUrH0vSPb9ZAj2AeGFuNUBzEj6AMdaZ2Wyh9NrBj10TBzsEVCj7Y1rUf1Vv1mi9Kg8yNHu+exnoM7AeP/e16XHaQZCGzUZ+4B9uHXQhZrN8mTk6wS+7SEZM7pgyRu59gumJMnY6oW3hyxdrfz416/KDBpO6StLp2oADAGZ2J6azmMRFIAlhXWjLsV9vEiDQRxepWk4Xu5OaxTXLm9qmaJPu08F6/0QT2SaqLqRg78e5FGtSLQm61qPLlOSj06B3c7AIhRftSoG/L0q9nus4Xgk8XFe5XNoopoufJYhfVN9Z0GR1wWqoGOzu9UGYVpffg/36twF+gq3taTBbvnMs0aMQDUyP6WoEIAaag5OixN+l+6xtCrPGYdBdsP16CksR5qo4OeARiN63N/PQFlxTEGSrD6KsuG+ZCSo6 ndgucGJv 9lvpkgIdiVXwAqt8mbTse509fwn2x8H2n8c2VIbHpcal3C9IcS3GA7D82Y9KgMyRsN4BNqX/88s/k8kOW8QF5iUse9mq1yc4ro/ZepjeiG/tCBwidoHXCIAWwsgp02sOcMoc07gCUU7AX8kNinOUcFveK1r61J7Cbj4cNWeLjOXOuxBcNPsuloz7v70oAWAM1xWnw7F+qOiPuSwbCE9Rrm2kHkiVNK+MtMpy62d0XXiM6Lch/szpC66UEDCELJD+92WJmVNydhsVBIQGtB1QXy5ahb9H6OQcN+Vc4VRDfebGFFEQ2E+693LHihWbMxV1A/iP8YSeeU9s/+neV7z2oJPh/WmDsWyyIWU4O X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If ractl->uncached is set to true, then folios created are marked as uncached as well. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 1 + mm/readahead.c | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index bcf0865a38ae..72b03b37c265 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1353,6 +1353,7 @@ struct readahead_control { pgoff_t _index; unsigned int _nr_pages; unsigned int _batch_count; + bool uncached; bool _workingset; unsigned long _pflags; }; diff --git a/mm/readahead.c b/mm/readahead.c index 8424bf1b67e2..33a2d0feae14 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -191,7 +191,13 @@ static void read_pages(struct readahead_control *rac) static struct folio *ractl_alloc_folio(struct readahead_control *ractl, gfp_t gfp_mask, unsigned int order) { - return filemap_alloc_folio(gfp_mask, order); + struct folio *folio; + + folio = filemap_alloc_folio(gfp_mask, order); + if (folio && ractl->uncached) + __folio_set_uncached(folio); + + return folio; } /** From patchwork Tue Dec 3 15:31:41 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892622 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AE9D5E69EB3 for ; Tue, 3 Dec 2024 15:33:01 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id AD7FA6B0093; Tue, 3 Dec 2024 10:32:56 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A5E056B0095; Tue, 3 Dec 2024 10:32:56 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 926EB6B0096; Tue, 3 Dec 2024 10:32:56 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6D5506B0093 for ; Tue, 3 Dec 2024 10:32:56 -0500 (EST) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2679AA0A93 for ; Tue, 3 Dec 2024 15:32:56 +0000 (UTC) X-FDA: 82854040458.18.2CC4AC5 Received: from mail-oa1-f50.google.com (mail-oa1-f50.google.com [209.85.160.50]) by imf29.hostedemail.com (Postfix) with ESMTP id 1BF1E120024 for ; Tue, 3 Dec 2024 15:32:34 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=OfQO92qZ; dmarc=none; spf=pass (imf29.hostedemail.com: domain of axboe@kernel.dk designates 209.85.160.50 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239968; a=rsa-sha256; cv=none; b=CsWzsw6gQ26YxT58nz0ZwOztwJGhG/CUAhjYFbZesECH7oO7BDk4dhgNWIPXgP/vWkYYzX 4nZFV9w6DWVSpzwlUCKZ1a/taKjm9vIDtIIAf0cHngrAe+Zn2N2Xmlf63hQHwNiVad1bVP f4NqbpNqzpf8DHkdKgvD0ZlMsq1RXzk= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=OfQO92qZ; dmarc=none; spf=pass (imf29.hostedemail.com: domain of axboe@kernel.dk designates 209.85.160.50 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239968; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=elv86DXnTmRzX7BzNNN54cJv91/ZhMhRandVuyh+78M=; b=u3LeYcBMGxyJ2z3p6vMm+zW/wsuRo6fBSYbux/vtFJm/zwMIHp5QjMWwXYSMSpiuI+JDWN 7Qs1M0ODI+EEPeRm+yUbLf9NRO0rKAtl/QbUOYKElHLZ/ZhbUB+TsB4cwD8HbEYcgxdROc ld5A9vsH1d7llfwCNcOUFNk3DdTWtNI= Received: by mail-oa1-f50.google.com with SMTP id 586e51a60fabf-294ec8e1d8aso3783760fac.1 for ; Tue, 03 Dec 2024 07:32:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239973; x=1733844773; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=elv86DXnTmRzX7BzNNN54cJv91/ZhMhRandVuyh+78M=; b=OfQO92qZagdJjqhgQdl6kb6drpmxiw1UjqZf0DF8FKAzdBw7hfCxllej0C95a8/EIA bxDWVUZbr4xcsm/FqRB9oDhAVyn2JJp5F94Nd4qbjgJ/B+WqC5TF1e98C5Y+k1FZcxAS a4UFkSqjoxIVGQnwGWmt3ra1H1zFrVEno2fNdOfbf+jxdBacqkqgcxzoOPcFHakAKhUa qZT42ZWImnnHp0ZbgmFzp+dKKtaXzCzLWD+vzo87CnRhgdYPWntnw9DMCEKNCcQAP6wA jxUXuOcECOM4ILRuUKeAPzlJgnW/dHMZxKiCFq+9v0Gxq8Q4ydg0UmqoLtQV4RF3/ZcE NYgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239973; x=1733844773; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=elv86DXnTmRzX7BzNNN54cJv91/ZhMhRandVuyh+78M=; b=dK8Mj0xbaXfc95O4iavLb2m+LQv9SAcJhSNdLp1Z3tZcPDNBxJWmVLOW2EQIG1qKNe RLtnikolP71m+rOZCX+pyrSKH/S6N4oOFrkB0KXlgTF6qeKdWCtApAOYJRMujl8yTgWm ONM0tcAl+TlcIQZzXxo+pYcuNiD0Px2YRN33cEVE8tb3uvfF6lj1qnlxwF91luJRsJfW mXvFhRrjQvx/IDfHy1Z+x7LOczmdy+hQNKO2jjG/sxjD38+julzVN1VXWLBhJzmAhmCv ZB4HmFDrGG4oO5vaI00/ceUitovWPOWYV79UKmcllD8r5zSM4+RE6TnPEBFELLxPd2h3 toPQ== X-Gm-Message-State: AOJu0Yx+6iC2wCYz4uCN/Iq6KMn7j+CmF6RpbxsxUGEq2inbBytjU6ol MZSoC3pQfbjzg6g7U+GUk5th8hPtpVDmJ76ljT8ePzsfcGIuabp0Rf/T5KbXMeRbXfkvkVhOl/K R X-Gm-Gg: ASbGncv6QKHaHQ52xdl+V4W18/iisn+Z/VkGqhow6EZAnmhMUcZTiMq70v0Hwb/nTf6 DJjdP+8kZCN42ztKnwGgkyC1JgOxDma8J5HVjVK7KUcZOpOkU8f9/qgchfeEFYs0GEqqSWtiG3H R/AAZs0A0eaAwFswHhPkQOqr+94g0D4EiypxuTK+1Fi/YEfunZWmljgaoCZXO7uM9s9KwjBiZxB D2K5R9gjiVVDb7x6sW1z8AR0NVPcaXvfDdI1iGwr/dxNl/Vjc66YnemkS8= X-Google-Smtp-Source: AGHT+IGEqY5mPPeclzVfJP1sqwpK2NEo6ZfmI2zg3SEWdjZlSMpJ+brY6rhQdeJiLlo5jFiEsUF9Rw== X-Received: by 2002:a05:6808:1795:b0:3ea:4c23:daf5 with SMTP id 5614622812f47-3eae4ecb2camr3172373b6e.8.1733239972824; Tue, 03 Dec 2024 07:32:52 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:52 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 05/12] mm/filemap: use page_cache_sync_ra() to kick off read-ahead Date: Tue, 3 Dec 2024 08:31:41 -0700 Message-ID: <20241203153232.92224-7-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 1BF1E120024 X-Stat-Signature: zc56bxtcho84zre98w4z8wu5ntuq73zz X-HE-Tag: 1733239954-822454 X-HE-Meta: U2FsdGVkX1/RUS0gVZPPIZl4rW5Ony8r2pjFW0CZlJ/Ya3a7l4+4oxh9Ls/TqTAdUQ0CCsHGICjZ50hMH6Z0SRSMPt8ruo48881P/ISEVsNg24BJ/WfH8OTBAFtNi+nEQruna38TPQ3K5sjuXeCDDU0Jm6+kHktbyNSNk6e7QQT8GuH0fXDtpUNb6//uBpZH30PWWRyv8RovpYEJOPEazr701QxjW4vNzezzH9QG6whhwNyF7Iwcnd52LtwlHBudT3lfubZ3me2VNEjq8hH6y2wJ8+MQJVL46Avdv1Vb/ArmayOsZ/m+b9VNz40OT9HFya93jHg2mrb7j6CbbmaVTUuLcBWLEp79st+2QLLmYk4PWWlfkY82wa1cEK1iryfMK2/RHJy3069CRB1b1ejKfq+niFo41dloxkT/YN9GwSIFY3bx20xM54WFL5qhNQMrOY92gOz0Zp0XMD4zFuWijCZ91J0xM7Emz/xLVDYU6bYti4pquQfQabvdlnFdW8BTWuCfd+yNbkXLmfkYb5M/lvWLTkGLb17FwsVyfBF5WKy4c+Am1mnq2vfmIFQv/Y91pbkSF4LJY8tnvneup6XJYc4+5kQ4Uu1/gfXv9yAOCsqTt5iYtN2aKLdKlXOqC7/oS47bIhBEHzE+/iVBsHJbytlomSbcdh+mFZTQXXLldWeSLo3DFkyfdi+u4enp+hUB53JmPRO2e4pid7KqQc8A74ImxyjO+RvHhTcZ7KC2CULPu8PDzgfpoOXqdrTHltBn4QZdSvt9WtgL3Hgz8Pfm5BCpzYcd667TtvO3RociGGRwpasYMKOnNv8bO55JQiCmHmKDyVLis8cm/EDyGCmJOvMPNZNN840uZxEukhIT8RzUF+fbTGVuZERMtlnopLgOsc+IBFD0F2KLkIuTBxXaW0Ur2eSR9TqxQtM++iyHSUefYzgHkzw+aedeLVZ3SPGKL3HSR9IToPKX+4aXgZ/ C3unSdWN P32sWJ4CP9HvTYxcSbMVYYFbzxIQaa4e7kHCJ3flx4wnd+2jKtx41wMt0iCLNQjKakFR8V7G4GLmkjhGRGesVY6y7CiLQqLA3O/ms5QSZ2LYzHTRVVNyXjASVmvy9DbpHT0mwl/auQI9vJepEZXqEZ5oR0utu15KtJYYyQ3hkhvznY2Lfbtj8xMKBKDrZtld+X7xzsb+yPyGCHGQEzIKfNEjFhRFeX+lPYTXuj/RNRNCqgvhx6AWzqkb66xyPls5tN3Z9GMM70oXl+OLa1hx5LF/5idVesO/ifipCgRZIWr3xyYzjYsAu64PLZygCOeJOSN2B2rdMxjqMqd/XqsOb2z+YezKRYgBYBZgB X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Rather than use the page_cache_sync_readahead() helper, define our own ractl and use page_cache_sync_ra() directly. In preparation for needing to modify ractl inside filemap_get_pages(). No functional changes in this patch. Signed-off-by: Jens Axboe Reviewed-by: Christoph Hellwig --- mm/filemap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 898e992039e8..dd3042de8038 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2527,7 +2527,6 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, { struct file *filp = iocb->ki_filp; struct address_space *mapping = filp->f_mapping; - struct file_ra_state *ra = &filp->f_ra; pgoff_t index = iocb->ki_pos >> PAGE_SHIFT; pgoff_t last_index; struct folio *folio; @@ -2542,12 +2541,13 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, filemap_get_read_batch(mapping, index, last_index - 1, fbatch); if (!folio_batch_count(fbatch)) { + DEFINE_READAHEAD(ractl, filp, &filp->f_ra, mapping, index); + if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; if (iocb->ki_flags & IOCB_NOWAIT) flags = memalloc_noio_save(); - page_cache_sync_readahead(mapping, ra, filp, index, - last_index - index); + page_cache_sync_ra(&ractl, last_index - index); if (iocb->ki_flags & IOCB_NOWAIT) memalloc_noio_restore(flags); filemap_get_read_batch(mapping, index, last_index - 1, fbatch); From patchwork Tue Dec 3 15:31:42 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892623 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3DEC2E69EAF for ; Tue, 3 Dec 2024 15:33:04 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1ADA56B0095; Tue, 3 Dec 2024 10:32:58 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 161486B0098; Tue, 3 Dec 2024 10:32:58 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EF0846B0099; Tue, 3 Dec 2024 10:32:57 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id CFF156B0095 for ; Tue, 3 Dec 2024 10:32:57 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 870BDAE665 for ; Tue, 3 Dec 2024 15:32:57 +0000 (UTC) X-FDA: 82854040416.24.E8FE128 Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) by imf13.hostedemail.com (Postfix) with ESMTP id 0A4A720020 for ; Tue, 3 Dec 2024 15:32:40 +0000 (UTC) Authentication-Results: imf13.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=UVyAIebX; spf=pass (imf13.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.171 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239969; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=DRb+N8QQBu8dVr+Sad3apJL8CLp+dfOfMlp0JKFijps=; b=YdUGpW63q30m8jDl7RwSvRYZVtF5fCbeUTvRZ+L8qqM9h5PA3osRhBPRCuyg/OAHoJpyKd cvYQ9jCzCoy5lYMsy2S0e6BnB/Z58tNl7zNe9rvGg4eHMSXRNr02XlFM4CWO+66xKm/Onz Bk2r3+BKiG8fUh6trEQHq71t9vsWTIY= ARC-Authentication-Results: i=1; imf13.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=UVyAIebX; spf=pass (imf13.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.171 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239969; a=rsa-sha256; cv=none; b=aO1UXutoxRMY4XL8+trGhDNEXgQn+s0z63gJr1KLj32zfHD40HsFxmGParb+K7sfvzaqH7 PtyDaMYxDFq1XZFFAepCJERwk/LQwblkEpaRrgPSB3CCG3zV2UapMasJAVL4cwHMWqrqji LRnx/+zlEY5J1zVBzteXn2VBWAunhSI= Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-3ea4c550a3fso2766295b6e.2 for ; Tue, 03 Dec 2024 07:32:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239974; x=1733844774; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=DRb+N8QQBu8dVr+Sad3apJL8CLp+dfOfMlp0JKFijps=; b=UVyAIebXLvrJq6nq92bE9/Xk1Q9FydCpQ0r9Lf284i6C80hI2ssrMUEflitf5zWqSh +M3sewANNc+j+nnfWXutjV+AzK36zwkeEtt63g9TGvSJqqQJTf/ZjIMWE+KyBkhZPF1e sw4Uroa2qtNUfzpqYitZPuCo43dVR75yDClzIdjCxvxWsgWnoc6KdkmBB3+Gs5n6D5If O87E4mJZ32CvgVnbvD+nImo+LXE35vWI7x4+Eux/sn7KsEKMsPw++py3a7Zv+8PV+4ku xKL4bTo3NDTh48kqjw7BeaqFDKn934YWhhiv7SnRWqfPEn11QDSrQMG9Zo4DzYMruaux mkjg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239974; x=1733844774; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DRb+N8QQBu8dVr+Sad3apJL8CLp+dfOfMlp0JKFijps=; b=iHjFH5jbiGD0LFKkVr18BmcnQ0zf1mAYvvxTHwWUg3f51paTjxVIoqWosSihY9Qx+2 Q9nG5Ay2WdVkiCRir/PG8UFPFWfaAKEfVAy3Lvhge3LDGKnO9Qhze89hgbugjOvBPbHW 8Nu0JeTDg/ocPcOtMVgdGl8Tr6BEfJTTOJZczntnCIKiXp0gMEyRRKlP+Z3CtlCbbFsD pTqSgSgROKCklFXIeUNC4cHc13/l733EjlB2Z3bNy2m4phY8TQMO4JPnPRyqnihZYuPU uSXHDGYpxX4J1FN4qburK3oVoP6TzFnULigyRdsdDy2/ajTTr4FJa/38nmSqCf/61uVs 7YYA== X-Gm-Message-State: AOJu0Yy2NXRI7v8Hjz2A37xjuRjOkZK8+SwxTHiaOHHsK00MJdqBIJQr +85Uf9+/jXT7viWEpNV9/HWtkqKFZ+bMl0ewvZqS6Vl5Ze/wytTZA1dqRzIG80R5VRaFlrnL2K4 e X-Gm-Gg: ASbGncsH43dmyJCL910BNpTmcztOme1nJ7DEg+8ESrcSC+cT5EAw5nRSGLj0sEZnYKV ukOAK6zA8qxkUItzZV5lYwZUkqfxRMtPct0QBnqLbmPEtgt0GX2p27yRyUd1IMSlHOnKG5/AsNh iV/pggEcxi58+i6XL83KnmLP1pfKCdZKuyYhWUqXcyQJE2xCLbzhFuG+Yxm739yfw//I6XyDjrg qkMg8MUQpT35Nuc93r9y3qrgASlP5czDaAZbnOKuAzl33L+kFUpHldkdY0= X-Google-Smtp-Source: AGHT+IGHTQ5LeGksRhzqMID0Kqh5YSoxN8Gylam5Ojf1AQcflOZOTzp2gKozYR7Jx08uJ7hl5NcXoA== X-Received: by 2002:a05:6808:3843:b0:3e8:1f5d:b7f8 with SMTP id 5614622812f47-3eae4edadc4mr3375768b6e.1.1733239974385; Tue, 03 Dec 2024 07:32:54 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:53 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 06/12] mm/truncate: add folio_unmap_invalidate() helper Date: Tue, 3 Dec 2024 08:31:42 -0700 Message-ID: <20241203153232.92224-8-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspamd-Queue-Id: 0A4A720020 X-Stat-Signature: 6fuun7if97d6ji44qots39h5uji78w4q X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1733239960-505125 X-HE-Meta: U2FsdGVkX1+oxr2xOPifHpd16hWOQOyHRLhV5PpdDNsU+kOOWdk/uMrUjxsIVqh7cKJ70GfkbXzyZ1OFVs994RVPYr21tOULIw96sZx4nsep6Zyc5YjtunZRIFlBpqp0KfTgIlFMcP3hU36hgLqFnxm4iPFs9hFHTMQ5j8p87f/TNapANoZB1XAG6IgynRcTWaSNVS3KtaiTU2Kk1jB2XikqZQL+0kxFRbuY/LL9Xgx+23OKOGGjgWEjixD0NYePDyNaKNJBbkNmJs5BF0ReQxxUKV0wMc9VijBadkcGq8MvrNWnlb5uyyOYZMydeXfIDOowcbL8ypH0v16bP+0uu+VtZvN44+sNkdC+ccM14jR5An2SxrM0ALBg/iIUPe+3TjCOZ3Su6ULvyEuHoXU4Np2/34PZkIEq/fpfSSWP/ePUfIBiSNCoLBebCjlc3NUGFzdEymhCPphhPRJieWfA5smSncQ3KSqR1bJbntRJN7qBX8evPXUQZ3PuEYJRmvRkvQ/fvYge3JE0ebsdS8Oj8naOtYzzG2b5KDRYAiVrwS2wKlpPsVwreaiNtYRHU6bnEbqJzBmGcaLiSI5SgFvQEzxhyPo9YTCzj8eiGZxG5GRxp9Ss4ZJ7kPvlmKXl5q0aie5/OBWQ86B57DqugeFUYUIlV4WEzqViTe/lWaMUOSNX7JPyz/7Ey3fVV18daJTXMNmghfYukOs7cGkemmSAMe6lAQOVitimdUR/k44neE/oNafD5ApUj3wRANsSk8wBRkgIlOaL7nkDARd9vCJU2PFbEejx1BE/iUYyJEcvsm5XukeDu1/HL/NmH1LtEenFVYZ5+a102cVRS5ruV5mVaMcPvgP9wU6HIM0TM2zZQE2UrGC2mEM1SWd/C54g55GG0V8BgvRgmFwJMnqPxHv4TBhQrZddXkIXNXcOzoceSPVUzCDIvDV62CK2XnOyaSOOLzM2T6tF062LNoFXhNb /bm164Lk L6Ie0TreXkVlstQ6kN4QPiYpBFGIDabhRFXiclnM5KXf4Dfxojai01CzIwC5pR7hhuI93udL2pe3FQfudEvSnbClGgrLWLiBUKlxKF6F3h4Gl0Qdh+2RWB/jFvGVZHdA9WftH8dl9DvrjLnHz3pNzWN6gWmAaM97ZcWZ3d4YdFvdUvGwLgjGlio1RIKjVUMMsRau05IECk9l2nJxgBt7z/yvNf2NER7bqcDW9+WokAkEjURqMz7XrGsBQ7tIhe2l63f89qblQ6LZW3ERs/qWW7sL3VcCYC7d5IxTGctR9upZaxonaoDKWLDt19/RBJwo3OKhIwUgT/Y60JVwiv6z/iAvtNIjgCXIu7fLT X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add a folio_unmap_invalidate() helper, which unmaps and invalidates a given folio. The caller must already have locked the folio. Use this new helper in invalidate_inode_pages2_range(), rather than duplicate the code there. In preparation for using this elsewhere as well, have it take a gfp_t mask rather than assume GFP_KERNEL is the right choice. This bubbles back to invalidate_complete_folio2() as well. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 2 ++ mm/truncate.c | 35 ++++++++++++++++++++++------------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 72b03b37c265..f2d49dccb7c1 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -34,6 +34,8 @@ int kiocb_invalidate_pages(struct kiocb *iocb, size_t count); void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count); int filemap_invalidate_pages(struct address_space *mapping, loff_t pos, loff_t end, bool nowait); +int folio_unmap_invalidate(struct address_space *mapping, struct folio *folio, + gfp_t gfp); int write_inode_now(struct inode *, int sync); int filemap_fdatawrite(struct address_space *); diff --git a/mm/truncate.c b/mm/truncate.c index 7c304d2f0052..c1dfddb1122a 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -533,12 +533,12 @@ EXPORT_SYMBOL(invalidate_mapping_pages); * sitting in the folio_add_lru() caches. */ static int invalidate_complete_folio2(struct address_space *mapping, - struct folio *folio) + struct folio *folio, gfp_t gfp_mask) { if (folio->mapping != mapping) return 0; - if (!filemap_release_folio(folio, GFP_KERNEL)) + if (!filemap_release_folio(folio, gfp_mask)) return 0; spin_lock(&mapping->host->i_lock); @@ -570,6 +570,25 @@ static int folio_launder(struct address_space *mapping, struct folio *folio) return mapping->a_ops->launder_folio(folio); } +int folio_unmap_invalidate(struct address_space *mapping, struct folio *folio, + gfp_t gfp) +{ + int ret; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + + if (folio_test_dirty(folio)) + return 0; + if (folio_mapped(folio)) + unmap_mapping_folio(folio); + BUG_ON(folio_mapped(folio)); + + ret = folio_launder(mapping, folio); + if (!ret && !invalidate_complete_folio2(mapping, folio, gfp)) + return -EBUSY; + return ret; +} + /** * invalidate_inode_pages2_range - remove range of pages from an address_space * @mapping: the address_space @@ -629,18 +648,8 @@ int invalidate_inode_pages2_range(struct address_space *mapping, folio_unlock(folio); continue; } - VM_BUG_ON_FOLIO(!folio_contains(folio, indices[i]), folio); folio_wait_writeback(folio); - - if (folio_mapped(folio)) - unmap_mapping_folio(folio); - BUG_ON(folio_mapped(folio)); - - ret2 = folio_launder(mapping, folio); - if (ret2 == 0) { - if (!invalidate_complete_folio2(mapping, folio)) - ret2 = -EBUSY; - } + ret2 = folio_unmap_invalidate(mapping, folio, GFP_KERNEL); if (ret2 < 0) ret = ret2; folio_unlock(folio); From patchwork Tue Dec 3 15:31:43 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892624 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B5D3EE64ABB for ; Tue, 3 Dec 2024 15:33:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 94F5E6B0098; Tue, 3 Dec 2024 10:32:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 8AE916B009A; Tue, 3 Dec 2024 10:32:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 6DB386B009B; Tue, 3 Dec 2024 10:32:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 4E35C6B0098 for ; Tue, 3 Dec 2024 10:32:59 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 0E0CB1C7479 for ; Tue, 3 Dec 2024 15:32:59 +0000 (UTC) X-FDA: 82854039912.17.C774BEA Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by imf23.hostedemail.com (Postfix) with ESMTP id 5A67214002D for ; Tue, 3 Dec 2024 15:32:48 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=1TImmsmk; dmarc=none; spf=pass (imf23.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.176 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239971; a=rsa-sha256; cv=none; b=FsG1UGLDItdrdaLgDtxrUC6q8mwt3TxQnmVcmQsJ2l/T4KOvD/WSNZ+KWQl++aJwA2aCJC 241cz5zI3kZX3IWEZq/VJ4iPzDo4P12xD1GBRVN0ETibdurMiQby1nERKtCUPEbbVvNqDv okGeDZJPFuMKrkamIvxM5JlFgiT4dNI= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=1TImmsmk; dmarc=none; spf=pass (imf23.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.176 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239971; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=99ZtQ6Npm+o/kkpmwDDPCgpJncZgiN/IMlRLftnmHuc=; b=tRmpiqTn7HVU579T2EaUhJv2IGtGSVRmVaJFPsFieOA7BR8AOtX24tIlBk6b5up1QmEoYh L+vfvKuWzWvUGE4pRKSH1MBEjZVo6yWRRkeY+MDq2GjwDNoiAUrdJ8pl/DFgW7G6kF2ksS Nyb5peGooz+bOqs0C03L1dM/OzsQoTg= Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3ea55a2a38bso3014476b6e.1 for ; Tue, 03 Dec 2024 07:32:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239976; x=1733844776; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=99ZtQ6Npm+o/kkpmwDDPCgpJncZgiN/IMlRLftnmHuc=; b=1TImmsmkuy27vqjtxe8RTBh0uKldqqYlJcmBwynwiWNb533yq//HH+VflbxQsFAjtL NSV8jMw/5P4HD7R+ftzO7+F0mRkStdTfaO3PlO6vhns59kaNlvK25ahu1lTXjvAyLMwP WzATfVIMDGEeSzlnQoxdotutpWbiQasSufbhmW7ImyDTfwXPZ5yxMLAlMiAv2EwedWlU z5KnWeGwIC5hYkXc627b13KLea+dhhIRlzaXhZvSR19JT6pJotWv2aP8CHIpMTDebifC sa3G7eQUCe7FIZrZa9IdOjq3Ae5ykGdPG//8E1C1nCtCeYM0yng6ANkyYMNb5RIRzTWs oXkg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239976; x=1733844776; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=99ZtQ6Npm+o/kkpmwDDPCgpJncZgiN/IMlRLftnmHuc=; b=nFmAPW5aJvOpXmYbgcMPK8+u65e93bqSG23c8Gg6YJ9IZaqNYS88mqdBZDyW6bOgJK n0lAsDzU/BYGWJ+BBiZK/s0e4jwX+4i5DV6YeCPyfV/Y0T2WiSm2cpC+uv6Hc3tcdWbr QNtZ/zOdsgUwbpoJ355x1rr5bWp70GIbv7TzqFGB8TGRaionySce9H+ZzhtirUeaEvhZ xxNqBsc+fsknPoTecDIad0w9Nv2zdgvUAAQ13s5wfHejV7WHrTrufz/Doi7Vpr2s2IOj dYAn/48Zl9NrdA/8GsknKwCkn2YCJarVSx6GaXUlpCDX7kETNa8em+eK807/ZWtJK3OH eRjA== X-Gm-Message-State: AOJu0YxfdRmIw4rD4QkjvpYE2PpZoOZGTlUSinJQwlYQ5UIuXfzTTf2v uULVzNk64B61/PZlrwUq7E1TFJUbNSbqcRRgNPLb556fENCyqH8v/eoQzxKxQoXfniFksKmsHNF 4 X-Gm-Gg: ASbGncsZnS/5IEM2F41cvu3t3n6MCasf0DZzekHiqIIWZctzjMI3dl3YnU2gzQ4x4TA l/+9PAsON+XLe3tS5/mPrDlqJwpn8o3WATYx+D736JqeXC39ZW6vYl6U67+2h0Pp6xu2unEo9SI Oubpsk130o7EPJhn5LpqI/wyh5W7qrHWDgrWE/LLt7ZVZ6M16IERbIpfUXofrhMFWi4DkwC3290 u8JrD+licyO377mKYhj2l3HpWq/MocbCqOwVXaaSJJOvTUcfd2HHi1AHrY= X-Google-Smtp-Source: AGHT+IHMJnWLWXZesvRG8H115i2+SnEyGT/XbMmAgLxKcYoNaa8g7OGxqlE3KOV7Yd59LM9TrdKc0w== X-Received: by 2002:a05:6808:152c:b0:3e6:14a6:4288 with SMTP id 5614622812f47-3eae4f366f8mr2926476b6e.11.1733239975822; Tue, 03 Dec 2024 07:32:55 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:55 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 07/12] fs: add RWF_UNCACHED iocb and FOP_UNCACHED file_operations flag Date: Tue, 3 Dec 2024 08:31:43 -0700 Message-ID: <20241203153232.92224-9-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 5A67214002D X-Stat-Signature: 8cwqd466m8ageafruju8964u1s4zzrt7 X-HE-Tag: 1733239968-838859 X-HE-Meta: U2FsdGVkX19CSCMAiDtkk/wP4+bfVMJYIUDLnmkkfeEvwJfLEprA1+LaL4zHc3jWULa+vz611uGawOO0moI++yTMfd54kTlm3TJ1gPZFNVIuEf5KyuoD3wEdtTvxHHkHrAv9rmlDqU5rc8GJ0YzAHijTkEV7a4r4PvgM+j593KZsHSP9Jl8mnahDLqxQO9wRY+eU8hDAQ5wjRLsnP4ZeErPQy6WWC8D/XetkgrHpyCwXjEvWtFx2kFcf9VWntoBLroSBdhApp/sXDtpCY/F1pOQdfWX+J8Itbb2+qlTuBrEtJBZ09muf3izh4cB5WSpnhGV8ZsDM+bQ3AJhy5ffwmFrG0mrI8QZmgTGu8j359Nfp0PV0HOLq+Y0L7Tlpbf3vk+V/hb4QMNlg/TdZDXJHUTRDgm9jgQStoCi9NlMdMQAoRWwL3Dm+C3m+j6uaXCPrDyjjRhU9aFHeGyEbhZ8ILoDQpO3Opjq6ZUBhXN3aRW/ooEPI7rMCucJKn4B7aE1ZB8AKw+76ar33ZwuX6d9bo1RbLWqzKuDkkdgtH2azflh5pSfjkeAhRpr71PFODMeNosR+RZTBLsKA+e3/ahpSksCsuzhw0WPgVehG+Zf31PjlHM5ytCHwtVqqjedlMTzgzB7CJTZOYwWL+1redw8PjYsFYZ2/+Fs8yHQ1qyp1nnys3xFj2vVbMVoU5JSsbZU5bInxYcYjRVha5Jdda95GUKhgQVdxEJfpwedsC1AbvHE3jvH3eB5gEN5kK9n8jsKRqltWzhfbBn4h6FuEUaglBboMnvmlf0VQXePDiONFNBZLe8ahbx/3LB9W8yoiYkOZ4tHQsv8GvyJ87GFzMxjBSTncXky53KL8KjnI42OuIPMP5vcDZ7s4l9Ys1kLElB5Zla7CDyjYz6ZM6pDXF7KJmTdoBcPopl+nIAv9dZriBgRh6jZm2FBcc9+/TzbxripQWGvzTN3l6iE+dQPNqxe ksULcsse Hq4jtD00RRSl5OTFq+ovXEkK5eykcaNciVrlKB4DCoW8tHpN6aeisADIa9xluKBA/qMNgV10pX2jGAEad6cxEpdd7ZLYf0/Pe/SJcifGK8/6qFHwpGGI6ghwG0fgeWYkNgBatKWf39SfdGrGqwbbJXa96BdTW7j0/94UCX4EXA7HUgGK7yR9RZcn04MAg5ZzRKAxvdmyW3+kgc8nqclK95d/1DSbZknhO1MK3mXVnkM6MPQndjDGiAh346DabeoypBPyjBM3X/NDS3Bt2Vzhiwq+SMPPy/wKOgXtRi2AYSGxDU316JldGkS0U++nl7KFgd1sQg61+2V7Hlmi1CWdaHG2ucXNAwNoWnI88 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If a file system supports uncached buffered IO, it may set FOP_UNCACHED and enable RWF_UNCACHED. If RWF_UNCACHED is attempted without the file system supporting it, it'll get errored with -EOPNOTSUPP. Signed-off-by: Jens Axboe --- include/linux/fs.h | 14 +++++++++++++- include/uapi/linux/fs.h | 6 +++++- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 7e29433c5ecc..b64a78582f06 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -322,6 +322,7 @@ struct readahead_control; #define IOCB_NOWAIT (__force int) RWF_NOWAIT #define IOCB_APPEND (__force int) RWF_APPEND #define IOCB_ATOMIC (__force int) RWF_ATOMIC +#define IOCB_UNCACHED (__force int) RWF_UNCACHED /* non-RWF related bits - start at 16 */ #define IOCB_EVENTFD (1 << 16) @@ -356,7 +357,8 @@ struct readahead_control; { IOCB_SYNC, "SYNC" }, \ { IOCB_NOWAIT, "NOWAIT" }, \ { IOCB_APPEND, "APPEND" }, \ - { IOCB_ATOMIC, "ATOMIC"}, \ + { IOCB_ATOMIC, "ATOMIC" }, \ + { IOCB_UNCACHED, "UNCACHED" }, \ { IOCB_EVENTFD, "EVENTFD"}, \ { IOCB_DIRECT, "DIRECT" }, \ { IOCB_WRITE, "WRITE" }, \ @@ -2127,6 +2129,8 @@ struct file_operations { #define FOP_UNSIGNED_OFFSET ((__force fop_flags_t)(1 << 5)) /* Supports asynchronous lock callbacks */ #define FOP_ASYNC_LOCK ((__force fop_flags_t)(1 << 6)) +/* File system supports uncached read/write buffered IO */ +#define FOP_UNCACHED ((__force fop_flags_t)(1 << 7)) /* Wrap a directory iterator that needs exclusive inode access */ int wrap_directory_iterator(struct file *, struct dir_context *, @@ -3614,6 +3618,14 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags, if (!(ki->ki_filp->f_mode & FMODE_CAN_ATOMIC_WRITE)) return -EOPNOTSUPP; } + if (flags & RWF_UNCACHED) { + /* file system must support it */ + if (!(ki->ki_filp->f_op->fop_flags & FOP_UNCACHED)) + return -EOPNOTSUPP; + /* DAX mappings not supported */ + if (IS_DAX(ki->ki_filp->f_mapping->host)) + return -EOPNOTSUPP; + } kiocb_flags |= (__force int) (flags & RWF_SUPPORTED); if (flags & RWF_SYNC) kiocb_flags |= IOCB_DSYNC; diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index 753971770733..dc77cd8ae1a3 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -332,9 +332,13 @@ typedef int __bitwise __kernel_rwf_t; /* Atomic Write */ #define RWF_ATOMIC ((__force __kernel_rwf_t)0x00000040) +/* buffered IO that drops the cache after reading or writing data */ +#define RWF_UNCACHED ((__force __kernel_rwf_t)0x00000080) + /* mask of flags supported by the kernel */ #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ - RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC) + RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC |\ + RWF_UNCACHED) #define PROCFS_IOCTL_MAGIC 'f' From patchwork Tue Dec 3 15:31:44 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892625 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99C5EE69EAF for ; Tue, 3 Dec 2024 15:33:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CABA76B009C; Tue, 3 Dec 2024 10:33:01 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id BBCF56B009B; Tue, 3 Dec 2024 10:33:01 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A0E006B009C; Tue, 3 Dec 2024 10:33:01 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7FFC06B009A for ; Tue, 3 Dec 2024 10:33:01 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 04135C0226 for ; Tue, 3 Dec 2024 15:33:00 +0000 (UTC) X-FDA: 82854040542.22.18A6BCF Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) by imf17.hostedemail.com (Postfix) with ESMTP id A887440028 for ; Tue, 3 Dec 2024 15:32:48 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=WD6XNqMG; dmarc=none; spf=pass (imf17.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.174 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239966; a=rsa-sha256; cv=none; b=cyZVKO7vVF1Ioo+PuYZ2sN4Wu7oCS6Ejr81QGJM9ZKYmB8LjyMsaBT8Cd7wwFcObrt/gS0 49adj26L26nIkKIyhAj56OkE4WYWTgnM3qM8K/GLq6yKoeNxVTxv07b9XI90ZOp8U5CEJM c2ruvUXju4pVlUyIEaSeA05fM7FtKpc= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=WD6XNqMG; dmarc=none; spf=pass (imf17.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.174 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239966; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=854g8zvMX205jFWe5CwR3kRcBo1TfCPiGvzDJnv2dMk=; b=icK0L79alMIZi0St1bhDvFZTqTbw/FVZF6/yOGOoG30FNGktZmVgKNKtKWwdoMBng5q6Vq /9+4eCwds0AjlbklDzmCAxBLGi4Y1imkVUSggPNv0Q21AIa/a1vYqcMpUBltCDYoH9wZbG ecmDUrzFQoF43uMTXG1/ON2jN+TUcks= Received: by mail-oi1-f174.google.com with SMTP id 5614622812f47-3ea60f074c3so3240297b6e.1 for ; Tue, 03 Dec 2024 07:32:58 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239977; x=1733844777; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=854g8zvMX205jFWe5CwR3kRcBo1TfCPiGvzDJnv2dMk=; b=WD6XNqMGYdso5u3YPAHDxxwnUjwwB5bYpobh2Zdy/AQ5YvWMJXZqHN/iEuOw071xgp pYE9njA6g+noqsYoD4X/Yu0Zq8o7/z9b3NsGYX2cNpVqTpjKL/Ql2Da1OyrXE5x7E6Yt JXQw2r1r1GgOv1xVtQiEyipxoS6tX5HenAaRThr/P3pKr5q5l4ZRXsJRM8LloFKmTkyt 5nGKRbUUiRnMMF+S+Op4alSiIrcVUQ1XPxqn3ZQcrB6J7PYo4spEICdSymlsNbiqR3WP omxKQ8xdvSmpyuTfCnKoeP2pOLOgUZ7GtZDZW5u6J50RJFRpoYC2GKEZGKX9IEfBq8Fa J3zw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239977; x=1733844777; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=854g8zvMX205jFWe5CwR3kRcBo1TfCPiGvzDJnv2dMk=; b=oNrOPfIbSGZHqjOiIO1O0vQ+Agd+/kGoxkQPpyiuuIYhEcukhHv5bt5W9QRA9sN81d mE/0ifz02GfF8QlsHYDKB/uers2Uv2r1ffViUrNNXJ6mN+PZHU6EPTzE3N6qi3SxggM3 GQAbBtEqS2O9dz71YhWXEVLpEUoXOoj9Kab4EGxhQz/wwCgnk2pMSiQKaNadyeEWNhro st+hsfc63c+lp3q7kFThZ9WyJFjR6djA1zfxZQtfqPnjzAt3NVnHGZLeKtasFE+hVgc8 gut2OBxj5kGvzbNRtrFEadWMKlDSdTAJCE60KQuA0kHL4RTI4XRAEzU5fU7j6Zz8FrWl yWgw== X-Gm-Message-State: AOJu0YxiJabl1TG142fVxD7kqDoakIvtx1hSyTVaFsNv2r6hpnlmBZhq frR/oWjtsAyl+mqWoRmRq+fflfRDQbie4J1bBDcDlucka9L5KKT5D2eVxlSIU1xOUJKemIaZXYQ C X-Gm-Gg: ASbGncsylXH/Oyd9PC9+peMXCnv3iWAWefBTS0dYYI6BYCliRvfmU7qeRYOYufA3guB sqf1YuMBGvNlDQAELNDhli/vfIkx1LPZ5QL/H8SGuxqENeD2xvIq3QdKA5FlgIWdfwfcCUrhPqz 1Dw/SbtZAGg68rUqXcHQvTA8xPTVlr9ucACASZxXnV1yBj7WJV/CEXa/cLVXEIbFI5bNzyDwc+J pA1EKSMODR3KKGBYQP9S9x36P+2mcgqpf1bJg7IHX9mMoO0C2UEtB2RLiI= X-Google-Smtp-Source: AGHT+IEn2a1k9EtjOpfVet1ysm6MWr9jkiUyE2X2SH8cUoAjtJnT3D5sVQSXdc6UVYtxwSGjKa8I4g== X-Received: by 2002:a05:6808:2e4e:b0:3ea:aa8a:c115 with SMTP id 5614622812f47-3eae4f8c2c0mr3705224b6e.21.1733239977325; Tue, 03 Dec 2024 07:32:57 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:56 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 08/12] mm/filemap: add read support for RWF_UNCACHED Date: Tue, 3 Dec 2024 08:31:44 -0700 Message-ID: <20241203153232.92224-10-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A887440028 X-Stat-Signature: wxttmk5h6twx1xoosjtrrgiqm6snt5o4 X-Rspam-User: X-HE-Tag: 1733239968-304518 X-HE-Meta: U2FsdGVkX182dA49Z4E0wVIsxx1Fv/n2UQDw0+IZTMaFOA5NZeQm81wBB32/jv09kYmplRDRtzgV+CJh+ndkz283a4W6+4lfktaQHwxtKli/3rDEqYHdrj1rQy9F1pMWrbXE1rBKF83Lk1645fPGNHmtVA9JyyYxfQtyTfefLrq+6Zx1jriLfSUWQlcxzvrIVeSAywtcN1OFag/FJxpoF7e9BrBkKSsgjrEKDGZs1FtVpUd/3UEJYaUflrIXpzW837u3GIxc7SUo/xWh2EID5BnhfMNl/nc5M/UkPFGOhLlmWFv1/8E+F5ED0Unz6EJXYceuM8Lwv8JyF+KrWSRZBgNKaJLgyjtAbn01wp+yfwqPm8lQehmh+cjNeniiW9hRJZeWv9kK1ArFRPTwOucnOvgJHKRW3sVYomy6k38seR5ZC3BTli4hek94j1r8IyiRM9iwrZzluhUEDueeTP4TWbwaTTvThmcGjxR+gTmfRq0U68aik2l6HrZc6I0GohvzfbHZOeCFODKqnlYJFdi4rsMaiG1fdXypOa7XSfxcimxVLfnKOy1uAMm40KkEbioZfvbB4vIdAkb+M1AvjKQDDov9U198icN0y/nOzLXclxKb/i2Icb/fsnQRtYlnbbxOaR8mnvjzgj4CVyiGzTqXMaHG0uhUZujIP72tR6DGQNsFemVRdAPzF4pjKLD7qK4pNocmPbsoE2TWhofc83QytSjkTf5sRs1rsFlSdNB/Vp6PEVJ5918ej7HEDbPPUJ2c1T9/OSyUWLLs1PpiLP2CfCSlEqvalvZXDSCnAnl8o40VQBIFoaf/xf+MLH//5Ae6TVAxdkl/uJaPHFTCbi9U9Vk67apEYSBz437Iz9ijGxgoaTBATWvnTJBCz45dmxiYDekYf8UJr5vJMBrUVyRbU+0yvGtkMM1FtMPsupt1gsaZLU23W/PjiKFUDEC813D3r1NzGNCKjkigez2Thj1 svhoTAqT fqwKDCF1SXW9AJW/IomzV7nyNgSav2xbWMTgOHGxeAUVvJDkLtDbFBx6ucjaRnejSIsFoVAOhO+WH5EsEQRCp8rw0WM3/FM6Sl0PMm8xCb6y6rpKopuljDmXvvDhuhO5v1AuEmkYdWcN0iGkFsee3w1oqbs52aWRDc2WyL65oZN8V7nqzZ0hVrEZpE91mkBuB14AuLag1Z5mACCg37fdHk3fGmgWOgLf9qV6CobrtPTBT+9nLoJ6LaZzCHhJco748FoLmYHewvn4av6yjOIthzFhQFRKU048bjmEOi479ymY2WHL3HhosRDGTaxH8B2rqp0q++igkQUjNvP0drJvcFPniDVevgsH+jLTr X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Add RWF_UNCACHED as a read operation flag, which means that any data read wil be removed from the page cache upon completion. Uses the page cache to synchronize, and simply prunes folios that were instantiated when the operation completes. While it would be possible to use private pages for this, using the page cache as synchronization is handy for a variety of reasons: 1) No special truncate magic is needed 2) Async buffered reads need some place to serialize, using the page cache is a lot easier than writing extra code for this 3) The pruning cost is pretty reasonable and the code to support this is much simpler as a result. You can think of uncached buffered IO as being the much more attractive cousin of O_DIRECT - it has none of the restrictions of O_DIRECT. Yes, it will copy the data, but unlike regular buffered IO, it doesn't run into the unpredictability of the page cache in terms of reclaim. As an example, on a test box with 32 drives, reading them with buffered IO looks as follows: Reading bs 65536, uncached 0 1s: 145945MB/sec 2s: 158067MB/sec 3s: 157007MB/sec 4s: 148622MB/sec 5s: 118824MB/sec 6s: 70494MB/sec 7s: 41754MB/sec 8s: 90811MB/sec 9s: 92204MB/sec 10s: 95178MB/sec 11s: 95488MB/sec 12s: 95552MB/sec 13s: 96275MB/sec where it's quite easy to see where the page cache filled up, and performance went from good to erratic, and finally settles at a much lower rate. Looking at top while this is ongoing, we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7535 root 20 0 267004 0 0 S 3199 0.0 8:40.65 uncached 3326 root 20 0 0 0 0 R 100.0 0.0 0:16.40 kswapd4 3327 root 20 0 0 0 0 R 100.0 0.0 0:17.22 kswapd5 3328 root 20 0 0 0 0 R 100.0 0.0 0:13.29 kswapd6 3332 root 20 0 0 0 0 R 100.0 0.0 0:11.11 kswapd10 3339 root 20 0 0 0 0 R 100.0 0.0 0:16.25 kswapd17 3348 root 20 0 0 0 0 R 100.0 0.0 0:16.40 kswapd26 3343 root 20 0 0 0 0 R 100.0 0.0 0:16.30 kswapd21 3344 root 20 0 0 0 0 R 100.0 0.0 0:11.92 kswapd22 3349 root 20 0 0 0 0 R 100.0 0.0 0:16.28 kswapd27 3352 root 20 0 0 0 0 R 99.7 0.0 0:11.89 kswapd30 3353 root 20 0 0 0 0 R 96.7 0.0 0:16.04 kswapd31 3329 root 20 0 0 0 0 R 96.4 0.0 0:11.41 kswapd7 3345 root 20 0 0 0 0 R 96.4 0.0 0:13.40 kswapd23 3330 root 20 0 0 0 0 S 91.1 0.0 0:08.28 kswapd8 3350 root 20 0 0 0 0 S 86.8 0.0 0:11.13 kswapd28 3325 root 20 0 0 0 0 S 76.3 0.0 0:07.43 kswapd3 3341 root 20 0 0 0 0 S 74.7 0.0 0:08.85 kswapd19 3334 root 20 0 0 0 0 S 71.7 0.0 0:10.04 kswapd12 3351 root 20 0 0 0 0 R 60.5 0.0 0:09.59 kswapd29 3323 root 20 0 0 0 0 R 57.6 0.0 0:11.50 kswapd1 [...] which is just showing a partial list of the 32 kswapd threads that are running mostly full tilt, burning ~28 full CPU cores. If the same test case is run with RWF_UNCACHED set for the buffered read, the output looks as follows: Reading bs 65536, uncached 0 1s: 153144MB/sec 2s: 156760MB/sec 3s: 158110MB/sec 4s: 158009MB/sec 5s: 158043MB/sec 6s: 157638MB/sec 7s: 157999MB/sec 8s: 158024MB/sec 9s: 157764MB/sec 10s: 157477MB/sec 11s: 157417MB/sec 12s: 157455MB/sec 13s: 157233MB/sec 14s: 156692MB/sec which is just chugging along at ~155GB/sec of read performance. Looking at top, we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7961 root 20 0 267004 0 0 S 3180 0.0 5:37.95 uncached 8024 axboe 20 0 14292 4096 0 R 1.0 0.0 0:00.13 top where just the test app is using CPU, no reclaim is taking place outside of the main thread. Not only is performance 65% better, it's also using half the CPU to do it. Signed-off-by: Jens Axboe --- mm/filemap.c | 28 ++++++++++++++++++++++++++-- mm/swap.c | 2 ++ 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index dd3042de8038..139d1db79ff8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2473,6 +2473,8 @@ static int filemap_create_folio(struct kiocb *iocb, folio = filemap_alloc_folio(mapping_gfp_mask(mapping), min_order); if (!folio) return -ENOMEM; + if (iocb->ki_flags & IOCB_UNCACHED) + __folio_set_uncached(folio); /* * Protect against truncate / hole punch. Grabbing invalidate_lock @@ -2518,6 +2520,8 @@ static int filemap_readahead(struct kiocb *iocb, struct file *file, if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; + if (iocb->ki_flags & IOCB_UNCACHED) + ractl.uncached = 1; page_cache_async_ra(&ractl, folio, last_index - folio->index); return 0; } @@ -2547,6 +2551,8 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, return -EAGAIN; if (iocb->ki_flags & IOCB_NOWAIT) flags = memalloc_noio_save(); + if (iocb->ki_flags & IOCB_UNCACHED) + ractl.uncached = 1; page_cache_sync_ra(&ractl, last_index - index); if (iocb->ki_flags & IOCB_NOWAIT) memalloc_noio_restore(flags); @@ -2594,6 +2600,20 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio *folio) return (pos1 >> shift == pos2 >> shift); } +static void filemap_uncached_read(struct address_space *mapping, + struct folio *folio) +{ + if (!folio_test_uncached(folio)) + return; + if (folio_test_writeback(folio) || folio_test_dirty(folio)) + return; + if (folio_trylock(folio)) { + if (folio_test_clear_uncached(folio)) + folio_unmap_invalidate(mapping, folio, 0); + folio_unlock(folio); + } +} + /** * filemap_read - Read data from the page cache. * @iocb: The iocb to read. @@ -2707,8 +2727,12 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, } } put_folios: - for (i = 0; i < folio_batch_count(&fbatch); i++) - folio_put(fbatch.folios[i]); + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; + + filemap_uncached_read(mapping, folio); + folio_put(folio); + } folio_batch_init(&fbatch); } while (iov_iter_count(iter) && iocb->ki_pos < isize && !error); diff --git a/mm/swap.c b/mm/swap.c index 10decd9dffa1..4019ab371759 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -427,6 +427,8 @@ static void folio_inc_refs(struct folio *folio) */ void folio_mark_accessed(struct folio *folio) { + if (folio_test_uncached(folio)) + return; if (lru_gen_enabled()) { folio_inc_refs(folio); return; From patchwork Tue Dec 3 15:31:45 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892626 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5C9C5E69EA6 for ; Tue, 3 Dec 2024 15:33:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B4C3B6B009A; Tue, 3 Dec 2024 10:33:02 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id AFAAD6B009B; Tue, 3 Dec 2024 10:33:02 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 94D206B009D; Tue, 3 Dec 2024 10:33:02 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6C5A26B009A for ; Tue, 3 Dec 2024 10:33:02 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 0C891A097C for ; Tue, 3 Dec 2024 15:33:02 +0000 (UTC) X-FDA: 82854040458.07.219640D Received: from mail-oi1-f174.google.com (mail-oi1-f174.google.com [209.85.167.174]) by imf01.hostedemail.com (Postfix) with ESMTP id D69C740020 for ; Tue, 3 Dec 2024 15:32:49 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=2t1XGnRN; dmarc=none; spf=pass (imf01.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.174 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239974; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oKUI9o4nURcpztCkRpa2upslaug1l9SMNKWGbNjNG00=; b=8B2cYaliZcBL7AcEchjruXroq36XAiigqZjV1VNiO7br8KL/AE91Ii6h0CprG+/Bf1zIIS jPc9bjYmarn9Lk3gahS/Kd4bqrXAFYkpj+KjPhDYxb3+cJ6jZldGnuWNRhI7KKF5HQJTLZ hejSj5b+QQxpVZ9Nqvf1hO4N/sFJ4cc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239974; a=rsa-sha256; cv=none; b=X396uSavVCJI3qKLAeSh4BXxP+bltP4lL+D4z6feglUwuqcw9yKMX9ennVxXC7ppNupy+o kzR1+r4qruKLwAiSNAt4igin3J4FA4l/amaAMDLBRPRjuYYR6z6oaZPlmS53kijQtsEz6R omY45dhxiCt3Ps76+mSskz32Ate58Cw= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=2t1XGnRN; dmarc=none; spf=pass (imf01.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.174 as permitted sender) smtp.mailfrom=axboe@kernel.dk Received: by mail-oi1-f174.google.com with SMTP id 5614622812f47-3ea60f074c3so3240316b6e.1 for ; Tue, 03 Dec 2024 07:32:59 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239978; x=1733844778; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=oKUI9o4nURcpztCkRpa2upslaug1l9SMNKWGbNjNG00=; b=2t1XGnRNU4m7g4x3VT4RjUo/z03GvtVlJ2qOimY+xPQcHBTlwLvwNlwADaYcRRcDxn keBE4cOyCLG1Lki/z7aYNPuIRX5u47EB05tE0oFYY9IXx9BkQXKYuRjPM4UHWaJwHtow TCpJt9HlS09h38nlzZ3NTJsBc0I2hK3S41072dJYDMuHmOBNiH+Q/Uv7Prbb3XsJinkp 0n+6Ft+9xhlB9TpcILQGE1RLzNEiQFCRXeC7jEWOmx+Wi2h+P3j9pNAR23vqWQwk98Sy HqTqaKaXz0S9q2OKvxAFNHBakUwMNEbqXmaaKq7JPxcVjRiHzoxN2GMmvDuIUY8NgjGZ 7rVw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239978; x=1733844778; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=oKUI9o4nURcpztCkRpa2upslaug1l9SMNKWGbNjNG00=; b=one4/GKFufeJ6Pyz9agm3LtmROxJ2DqFSL/aeoqz4WVVFNVSkrODRLpKJtA7zyRmCm m1uSadTlyol1KGW9vhZvDz8OiKY5TCljVekOoD+7fS7i1w4nH97HdMOn6PA7bTLSd+Nr SV5zI9/C/5wcUX9XcEc+MrC1DQ/Xbf0MVNr7SqHcEj8NYEoVGvtVdTP6MPaWzI1gEXZf Y81PhTjg4Uw9PVJmbcViAVeBSCYW5oGyfDaQx8Z0imOImRMDvwlZ+EPdIf1NXGbCYT3o Lt+9I0VB05huaqV7tX/tayOqnZBeEVLOwMjFkkqf89Cj8rVBS/9MxGjrW6efTn1T6gvL yWPA== X-Gm-Message-State: AOJu0YzPEk8BJlojJOBl5Mg+E6aKmRZhFbsV5B/3FuVU1+Jl7uDuy1ig AbFKV+1Qx7DTgWa0O9T1RliQqZ7h+xQf8yGJVDog05XmLEnzuJPg1Uo9LUSWnUbrCQXNwCJJS++ 0 X-Gm-Gg: ASbGncuwn3EixyzQidwrpfyoE03S9VN80SncdliaUOA0jAYl3MHwWZdUQ1B5ny0wgLY hFGc83kcACD47uCdr90WfqD1W0mEukNFBLvxr5liMGS2nHvCh7H44QZaHCZGnS738LCbdf7nDnF 7y08k2dJlfBv+VCrQKu8VH+05d5Iwk8b9Q9jnSuDLoVITnDhZ4YcPXGc7+tLH/bplKxmtl1xyZK 1CqxjVsuSG587w5VQgiQq6cS9k9zcYwXEOGXAh0na69kafKQQBfp8Db+CU= X-Google-Smtp-Source: AGHT+IGmoU9sxAuWmjwWvkKLQ0cqG9D7TpsEEAnWsGzkgwInxbMdKTegKVNOgiZahNH2vDY40Qhwdw== X-Received: by 2002:a05:6808:218d:b0:3ea:4aad:a877 with SMTP id 5614622812f47-3eae50d7cd4mr2734954b6e.43.1733239978709; Tue, 03 Dec 2024 07:32:58 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:58 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 09/12] mm/filemap: drop uncached pages when writeback completes Date: Tue, 3 Dec 2024 08:31:45 -0700 Message-ID: <20241203153232.92224-11-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Stat-Signature: 6ewwju34th141kw8qz47j17b6nxrcpiq X-Rspamd-Queue-Id: D69C740020 X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1733239969-33832 X-HE-Meta: U2FsdGVkX1+Y/A1/Zfb6OyspBk7aEPSy2QUHceVlKzihcnd9K/jXJWKTJ9v27jmBJbolZmmuToz+NioEEvfpK9U6Nu95SAf49rDkh4rNqaSUJI9L91RrqzWYfvmBYYKAGYEnMQ3KLdp1uItItofj2ge7pUptyFH7vjih4nu20MO4OpB5tKq1O4oKxKGBGn2s9hev93t19rE7ZCL9FlI8OuSCoo9V8WAX0x99V0ql20Apqi4r00sxDU2nGdrJmmTWv/GS9QRmqAFkf3okJy8SS8nRKPxzh9TMsQk2CAcRr2CoGz+SnNfgqqy+3U5FKY5tHwG8cogwLgDsF3wV8gki8aA2OY13QzKeCNzGDhT4Kon1K6yHiqxXGCuIg44r8CHHinyCQuVnCZm4zeRyc2DgPHm27wjQ5F67bAsH6jNYbQmS0LP5OxeH9GkABgC1fsxCVXA612kNZRAzOhT6ti9qzcfNL0aY9ZwnIsKx3z3Qtu5PxTlco+zYvPUrqFebxv/LMFTGMmFRWNq4JOFofnGBkZyyWZ4R26dhxgchfTjTTy5Fy0ZvpifMTaKkgX44q5t9TFvzFlRRaG7knLvF1uRjuIUP4FCZKnXh/snQLT98EleYnCM+ixazmMJEwTSs0Qt6nbxCNrBmQFAwLhrBvOu2F3+QCSshVj70ILjNpq/eYuXGohqkRUlPQGaUvVbMriS0lkx+gjRwBQJzF79/8hr9BNs6QVRw5ZfcGyo8O14hoDWs0ic2+HClzNhoqa7y03rgEAkgJSQdMGL0XpVi/3dmqCa9dErusF7bZ6asz54p0OAwXdXWO9QoJlkzVtAJ0CNA5FoG+ZOukJ57BY6mCaKsR1sljFja4ZdUTzQivIniQtOxJWivijR5FBBf2gp5NAPhjkkMVKdiHVvAhef2yOa3PnzuLcXyvC+OG8OKCmpeFbn9SdDBXFqhvJ7sx6kVDmKUw/uz2BOocFhJUoTumdJ 5fzFINCB rImVQsXpMHzj2P3k0k0qJPoYBQwNfxeWH/dr4uxe5jUWeQCRhDk3dVMCJQDgr4xAthlz6y7xfa6Q9R+HQ1voXWhbmSnMzCTXQAEdn3yldadAidSa7AyU+hkUW56RFjsp+fcE0ZZPsw3oXVvsrPIPkTU3QuAWIazpfwa4JH6aQqZ3xUU0GV113k5LSKPXy3/AhmJKTNvV5zEP+Env2CkxKFGHLUaHozcutqx3FuqcUHrPFEgb34wTBBlB3bR7eVT1hDxMINVlx8lHZ4h/vAVJHpb/EARYU96toY4w3QyeHgX5wSAw= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000002, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If the folio is marked as uncached, drop pages when writeback completes. Intended to be used with RWF_UNCACHED, to avoid needing sync writes for uncached IO. Signed-off-by: Jens Axboe --- mm/filemap.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 139d1db79ff8..eb6a8d39f9d0 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1599,6 +1599,27 @@ int folio_wait_private_2_killable(struct folio *folio) } EXPORT_SYMBOL(folio_wait_private_2_killable); +/* + * If folio was marked as uncached, then pages should be dropped when writeback + * completes. Do that now. If we fail, it's likely because of a big folio - + * just reset uncached for that case and latter completions should invalidate. + */ +static void folio_end_uncached(struct folio *folio) +{ + /* + * Hitting !in_task() should not happen off RWF_UNCACHED writeback, but + * can happen if normal writeback just happens to find dirty folios + * that were created as part of uncached writeback, and that writeback + * would otherwise not need non-IRQ handling. Just skip the + * invalidation in that case. + */ + if (in_task() && folio_trylock(folio)) { + if (folio->mapping) + folio_unmap_invalidate(folio->mapping, folio, 0); + folio_unlock(folio); + } +} + /** * folio_end_writeback - End writeback against a folio. * @folio: The folio. @@ -1609,6 +1630,8 @@ EXPORT_SYMBOL(folio_wait_private_2_killable); */ void folio_end_writeback(struct folio *folio) { + bool folio_uncached = false; + VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio); /* @@ -1630,9 +1653,14 @@ void folio_end_writeback(struct folio *folio) * reused before the folio_wake_bit(). */ folio_get(folio); + if (!folio_test_dirty(folio)) + folio_uncached = folio_test_clear_uncached(folio); if (__folio_end_writeback(folio)) folio_wake_bit(folio, PG_writeback); acct_reclaim_writeback(folio); + + if (folio_uncached) + folio_end_uncached(folio); folio_put(folio); } EXPORT_SYMBOL(folio_end_writeback); From patchwork Tue Dec 3 15:31:46 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892627 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1D57CE69EA6 for ; Tue, 3 Dec 2024 15:33:15 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id C865E6B009D; Tue, 3 Dec 2024 10:33:03 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C31B76B009E; Tue, 3 Dec 2024 10:33:03 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AAEF46B009F; Tue, 3 Dec 2024 10:33:03 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8A68A6B009D for ; Tue, 3 Dec 2024 10:33:03 -0500 (EST) Received: from smtpin10.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 41804140AFD for ; Tue, 3 Dec 2024 15:33:03 +0000 (UTC) X-FDA: 82854040626.10.D070C8F Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) by imf11.hostedemail.com (Postfix) with ESMTP id 9226540017 for ; Tue, 3 Dec 2024 15:32:48 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=fdLET6XF; spf=pass (imf11.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.176 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239967; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ckrp5/MogNB2HvhzJuNOPyu1ag5BruWn5XeKlERZse0=; b=w7ITQo+UswNuNo5u0bs0LFZav8+eHw7+d/Agv+49Y0G0TYpmlmPcT5rvEn94Ew3S4uPjte /GA0tPtBS27YNMgE0j7HSa2/1XaS1p7xjxmCzIC36xTm5KnwF8ox84EO44Zp4crJnVMhfJ Sr/bjrGLB00cW59EK6f4dAC8oK1daoM= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239967; a=rsa-sha256; cv=none; b=vPh5oHVzWjqJJvEp3iwmk0L/0kerFlQ8GHnpqqIIHKnkhYvVyPARBFFmg/7+9p4arlct6/ nE9Gb9PuFAqNy1sW8+YrTTrDEHyWdsu+7L/1TOk/eeVbYcwv0tq4uPLKg1k0y+ZAYko88t ergt47mxhZczKF+6JNshPdufDD+RUHk= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=fdLET6XF; spf=pass (imf11.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.176 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-3ea4c550a3fso2766381b6e.2 for ; Tue, 03 Dec 2024 07:33:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239980; x=1733844780; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=ckrp5/MogNB2HvhzJuNOPyu1ag5BruWn5XeKlERZse0=; b=fdLET6XFAdUxlQKyWs51WOE13dkUX3j4XmObPWhTEUGlJX8U7kFPH0duXqlzjpnBXN FMDmmNYBED+JH5FO3nCe5bMtdKZo0c5SwySkwqaZDeb590iZmqoPLHcYAm6+TqPMitBW 0r50cArkWOlIORros03YtXXll3zsbS757Ad67Z+ytB1PK/SA3b/Q+DLJ2uOYzZHT5GOt 1RQFAA9f73U95Dyjsg0aeTJgRjHTII8+HgNepuIApgldBqjRFbP/JqdVmArmqh8mrMPt lvQ4aX3MT0+YrV+0pXLtbJ9WCwdcSs/oHZ4ewwELDNNzmgGiU6cdRErRLW0JWutEIlQ0 BO4g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239980; x=1733844780; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ckrp5/MogNB2HvhzJuNOPyu1ag5BruWn5XeKlERZse0=; b=rN3r+7ZXwIV8t2lyVZQOZgOj5xm1e1U6MWkIkANI7X3uDJwIhLCuf0w5O81HTJihZL Z2OA+gx75AqFWIbWuVOcWzz/VOgjXXuNuOMUo+Lc8iiTei4BeyAjTde6f4cDdrpaEuDz iTcFgTi4ne8uK+wSMx3U0jmByftb11IwYuZFmZerUGV9PQmMaVRvNZJyW4IneoYeT08O k14r1hm94t7PEoTxvq5asVdU2NPn33hYAmzv4iKWV9JGWwLa1uAzU7ZPcC/s7XEXL/dd jS8WR/LkISvzeb90vUD5F9Y959Ca4suC6aovsGQgH5mQtacKb+dqBGQ6x2Rv6g5TAIXz HXTg== X-Gm-Message-State: AOJu0YwK3kARMeDgHwKAyQ2z5/g3zZw34Z5KuyGJ290jTeZisiRuJkTF TwLjt695EwreLZdyeJJk1R2p/iWa7M9c1l/lFiy3Rf+g45l/d8On2It7XNghridMUY1NPymAKcr C X-Gm-Gg: ASbGncshArrdCccQTVSzurbcQqqFAxmXMd82GkE9olzfP1Iq7TObvkwZY3SbjpB4uTR +z0VtWyCsP3yfPAcbkLo8rHrKw3wL1ms0tQutIErDbniPk2uKJdaXZf05DQOF7AkMVygtyTmvss wjcfT3rJwF9PX3DYecSJSib3+Pa98OnGBgRcVViOBX9Qm9FJ8+BcNKJJ4IvzULPJXmoUkrIe61+ SIM3c73iREwaHkFOBP/kbrpJnFc/dNeRfFcb8c5Odo2t5LG0tYPnEjzGoQ= X-Google-Smtp-Source: AGHT+IG43FQHTFqL4PqM6F3qojzARV1JxsiCZ9D3Zgsdfa2TEHjYK26CmkPy2H7KTh+zkEYZWpQaaw== X-Received: by 2002:a05:6808:3085:b0:3ea:52b7:ebdd with SMTP id 5614622812f47-3eae50c51b8mr2904392b6e.42.1733239980115; Tue, 03 Dec 2024 07:33:00 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.32.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:32:59 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 10/12] mm/filemap: add filemap_fdatawrite_range_kick() helper Date: Tue, 3 Dec 2024 08:31:46 -0700 Message-ID: <20241203153232.92224-12-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspamd-Queue-Id: 9226540017 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: d1wnenu3qc7okcuhst1rzbau9hz5xs3e X-HE-Tag: 1733239968-774086 X-HE-Meta: U2FsdGVkX18C2UpC8gPPYcz7YRlAiQM3m5pykSZSN/viUvvvdq3QrR0rV8XwqDm7Mmbopgp2wVctjPDZ28CPBhThExa+919ow1DKT2etOeSLStk9tGPLV+uUqE2spLDVAdOsZTQi314S5S+G7EVwHZvw3JWBxWqydyS31C4EuWitQeQkXqjz4PHijmI3aVEduJW9tptPMlArJzoSzRpu1ccV/GxWcc4E0gUc+BsCFhxJCqjgHHpzn+Nqs2BiQwoABir5DfjXwaEbCr7fPRDR8zmmFUKNluah041sw0/psoY7KjKStPfzHoM8InXSCvziDeGuvhfGh/Rv8uvuy6daNyh46pw15bPQ+d2l6Z/NvJF6iOmf8d56W1q9zafeW7rgkuuj+FuPOwKGLfRMoNU8L2ktiLu5pL0ibkxDkImaTvfL2LhGR48y7R7aMTwFYWB5Pxab/FqOom0Vf1m8qSnoeUDKyVu4U+qqpf4GlacrqEGBFa8Rmk9lFAVJGL6cdwdijGN67+NQRYmIB77nLuAGvKHUYzMZxKefnr8vLIyGW3cE0xtgZmSdC5I4nGXkDDWIvmK1YgAYRTnGL9ddDBSTPbD1Dc39z/zgXWHnHwKlXxBlIRboZbjFDtKpduFlbfgXu/POpP9X7RPqLFN9gqlMGu7yG3pwurf/S37FfRqSXv3YIQSGF87kxVB95+rqPnRa/XPG59rTcK6ZbSlAIfERuTPvnmbtUCMsiVAVwvHsdOZTv0J3sNPJ3g9XoX/ESS9WwVKu96SMsX9imPWJFTanwsYymKPNuIpg2SxbMRhHkUI9WtkTgxRcQFQz+VgRNSEGsJT65eatEl9kwszof4Iu/Jr6x57XXqOq99R1pU0kh9++19VGG5C0SWjDq1NAaaX8ZfxXYSBNbiaabxnwGxKekKPH6AbbiJ5yukTRQWIP6x5Xy1RJhOO5USkA4QyuD8WSRaC3imqhvZAGjqah80y qK/uH2Zs hjtXTFwtNjkRWsPF1W5DXjf0wvynkrZ4G3A4+rSwiCtpKptXrWZl2jDXs9Kl350z53rISK6Ebw0ZKYDi3jHHkFJAvD/UJC56Bx8r7to8DKe8EAAWcN53u2FUq0IxbLHadsaprAUojTRqG8Xri4vZfEHbZQGMnfBjTIuDuBTWtQWQBGenGviLpGqEawDQkM+b1DzusH6BbazuErqv5aXgjr72hBM0DFab63mVb7C4xXI2UU7bTmHbngFIZnXNqJ+8pPpNVU9k4jleVB5EHhANkJMDmsduxxB/EuW5JQ5Dr1pTS5Yu8tw6NuGz5xSowBAFfacrhVQyDz4tLPOJT8bMFDVk5sxdJl+Bac+Ep X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Works like filemap_fdatawrite_range(), except it's a non-integrity data writeback and hence only starts writeback on the specified range. Will help facilitate generically starting uncached writeback from generic_write_sync(), as header dependencies preclude doing this inline from fs.h. Signed-off-by: Jens Axboe --- include/linux/fs.h | 2 ++ mm/filemap.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index b64a78582f06..40383f5cc6a2 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2878,6 +2878,8 @@ extern int __must_check file_fdatawait_range(struct file *file, loff_t lstart, extern int __must_check file_check_and_advance_wb_err(struct file *file); extern int __must_check file_write_and_wait_range(struct file *file, loff_t start, loff_t end); +int filemap_fdatawrite_range_kick(struct address_space *mapping, loff_t start, + loff_t end); static inline int file_write_and_wait(struct file *file) { diff --git a/mm/filemap.c b/mm/filemap.c index eb6a8d39f9d0..826df99e294f 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -449,6 +449,24 @@ int filemap_fdatawrite_range(struct address_space *mapping, loff_t start, } EXPORT_SYMBOL(filemap_fdatawrite_range); +/** + * filemap_fdatawrite_range_kick - start writeback on a range + * @mapping: target address_space + * @start: index to start writeback on + * @end: last (non-inclusive) index for writeback + * + * This is a non-integrity writeback helper, to start writing back folios + * for the indicated range. + * + * Return: %0 on success, negative error code otherwise. + */ +int filemap_fdatawrite_range_kick(struct address_space *mapping, loff_t start, + loff_t end) +{ + return __filemap_fdatawrite_range(mapping, start, end, WB_SYNC_NONE); +} +EXPORT_SYMBOL_GPL(filemap_fdatawrite_range_kick); + /** * filemap_flush - mostly a non-blocking flush * @mapping: target address_space From patchwork Tue Dec 3 15:31:47 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892628 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E67B5E64ABB for ; Tue, 3 Dec 2024 15:33:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3593D6B009F; Tue, 3 Dec 2024 10:33:05 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 30AF26B00A0; Tue, 3 Dec 2024 10:33:05 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 10E2C6B00A1; Tue, 3 Dec 2024 10:33:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id EB1EB6B009F for ; Tue, 3 Dec 2024 10:33:04 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id A8DE81C74C2 for ; Tue, 3 Dec 2024 15:33:04 +0000 (UTC) X-FDA: 82854040710.22.2484658 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) by imf29.hostedemail.com (Postfix) with ESMTP id A2FA2120012 for ; Tue, 3 Dec 2024 15:32:43 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=RnOpZfH8; dmarc=none; spf=pass (imf29.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.182 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239970; a=rsa-sha256; cv=none; b=yZB5qfiQ5oh9Vo4oHH3LhKAtLpWBmKdiiH/0mUDzlNbVPTKCGwlI76HacCbAvwBwzefMi6 JLtCJq6msm5nvlLW1tLmdxneR7c6J2ZYLgnkco78tpyy6ns20RD3Cq/Oumjgzm1y9/ZvHK GOYRjFGr5SeoqgBpeu32Hu48sr+Vj8A= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=RnOpZfH8; dmarc=none; spf=pass (imf29.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.182 as permitted sender) smtp.mailfrom=axboe@kernel.dk ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239970; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rNRkYctWaLbO70YbebeuOKFfbUg5iKQfEN1xFJYmHB8=; b=xCW1LraQxeaA7IBkF9g0ujHPm1u2u5NEXQxorjHPS70/sJ5oYXPtCze2OhAopcloi8xuwG pJTjEqb5pHFKbaHIC0V+XwlIo1ljzvNQZW+9Fv1I6IlkT91jLjne0gz2NH7J2L8aIbVLUD e2VquNx68pdOSo7CehvuonfEey258rw= Received: by mail-oi1-f182.google.com with SMTP id 5614622812f47-3ea55a2a38bso3014524b6e.1 for ; Tue, 03 Dec 2024 07:33:02 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239981; x=1733844781; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rNRkYctWaLbO70YbebeuOKFfbUg5iKQfEN1xFJYmHB8=; b=RnOpZfH8BuL3g1dSFP6XELBS3E5zKDRlQ6yz2RUNlcLwhtIGyBYz+MiykvhnClN//y xSk8lQ9wGSgVO/NfzpH2/CYIdj0gnfOKFbbq9BVUrelcfwf1wVP4jAtAMClLMic11EVW AO0gMuLUzWcIkuAnMZMJq97EWEVtOXzLIhY5zbnbU8gv4oLnW2LeFWpRlF8pJHEM4WrY GSTQSGI4e2Qd0vVY3l/kiF3GKPfnsr5XGWyfukB5rj6g5Kj3rDexC/VjL89BE5oq5Hj+ T9yg+PT5bzyo9U4RHibQVut06LWrSvB6GVPtljF3dI85Hs1Mpktv+gLZZgFsM75FbIQy reFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239981; x=1733844781; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rNRkYctWaLbO70YbebeuOKFfbUg5iKQfEN1xFJYmHB8=; b=kpc4tn9KpNOt9Dl0WLv27N9ZhHtccA0EcHFZVYscZthjvXnuJ2CO2L3O51ppcqFxvu 06FczNlfauFRTnT53pUosF5aYoPbT+1lky/Iw8smYHFB9TlDDZKwtO7GEjR9N1nB5cJd LWuM4gGKljK6RlicbYQEHmM4RUhOamzn2B7/N6dd0f+8IOLeeBHr++izXfBSL2meTiHt 2H6zO+DU4oThOXSHj9q32SxaMqCrem6OoOIiMvnzMCSFeBoUga59MOaWkttsE5Q2pLcR qrcL8g4J+5MgQYhv7hv4XZAPwm00KFjSZXsP8aExZxoYxPHRUFUVa7JF3jJQ0G5sIsTW 2oXQ== X-Gm-Message-State: AOJu0YyOZy/nXIpXuQNEdyu+KZoq8B4qKgJHoP3qFbBZcUTLHyM8hDJA gegJjfn/0NTHgpULi82g2QUIQuNcpmwwKil9/fHqphqLUZdOBYgv0VOGtNWZb6C66wGZgG87i0d 3 X-Gm-Gg: ASbGncso38BxlB8IWrEDeaRoZb4FIpfDaJFJFiugth82nxj+MdL+ZP98m+cm/ERuCMw uvaU5tPzhqjXwjHQCw9lfp7E8jVcu4OwOK2t9CiymgEyfc1iMJ5mg+Rh1Ld8wDVxL6cIeDwZs3o 5vTyGcFxPPgzh6uTwa/8qKZYSMIEHzgmNqA8L/8QsIowV1asZqc4tPZdbmSnx6dYJpzcBUn3hSM cW6ON7M5RVwvGIhoXYoR4KWw5QSruYY18uU4QDMUb8pUsEFSckUDfR3udc= X-Google-Smtp-Source: AGHT+IGnmoFBWMRz09BuY3oNVVI/1OH4qyI0X8Y48goC/ICz9U7NJeNOcoRPtjmNOo7azkeA+JUnjQ== X-Received: by 2002:a05:6808:200d:b0:3e6:6208:b737 with SMTP id 5614622812f47-3eae4f3bce5mr2757240b6e.17.1733239981545; Tue, 03 Dec 2024 07:33:01 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.33.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:33:00 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 11/12] mm/filemap: make buffered writes work with RWF_UNCACHED Date: Tue, 3 Dec 2024 08:31:47 -0700 Message-ID: <20241203153232.92224-13-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: A2FA2120012 X-Stat-Signature: mjrnjurpha1h3sfh8y1k9b6tnz3qax5b X-Rspam-User: X-HE-Tag: 1733239963-394803 X-HE-Meta: U2FsdGVkX1/4f0siUSZhHZ0HLDXgVTH5RW+GZKuGXbzDfnsByNB8ZK2jHN45a38/917oDnDFTYcUHLIw4ucB6BgcDTAsyWuzhLl0a3O5FzTqOgD6NusznNeQQQb5N1HQKktMr+1HhXckkwR9THaN/ViXKWsa/e+NvmUzQJir77SmzxXff2JQwxhaSnghxgWItp9xTQqk1DBaZR5Wt5Am4MeQnoz1S57VJgSBEYcneXLmjytSZhqu75GuowO8jHCl+lo/5v2kEqpajjNhx9E/JBwpimNy3gK5UfgAuunwB/g2qah3Orm1qiq8aSkXMgaiBNRbyiOiF27w0oJXGLg8NVh3z3U8U/QN8WxHNbsjo66Q1yYPh7jsLr7+4iGlz3iZ/OGgOLBzU5M71UTI5BLY1FD8bS1Xo08ROlWUF0cfwZEh/J/qUSyBFVvU7e4CoP217BYB/Sn1Yw+aqSjIkUngnP52IfORe8HN8xC3/EebIGADAWLvmzDk2NSh5BUEHjF00OjLMiceshKHhTnQHeq2PgUr3OQ7H6SxusMJWqBMo9kFrctfbeQqyFdd8jnErZ1PJ5oJGKglzfwB1uOM3gjz3iHb6Ctl8nPhACNkh3JmEeO6lnrAovL91YZoBKGwA0ZMzApPglrI8yZOzI+ZuL+VOJhiN3AGTKc3Xp8WIp3jOZZS/Kk02goj9bBENW8I7Oq9mHTPKNjl2QzttV2DBbPclHkmaVYpzbUBvSWwIIuJSCceWSh9Vna5Y/HX5dIUA4ugRHD0NyMiOKGQOQQ8btzUolrVCNDI7z8Q0GdaURnf4uHidskFP8SYZ5QSNGI+Ade44i/QESlFrc3v4DCH05YbAJKO6/Ws1KLwJki1+loNp6AyB2pELKpJ9Z/bilYje5s/ylyK6EQgcqQKPis4+F0ZHt38ss5E3eRaZwfWECzHHFKtr/bImB2upZaYR/4lnKKRuZYGQ0ZEix9KaKjYLmz VHY7KK4C bvcQaAl7fq0FT/L0geWoIe5l4hwtQVZC4VdupE9RGX1TFbjr2zl+/J6Ppn7PemtzcweBmMvZHVUBRHc5MvnQDosrq671L33YKsVi4VsQYHbtNEF9DKCxebuizPfOn41uUvp1d56E9uF9iAG+ffxYd2BJId0V+jG77srUJ+NZWMdZODb8ubhEQ7UenRo0Wd8kkunf0oiQEjNMml3Xd/boinyMpAwEp/U6t7Kp2lADoVuXFUnqRafpgHpc/Sautm0HYma72XeS2M7LA3TFsDlZwxMfK1wXt7mCTtIfmuN/itRqp0Big8kop/UpYFiZwk1FARKkmxy1Cwai6skRIopMncnhDgNjN/6xgosqg X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary. Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached. This provides similar benefits to using RWF_UNCACHED with reads. Testing buffered writes on 32 files: writing bs 65536, uncached 0 1s: 196035MB/sec 2s: 132308MB/sec 3s: 132438MB/sec 4s: 116528MB/sec 5s: 103898MB/sec 6s: 108893MB/sec 7s: 99678MB/sec 8s: 106545MB/sec 9s: 106826MB/sec 10s: 101544MB/sec 11s: 111044MB/sec 12s: 124257MB/sec 13s: 116031MB/sec 14s: 114540MB/sec 15s: 115011MB/sec 16s: 115260MB/sec 17s: 116068MB/sec 18s: 116096MB/sec where it's quite obvious where the page cache filled, and performance dropped from to about half of where it started, settling in at around 115GB/sec. Meanwhile, 32 kswapds were running full steam trying to reclaim pages. Running the same test with uncached buffered writes: writing bs 65536, uncached 1 1s: 198974MB/sec 2s: 189618MB/sec 3s: 193601MB/sec 4s: 188582MB/sec 5s: 193487MB/sec 6s: 188341MB/sec 7s: 194325MB/sec 8s: 188114MB/sec 9s: 192740MB/sec 10s: 189206MB/sec 11s: 193442MB/sec 12s: 189659MB/sec 13s: 191732MB/sec 14s: 190701MB/sec 15s: 191789MB/sec 16s: 191259MB/sec 17s: 190613MB/sec 18s: 191951MB/sec and the behavior is fully predictable, performing the same throughout even after the page cache would otherwise have fully filled with dirty data. It's also about 65% faster, and using half the CPU of the system compared to the normal buffered write. Signed-off-by: Jens Axboe --- include/linux/fs.h | 5 +++++ include/linux/pagemap.h | 9 +++++++++ mm/filemap.c | 12 +++++++++++- 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 40383f5cc6a2..32255473f79d 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2912,6 +2912,11 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count) (iocb->ki_flags & IOCB_SYNC) ? 0 : 1); if (ret) return ret; + } else if (iocb->ki_flags & IOCB_UNCACHED) { + struct address_space *mapping = iocb->ki_filp->f_mapping; + + filemap_fdatawrite_range_kick(mapping, iocb->ki_pos, + iocb->ki_pos + count); } return count; diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index f2d49dccb7c1..e49587c40157 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -14,6 +14,7 @@ #include #include #include /* for in_interrupt() */ +#include #include struct folio_batch; @@ -70,6 +71,14 @@ static inline int filemap_write_and_wait(struct address_space *mapping) return filemap_write_and_wait_range(mapping, 0, LLONG_MAX); } +/* + * Value passed in to ->write_begin() if IOCB_UNCACHED is set for the write, + * and the ->write_begin() handler on a file system supporting FOP_UNCACHED + * must check for this and pass FGP_UNCACHED for folio creation. + */ +#define foliop_uncached ((struct folio *) 0xfee1c001) +#define foliop_is_uncached(foliop) (*(foliop) == foliop_uncached) + /** * filemap_set_wb_err - set a writeback error on an address_space * @mapping: mapping in which to set writeback error diff --git a/mm/filemap.c b/mm/filemap.c index 826df99e294f..00f3c6c58629 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -4095,7 +4095,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) ssize_t written = 0; do { - struct folio *folio; + struct folio *folio = NULL; size_t offset; /* Offset into folio */ size_t bytes; /* Bytes to write to folio */ size_t copied; /* Bytes copied from user */ @@ -4123,6 +4123,16 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) break; } + /* + * If IOCB_UNCACHED is set here, we now the file system + * supports it. And hence it'll know to check folip for being + * set to this magic value. If so, it's an uncached write. + * Whenever ->write_begin() changes prototypes again, this + * can go away and just pass iocb or iocb flags. + */ + if (iocb->ki_flags & IOCB_UNCACHED) + folio = foliop_uncached; + status = a_ops->write_begin(file, mapping, pos, bytes, &folio, &fsdata); if (unlikely(status < 0)) From patchwork Tue Dec 3 15:31:48 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13892629 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6E827E69EAF for ; Tue, 3 Dec 2024 15:33:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A87CF6B00A0; Tue, 3 Dec 2024 10:33:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A368C6B00A1; Tue, 3 Dec 2024 10:33:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8D6266B00A2; Tue, 3 Dec 2024 10:33:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6F7546B00A0 for ; Tue, 3 Dec 2024 10:33:06 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay02.hostedemail.com (Postfix) with ESMTP id 2965E120B54 for ; Tue, 3 Dec 2024 15:33:06 +0000 (UTC) X-FDA: 82854040206.20.C459038 Received: from mail-oi1-f178.google.com (mail-oi1-f178.google.com [209.85.167.178]) by imf23.hostedemail.com (Postfix) with ESMTP id 6BC67140018 for ; Tue, 3 Dec 2024 15:32:55 +0000 (UTC) Authentication-Results: imf23.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=jPt3bW9L; spf=pass (imf23.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.178 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733239978; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=45ZT5fvdDImmdgAdnvc6z/+AzY3WZJ2Fl5NYIbZiSAo=; b=1siB5WCq4sbkWhGmbU5dmqBQKer2QZHX0dL6QzhPU6A1MCxSF7csiko1IqiD/9PeMmAY7w JJoolJeQxnmqPuj4wjm/aIYs5W/DGLd5UsLrzfsC9pn/RRQ+vVHwHk1yhyB7N51iFGkWRr dnfk+K2npPn14GsHpDQbnIF3HTVYfXk= ARC-Authentication-Results: i=1; imf23.hostedemail.com; dkim=pass header.d=kernel-dk.20230601.gappssmtp.com header.s=20230601 header.b=jPt3bW9L; spf=pass (imf23.hostedemail.com: domain of axboe@kernel.dk designates 209.85.167.178 as permitted sender) smtp.mailfrom=axboe@kernel.dk; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733239978; a=rsa-sha256; cv=none; b=whUi4aQKkYOF+6XN5/77BHRkU2h6V1ObTBMDOcFZMVRHA+5jghs9cpSLQ2faE7jeCJk8LF v63xe/abGInWZISpzAnZWQY4Ug/ArZfnCJCxmlfWKpPzd08oCPJ+QpXMPVD9jkjaqNQrOH iSx3kDbWWt3K4DFJcJ+u5sIt8NjTrAU= Received: by mail-oi1-f178.google.com with SMTP id 5614622812f47-3e786167712so2728175b6e.3 for ; Tue, 03 Dec 2024 07:33:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1733239983; x=1733844783; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=45ZT5fvdDImmdgAdnvc6z/+AzY3WZJ2Fl5NYIbZiSAo=; b=jPt3bW9L3CFp6HWDaRnqqjgbWfYU1VAqw1gE6xE7FNm4RBPFPTUQrHdygjFXDPrebe 33Odk4FbGzHEtldbeR3h63MqrtBZ9nU1iw6pRQr5VZZuj19jB+v1MRTJlSBdC/sn9oOU 5beGrse4nuYwAHJDgno1bKt9NLDHBD+sX2LIxSQigSTluT6mpC//xkBJznoz7lOgTFeM o5FwPrA6b+q3DpjYA8Xxn3AfM2cL5WSP72xWXBFfToa84evZhxqzHzjdmTQV4GYuDOg0 Okt2jESeGIlYEDrYUVwmsWyU/ImjbUUY0pPfhv+XHIZLCXaW2N+CpQQM7LJjmxpQRBb/ iYHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733239983; x=1733844783; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=45ZT5fvdDImmdgAdnvc6z/+AzY3WZJ2Fl5NYIbZiSAo=; b=SJA8pdkaM55Uebi6QFfSZ4x3GeNE4fo7addBcOxkbIWBiYRom1rEburLV2vHkE6gXf 12hH0NJlMMJOQ4I0+G5/HAcV1bcHu+PHXEDG4746iE7L0SrBFpmbOn/ign23gjuT8Kfo uCP3gQODRTKUh2z/DCQ8xPpeH/H45/pDWBfkUWY+T9eTZKj2nlKhdFr9dt8SDKMK/Lk0 8Dl8u/hO+vTKXczuWUhmwDp9PjRDRdNePJyhLkDFV++7MqhUi+Uvel2p0mFjcATXWSlt X6xXObxP23vThSqoELroHr3O3jvCTZeZCBDrIKRubbPvWSX6aCMKyb8+aeYups9twss5 4KTw== X-Gm-Message-State: AOJu0YxgAcR2/Yi3uFpGFthOBYQXXGk+GMQo1Qd+9Gd1guON8BQe4q4z sjVYpXp8W2UO+5yhEzSZAzNS/geIDUFfPHmAJ11N/TeWvvQcBuYCgGBsfJ1ALVsG0X8UgqPA4fw n X-Gm-Gg: ASbGncvTy1TcnoViYwvLlQ2uksbrF0DxFNj9t94tZ1YX5slnnirKiA9fsFGtgRouoxj lmmyeL4BpjwvYiPArJon1kDVxwVDPbK8fhCIE/I5PS9DaZwRBnBF+7wRZ1kne1UDRP2WVBV2AX/ uZ5+ooUY3pDy1B1gGpbQ7iVaaP954EUcsXXcekaNEyCfOEqOG93NAqRibrHOshRqJSp6puoxu4W m/Vtuhw89lBiK5wI6TyGLilp1LC+bGYfzhZBqP3mYD0Y3Gr0xmmB6itDCU= X-Google-Smtp-Source: AGHT+IFgNffB59kRtGSa279D+oxFDG6GGmZlBVzfslJYY485yS6Rp8valFBlgcp9p+EP83czXVtQIg== X-Received: by 2002:a05:6808:199e:b0:3ea:6533:f19d with SMTP id 5614622812f47-3eae505a0femr2960000b6e.30.1733239982974; Tue, 03 Dec 2024 07:33:02 -0800 (PST) Received: from localhost.localdomain ([130.250.255.163]) by smtp.gmail.com with ESMTPSA id 5614622812f47-3ea86036cbbsm2891878b6e.8.2024.12.03.07.33.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Dec 2024 07:33:02 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, bfoster@redhat.com, Jens Axboe Subject: [PATCH 12/12] mm: add FGP_UNCACHED folio creation flag Date: Tue, 3 Dec 2024 08:31:48 -0700 Message-ID: <20241203153232.92224-14-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241203153232.92224-2-axboe@kernel.dk> References: <20241203153232.92224-2-axboe@kernel.dk> MIME-Version: 1.0 X-Rspamd-Queue-Id: 6BC67140018 X-Stat-Signature: jgywzckce1xeswyytdkiw5s9nzensxgx X-Rspam-User: X-Rspamd-Server: rspam11 X-HE-Tag: 1733239975-322383 X-HE-Meta: U2FsdGVkX186Qzrg8lLkTlJOfoQDLBJ6b52b3G/zt5WKj2rv5ztQR/OtKkTllRgIHWq7ssz66+yWJ6MBnmqp7Pkz8W3zWm7DoKLYhInDC0qU+Ie1ueyX6SX88Xal57/KtqHK0DzfSsKgczBX3lmBykkZ6NbrLnRbccECF46A+MQhGW6ozD2+uk1q8eVlC0j21TmT9uxNwFN7lP9osOCHdOM99puCL9k1rdhLAzBlSl0aP2CRV8EXYahQk7um0S+d7rNFglrjNP24UNPTFcpm9vXYjA9cP2Auh+xOM263wQlVZ+vty82ntUf4esSr72+1z544rRqaqZUuRLfrMCDKjPo2fu0zE0DNnzyMTwbXguUYczU4Q6P+n1WeagzaDJXQhQkv0R1r2ohDPASQXfu8fmmnR00tWzY9D/jQCi/vbYNas7PxDmPuOpb6ysRNs5fVLtSju08gEN0KQZCx3XQbHNlWU3glN8YHk3H/FixcPM0fyJ5Cnz6ZnBF5fmUs0CnyfYW5QeMw5YOQK6okaubBbI8akBCjvc/D8nX2tJ/NqrZrZvIZxhSMEo2V7p5fQF5STMHhuxssDsSee2T/71TTj8vk2Dizy12QmlwbNRzDjKx+NiaHx3StFquQ+zKnYggjoZr1Z7sN0/BrmZvvd6USboJe2dn3Qyv3dB2vvPWGfnXdgcCH4O2HjZGgXGxsbZlsomrtmFec6YPnbT8kYTmSmKR+Gi34EUJEABnxFCtX5UHmEH+D8p3QIKi4iUBJ5P5rgXCbYQ1SPk0SakhFhM2w5LnwnEE47L2gUvELsrbsi4zZqliJA/1bSDB0cA5waZGhLNwHrKJGHmthvwyy3IPKPZSjRwKvk4sgKkH044l8D0yor53bOs9+CX18H+uLrmp3xngGkg/z/X9AcaIhWTWiIVaiIrqyLo0uVNzL545lkmUlIRXY7LJxROb2Y3BoEc4c6a/VwV09sJGc0mWUtPf QT3JUPAz +/BmZFnSWkj1mMtfVmHNVjtlSOPioKup9Yi8nT0VBISuOa9H+I+RUolgev4VQbcwjN6lLg5UIvxXP8IlQImKgermgGpgNFyZp+xEwJhz81DNdX3/dyxhRwMv1a6Yyxyyc/MmnlKFwaBKDBEM20Qe9XptLUmXcKxcGtBgV+jmOEHK5lqAn6mlUz6bKu1E676NcrGsePE4/qb3W8te82WFQm1Aq/8udsW13VhurxLoZeM5jo0cfjxxllUkr5jCa7j4fnMpFVIfCuv+kXfWU5GZxBV/Z8oSS1BPSq5r/urzoggwmUZZJ7tLFJl0Ad61jUT90eVD2ucu5xXjbZEaFWnXJMHdB+ID7yM7iww85 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Callers can pass this in for uncached folio creation, in which case if a folio is newly created it gets marked as uncached. If a folio exists for this index and lookup succeeds, then it will not get marked as uncached. If an !uncached lookup finds a cached folio, clear the flag. For that case, there are competeting uncached and cached users of the folio, and it should not get pruned. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 2 ++ mm/filemap.c | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index e49587c40157..374872acbe1d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -721,6 +721,7 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping, * * %FGP_NOFS - __GFP_FS will get cleared in gfp. * * %FGP_NOWAIT - Don't block on the folio lock. * * %FGP_STABLE - Wait for the folio to be stable (finished writeback) + * * %FGP_UNCACHED - Uncached buffered IO * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin() * implementation. */ @@ -734,6 +735,7 @@ typedef unsigned int __bitwise fgf_t; #define FGP_NOWAIT ((__force fgf_t)0x00000020) #define FGP_FOR_MMAP ((__force fgf_t)0x00000040) #define FGP_STABLE ((__force fgf_t)0x00000080) +#define FGP_UNCACHED ((__force fgf_t)0x00000100) #define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */ #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE) diff --git a/mm/filemap.c b/mm/filemap.c index 00f3c6c58629..a03a9b9127b8 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2001,6 +2001,8 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, /* Init accessed so avoid atomic mark_page_accessed later */ if (fgp_flags & FGP_ACCESSED) __folio_set_referenced(folio); + if (fgp_flags & FGP_UNCACHED) + __folio_set_uncached(folio); err = filemap_add_folio(mapping, folio, index, gfp); if (!err) @@ -2023,6 +2025,9 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, if (!folio) return ERR_PTR(-ENOENT); + /* not an uncached lookup, clear uncached if set */ + if (folio_test_uncached(folio) && !(fgp_flags & FGP_UNCACHED)) + folio_clear_uncached(folio); return folio; } EXPORT_SYMBOL(__filemap_get_folio);