From patchwork Thu Nov 14 15:25:05 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875192 Received: from mail-oo1-f44.google.com (mail-oo1-f44.google.com [209.85.161.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D99B215FD01 for ; Thu, 14 Nov 2024 15:28:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598085; cv=none; b=Wi4793F6A/wNttBM4r8f1KL/gnx3LPecsEMbjO5dD8x/qYNmfd6knXn3/XiDxXCCPKqfjYfpD4kVQVhOHnI0k8+c4EkhqVXvXN7sTVBu5DS/StDDFCGk6klBwOvyGmCfgIwXpLGSrUmq1HJ/0VywcdIg3C0e5wJDXVTEMr6t244= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598085; c=relaxed/simple; bh=8sfKwyActgC+UFgrCv8AbW1Vz+K4ykiNjF4V6I33E+c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=HS0CWPZVXv2oOkjQPV46pCo89gcqXcG8tGqy8n8cQ0x3lujq2RVgkOznqhGKt1O+gfaVm2TEfAXeadan4VotzLP/imS+1HOup90otl06GVe7LWlzJDVleSyVRE+xPSXOi2GYh7kd+FZqu/PoOTF86BxVwPZuN+EriSO4nLXttv8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=nvnVM+7l; arc=none smtp.client-ip=209.85.161.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="nvnVM+7l" Received: by mail-oo1-f44.google.com with SMTP id 006d021491bc7-5ebc9bda8c8so358609eaf.0 for ; Thu, 14 Nov 2024 07:28:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598083; x=1732202883; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=QzX2IvmAROGBANn6HEgih/p/Joy2MXPhit7EzvqeXZU=; b=nvnVM+7lRPjxgUobjcY29KJyrT1BRlFyRd1jjNUjZlCdoUW/FrYmPGYqQ2yhomt746 53/sbldwX6FjbV+u0r6hLjGtYQrlnKXLFBmIDDZjJQy5ywl7rRb8B6AkUIJQlO/7/N3v uNIcTAwj1jOJ9X9MR8Q0Y2afUODEuhc1Q2bElJxM0SiEQ/BJEErZc15QmDuGCcJTCfU+ fB1Vr39IPGPLJrYKtSr1bZFHZsIh7ff1m/ti5v4tJ9SdxwZEUk2LSWyx6WVHU3myK1Mb IwYw8yaazQ6AKYLAhJPhEUmyblI7oUUhlzYYfoH3mIn6Vp1R6OCtQhSzAtHnjW72EGx1 TO8w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598083; x=1732202883; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=QzX2IvmAROGBANn6HEgih/p/Joy2MXPhit7EzvqeXZU=; b=JBh17CvY5OSMAoTE21HxYvI2AQeBF8MZ9qSNvHPl8aYId0ZHf7RGkfLlkonDIqDBPo IYp3MAJFckPgYUOBLYyscEWvfYqncD+z3HC4y7uPTqWBYqqtonAbWqdLKmOZnurTYKMf X/FOTGXmoMTgM/ClNJ082CFkkHBVb4xvc2McmDqj/yKqJksYIQ+Gwaavr++qtFWoDnbM via7wRXkaqZOykcAKrpk2VXcwrt6ZL/HyZBWaKqAo4EUISrSzH4U0P2DRWOcechYO/Mc +HD6ijerTYku9nZqYGMZC6GwQIEmFJ0jF7eZ0CHVq/tSH6S6uWA0rWZNKuI4MEk1IyMk ly4w== X-Forwarded-Encrypted: i=1; AJvYcCX8rMH6ZYOPrkPDIPJpsILVqWQ7O4LRrfqULuFkQJBhM9+0Ci/5IM8qKgbsONX/5tzw7RIURzsVM5o3iVQA@vger.kernel.org X-Gm-Message-State: AOJu0YynSPFIcz0wVlKqRMCBEBiIOP4qLG7+gQ0MQdDsVfNe54gYT8sZ DQLZPJK5d7GBNXH5k2PvcjyKLmI+Quuw/QWduezoNCOqLZxwOAaqbWt4fRQC0zg= X-Google-Smtp-Source: AGHT+IFRnE0ztQRJK93idhhgrf7Bc5KUMDNOKjdutZVsJeSEeh/0aaiCxUu41HRdHQkrlLeWeSTtMA== X-Received: by 2002:a05:6820:8cc:b0:5eb:6a67:6255 with SMTP id 006d021491bc7-5ee9ec3bae1mr2680975eaf.1.1731598082894; Thu, 14 Nov 2024 07:28:02 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.01 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:02 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 01/17] mm/filemap: change filemap_create_folio() to take a struct kiocb Date: Thu, 14 Nov 2024 08:25:05 -0700 Message-ID: <20241114152743.2381672-3-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Rather than pass in both the file and position directly from the kiocb, just take a struct kiocb instead. While doing so, move the ki_flags checking into filemap_create_folio() as well. In preparation for actually needing the kiocb in the function. No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/filemap.c | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 56fa431c52af..91974308e9bf 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2460,15 +2460,17 @@ static int filemap_update_page(struct kiocb *iocb, return error; } -static int filemap_create_folio(struct file *file, - struct address_space *mapping, loff_t pos, - struct folio_batch *fbatch) +static int filemap_create_folio(struct kiocb *iocb, + struct address_space *mapping, struct folio_batch *fbatch) { struct folio *folio; int error; unsigned int min_order = mapping_min_folio_order(mapping); pgoff_t index; + if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) + return -EAGAIN; + folio = filemap_alloc_folio(mapping_gfp_mask(mapping), min_order); if (!folio) return -ENOMEM; @@ -2487,7 +2489,7 @@ static int filemap_create_folio(struct file *file, * well to keep locking rules simple. */ filemap_invalidate_lock_shared(mapping); - index = (pos >> (PAGE_SHIFT + min_order)) << min_order; + index = (iocb->ki_pos >> (PAGE_SHIFT + min_order)) << min_order; error = filemap_add_folio(mapping, folio, index, mapping_gfp_constraint(mapping, GFP_KERNEL)); if (error == -EEXIST) @@ -2495,7 +2497,8 @@ static int filemap_create_folio(struct file *file, if (error) goto error; - error = filemap_read_folio(file, mapping->a_ops->read_folio, folio); + error = filemap_read_folio(iocb->ki_filp, mapping->a_ops->read_folio, + folio); if (error) goto error; @@ -2551,9 +2554,7 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, filemap_get_read_batch(mapping, index, last_index - 1, fbatch); } if (!folio_batch_count(fbatch)) { - if (iocb->ki_flags & (IOCB_NOWAIT | IOCB_WAITQ)) - return -EAGAIN; - err = filemap_create_folio(filp, mapping, iocb->ki_pos, fbatch); + err = filemap_create_folio(iocb, mapping, fbatch); if (err == AOP_TRUNCATED_PAGE) goto retry; return err; From patchwork Thu Nov 14 15:25:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875193 Received: from mail-oa1-f44.google.com (mail-oa1-f44.google.com [209.85.160.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 273121779AE for ; Thu, 14 Nov 2024 15:28:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598087; cv=none; b=StYtOD6g2ZRyJsfQ4WJFbmk8SGdLaVuXbqjvMl8DY44y8Lt8l2o8iCIv8D3t175gQpVnLBxendzZFcYn5aMEcUrlBhWx8FL18ceUKkPSXEzWKmGtXTNSSEs6n8hnKn16UlxNrPNF5TjO/Z0EB/j9RbPS4Zlm9trQugAlQNS6AH0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598087; c=relaxed/simple; bh=wdE7j3SXsvf8mZ3WLeO9bDbqT1xBl9uoI5Ta1MifPnA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=POXPw8gE8M5qza2KAxYGcbIhGtusosCUnNm0nije6IupbLKWvPJzfhwF1Ylb37blVxUkIWpXuEO55bQa25CKtYmCa30exmVdkeswwLaJKUeMpLyDIgr14VuiglyDwtc+yW9pegUTt9YYMnqAjGUVjYf9bRdxqXznP5jtzoQMn1c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=WozBdVq6; arc=none smtp.client-ip=209.85.160.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="WozBdVq6" Received: by mail-oa1-f44.google.com with SMTP id 586e51a60fabf-28896d9d9deso297151fac.2 for ; Thu, 14 Nov 2024 07:28:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598084; x=1732202884; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=G7YIma1TSdS4GVKncfhre22V8DGIHalzmKUoqzgwI0E=; b=WozBdVq6+ts64ISVrUD0htZzkJRFsQUGcoy0p5TvGGvhIiOLyPBqdmNS+k2KHBZMO/ Hj9faRp/lVI0Rw6prHQam9qjedB5YN0Sfcv9tVELjWjt/D5TWHZBYf1vkmszMLaS9zTY MvR8g59YCwEU9EPMxOKfY47jRGapmfGsMeVz32H1rPtXP5MxzatQgQ0YGCMvWqZKYxCS 8YbTAGFJgAOlW7qaRFmWYGRMTlvQIEzCYB3iKGSbNqAM+V52Oar5wGGxtaadoriFkYqj vfnC+pgc2735Q5P2/7L2ShYY2bdsVZRFwpeSJMo1t1APojsbG1wG+NFlWYFen0B9YIZ3 T8Cg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598084; x=1732202884; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G7YIma1TSdS4GVKncfhre22V8DGIHalzmKUoqzgwI0E=; b=NqJbsnNs8CK6RirMz8CaNwFeQ0AbdOmObIIWnfiLN79mdNif/X6m13WvGxXW7WiZ5V 9SPygP9MaGGfQ1Brkrte0SLKoJvgCIbVrPj5rYLgEESw/dG0u+M/oKDVO2NBfQZeHUlE +NS140Nw0l0dSGu/0tnGfABfbw1U2vIlYUh90pVTN0ND5nHPSe7yjQOIY+9uKn3dalyJ aO64dFRX8XiArZDtTKziVQSPb52c7mxrcJTg2HejN65jGHGf7riyzo/zY/VXJth0Qxdm 0jX54UmOrQrvAErACWQrrmBXCmMuKL2f8gT3AJxAmSLUT7R8Mm1JPrSq4iHOFdW5Ewyz pACg== X-Forwarded-Encrypted: i=1; AJvYcCXgBRDkj2qL0YffojKdRXjil66r9z0ivTDf6N0I2I7Tktof8wIG5aj9UxUZR5pCu32lfx5E/txxm4mZ/Xph@vger.kernel.org X-Gm-Message-State: AOJu0Yw7pD5bhi2wPtCiqA0NjNjWHjpUbJbgv4X4KvXueHyEvBVw+uW8 wdps9vvex5LXORiKj3A+xd754AbD0RtqlVbTahjjClcUyqM3XhcLs0NZkh5LX4I= X-Google-Smtp-Source: AGHT+IH6odaDwIdA82zwSIWff/OaVLT9OGNonkILORb6hdBpcBvYTtZBZw+TFEiCAtvLZtie7sKouQ== X-Received: by 2002:a05:6870:5d8b:b0:277:e512:f27a with SMTP id 586e51a60fabf-295ccfd9242mr11462005fac.16.1731598084309; Thu, 14 Nov 2024 07:28:04 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:03 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 02/17] mm/readahead: add folio allocation helper Date: Thu, 14 Nov 2024 08:25:06 -0700 Message-ID: <20241114152743.2381672-4-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Just a wrapper around filemap_alloc_folio() for now, but add it in preparation for modifying the folio based on the 'ractl' being passed in. No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/readahead.c | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/mm/readahead.c b/mm/readahead.c index 3dc6c7a128dd..003cfe79880d 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -188,6 +188,12 @@ static void read_pages(struct readahead_control *rac) BUG_ON(readahead_count(rac)); } +static struct folio *ractl_alloc_folio(struct readahead_control *ractl, + gfp_t gfp_mask, unsigned int order) +{ + return filemap_alloc_folio(gfp_mask, order); +} + /** * page_cache_ra_unbounded - Start unchecked readahead. * @ractl: Readahead control. @@ -260,8 +266,8 @@ void page_cache_ra_unbounded(struct readahead_control *ractl, continue; } - folio = filemap_alloc_folio(gfp_mask, - mapping_min_folio_order(mapping)); + folio = ractl_alloc_folio(ractl, gfp_mask, + mapping_min_folio_order(mapping)); if (!folio) break; @@ -431,7 +437,7 @@ static inline int ra_alloc_folio(struct readahead_control *ractl, pgoff_t index, pgoff_t mark, unsigned int order, gfp_t gfp) { int err; - struct folio *folio = filemap_alloc_folio(gfp, order); + struct folio *folio = ractl_alloc_folio(ractl, gfp, order); if (!folio) return -ENOMEM; @@ -753,7 +759,7 @@ void readahead_expand(struct readahead_control *ractl, if (folio && !xa_is_value(folio)) return; /* Folio apparently present */ - folio = filemap_alloc_folio(gfp_mask, min_order); + folio = ractl_alloc_folio(ractl, gfp_mask, min_order); if (!folio) return; @@ -782,7 +788,7 @@ void readahead_expand(struct readahead_control *ractl, if (folio && !xa_is_value(folio)) return; /* Folio apparently present */ - folio = filemap_alloc_folio(gfp_mask, min_order); + folio = ractl_alloc_folio(ractl, gfp_mask, min_order); if (!folio) return; From patchwork Thu Nov 14 15:25:07 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875194 Received: from mail-oo1-f45.google.com (mail-oo1-f45.google.com [209.85.161.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A55F6183098 for ; Thu, 14 Nov 2024 15:28:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598088; cv=none; b=A6GhQmX5tUjmnJ5LO3x7mS9jyKIrg/llOU0lN4jGbFqCAUKEgT2pUjG+IHuQ0Eez8nFzqAhau5+4R+I/KK6gZBpiOWFxT8YL4UlMDLSWFMT4ijXp/4jdD7KueRMx+MPvNzRjGCgSOoIIZdyaDxjmJbtQ1/N1xelj8yhZH2By+P0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598088; c=relaxed/simple; bh=klEANlDo0QcivFe1NIYDNiOFa1lRCa2wf9JjLv/2My8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=PKsefGbFy3sxyWmhLpvoO/5hQVsjTCy8CL5kz81NCDuLyiEPjRxKY1/I5a+40kJBOoHaghtq7VK6BfdN+RpQdj1gFw+G3RJzcjUkm9pI9FBSJalO2nRE6cXZGkYClCtqJCMrA/rhmvtBbJ4jEvfSuWbcMcLUgN7LQaHXG17MJlQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=KoxCnn/F; arc=none smtp.client-ip=209.85.161.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="KoxCnn/F" Received: by mail-oo1-f45.google.com with SMTP id 006d021491bc7-5ebc05007daso325397eaf.1 for ; Thu, 14 Nov 2024 07:28:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598086; x=1732202886; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=9QDQ8u+XOjRX6rdX1KxWEpFxXqQ9HPCp5iTXnKjwIis=; b=KoxCnn/FfZycku2PMA2bkOnRGL+7wMjFuUNW399IDi7AkgoaONHvIcoqxIm4Mc618f n+G2NlDxEQyTBhHyTF1Q7icxPcS2tk6UMfe1IJ5BoLwNdx8zT+6d2TvipJAMQ91PT2by 5PmpYJyfO790CzxMQMtRgHw0GKGF90BzFMMuLRj55vzXkeaXrCSz3NBUbHcIp6AMikoC wVcwvVgAfmZV/vOeQx+NFcy1y3JIBsqKElVuysSQV3shkBRrrmAGt3+SiD8d45XaAIAJ 5xLmUmtMTlQWn8fnHj4rR4HIUCF4yCCKqzxBUy9N42SWgTb+nWXuaFowTV5FelV/QNKL 6Stg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598086; x=1732202886; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=9QDQ8u+XOjRX6rdX1KxWEpFxXqQ9HPCp5iTXnKjwIis=; b=bXg5mYcU3JZv8CT26cDNtow8QCV02WnMAOgJiHtFAn6/sO/D4UEUbJ/f3uti80HCBA TvLmxohP6d7RBkbog189czPntknC00vv5YSa2g+gnCAMQYWfg36T/Onjq34r76gLIrxL ErKQcH9diD2DwqUp/j01ezo2gBjke0jxfLeSW93pqjblk9trJssqZNYX+4xsVlP7He0V bsjtvb+uCDFV0lpJETVQnSDqa9AZL5CtMdS322npBFcMGZRPBS1vxTkK8JMXu5lqDRTK WKaVocm/Qbztm5MDQHyQYfDd15WBZY8aOHcDn4sowpkw9EaBcFoj+e6vifPAT1d2jqQI njeQ== X-Forwarded-Encrypted: i=1; AJvYcCW0v+BLv9pxVM3fr4HyCriNbCuGiZVnowyZHL/WeLTYAcTKRK18Lcdc27MClpB8u5N9bWZ4hnQoA/AzofZS@vger.kernel.org X-Gm-Message-State: AOJu0YzrhqKlL+SV3ueO6XEy/gfIwe8OHsSul+i9KLWQ2DwYjMrVYH/Z PHrX3S+D2SXQO2aSiExUFmDrHMLzIzq8NUT/e40LgHS2Pi5cqbrQnW0bb8bPL0c= X-Google-Smtp-Source: AGHT+IHYfzNU9jsYZVqq6BubIUmPINzkrBJpOzUKZ5nbA/vQYGCWpQqOD4e90GxHDczr5rmeIPTulQ== X-Received: by 2002:a05:6820:3088:b0:5e5:7086:ebe8 with SMTP id 006d021491bc7-5ee91feb02cmr6995384eaf.0.1731598085665; Thu, 14 Nov 2024 07:28:05 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:05 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 03/17] mm: add PG_uncached page flag Date: Thu, 14 Nov 2024 08:25:07 -0700 Message-ID: <20241114152743.2381672-5-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a page flag that file IO can use to indicate that the IO being done is uncached, as in it should not persist in the page cache after the IO has been completed. Signed-off-by: Jens Axboe --- include/linux/page-flags.h | 5 +++++ include/trace/events/mmflags.h | 3 ++- 2 files changed, 7 insertions(+), 1 deletion(-) diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h index cc839e4365c1..3c4003495929 100644 --- a/include/linux/page-flags.h +++ b/include/linux/page-flags.h @@ -110,6 +110,7 @@ enum pageflags { PG_reclaim, /* To be reclaimed asap */ PG_swapbacked, /* Page is backed by RAM/swap */ PG_unevictable, /* Page is "unevictable" */ + PG_uncached, /* uncached read/write IO */ #ifdef CONFIG_MMU PG_mlocked, /* Page is vma mlocked */ #endif @@ -562,6 +563,10 @@ PAGEFLAG(Reclaim, reclaim, PF_NO_TAIL) FOLIO_FLAG(readahead, FOLIO_HEAD_PAGE) FOLIO_TEST_CLEAR_FLAG(readahead, FOLIO_HEAD_PAGE) +FOLIO_FLAG(uncached, FOLIO_HEAD_PAGE) + FOLIO_TEST_CLEAR_FLAG(uncached, FOLIO_HEAD_PAGE) + __FOLIO_SET_FLAG(uncached, FOLIO_HEAD_PAGE) + #ifdef CONFIG_HIGHMEM /* * Must use a macro here due to header dependency issues. page_zone() is not diff --git a/include/trace/events/mmflags.h b/include/trace/events/mmflags.h index bb8a59c6caa2..b60057284102 100644 --- a/include/trace/events/mmflags.h +++ b/include/trace/events/mmflags.h @@ -116,7 +116,8 @@ DEF_PAGEFLAG_NAME(head), \ DEF_PAGEFLAG_NAME(reclaim), \ DEF_PAGEFLAG_NAME(swapbacked), \ - DEF_PAGEFLAG_NAME(unevictable) \ + DEF_PAGEFLAG_NAME(unevictable), \ + DEF_PAGEFLAG_NAME(uncached) \ IF_HAVE_PG_MLOCK(mlocked) \ IF_HAVE_PG_HWPOISON(hwpoison) \ IF_HAVE_PG_IDLE(idle) \ From patchwork Thu Nov 14 15:25:08 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875195 Received: from mail-oo1-f44.google.com (mail-oo1-f44.google.com [209.85.161.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 25E7C188587 for ; Thu, 14 Nov 2024 15:28:07 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.44 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598090; cv=none; b=kWrDIF+GyEBTrQxR+l64A5LF+dcSLdwYPL5Jq6HbvXU2Ni+7O9LWpTwqByCEEQvnDAT+Sr9mtD6HfVM0va8I7yiGwaSxX0UCj41eajM85C8ReKjiyNKjfzMiyUo5F1kfOJWk7qdxbsTdndQxxafWWgCzNY7ckuUFwET5T8Z+K4o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598090; c=relaxed/simple; bh=AFLrZl3fH+/hMhwZTjgZ0OWM0ndgzDi220n1zUDQffQ=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=I/HmA3rYuZKGzYFz3ND+oSN+N/J1MrYBPUxtbOx7ZmlGq9U6BbhTSRzWxJIazBOTOMEsh2ABn7KLJIrd1FsbDNZ39+4d90uZ0zhVAjz8a884h8A0XHqSv96xSGxNfTDxB1kYOJaoTHBRMWT5A1V4WKkpj0eOS8WEncd1yCv6d3I= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=arnRiepF; arc=none smtp.client-ip=209.85.161.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="arnRiepF" Received: by mail-oo1-f44.google.com with SMTP id 006d021491bc7-5ebc5b4190eso366714eaf.3 for ; Thu, 14 Nov 2024 07:28:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598087; x=1732202887; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jvfAlDie45/dO/q2Oa4nMUlSYRbUQddnQckYeJvFoiA=; b=arnRiepFOtCGKagziS6xtO4Vv/Jf+8LAVE0J8ryI70wI6+zLwxYTiedTsJ537HC4a2 LNlkH6zA2veKrY6gC1AV/soymGZv4dcjTFrRd401ey51Z2ZRNhbCWz1MrRG2VaHDZzZU CtXa6Q7kxE3rM7/IfGcF6ekYH/Sf/DSYhadei8uNuwK+yUUDnHirrUBAfcWJ607o30su d6FeQ+b3xv7erZkoT8Tz+ZkoIVPCu/Hde48Xczj9XrRav8FokUakSv8Xe5BFKkhjJUpF y0R/VAeybRFMVsO0vmqjYr9OX8GJ++AAgoqqGbbxdip/H442BYrrkZHwQK0MuurJgjD/ r0jA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598087; x=1732202887; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jvfAlDie45/dO/q2Oa4nMUlSYRbUQddnQckYeJvFoiA=; b=faIjs5h/+nhQvoxnxDDI1Tm1h3e59XqtkSF4o7DEiy6A1navNcqlUTewuQDYBS7NTL qle06/XqGaXlW8uvCH10WFGkyfOneKmJGsYZMd1bJKdlYY9DLfmw2Een2QB7LvQVe/9s K1Mq4w1Tal1tXSfAMFG1wZIrt6oNnN9g53/ocTDrEYk+zarDkDomIg5wCP5groQE0Ila ldDvaoQDRKgCreu7mvOfC6RdI52TmQmcbOeYwPppM4dODu4EFIFrzWXpR9dhiF0NTh9t nwhYS1QOkgpW7uDbrNwKbqiWrHoGbSkm0NKl5ZgIvfc36dBWXclI5uN5ws9oOtAK9HJN HTXQ== X-Forwarded-Encrypted: i=1; AJvYcCWyS9FgwqXTRtcThqPeflR9fGMfTOSZAKZs1JYMtFaeKu3Vdm/UiWZuDPRlJIsJCCTlAYXV7x9/LbrDArtO@vger.kernel.org X-Gm-Message-State: AOJu0YyRHb39fTIP7VdNjIwHzRE1iW8QAXsdqPgwSATk/cRRESlqGTiB aMNuI02f4JsQhaDT/CPiA5tyuZE9AO4WYhYIfHlAtRFJJ1+W6q1fFvbAXUxRB9U= X-Google-Smtp-Source: AGHT+IH7eepGaxoKX7XFVQSCKcZ5tiiQfUtPjwDWCbNdm4JNlvZQy9y+41dOTUM8EVcPpq30RA+88A== X-Received: by 2002:a05:6820:278c:b0:5ee:710:83fa with SMTP id 006d021491bc7-5ee922a81aemr6235784eaf.8.1731598087197; Thu, 14 Nov 2024 07:28:07 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.05 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:06 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 04/17] mm/readahead: add readahead_control->uncached member Date: Thu, 14 Nov 2024 08:25:08 -0700 Message-ID: <20241114152743.2381672-6-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If ractl->uncached is set to true, then folios created are marked as uncached as well. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 1 + mm/readahead.c | 8 +++++++- 2 files changed, 8 insertions(+), 1 deletion(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 68a5f1ff3301..8afacb7520d4 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -1350,6 +1350,7 @@ struct readahead_control { pgoff_t _index; unsigned int _nr_pages; unsigned int _batch_count; + bool uncached; bool _workingset; unsigned long _pflags; }; diff --git a/mm/readahead.c b/mm/readahead.c index 003cfe79880d..8dbeab9bc1f0 100644 --- a/mm/readahead.c +++ b/mm/readahead.c @@ -191,7 +191,13 @@ static void read_pages(struct readahead_control *rac) static struct folio *ractl_alloc_folio(struct readahead_control *ractl, gfp_t gfp_mask, unsigned int order) { - return filemap_alloc_folio(gfp_mask, order); + struct folio *folio; + + folio = filemap_alloc_folio(gfp_mask, order); + if (folio && ractl->uncached) + __folio_set_uncached(folio); + + return folio; } /** From patchwork Thu Nov 14 15:25:09 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875196 Received: from mail-oa1-f49.google.com (mail-oa1-f49.google.com [209.85.160.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 93C9418BBB4 for ; Thu, 14 Nov 2024 15:28:09 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598091; cv=none; b=Ap2Gu7voh02mo+vMXEafMFM8c5rjsiBoHKydT2oWbkt7hbcGHQ0aOVhsI4w2bLsUuVeB7MILLeBh8o0mnL8/DapYI42XZMaqfePD3dUPatpwyN5Wf15P4iTrckNrEmob/TQABx/gP2KJ2g0QgjHN7oFOfphn1I+/sFx/tagBjqc= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598091; c=relaxed/simple; bh=HNesIF4+NCKJRgMS8RJOzPdBmK2CQSC6p10zaqTN28c=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=lfhercN8anO47inWcSVVeejXYrrAJ4QcDsdv5UWCswJucGy9HexB52DrVIsLXqDc34jfyQ3VT7pH8NnPXo1vwqle3DTRGTAGUKzgbmqlxjrVJ4WvJ+93cqrN9slNIofbwWR49VxFIKbkDjDQPDuxA7Q/ej54iQTQOfvpd9pNLAM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=rYyOesV0; arc=none smtp.client-ip=209.85.160.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="rYyOesV0" Received: by mail-oa1-f49.google.com with SMTP id 586e51a60fabf-2951f3af3ceso447800fac.1 for ; Thu, 14 Nov 2024 07:28:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598088; x=1732202888; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=573A7VIJaN7Phef7EPro6/84rHuXaF2Ar2GUBUbETSc=; b=rYyOesV0QfwRWVm1PBYChCs1Y4zCFiNHzH9R1LQaPnj2I8tT1iPTBnN31qKM66drKa ztxK7fRNyFXXLM/ckR1TTL3eSG9kYJpCrCGc51hfexhpkcnHvLwcZutL3r1GsGUzsaOH dv+7AHgPXl1JMdeWYXi1svCXaQ5X6FSFMOiGdy5M0+ymXDb6scAwHTySb9Xr9WUFmcNC BESPCPmyvEuqPgrLOhMtZSKuLRYN0js4dOgSguoXXFLublShL5XNWGbjzgx9Xn58vGUI OI7bAdPajtubR/yBEm6NRIkX7iaRt1+zNKye4zI5q+6N/m2lbdzvOJ4IlOP3mrbR6B5V x/Vg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598088; x=1732202888; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=573A7VIJaN7Phef7EPro6/84rHuXaF2Ar2GUBUbETSc=; b=WVgkRX07pA7+hdZdGac76rDRoSoA5HRs9IYYJiRiRaFJ7b9wN6CWk8E452MZOnkRc3 IoMDsbZA8WyYqjGxIU1t6It0YAtnYX/5fwq1WidKp6cy2FAThGHVd9h+0S6vDFlPbWuQ ZJbpq6KIO23b9/f4++rP8kBl+EIwVYlVKADr7s/ggNvGUrladYwBDe9h4QqrADOZz/NZ 1yvGRI3YO7lbUyklaA3zev6RfeWYq1cySzZAAtVQvwkOgggB2eOLLWraT2+2ck4usNaF wc6pbD46t4fW0FlyTaJmWjflgoCV7hrQ4HTYcGEBE1AmqMGBcA6rXVaeepkZ2xTRbT+d NQEg== X-Forwarded-Encrypted: i=1; AJvYcCUbjTH9PrhvvJ23ZXFXEktO5do8PN9vG1XRSwBgbPcn4NtYJ36NrWJAgrdAI0zVWgSk40YTuqm9t9Rc5M+Y@vger.kernel.org X-Gm-Message-State: AOJu0Yw0EmiV2h+GmTkZO243D2IiiVowaSOM5qP2g7Y7TgEfwEQFyDN+ Onmg4rh38QjeQ1DbHz8jNKlvz8ptlUd4SJYyWoBotL//wdbOOJT4hnOAGMHOGCM= X-Google-Smtp-Source: AGHT+IGRJhbG3Iz8YyLsc6/0RWtRhZhYbcSZVTwigVtxhi/CGcrJEwq+4Kef8lO3NDoSjpzlANfJNQ== X-Received: by 2002:a05:6870:5312:b0:286:f24f:c232 with SMTP id 586e51a60fabf-29610696786mr2665053fac.42.1731598088623; Thu, 14 Nov 2024 07:28:08 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:08 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 05/17] mm/filemap: use page_cache_sync_ra() to kick off read-ahead Date: Thu, 14 Nov 2024 08:25:09 -0700 Message-ID: <20241114152743.2381672-7-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Rather than use the page_cache_sync_readahead() helper, define our own ractl and use page_cache_sync_ra() directly. In preparation for needing to modify ractl inside filemap_get_pages(). No functional changes in this patch. Signed-off-by: Jens Axboe --- mm/filemap.c | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 91974308e9bf..02d9cb585195 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2528,7 +2528,6 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, { struct file *filp = iocb->ki_filp; struct address_space *mapping = filp->f_mapping; - struct file_ra_state *ra = &filp->f_ra; pgoff_t index = iocb->ki_pos >> PAGE_SHIFT; pgoff_t last_index; struct folio *folio; @@ -2543,12 +2542,13 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, filemap_get_read_batch(mapping, index, last_index - 1, fbatch); if (!folio_batch_count(fbatch)) { + DEFINE_READAHEAD(ractl, filp, &filp->f_ra, mapping, index); + if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; if (iocb->ki_flags & IOCB_NOWAIT) flags = memalloc_noio_save(); - page_cache_sync_readahead(mapping, ra, filp, index, - last_index - index); + page_cache_sync_ra(&ractl, last_index - index); if (iocb->ki_flags & IOCB_NOWAIT) memalloc_noio_restore(flags); filemap_get_read_batch(mapping, index, last_index - 1, fbatch); From patchwork Thu Nov 14 15:25:10 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875197 Received: from mail-oa1-f48.google.com (mail-oa1-f48.google.com [209.85.160.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id F209818CC02 for ; Thu, 14 Nov 2024 15:28:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598093; cv=none; b=NCzWjWsUB5GuQ9NTH8bS+3/30ZX1EQypVfuGKthDZl+hBYWGxb8x1cjfHzILLwHqR6F8cA9TVI+nzAig9B7Z3Emh4PinwLtZddrmb1sElzSUMRpZk6aT/EiXXIsqjDkvsH2/puCv5KXcuLPsRKKFHHFgtXcJ7KairZYC5K0kG8c= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598093; c=relaxed/simple; bh=LKB5qmpxodZQRc7Oxubcf3TF1jie3lO8rGiyoHOU3+w=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=R8tnZroVtVNptU+/E9jYiJDpTKkwldRoxpXP4+Ym8B62/kibHBqf7gnyB9xwZlRrXibL1WRgZIUpgQIS73UOIphG8LYWwu2aBHryEVz0niOWSXsWFlFpO5ukdfKYPfGUjE091Pnvyq5eTajIbN7wd8GLWlkV1lTwkVXbhrSvi84= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=08YR1BRR; arc=none smtp.client-ip=209.85.160.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="08YR1BRR" Received: by mail-oa1-f48.google.com with SMTP id 586e51a60fabf-2951f3af3ceso447821fac.1 for ; Thu, 14 Nov 2024 07:28:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598090; x=1732202890; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zZIb5wvnGXtO89gOBSt/Tk4W0KJQpsBnaIBuXsS72Rg=; b=08YR1BRR7yCBt1JJYbgVC0Xty4xFN6eia0wi6XcpXntnDr49Q62c4zon5Qm9DAHzkp GH5nGsFMV2/o/Dr5EmvrtbleFFHcyNlcQnOeiMC0dPGIaaU/QBVGVnH4e/oORhDX2Tgj ul/UCZsvJ4OZcnROWf62ZTUetn58AWeDaW3PV2/xKkdnGzLzusJKs8Pvj8pFh3ZtcQWh oWsLks3kGL4HzhIc11x0p+JzK+rLkHwotvXwYQtJm4rA1RcgKAqAJbx9xV7poC3gpIhx oLcQk0py4rLNRStRwPHETNnGMECSRL5KP0D+Rz2q8Xsmvt8x1+2pOVkGEsp3eVV0m+Xm h1SA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598090; x=1732202890; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zZIb5wvnGXtO89gOBSt/Tk4W0KJQpsBnaIBuXsS72Rg=; b=V/hGYQg2+T/t6csr9/qP25sH9UFU93upnoT2rgw5TTkN/YlCRSP0ozWi3o6L1SGi98 FogZfAhiuS5F+gWFdyG9LWy7oEtgnUjRTh/bdLnxBiezRhiBPjmx7lOHdY4uudQqxOyi GRmUI5dCodDlIwkJ9gdoajxJ1h5MNmFa5gxvRoesZeAKPGnrp1Df+5yx/4p/xSXw6bQ1 ID8FQfBo9GJqkn+jOcCyd34jzoA1XIh93uXzE6jKV7aOZGvT//LTQMEhb1r8+pD+J/3S h0PoH3Fr5SrwFV2A079ZweDsot5jfC2rTXd1Im4Bz5lZALOBy5DSh+Le4f/1K3FygA9/ 52iQ== X-Forwarded-Encrypted: i=1; AJvYcCU2uiuzJ4+HZcWMIXrAU9RQ5rwj5o+Ka0zFcnAIwzs1zA83S/9X2nszsefsDkxyDJt/jT2pcH5FMoIhBbiL@vger.kernel.org X-Gm-Message-State: AOJu0YzmZTkwYrEvXQDwVC2ESmBsqu9RV2Y202M2rkT09bZGHBUM8zZg YiTpD2p9HWTw7UqgWS/0mNNPqv9kg1iKKMwwsxSZ571M72euV8crEJQX7kg7KgA= X-Google-Smtp-Source: AGHT+IF0BwKZuS0hD2QtJaajxUzrtQ2SwlZch/PHXd4jMsGrqLjoDD1qWu/LCzOrVtrsjrS2cDAn1Q== X-Received: by 2002:a05:6870:910d:b0:277:c027:1960 with SMTP id 586e51a60fabf-29610376b59mr2544315fac.25.1731598090064; Thu, 14 Nov 2024 07:28:10 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:09 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 06/17] mm/truncate: add folio_unmap_invalidate() helper Date: Thu, 14 Nov 2024 08:25:10 -0700 Message-ID: <20241114152743.2381672-8-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add a folio_unmap_invalidate() helper, which unmaps and invalidates a given folio. The caller must already have locked the folio. Use this new helper in invalidate_inode_pages2_range(), rather than duplicate the code there. In preparation for using this elsewhere as well, have it take a gfp_t mask rather than assume GFP_KERNEL is the right choice. This bubbles back to invalidate_complete_folio2() as well. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 2 ++ mm/truncate.c | 35 ++++++++++++++++++++++------------- 2 files changed, 24 insertions(+), 13 deletions(-) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index 8afacb7520d4..d55bf995bd9e 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -34,6 +34,8 @@ int kiocb_invalidate_pages(struct kiocb *iocb, size_t count); void kiocb_invalidate_post_direct_write(struct kiocb *iocb, size_t count); int filemap_invalidate_pages(struct address_space *mapping, loff_t pos, loff_t end, bool nowait); +int folio_unmap_invalidate(struct address_space *mapping, struct folio *folio, + gfp_t gfp); int write_inode_now(struct inode *, int sync); int filemap_fdatawrite(struct address_space *); diff --git a/mm/truncate.c b/mm/truncate.c index 0668cd340a46..6ea16c537534 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -547,12 +547,12 @@ EXPORT_SYMBOL(invalidate_mapping_pages); * sitting in the folio_add_lru() caches. */ static int invalidate_complete_folio2(struct address_space *mapping, - struct folio *folio) + struct folio *folio, gfp_t gfp_mask) { if (folio->mapping != mapping) return 0; - if (!filemap_release_folio(folio, GFP_KERNEL)) + if (!filemap_release_folio(folio, gfp_mask)) return 0; spin_lock(&mapping->host->i_lock); @@ -584,6 +584,25 @@ static int folio_launder(struct address_space *mapping, struct folio *folio) return mapping->a_ops->launder_folio(folio); } +int folio_unmap_invalidate(struct address_space *mapping, struct folio *folio, + gfp_t gfp) +{ + int ret; + + VM_BUG_ON_FOLIO(!folio_test_locked(folio), folio); + + if (folio_test_dirty(folio)) + return 0; + if (folio_mapped(folio)) + unmap_mapping_folio(folio); + BUG_ON(folio_mapped(folio)); + + ret = folio_launder(mapping, folio); + if (!ret && !invalidate_complete_folio2(mapping, folio, gfp)) + return -EBUSY; + return ret; +} + /** * invalidate_inode_pages2_range - remove range of pages from an address_space * @mapping: the address_space @@ -641,18 +660,8 @@ int invalidate_inode_pages2_range(struct address_space *mapping, folio_unlock(folio); continue; } - VM_BUG_ON_FOLIO(!folio_contains(folio, indices[i]), folio); folio_wait_writeback(folio); - - if (folio_mapped(folio)) - unmap_mapping_folio(folio); - BUG_ON(folio_mapped(folio)); - - ret2 = folio_launder(mapping, folio); - if (ret2 == 0) { - if (!invalidate_complete_folio2(mapping, folio)) - ret2 = -EBUSY; - } + ret2 = folio_unmap_invalidate(mapping, folio, GFP_KERNEL); if (ret2 < 0) ret = ret2; folio_unlock(folio); From patchwork Thu Nov 14 15:25:11 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875198 Received: from mail-oo1-f45.google.com (mail-oo1-f45.google.com [209.85.161.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5870F18E372 for ; Thu, 14 Nov 2024 15:28:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.45 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598094; cv=none; b=I/Feiew8rUn8oEdSyeOTMUdXhHfXUph7aT6YIok0NygLicH+llLdD7WUn6X/JN5F2l4abVyyIrmbryI3OItvUG5ycaHcC8aXHn5pin18uEAHWOrTtq+eLqskV1uxhT7NsJoPzK1z6hO3VsPIq15TaiQHntcfb2gu+Joqum73Yig= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598094; c=relaxed/simple; bh=gGKnnveMfZPAg4SgJ2FOiR86boymhVofVZdPadUM3k0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=THKASm2H00E3G0Og/BKvhtY6oskRX4nPTZiCyC0vTt6lNMZ2AqzvNtWaFh4dfIbLF5w/7MDZoeKLvdF1ls9Y+fXscNxarA9l51hgWceOCiSSXefxTptcG5N1UZ3lqc0QUZ4tConxxaL4dBZelGEQ7PmyMSZc096b6tz0R+ufj8c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=z2HqEI9Q; arc=none smtp.client-ip=209.85.161.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="z2HqEI9Q" Received: by mail-oo1-f45.google.com with SMTP id 006d021491bc7-5ec4a40e95bso321473eaf.1 for ; Thu, 14 Nov 2024 07:28:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598091; x=1732202891; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LyRvRhMBG1NsG1pjAu7RsorwHLATahyjt2W9R5OMw+k=; b=z2HqEI9QexiHndBsoUg/10+CpOQf0QjWppiL2AbY+qCvjAF2sEkPM+BxOECc8uUJYP ITB5KEBOakM51jI3hhi+wgtuKCBLO8unSpeLemgdjYbeP9omGt4OXmThSfGFTNzmff+m ySlzVp+v6Cuo65VBbCrNUN7HtWHIZlKpXuHVlrVuSfxmCIZNlztbNMklhaKpOBb49BcK gN8LuzXTsLq82DfHYF3q9nj7SKsBIWYEQk69vqjf1l3tF1khWXVCeNPHEOa6PYU+zwDD uUxT8igok79gqdh0Cjd3rwD8PzXyWGy8D0KOBKjUrMW7GQcn79aTvGRPJBZ+JJ2pxQn5 plhg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598091; x=1732202891; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LyRvRhMBG1NsG1pjAu7RsorwHLATahyjt2W9R5OMw+k=; b=AkpjgvWDkHFfPb/FpflILGlfIZHg094ABm37AHx90819uLm3iIoJZ2AFS6TdtwMYan nWcWparsXkOxuhQAK1FY5tcRuHUCMJciC8B7dGbUUosniOomn8N5aCI3BT1+0oQ11+qP JyanMFI+X7AiP/akbMkqXQR63Be+Ftr/B3Ta+N+zlTr8RB55RaRLWlvWD9rCGvgH8G5v Rh2ghqhVN9imNAUGpLTi0bn8mM7iP+js1SwPs3QDgFJZvxp4zdOJ/chWvkzKkiuR5dk+ rIIsrlcvygP1Em1wsgfi/F4VZRteR16A1IbAvs//scqPLsi/f9SWy8TPofUzRtuaGnP2 m7Mw== X-Forwarded-Encrypted: i=1; AJvYcCU2hDM/KExBg6uKFz6YZeQDct6ne6P5Uh6puhZD+zhBicCJ65xVPpDNF/HXuWcTJDiJR58WUjJ2uNIhYUHD@vger.kernel.org X-Gm-Message-State: AOJu0YzYE3KBFNfDXt18kharW4Yb71jgXEGiZmyCQ/SghH1tNlrLaR52 FHoBTt3ZwKBuqjaycD5Ln4iRphL45/WWBcbCfU7Z/kZulMmYvP2dAGhSy3Lq8M4= X-Google-Smtp-Source: AGHT+IHQ/GenKedH6rwQd+su0Ph/MRWIuzytWarVEOsqT+imV9hIASvSe13ZVKnr0H0iVbl20zcXwQ== X-Received: by 2002:a05:6820:4d06:b0:5ba:ec8b:44b5 with SMTP id 006d021491bc7-5ee57c4397bmr19027877eaf.3.1731598091501; Thu, 14 Nov 2024 07:28:11 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:10 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 07/17] fs: add RWF_UNCACHED iocb and FOP_UNCACHED file_operations flag Date: Thu, 14 Nov 2024 08:25:11 -0700 Message-ID: <20241114152743.2381672-9-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If a file system supports uncached buffered IO, it may set FOP_UNCACHED and enable RWF_UNCACHED. If RWF_UNCACHED is attempted without the file system supporting it, it'll get errored with -EOPNOTSUPP. Signed-off-by: Jens Axboe --- include/linux/fs.h | 14 +++++++++++++- include/uapi/linux/fs.h | 6 +++++- 2 files changed, 18 insertions(+), 2 deletions(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 3559446279c1..45510d0b8de0 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -320,6 +320,7 @@ struct readahead_control; #define IOCB_NOWAIT (__force int) RWF_NOWAIT #define IOCB_APPEND (__force int) RWF_APPEND #define IOCB_ATOMIC (__force int) RWF_ATOMIC +#define IOCB_UNCACHED (__force int) RWF_UNCACHED /* non-RWF related bits - start at 16 */ #define IOCB_EVENTFD (1 << 16) @@ -354,7 +355,8 @@ struct readahead_control; { IOCB_SYNC, "SYNC" }, \ { IOCB_NOWAIT, "NOWAIT" }, \ { IOCB_APPEND, "APPEND" }, \ - { IOCB_ATOMIC, "ATOMIC"}, \ + { IOCB_ATOMIC, "ATOMIC" }, \ + { IOCB_UNCACHED, "UNCACHED" }, \ { IOCB_EVENTFD, "EVENTFD"}, \ { IOCB_DIRECT, "DIRECT" }, \ { IOCB_WRITE, "WRITE" }, \ @@ -2116,6 +2118,8 @@ struct file_operations { #define FOP_HUGE_PAGES ((__force fop_flags_t)(1 << 4)) /* Treat loff_t as unsigned (e.g., /dev/mem) */ #define FOP_UNSIGNED_OFFSET ((__force fop_flags_t)(1 << 5)) +/* File system supports uncached read/write buffered IO */ +#define FOP_UNCACHED ((__force fop_flags_t)(1 << 6)) /* Wrap a directory iterator that needs exclusive inode access */ int wrap_directory_iterator(struct file *, struct dir_context *, @@ -3532,6 +3536,14 @@ static inline int kiocb_set_rw_flags(struct kiocb *ki, rwf_t flags, if (!(ki->ki_filp->f_mode & FMODE_CAN_ATOMIC_WRITE)) return -EOPNOTSUPP; } + if (flags & RWF_UNCACHED) { + /* file system must support it */ + if (!(ki->ki_filp->f_op->fop_flags & FOP_UNCACHED)) + return -EOPNOTSUPP; + /* DAX mappings not supported */ + if (IS_DAX(ki->ki_filp->f_mapping->host)) + return -EOPNOTSUPP; + } kiocb_flags |= (__force int) (flags & RWF_SUPPORTED); if (flags & RWF_SYNC) kiocb_flags |= IOCB_DSYNC; diff --git a/include/uapi/linux/fs.h b/include/uapi/linux/fs.h index 753971770733..dc77cd8ae1a3 100644 --- a/include/uapi/linux/fs.h +++ b/include/uapi/linux/fs.h @@ -332,9 +332,13 @@ typedef int __bitwise __kernel_rwf_t; /* Atomic Write */ #define RWF_ATOMIC ((__force __kernel_rwf_t)0x00000040) +/* buffered IO that drops the cache after reading or writing data */ +#define RWF_UNCACHED ((__force __kernel_rwf_t)0x00000080) + /* mask of flags supported by the kernel */ #define RWF_SUPPORTED (RWF_HIPRI | RWF_DSYNC | RWF_SYNC | RWF_NOWAIT |\ - RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC) + RWF_APPEND | RWF_NOAPPEND | RWF_ATOMIC |\ + RWF_UNCACHED) #define PROCFS_IOCTL_MAGIC 'f' From patchwork Thu Nov 14 15:25:12 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875199 Received: from mail-oo1-f48.google.com (mail-oo1-f48.google.com [209.85.161.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EAFDD19340E for ; Thu, 14 Nov 2024 15:28:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598095; cv=none; b=RNdFsaA4FMZbzrzEy/7UYj0IvqyTl8YaseJ2YFVtDmYPwJScOXkz6sDZFy85Rkcpu6LVl3/m1e9pTLEVsQcVxAW9Vth2eqNfr6U3FAp/T3vdL2YcTl5mKnVR/4iQQqd05BNdx1oNCnfhfwhiNyaDh9ky+Ms43VoTKFzpjvmnPZA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598095; c=relaxed/simple; bh=+LBI8Q/JaILudpg11KeqvJRORHez4jw0F0mptlAAtjg=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=hn7/za5P2ot1YSjJmslMyGYwQ02ugILnLR11hHbPFWYZsYzJQIRpbfMmR0UjoFACbg1KeM/YIeHSAL9IYEHpP897t4UOFN1n15ZytssnqYTxVC+fCwej8HZwCdmT4uFjANF3+tohUrsl3G22FHgHmR4kTjGJ6tKDou/wzZoOCnM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=qqHMLA/+; arc=none smtp.client-ip=209.85.161.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="qqHMLA/+" Received: by mail-oo1-f48.google.com with SMTP id 006d021491bc7-5ebc4e89240so309452eaf.1 for ; Thu, 14 Nov 2024 07:28:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598093; x=1732202893; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7749KjgMy6eatM2VSy2jowDIab8tvoATfJQmYOPl1pc=; b=qqHMLA/+W3MXqoe3ZgYdwBG52pjmWMZyuV8lwyc+PPBVuO95iwCtFK8QcXnHEAD7G4 BPOHvQNQDiuYxHHrey6bptacw0CysLM+9YltKZlw9rN/Q7sL1l7tzZqcaDOycr9ad9IG x+s7RUoPvXpmfRMUATWTgDCiqbxsUys1HKePsVFn02vGwPkXLT618oEbdgxmGKY8ERY7 e6uFsqBx49ueZZ58a1PftC3GzO0H2ld794uhYdjV3rnEWqQQIJ93GjnjY825XTESeCGS VAFJ4zW9XPoHevGcTpqJHI4AVF0PnV71G59h57bbIyd2pLA4ySu46pBP64POBRTYzHkd wD1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598093; x=1732202893; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7749KjgMy6eatM2VSy2jowDIab8tvoATfJQmYOPl1pc=; b=gQI/snR6El35rpulBeObsDAk7230zMx3oj9KMajDSkgvuU3K2XTlJrBbkQl27UK0nn FLt2EPeAL+fnnX1OtmJR3MyEXPxPgK37g1R0zUyWK0JFTaK/qkaPNtpWS+AKUryMb7a+ WLaMhnT42S87o7GXDeFu1pcoD+ag9LulNtqQErNZuzqUe4AufHunCQ+OlihEAj0U4M5l 0ADA8AVgcc6EA3M5yPN7VBLH/ZM6M118E/6JQrc5f6ROD0vCuYTz/k+u+4LfbjdQGvDF Wh4pu90KB06U9UjwxNltXfkIO2VHp6nl5KM4DsaqhP6TZONrBpPIZAH6OXXrjhxUh25m OPnQ== X-Forwarded-Encrypted: i=1; AJvYcCXCDbZxSnNNzOhX1GlGaWkh8RjeWGeZBGPfizZpfQnhe4+5c7o2HKnQq3xFo7HM/5atIukCWNv/16OxDzWy@vger.kernel.org X-Gm-Message-State: AOJu0Ywtuljal/hQfwQD3tdlJkQCHzji0L86r1i0n/oAdNNSL8bKOb40 0TF/Yl8/YKIiyKUaHLf3U75ONQFuVL/9JQr4YjnuYrUJbTMItSoB86os+yJ7G/w= X-Google-Smtp-Source: AGHT+IHV4oSrYqhQg8SNV67NfRcdIlRE7mzCITp5FLKpYfj2Lpw4MvYi01EsizY0Kl5ORLV8Nbza+Q== X-Received: by 2002:a05:6820:8cc:b0:5eb:6a67:6255 with SMTP id 006d021491bc7-5ee9ec3bae1mr2681521eaf.1.1731598092948; Thu, 14 Nov 2024 07:28:12 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:12 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 08/17] mm/filemap: add read support for RWF_UNCACHED Date: Thu, 14 Nov 2024 08:25:12 -0700 Message-ID: <20241114152743.2381672-10-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add RWF_UNCACHED as a read operation flag, which means that any data read wil be removed from the page cache upon completion. Uses the page cache to synchronize, and simply prunes folios that were instantiated when the operation completes. While it would be possible to use private pages for this, using the page cache as synchronization is handy for a variety of reasons: 1) No special truncate magic is needed 2) Async buffered reads need some place to serialize, using the page cache is a lot easier than writing extra code for this 3) The pruning cost is pretty reasonable and the code to support this is much simpler as a result. You can think of uncached buffered IO as being the much more attractive cousin of O_DIRECT - it has none of the restrictions of O_DIRECT. Yes, it will copy the data, but unlike regular buffered IO, it doesn't run into the unpredictability of the page cache in terms of reclaim. As an example, on a test box with 32 drives, reading them with buffered IO looks as follows: Reading bs 65536, uncached 0 1s: 145945MB/sec 2s: 158067MB/sec 3s: 157007MB/sec 4s: 148622MB/sec 5s: 118824MB/sec 6s: 70494MB/sec 7s: 41754MB/sec 8s: 90811MB/sec 9s: 92204MB/sec 10s: 95178MB/sec 11s: 95488MB/sec 12s: 95552MB/sec 13s: 96275MB/sec where it's quite easy to see where the page cache filled up, and performance went from good to erratic, and finally settles at a much lower rate. Looking at top while this is ongoing, we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7535 root 20 0 267004 0 0 S 3199 0.0 8:40.65 uncached 3326 root 20 0 0 0 0 R 100.0 0.0 0:16.40 kswapd4 3327 root 20 0 0 0 0 R 100.0 0.0 0:17.22 kswapd5 3328 root 20 0 0 0 0 R 100.0 0.0 0:13.29 kswapd6 3332 root 20 0 0 0 0 R 100.0 0.0 0:11.11 kswapd10 3339 root 20 0 0 0 0 R 100.0 0.0 0:16.25 kswapd17 3348 root 20 0 0 0 0 R 100.0 0.0 0:16.40 kswapd26 3343 root 20 0 0 0 0 R 100.0 0.0 0:16.30 kswapd21 3344 root 20 0 0 0 0 R 100.0 0.0 0:11.92 kswapd22 3349 root 20 0 0 0 0 R 100.0 0.0 0:16.28 kswapd27 3352 root 20 0 0 0 0 R 99.7 0.0 0:11.89 kswapd30 3353 root 20 0 0 0 0 R 96.7 0.0 0:16.04 kswapd31 3329 root 20 0 0 0 0 R 96.4 0.0 0:11.41 kswapd7 3345 root 20 0 0 0 0 R 96.4 0.0 0:13.40 kswapd23 3330 root 20 0 0 0 0 S 91.1 0.0 0:08.28 kswapd8 3350 root 20 0 0 0 0 S 86.8 0.0 0:11.13 kswapd28 3325 root 20 0 0 0 0 S 76.3 0.0 0:07.43 kswapd3 3341 root 20 0 0 0 0 S 74.7 0.0 0:08.85 kswapd19 3334 root 20 0 0 0 0 S 71.7 0.0 0:10.04 kswapd12 3351 root 20 0 0 0 0 R 60.5 0.0 0:09.59 kswapd29 3323 root 20 0 0 0 0 R 57.6 0.0 0:11.50 kswapd1 [...] which is just showing a partial list of the 32 kswapd threads that are running mostly full tilt, burning ~28 full CPU cores. If the same test case is run with RWF_UNCACHED set for the buffered read, the output looks as follows: Reading bs 65536, uncached 0 1s: 153144MB/sec 2s: 156760MB/sec 3s: 158110MB/sec 4s: 158009MB/sec 5s: 158043MB/sec 6s: 157638MB/sec 7s: 157999MB/sec 8s: 158024MB/sec 9s: 157764MB/sec 10s: 157477MB/sec 11s: 157417MB/sec 12s: 157455MB/sec 13s: 157233MB/sec 14s: 156692MB/sec which is just chugging along at ~155GB/sec of read performance. Looking at top, we see: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7961 root 20 0 267004 0 0 S 3180 0.0 5:37.95 uncached 8024 axboe 20 0 14292 4096 0 R 1.0 0.0 0:00.13 top where just the test app is using CPU, no reclaim is taking place outside of the main thread. Not only is performance 65% better, it's also using half the CPU to do it. Signed-off-by: Jens Axboe --- mm/filemap.c | 28 ++++++++++++++++++++++++++-- mm/swap.c | 2 ++ 2 files changed, 28 insertions(+), 2 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 02d9cb585195..3d0614ea5f59 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2474,6 +2474,8 @@ static int filemap_create_folio(struct kiocb *iocb, folio = filemap_alloc_folio(mapping_gfp_mask(mapping), min_order); if (!folio) return -ENOMEM; + if (iocb->ki_flags & IOCB_UNCACHED) + __folio_set_uncached(folio); /* * Protect against truncate / hole punch. Grabbing invalidate_lock @@ -2519,6 +2521,8 @@ static int filemap_readahead(struct kiocb *iocb, struct file *file, if (iocb->ki_flags & IOCB_NOIO) return -EAGAIN; + if (iocb->ki_flags & IOCB_UNCACHED) + ractl.uncached = 1; page_cache_async_ra(&ractl, folio, last_index - folio->index); return 0; } @@ -2548,6 +2552,8 @@ static int filemap_get_pages(struct kiocb *iocb, size_t count, return -EAGAIN; if (iocb->ki_flags & IOCB_NOWAIT) flags = memalloc_noio_save(); + if (iocb->ki_flags & IOCB_UNCACHED) + ractl.uncached = 1; page_cache_sync_ra(&ractl, last_index - index); if (iocb->ki_flags & IOCB_NOWAIT) memalloc_noio_restore(flags); @@ -2595,6 +2601,20 @@ static inline bool pos_same_folio(loff_t pos1, loff_t pos2, struct folio *folio) return (pos1 >> shift == pos2 >> shift); } +static void filemap_uncached_read(struct address_space *mapping, + struct folio *folio) +{ + if (!folio_test_uncached(folio)) + return; + if (folio_test_writeback(folio)) + return; + if (folio_test_clear_uncached(folio)) { + folio_lock(folio); + folio_unmap_invalidate(mapping, folio, 0); + folio_unlock(folio); + } +} + /** * filemap_read - Read data from the page cache. * @iocb: The iocb to read. @@ -2706,8 +2726,12 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, } } put_folios: - for (i = 0; i < folio_batch_count(&fbatch); i++) - folio_put(fbatch.folios[i]); + for (i = 0; i < folio_batch_count(&fbatch); i++) { + struct folio *folio = fbatch.folios[i]; + + filemap_uncached_read(mapping, folio); + folio_put(folio); + } folio_batch_init(&fbatch); } while (iov_iter_count(iter) && iocb->ki_pos < isize && !error); diff --git a/mm/swap.c b/mm/swap.c index b8e3259ea2c4..542f298d3dcd 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -472,6 +472,8 @@ static void folio_inc_refs(struct folio *folio) */ void folio_mark_accessed(struct folio *folio) { + if (folio_test_uncached(folio)) + return; if (lru_gen_enabled()) { folio_inc_refs(folio); return; From patchwork Thu Nov 14 15:25:13 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875200 Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4213A19B3E2 for ; Thu, 14 Nov 2024 15:28:15 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598097; cv=none; b=JX8QgOcyHDOvnsltJ+O/D1pcuNmRKBUzUInGXIshqDCa2qIosQgr+ZkcCsocSel6CpmOsjnxL5HWSVOwF4x59qIE2pfKggOVQtMo8MVytT/vLTtbQp+nO+O3SGQEoT5B+L4kTx9CFyhzlH81HTwoCCfAuNiq+Ryf4J1cby854ug= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598097; c=relaxed/simple; bh=DvaplPoN0uFc8s2haQH1BvU7ppNotoZgida5jWixk9M=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=AebVdmtm1Pt5uM7NzVojX0u45KhjmJZcgtihrSV8ffRVgWmzXc2g0KtgPB9dqloc49/18LdcnRLHkXn+v150B/uzHS5rbz1xet2UKv4ljKqOGANlLEWNZLQ4bNCcSv/yRTCdD4mqqxRjstSjS+PNH4v7lGxUAiXS63Wcp1YUi60= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=AazWmzX5; arc=none smtp.client-ip=209.85.161.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="AazWmzX5" Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-5ebc27fdc30so329596eaf.2 for ; Thu, 14 Nov 2024 07:28:15 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598094; x=1732202894; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=vSo08QFzm+JiqHz4hWmAqopvA0C/7UNivlhAphA9PAA=; b=AazWmzX5eeAsU8Xu0NaiseiCfM42rfUilq+4ewe2Z21r9ikwkCoF/e4T5odgOi3+88 4GYpCMogj+vWesfwJjiXFbim6l8Rd8bweRI5SuJG2D/dbrxtyJrYgzplEUqa2P+aG3Cu s0pGffc/WwmWF7wQzCEMRxHOhQh9IsUYCgfGCUyyxovEqS7pdjQG6WpLOyBRh1MptgjP UWF4/FKRJxDG0QwTGM7Xu6UTpPldaAZgW/rym6fwyJ+M1hVdhpun0RUI4HHcyGmhdqZE ZyDOji3OJJYQ/FVTrNFZs2eUxQ8vsbtIb7Iczfvi+42dMteJRYIRvhChElqd2z9RFoBB nYkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598094; x=1732202894; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vSo08QFzm+JiqHz4hWmAqopvA0C/7UNivlhAphA9PAA=; b=hMHF5yOL5Je4tuasg18HrLQNhci8iTFo8Fj8AmzrKyYWNf4WRyckotHVaKRCrS2pNb 2JtQEaDfI2SqCuZrbwEKHvj3pjktzkdDVMazhnG0wE+sMLPJIw4kg3fhvwEnkLJxzi95 YQbEurhjCKPl8QUTh5mhAtVV34M5ajFz5TNWJnyPN3XC4+xvKy5M7IfxM9q/xyFuP8oo i+HWTtrOLJRmEZyAbGKvq80CHchWkFTO2RW3HwyQg/seXIFwQsyhuL6HCWl7VUTgR/U3 mkfmjJZsBCxpNQN4wU2CngohD+qYG7lvUU4njwzdFsusFgBKy46stmen7r99hNSA/E6h IqDg== X-Forwarded-Encrypted: i=1; AJvYcCX4agYBlWabMNpDONArPq7GSZLuA1EFBA5RhDbMnFV1CFEBAyXj5zVHLm8kAe39kXSWLOLu5NahDS3lojdh@vger.kernel.org X-Gm-Message-State: AOJu0YzxcNvszSvMO7ErMNmiJ1/v2QgyuCtx7F4DfBWjdIF6mI09mae8 amhvwqumxtVau+/t8eTRj8exu0Sf6XdOk8oXsFCa88iMEfbFZG4q1gT92CAXIxc= X-Google-Smtp-Source: AGHT+IF/wU8hdLitBYUw1Xm5/VEAvnGHF/Lmj1TdlA4Dk/xHse2V/nkks/Gl33gMzxUMuX6RfmQ7hg== X-Received: by 2002:a05:6820:983:b0:5ee:bb2:bdd4 with SMTP id 006d021491bc7-5ee57b96aadmr18195949eaf.1.1731598094397; Thu, 14 Nov 2024 07:28:14 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:13 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 09/17] mm/filemap: drop uncached pages when writeback completes Date: Thu, 14 Nov 2024 08:25:13 -0700 Message-ID: <20241114152743.2381672-11-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If the folio is marked as uncached, drop pages when writeback completes. Intended to be used with RWF_UNCACHED, to avoid needing sync writes for uncached IO. Signed-off-by: Jens Axboe --- mm/filemap.c | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/mm/filemap.c b/mm/filemap.c index 3d0614ea5f59..13815194ed8a 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -1600,6 +1600,27 @@ int folio_wait_private_2_killable(struct folio *folio) } EXPORT_SYMBOL(folio_wait_private_2_killable); +/* + * If folio was marked as uncached, then pages should be dropped when writeback + * completes. Do that now. If we fail, it's likely because of a big folio - + * just reset uncached for that case and latter completions should invalidate. + */ +static void folio_end_uncached(struct folio *folio) +{ + /* + * Hitting !in_task() should not happen off RWF_UNCACHED writeback, but + * can happen if normal writeback just happens to find dirty folios + * that were created as part of uncached writeback, and that writeback + * would otherwise not need non-IRQ handling. Just skip the + * invalidation in that case. + */ + if (in_task() && folio_trylock(folio)) { + if (folio->mapping) + folio_unmap_invalidate(folio->mapping, folio, 0); + folio_unlock(folio); + } +} + /** * folio_end_writeback - End writeback against a folio. * @folio: The folio. @@ -1610,6 +1631,8 @@ EXPORT_SYMBOL(folio_wait_private_2_killable); */ void folio_end_writeback(struct folio *folio) { + bool folio_uncached = false; + VM_BUG_ON_FOLIO(!folio_test_writeback(folio), folio); /* @@ -1631,9 +1654,14 @@ void folio_end_writeback(struct folio *folio) * reused before the folio_wake_bit(). */ folio_get(folio); + if (!folio_test_dirty(folio)) + folio_uncached = folio_test_clear_uncached(folio); if (__folio_end_writeback(folio)) folio_wake_bit(folio, PG_writeback); acct_reclaim_writeback(folio); + + if (folio_uncached) + folio_end_uncached(folio); folio_put(folio); } EXPORT_SYMBOL(folio_end_writeback); From patchwork Thu Nov 14 15:25:14 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875201 Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B25981A265B for ; Thu, 14 Nov 2024 15:28:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598099; cv=none; b=IFIe+50FiJ7q+z8A+no9OcelDzefKpp9yxNowtkUAckaNxMRdBjQXO1Xh7JmPWow49WGCLV0Fhyc68Jzsdv6QBrhmjQlDEBtDTug6src1V4KM38aW9zE60oCqRhcfknRBwV9U0DX8+TevjZKheHK4XXQ2UfVqZmbiiDBGWtxCiw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598099; c=relaxed/simple; bh=4ADL7kFFYjVJou6EGB93OsIZhCYEyfOeIONxeYUu3bs=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=mLEAMmU+jbxedDxZ2SBHbsBoVoM3xnJiz1S1IAvuIj3/+denIgKtIkFizriT3cC0ISc7CnOTtRbnOnTi75eoUkicSPm5HNssW4I+bHRZUZBYrI07cmbKPqZDzfsQzzxywL+jR8Uvy1/np/jM5IMIm51hQXn5p9SPcbMTKW4tDWI= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Y/1STsAJ; arc=none smtp.client-ip=209.85.161.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Y/1STsAJ" Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-5ee53b30470so398359eaf.3 for ; Thu, 14 Nov 2024 07:28:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598096; x=1732202896; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=xBThbwgpFnDYguERwRJ3Nd3OfeHrc6kNH6KxzsoIqNs=; b=Y/1STsAJbfgjZWLNjNVPEb2T1+Tkc+R0bkZw+xEncWocTemeDgFAq8hag/d4xZ23Gw LSXgXhX7eUX54UxAHmT1/2Hsvm+roRe0baboYnuFC2gBibn9Hozis+66CEMRck5bWKGQ VKRULUlDh9jkAJ8E2fxcOlDoY5eEI5kUB8wFcV8r49dE91ltSRZ/LdxRWIRLGNy9LgbT 2OSo0ofWUrxG+GfKgo+r+J2lZL8RNuDtdhuSsyC6b/U7+qsAG1ZeFwwouavPzZnzUj6U K8fwZ0KLqWfSHYyT1kjZog9HorPus4/7qfRQb5pSNhkL/yi2ncA2b3S5ePj/+Dk0YFbc IIDQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598096; x=1732202896; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=xBThbwgpFnDYguERwRJ3Nd3OfeHrc6kNH6KxzsoIqNs=; b=Cq2lLBdzmrsH4DJ/M9y0dyERSH7Oibds/ldA0/bX/C5PqIjDq2sgF627y4GZCKKOYa G3a0jmsBdE8h0boAtziS9zc5FisY+ALqBtZjykaMN+gVzd7x0bbWnwWc58ZSADSZVmrk Ms08Y+fhpFMYSbqSC2ZanS78ExGa4T4tHnlpB9NdobVRHUy5UTZK1UT2FjEHYe/WvQ0h H+IZSqkZ2DxI+G2CvIiL48+9J6szD0i1BR1YrXU0KEWyKmceR+6JmPWWzXit8nVk6H/y CD9hPRwgh3FIu1ewy95dhsIbPEoQ6jqL0qa/BHrnGccd1gPL1zmea7tZ74p46b/pGJHl tggg== X-Forwarded-Encrypted: i=1; AJvYcCUeWN4tVAV0kXtt3MNhZwsq/Y+nak/H/Q0hjU1+sHN2YEweu2xBXinPesAEPfFa2LgiVrVg2H9PtwYbK9pM@vger.kernel.org X-Gm-Message-State: AOJu0Yx8J+xBpwF+s9SvVryncBGriQsphTScE12MJ4Nv6YvOGBB8vDaR KgF1QaKoapX8cof9YdVUEzBuHQRP7G9hR/vKzih/QFiXBgb06H0WCzTFpx1KJubrOKTuAxKiqut /gbw= X-Google-Smtp-Source: AGHT+IHjZdeOSBooCNtlJS5s8uHCzSyESjo+TDYveTg1n/VGl8pzguc4+c1Edpma/7ckXWFt8H0gGQ== X-Received: by 2002:a4a:ec4c:0:b0:5e1:cd24:c19c with SMTP id 006d021491bc7-5ee868b0bf0mr9239441eaf.0.1731598095749; Thu, 14 Nov 2024 07:28:15 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:15 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 10/17] mm/filemap: make buffered writes work with RWF_UNCACHED Date: Thu, 14 Nov 2024 08:25:14 -0700 Message-ID: <20241114152743.2381672-12-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 If RWF_UNCACHED is set for a write, mark new folios being written with uncached. This is done by passing in the fact that it's an uncached write through the folio pointer. We can only get there when IOCB_UNCACHED was allowed, which can only happen if the file system opts in. Opting in means they need to check for the LSB in the folio pointer to know if it's an uncached write or not. If it is, then FGP_UNCACHED should be used if creating new folios is necessary. Uncached writes will drop any folios they create upon writeback completion, but leave folios that may exist in that range alone. Since ->write_begin() doesn't currently take any flags, and to avoid needing to change the callback kernel wide, use the foliop being passed in to ->write_begin() to signal if this is an uncached write or not. File systems can then use that to mark newly created folios as uncached. This provides similar benefits to using RWF_UNCACHED with reads. Testing buffered writes on 32 files: writing bs 65536, uncached 0 1s: 196035MB/sec 2s: 132308MB/sec 3s: 132438MB/sec 4s: 116528MB/sec 5s: 103898MB/sec 6s: 108893MB/sec 7s: 99678MB/sec 8s: 106545MB/sec 9s: 106826MB/sec 10s: 101544MB/sec 11s: 111044MB/sec 12s: 124257MB/sec 13s: 116031MB/sec 14s: 114540MB/sec 15s: 115011MB/sec 16s: 115260MB/sec 17s: 116068MB/sec 18s: 116096MB/sec where it's quite obvious where the page cache filled, and performance dropped from to about half of where it started, settling in at around 115GB/sec. Meanwhile, 32 kswapds were running full steam trying to reclaim pages. Running the same test with uncached buffered writes: writing bs 65536, uncached 1 1s: 198974MB/sec 2s: 189618MB/sec 3s: 193601MB/sec 4s: 188582MB/sec 5s: 193487MB/sec 6s: 188341MB/sec 7s: 194325MB/sec 8s: 188114MB/sec 9s: 192740MB/sec 10s: 189206MB/sec 11s: 193442MB/sec 12s: 189659MB/sec 13s: 191732MB/sec 14s: 190701MB/sec 15s: 191789MB/sec 16s: 191259MB/sec 17s: 190613MB/sec 18s: 191951MB/sec and the behavior is fully predictable, performing the same throughout even after the page cache would otherwise have fully filled with dirty data. It's also about 65% faster, and using half the CPU of the system compared to the normal buffered write. Signed-off-by: Jens Axboe --- include/linux/fs.h | 5 +++++ include/linux/pagemap.h | 9 +++++++++ mm/filemap.c | 12 +++++++++++- 3 files changed, 25 insertions(+), 1 deletion(-) diff --git a/include/linux/fs.h b/include/linux/fs.h index 45510d0b8de0..122ae821989f 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2877,6 +2877,11 @@ static inline ssize_t generic_write_sync(struct kiocb *iocb, ssize_t count) (iocb->ki_flags & IOCB_SYNC) ? 0 : 1); if (ret) return ret; + } else if (iocb->ki_flags & IOCB_UNCACHED) { + struct address_space *mapping = iocb->ki_filp->f_mapping; + + filemap_fdatawrite_range_kick(mapping, iocb->ki_pos, + iocb->ki_pos + count); } return count; diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index d55bf995bd9e..cc02518d338d 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -14,6 +14,7 @@ #include #include #include /* for in_interrupt() */ +#include #include struct folio_batch; @@ -70,6 +71,14 @@ static inline int filemap_write_and_wait(struct address_space *mapping) return filemap_write_and_wait_range(mapping, 0, LLONG_MAX); } +/* + * Value passed in to ->write_begin() if IOCB_UNCACHED is set for the write, + * and the ->write_begin() handler on a file system supporting FOP_UNCACHED + * must check for this and pass FGP_UNCACHED for folio creation. + */ +#define foliop_uncached ((struct folio *) 0xfee1c001) +#define foliop_is_uncached(foliop) (*(foliop) == foliop_uncached) + /** * filemap_set_wb_err - set a writeback error on an address_space * @mapping: mapping in which to set writeback error diff --git a/mm/filemap.c b/mm/filemap.c index 13815194ed8a..297cb53332ff 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -4076,7 +4076,7 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) ssize_t written = 0; do { - struct folio *folio; + struct folio *folio = NULL; size_t offset; /* Offset into folio */ size_t bytes; /* Bytes to write to folio */ size_t copied; /* Bytes copied from user */ @@ -4104,6 +4104,16 @@ ssize_t generic_perform_write(struct kiocb *iocb, struct iov_iter *i) break; } + /* + * If IOCB_UNCACHED is set here, we now the file system + * supports it. And hence it'll know to check folip for being + * set to this magic value. If so, it's an uncached write. + * Whenever ->write_begin() changes prototypes again, this + * can go away and just pass iocb or iocb flags. + */ + if (iocb->ki_flags & IOCB_UNCACHED) + folio = foliop_uncached; + status = a_ops->write_begin(file, mapping, pos, bytes, &folio, &fsdata); if (unlikely(status < 0)) From patchwork Thu Nov 14 15:25:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875202 Received: from mail-oo1-f43.google.com (mail-oo1-f43.google.com [209.85.161.43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 19B3F1AAE30 for ; Thu, 14 Nov 2024 15:28:17 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.43 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598100; cv=none; b=SwQW2ZMQBUHqlDPtNl2Ha6ZuQCpHCCi9rcRR74J4am8zL44/3FM/+NT7h/CLwZAaa0rtICnMpC3mrx4LzPZoqfWiar9n5E7p5YyCIo47yFhAb7t8/ogL0GekEUb4zs3QdOWd0IVCT415+RjLEaMxlf580nifMhk6LlEUvGWuxIA= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598100; c=relaxed/simple; bh=Cprr5Wztuokl5alQc44L4Bz+weZZLTHr6zOlAD5+Dc8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=nuiGeeyOIdKuTJY5+nYwZZLepuc48/EQec/j5kPZvQE0erMTLlApz7d18uERh+gmdg2dnN6UZwSi4etLWJlU3TQBl+X6w2J+UPZPA0/km7Db1Yv6EDrlwJ8rBjW7IlFndO+w6FMvP3M8YxwFGe8WRKxntoj1bShD8B2jAmh6XDo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=FD0BMbPu; arc=none smtp.client-ip=209.85.161.43 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="FD0BMbPu" Received: by mail-oo1-f43.google.com with SMTP id 006d021491bc7-5ee1c3dbeb9so338449eaf.1 for ; Thu, 14 Nov 2024 07:28:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598097; x=1732202897; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=Wvrv44z8Apd8qqgxqZBS6Do0FSmkBjZMAMWmq/c36cU=; b=FD0BMbPuTuxr0syAkrOqgbeVJ3V1KdUfTA6rPfo3c4tK6Xus72+a2OHKSE1lgXyCmX 2DDppcK7C5D7w39z8m4Lvoy77ByuELQBlYoBLDtmlzEOZy1nLCR6zrg6gnm49AaEQdnL 6OAJRAJCDHhGXePHms7IiKttN36ZhLmru1Mbc4AAZUFBrn7OaFD+ePREa+gIMhk5zd/l tq5C96H4cl8sJml5vZNE/34IhvhzaEAFObLQCL+QNThkViFGRXDGQ3Df3Gam6FU7T3B2 mw3J1ORn9FkMY1S9nHVRb6nsd14/Egb5HPjsu3AYAkDO/ldpeio+5hgXjlU0R3dyBqhT eltQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598097; x=1732202897; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Wvrv44z8Apd8qqgxqZBS6Do0FSmkBjZMAMWmq/c36cU=; b=L/2fZExMxPFTY8qkOmu3xKh0ztyxCg4comdFnRVD7CxB7UJK5uxK4NqLz5IQqy4BPG /bYk2ncItIcwidDEd+rbqcYVqbrBUKowqDq3LfIQwGMU1cxGf/BcV8XHsUUa/T0PV6BU zNvfX/sBijZUqzAsgQYj8U0YguKADfij2S4qwlSaDpOpuwfoqwAPQuMiTiJjqwBY+QOv wBnmZWR9Avz8mw2YAxZLtXR1Sqw0NZT3hITmA0jtc8XWvs6C1T1gYteA4hJ74bx2Pa/r hFJ0IIsbajkPJ+dNlFyLPqoyImGB0sjki3Yf1XaH/tekX8EiCoIvQnnouu0B6bfTYqHp K0gg== X-Forwarded-Encrypted: i=1; AJvYcCV6GqbDveGqfzoeLEeAOLmvSGDD9CJxuFGXLJQoAbwy26xBqXQztxUE3UHYJiAbezqnovCBXEa8yh9tsCBk@vger.kernel.org X-Gm-Message-State: AOJu0YzmsHBwyOn9wEXc/Okaz2aH9O4HvT47wx+ZxyZQ5DJy29RWpHbL SpIoNENXxQCMsv+4syLFs4FqLJbw/76oteug6A8LpKK0usk3DQFV+g1oIe3/ogo= X-Google-Smtp-Source: AGHT+IEaTjdnOMAkEmoEcJnRXLnfxDNshZw+Ax439RAi/pKdYn0q0me9qe1oslEKsbvc7WBkYDa4cA== X-Received: by 2002:a4a:e906:0:b0:5eb:6b3e:ce7f with SMTP id 006d021491bc7-5ee9c6c80d1mr2431138eaf.0.1731598097161; Thu, 14 Nov 2024 07:28:17 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:16 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 11/17] mm/filemap: add filemap_fdatawrite_range_kick() helper Date: Thu, 14 Nov 2024 08:25:15 -0700 Message-ID: <20241114152743.2381672-13-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Works like filemap_fdatawrite_range(), except it's a non-integrity data writeback and hence only starts writeback on the specified range. Will help facilitate generically starting uncached writeback from generic_write_sync(), as header dependencies preclude doing this inline from fs.h. Signed-off-by: Jens Axboe --- include/linux/fs.h | 2 ++ mm/filemap.c | 18 ++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/include/linux/fs.h b/include/linux/fs.h index 122ae821989f..560d3ee1bb8a 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -2845,6 +2845,8 @@ extern int __must_check file_fdatawait_range(struct file *file, loff_t lstart, extern int __must_check file_check_and_advance_wb_err(struct file *file); extern int __must_check file_write_and_wait_range(struct file *file, loff_t start, loff_t end); +int filemap_fdatawrite_range_kick(struct address_space *mapping, loff_t start, + loff_t end); static inline int file_write_and_wait(struct file *file) { diff --git a/mm/filemap.c b/mm/filemap.c index 297cb53332ff..a8a9fb986d2d 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -450,6 +450,24 @@ int filemap_fdatawrite_range(struct address_space *mapping, loff_t start, } EXPORT_SYMBOL(filemap_fdatawrite_range); +/** + * filemap_fdatawrite_range_kick - start writeback on a range + * @mapping: target address_space + * @start: index to start writeback on + * @end: last (non-inclusive) index for writeback + * + * This is a non-integrity writeback helper, to start writing back folios + * for the indicated range. + * + * Return: %0 on success, negative error code otherwise. + */ +int filemap_fdatawrite_range_kick(struct address_space *mapping, loff_t start, + loff_t end) +{ + return __filemap_fdatawrite_range(mapping, start, end, WB_SYNC_NONE); +} +EXPORT_SYMBOL_GPL(filemap_fdatawrite_range_kick); + /** * filemap_flush - mostly a non-blocking flush * @mapping: target address_space From patchwork Thu Nov 14 15:25:16 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875203 Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 688AB19AA5D for ; Thu, 14 Nov 2024 15:28:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598101; cv=none; b=B1Iw048bu3M+IRrsX705g5mrSrgmHyG5aXRUeKuN3HzFOphXEMjlADO4+9xycYKp88R0V2zOAJtOkD2mULjlpB3lPm3iLBqxBwfzmHKpwbc8sOUXiIAzCTzfyXUvzzs+NroaYsFSd3+JQlJjCjxi6eVAEncWFsYgUYUm42YWLiM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598101; c=relaxed/simple; bh=lNhwCmzwFphLkSB9taIvs4WJ/6CHD1yuvvg662y7/aU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=KpZfS/wQtrYlgOPauOoHS1T8k01R+WT3M++Hw4oYArS53Dy0uTKdAwTa9WxqKyhOHF3CIfOfrKELUMnT+A3q7NCbTfHPXz7MBeelaxG7mdtYvxE6Jhj1y/vIBQbgbN72qZIlcAwzrz9jAP0wQvfG3PKPlHsmIGgB7j1scwSvikM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=SVrMbxI5; arc=none smtp.client-ip=209.85.161.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="SVrMbxI5" Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-5ec4a40e95bso321544eaf.1 for ; Thu, 14 Nov 2024 07:28:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598098; x=1732202898; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bh4vHLcSKYCCRkbDjlhjn1EJt3wp9QgKXbTvV2Vr6u0=; b=SVrMbxI5HUdJ+oGYj469+T374H88/9gwfszmBpWundKS7Y43s6hygEFdIBbqJzVbbl SbQu0RNUsOk/eju4spW3OuP2ZOiOFKEJBKgaUppYsVUy3IAhsR/qnzoXWClNIJWMbr6l KNM84FWUZYttbT0yYFVwyE3DaFda1vz+18ywpMEpT3mNcEZrrwP6oC2UbW84l4cOuhlz 76nMmzfJ67XOHJ8TvfSKs7x8gaqc7pg+eU4MJhKPp7gGQvO/gNiwLHFAZZV1kSRfBt6o AADvjhk/iSnKTkXutc/JyULDFSc1N4QnVNkFNWDcw3OHGy/qxG6aEEXb1O93Jkdi26zm RNIA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598098; x=1732202898; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bh4vHLcSKYCCRkbDjlhjn1EJt3wp9QgKXbTvV2Vr6u0=; b=F1lX9jMpv9DWwVWUiRBAZ0Kbt1hBOixZZZV1NWNh5rziePvX57nbBSy/l/TWy5iuEE SG8BRU/+E23O63P82mZLgFs7p8h8ZYgsa7CVO0TPSZ4cG0yDltUVBs52OeX9vfEfDUVw hBcYCwrF5st8wJdEd6GRoqbZf1fonMWrxAH3/uznYbFfkE06lPLe5srnqXidn4ZPxfYw Z5Nu9bXDnRQ69po02MiWzjraPfmdJVTTXJiLdGdiU2pXWAyODlx4LE1O75PVlcSZE3Kt 3R21ZcSY3R3m+hBtcBCgpd+SE0Lt6ecPfwGw8sReiIZKmGIdwBwGBLqVD709g4Qbep/1 EdJQ== X-Forwarded-Encrypted: i=1; AJvYcCWSn4lT6OnMrDi6GgVg6HbsF219zcAOJhrmGyrHurlJ0m8h15Z9DZEH6iTTDsKnrNh2jo8RFX6NjMsvzG4Z@vger.kernel.org X-Gm-Message-State: AOJu0YzTxbbzxVjwhfyR70uIQI547YBY6x/F4a6V/iYivb0QG1xd10Bs DaiwOe0WXw+7XO+4ErxnfFQo1UpkFC9UwoWsQh2A1Hjf13q+5LytPSSZS3+z0Dk= X-Google-Smtp-Source: AGHT+IECtJqHxvsDZk559Jqz444iSDToPVSz/2QlEeAwby3Gwu7O3Iv2UCv4dhdd6n28ENyvjODyaw== X-Received: by 2002:a05:6820:1e0c:b0:5e3:b7a6:834 with SMTP id 006d021491bc7-5ee57b9c8e0mr20035625eaf.1.1731598098551; Thu, 14 Nov 2024 07:28:18 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:17 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 12/17] mm: add FGP_UNCACHED folio creation flag Date: Thu, 14 Nov 2024 08:25:16 -0700 Message-ID: <20241114152743.2381672-14-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Callers can pass this in for uncached folio creation, in which case if a folio is newly created it gets marked as uncached. If a folio exists for this index and lookup succeeds, then it will not get marked as uncached. If an !uncached lookup finds a cached folio, clear the flag. For that case, there are competeting uncached and cached users of the folio, and it should not get pruned. Signed-off-by: Jens Axboe --- include/linux/pagemap.h | 2 ++ mm/filemap.c | 5 +++++ 2 files changed, 7 insertions(+) diff --git a/include/linux/pagemap.h b/include/linux/pagemap.h index cc02518d338d..860807e34b8c 100644 --- a/include/linux/pagemap.h +++ b/include/linux/pagemap.h @@ -721,6 +721,7 @@ pgoff_t page_cache_prev_miss(struct address_space *mapping, * * %FGP_NOFS - __GFP_FS will get cleared in gfp. * * %FGP_NOWAIT - Don't block on the folio lock. * * %FGP_STABLE - Wait for the folio to be stable (finished writeback) + * * %FGP_UNCACHED - Uncached buffered IO * * %FGP_WRITEBEGIN - The flags to use in a filesystem write_begin() * implementation. */ @@ -734,6 +735,7 @@ typedef unsigned int __bitwise fgf_t; #define FGP_NOWAIT ((__force fgf_t)0x00000020) #define FGP_FOR_MMAP ((__force fgf_t)0x00000040) #define FGP_STABLE ((__force fgf_t)0x00000080) +#define FGP_UNCACHED ((__force fgf_t)0x00000100) #define FGF_GET_ORDER(fgf) (((__force unsigned)fgf) >> 26) /* top 6 bits */ #define FGP_WRITEBEGIN (FGP_LOCK | FGP_WRITE | FGP_CREAT | FGP_STABLE) diff --git a/mm/filemap.c b/mm/filemap.c index a8a9fb986d2d..dbc3fa975ad1 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2002,6 +2002,8 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, /* Init accessed so avoid atomic mark_page_accessed later */ if (fgp_flags & FGP_ACCESSED) __folio_set_referenced(folio); + if (fgp_flags & FGP_UNCACHED) + __folio_set_uncached(folio); err = filemap_add_folio(mapping, folio, index, gfp); if (!err) @@ -2024,6 +2026,9 @@ struct folio *__filemap_get_folio(struct address_space *mapping, pgoff_t index, if (!folio) return ERR_PTR(-ENOENT); + /* not an uncached lookup, clear uncached if set */ + if (folio_test_uncached(folio) && !(fgp_flags & FGP_UNCACHED)) + folio_clear_uncached(folio); return folio; } EXPORT_SYMBOL(__filemap_get_folio); From patchwork Thu Nov 14 15:25:17 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875204 Received: from mail-oa1-f46.google.com (mail-oa1-f46.google.com [209.85.160.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D33B1AE018 for ; Thu, 14 Nov 2024 15:28:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.46 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598103; cv=none; b=qvY5zqZ0NrZcVYc/zRIApyDv2x6z++GhfLWiqzaG5EdD85cRigILRspAKIUOcuXITiTSagrHKP7C2WB4TTz0oTmcuyhWAj0sGbJDt+jXPwv1vDV1YxGG9KBS4fSCHsnCy+QN903EoTADHCgbeiF6U3Aaaq13U/eXC4i6jaRdL7k= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598103; c=relaxed/simple; bh=uwlz9Ym4TNnGPdEk5eejNri0P+bccIlHRZuVyJuJRbM=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=o48I+UE4XC/nH9NIvSoekcnD02JLnBmlYCXb09rI0M8tzHbJ9bCtJCsnQ/8UswFyF+InC5JHwGF9Pdz5gcZMpYMtzuPSmLmSqX8dBe2vvpptKCITYlE+rGf4LDJDSrKwlREsHyrgB+mULblG+kvfgyY9/5VcuusqYd6Z+sFfFKM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=T+CtWO0k; arc=none smtp.client-ip=209.85.160.46 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="T+CtWO0k" Received: by mail-oa1-f46.google.com with SMTP id 586e51a60fabf-2890f273296so359257fac.2 for ; Thu, 14 Nov 2024 07:28:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598100; x=1732202900; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=rF419cnvCI1sSo/lAda6/Cm28hf6CisfTzw5UCKZsOU=; b=T+CtWO0kMypsUNMiZEYgaw3U/o/nnP+YOpm6OWBCUljZr1TEMhQ6UzDDHhA0kYIYVe myBepZvo2h1tY1aMzxiQbXqqA1Ubms/q/xrtWVYPZAcslVw0yF83JDJEI1+u2ngwfthu 6lCHu738WR2LRIEyYQIkTJFvrz3DJTFvmxtaVm2yCd072nyoIOPJbn1ATjhZWoW0jukJ CsfM1CdtcDRnkmoYZf5djvw02r9q/Mrx+zRUk1j8sbMsj3lfipDbVgzcnLa+EQRkrGhl PtmxUFmaX+THWwlpwPrOAiDBFcOtZRMD3dQlHPmsrqBu+WhLZNIoyboFl6f+h1cJj478 uPrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598100; x=1732202900; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=rF419cnvCI1sSo/lAda6/Cm28hf6CisfTzw5UCKZsOU=; b=E13kB4vq+tEB70PKIncuxSTe0WB8O/T/0wj2X8hXMF9bKU7BiqleHrZmyoK0jRKEZT zJn1JfBy52R47M66y5iG1m0guvdrfU6JyPT+a4b34TnzfkdeLs97XFWOTw/LDxEF3msx pZZEp44Mg2edYyiLt9m7ThNbZc8+hC0lNCVvxE+yNX7k6e3P6VPeiFP7dSNfqUa6I8bg O6M7t9kkOKG2Z7fGCi6eKaNp76XUWjac2mqEG7QdSloauKx06ZxY+X/p2PRm9c0mJcB/ +K/pO9rM3K10WC3AeguCRpKUUowkGy8532OYcQhIWq+PZSwvbS8l7ZaVe22qBFjCG9KK azyg== X-Forwarded-Encrypted: i=1; AJvYcCV6rhYyBlk6DcdjqI56cnEsX9y7CYAxPgCvZ+cxLKcfh+MgT8uigDe1dyxcper89JfZ7aEtAURzBGsVvgrp@vger.kernel.org X-Gm-Message-State: AOJu0YwE0l1CV9wgbZSAi13O5pq/vhyUOP74boa0n2PmNgI3OZ4cWPP8 cZA5SEd5I3DnC90dyjFKo65jvimiHKaNtT0W9vaq8eDwKSLeNMjH/BXqQAHIl9I= X-Google-Smtp-Source: AGHT+IFrxjJdmj36oQy6UzPIhY6HkDZabl7PIZsT4CTZLD+yls1jl9nPP1SVyU+w5pEmbGgfnynQrw== X-Received: by 2002:a05:6870:e0c6:b0:261:b48:3c99 with SMTP id 586e51a60fabf-29560143149mr21908350fac.23.1731598099964; Thu, 14 Nov 2024 07:28:19 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:19 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 13/17] ext4: add RWF_UNCACHED write support Date: Thu, 14 Nov 2024 08:25:17 -0700 Message-ID: <20241114152743.2381672-15-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 IOCB_UNCACHED IO needs to prune writeback regions on IO completion, and hence need the worker punt that ext4 also does for unwritten extents. Add an io_end flag to manage that. If foliop is set to foliop_uncached in ext4_write_begin(), then set FGP_UNCACHED so that __filemap_get_folio() will mark newly created folios as uncached. That in turn will make writeback completion drop these ranges from the page cache. Now that ext4 supports both uncached reads and writes, add the fop_flag FOP_UNCACHED to enable it. Signed-off-by: Jens Axboe --- fs/ext4/ext4.h | 1 + fs/ext4/file.c | 2 +- fs/ext4/inline.c | 7 ++++++- fs/ext4/inode.c | 18 ++++++++++++++++-- fs/ext4/page-io.c | 28 ++++++++++++++++------------ 5 files changed, 40 insertions(+), 16 deletions(-) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 44b0d418143c..60dc9ffae076 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -279,6 +279,7 @@ struct ext4_system_blocks { * Flags for ext4_io_end->flags */ #define EXT4_IO_END_UNWRITTEN 0x0001 +#define EXT4_IO_UNCACHED 0x0002 struct ext4_io_end_vec { struct list_head list; /* list of io_end_vec */ diff --git a/fs/ext4/file.c b/fs/ext4/file.c index f14aed14b9cf..0ef39d738598 100644 --- a/fs/ext4/file.c +++ b/fs/ext4/file.c @@ -944,7 +944,7 @@ const struct file_operations ext4_file_operations = { .splice_write = iter_file_splice_write, .fallocate = ext4_fallocate, .fop_flags = FOP_MMAP_SYNC | FOP_BUFFER_RASYNC | - FOP_DIO_PARALLEL_WRITE, + FOP_DIO_PARALLEL_WRITE | FOP_UNCACHED, }; const struct inode_operations ext4_file_inode_operations = { diff --git a/fs/ext4/inline.c b/fs/ext4/inline.c index 3536ca7e4fcc..500bfb6d4860 100644 --- a/fs/ext4/inline.c +++ b/fs/ext4/inline.c @@ -667,6 +667,7 @@ int ext4_try_to_write_inline_data(struct address_space *mapping, handle_t *handle; struct folio *folio; struct ext4_iloc iloc; + fgf_t fgp_flags; if (pos + len > ext4_get_max_inline_size(inode)) goto convert; @@ -702,7 +703,11 @@ int ext4_try_to_write_inline_data(struct address_space *mapping, if (ret) goto out; - folio = __filemap_get_folio(mapping, 0, FGP_WRITEBEGIN | FGP_NOFS, + fgp_flags = FGP_WRITEBEGIN | FGP_NOFS; + if (foliop_is_uncached(foliop)) + fgp_flags |= FGP_UNCACHED; + + folio = __filemap_get_folio(mapping, 0, fgp_flags, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) { ret = PTR_ERR(folio); diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index 54bdd4884fe6..9b815137fb2c 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -1138,6 +1138,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping, int ret, needed_blocks; handle_t *handle; int retries = 0; + fgf_t fgp_flags; struct folio *folio; pgoff_t index; unsigned from, to; @@ -1164,6 +1165,15 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping, return 0; } + /* + * Set FGP_WRITEBEGIN, and FGP_UNCACHED if foliop is marked as + * uncached. That's how generic_perform_write() informs us that this + * is an uncached write. + */ + fgp_flags = FGP_WRITEBEGIN; + if (foliop_is_uncached(foliop)) + fgp_flags |= FGP_UNCACHED; + /* * __filemap_get_folio() can take a long time if the * system is thrashing due to memory pressure, or if the folio @@ -1172,7 +1182,7 @@ static int ext4_write_begin(struct file *file, struct address_space *mapping, * the folio (if needed) without using GFP_NOFS. */ retry_grab: - folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN, + folio = __filemap_get_folio(mapping, index, fgp_flags, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) return PTR_ERR(folio); @@ -2903,6 +2913,7 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping, struct folio *folio; pgoff_t index; struct inode *inode = mapping->host; + fgf_t fgp_flags; if (unlikely(ext4_forced_shutdown(inode->i_sb))) return -EIO; @@ -2926,8 +2937,11 @@ static int ext4_da_write_begin(struct file *file, struct address_space *mapping, return 0; } + fgp_flags = FGP_WRITEBEGIN; + if (foliop_is_uncached(foliop)) + fgp_flags |= FGP_UNCACHED; retry: - folio = __filemap_get_folio(mapping, index, FGP_WRITEBEGIN, + folio = __filemap_get_folio(mapping, index, fgp_flags, mapping_gfp_mask(mapping)); if (IS_ERR(folio)) return PTR_ERR(folio); diff --git a/fs/ext4/page-io.c b/fs/ext4/page-io.c index ad5543866d21..10447c3c4ff1 100644 --- a/fs/ext4/page-io.c +++ b/fs/ext4/page-io.c @@ -226,8 +226,6 @@ static void ext4_add_complete_io(ext4_io_end_t *io_end) unsigned long flags; /* Only reserved conversions from writeback should enter here */ - WARN_ON(!(io_end->flag & EXT4_IO_END_UNWRITTEN)); - WARN_ON(!io_end->handle && sbi->s_journal); spin_lock_irqsave(&ei->i_completed_io_lock, flags); wq = sbi->rsv_conversion_wq; if (list_empty(&ei->i_rsv_conversion_list)) @@ -252,7 +250,7 @@ static int ext4_do_flush_completed_IO(struct inode *inode, while (!list_empty(&unwritten)) { io_end = list_entry(unwritten.next, ext4_io_end_t, list); - BUG_ON(!(io_end->flag & EXT4_IO_END_UNWRITTEN)); + BUG_ON(!(io_end->flag & (EXT4_IO_END_UNWRITTEN|EXT4_IO_UNCACHED))); list_del_init(&io_end->list); err = ext4_end_io_end(io_end); @@ -287,14 +285,15 @@ ext4_io_end_t *ext4_init_io_end(struct inode *inode, gfp_t flags) void ext4_put_io_end_defer(ext4_io_end_t *io_end) { - if (refcount_dec_and_test(&io_end->count)) { - if (!(io_end->flag & EXT4_IO_END_UNWRITTEN) || - list_empty(&io_end->list_vec)) { - ext4_release_io_end(io_end); - return; - } - ext4_add_complete_io(io_end); + if (!refcount_dec_and_test(&io_end->count)) + return; + if ((!(io_end->flag & EXT4_IO_END_UNWRITTEN) || + list_empty(&io_end->list_vec)) && + !(io_end->flag & EXT4_IO_UNCACHED)) { + ext4_release_io_end(io_end); + return; } + ext4_add_complete_io(io_end); } int ext4_put_io_end(ext4_io_end_t *io_end) @@ -348,7 +347,7 @@ static void ext4_end_bio(struct bio *bio) blk_status_to_errno(bio->bi_status)); } - if (io_end->flag & EXT4_IO_END_UNWRITTEN) { + if (io_end->flag & (EXT4_IO_END_UNWRITTEN|EXT4_IO_UNCACHED)) { /* * Link bio into list hanging from io_end. We have to do it * atomically as bio completions can be racing against each @@ -417,8 +416,13 @@ static void io_submit_add_bh(struct ext4_io_submit *io, submit_and_retry: ext4_io_submit(io); } - if (io->io_bio == NULL) + if (io->io_bio == NULL) { io_submit_init_bio(io, bh); + if (folio_test_uncached(folio)) { + ext4_io_end_t *io_end = io->io_bio->bi_private; + io_end->flag |= EXT4_IO_UNCACHED; + } + } if (!bio_add_folio(io->io_bio, io_folio, bh->b_size, bh_offset(bh))) goto submit_and_retry; wbc_account_cgroup_owner(io->io_wbc, &folio->page, bh->b_size); From patchwork Thu Nov 14 15:25:18 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875205 Received: from mail-oa1-f53.google.com (mail-oa1-f53.google.com [209.85.160.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 770F61AF0C4 for ; Thu, 14 Nov 2024 15:28:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598104; cv=none; b=OI6p9A30EUtIOIyz6O2wFGCL4LXK1ytntlv0hxH2xFsd8wA8EEn2+QOnhF62Un3GwcCps5NCDFMPtxGvR/Xcuo9MGgZgzTeGSiCVdn5lfopbl6N/l6JStMuNCX0RhGQwj+l6gui/Tspn8mYQd7X1ucqxaF4bDSvpBj0oLtJofrQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598104; c=relaxed/simple; bh=4XXew23gq3OoMMOZOjmCtGgMeRxntUp+Fg61kUdG93Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=NFGD3kocaTobHDiD1nFCM+tTH6VGI0b2McX8lCwHzR70vY8tM5peFirRJ2TlB8EtmDjkGXfQKuclNl5rBKqVqS0TqFSX18++TjFZQNfNEKkSrpnwIGW0lK6U7nfoWATgCllMvLlcoI62ly/wHhwLl3y2vD3SWiZ9M9WjApR1aNE= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=rR9DGWuk; arc=none smtp.client-ip=209.85.160.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="rR9DGWuk" Received: by mail-oa1-f53.google.com with SMTP id 586e51a60fabf-295cee3a962so418719fac.3 for ; Thu, 14 Nov 2024 07:28:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598101; x=1732202901; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=JnydZ5ahnoTDPrjopxRhuD9AlpgfRxt1ZJ34FZKhxI0=; b=rR9DGWuknEzGNqxGZH8Mcf0el13yrZ8xjZUBdN2I07qYtdGuKwk4Swbs+uU4K5pFuD GDj+muRTfHfnOkFG+a5VLKPByoEelfPv4ch4QaA1TOviM3IXSVId0fAd4+foDltKO9HR v/izHstC3jCHfbc0gbMDAeOBCBaMaXBZ+CD987zHFcNm2f9qJxhXsNYUIizvt+oWGjxC OKiJGwLBqrOfmwAPv9tDY82rPiK1HjH/PPkL45+Wl6MOPvAdxuI/rnnTGVDgtbGT2iov F0fSvTrNi3gfhu44gZrTR4SRFx/a7uwtINZpmAvRau03e8esVC7cXBSPuuAahtXvS0+u DCkQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598101; x=1732202901; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=JnydZ5ahnoTDPrjopxRhuD9AlpgfRxt1ZJ34FZKhxI0=; b=tj22V2s8FuGhqCQzo7AmyCCu6zYfTyJ/0FS1ctZ9qdzCzn3Kzq0qSMQNFxu0AMRNJQ pBJ/wt2q4CXwlSt9O1y5+VQYiwpXd+u/kNAiHT6suLggQluBkJAkwySnLkysyytI2J2H ri365WeyTIy0xnc2/1E6yooWWSjUVP1bvC96uoQ3ARbO6dFxjbH96s3hW3zB5ik0gNQe 51TjxQliNJHt7Ne94vtMDdqyBSYubEYzbcEJqzKzvaUOXsL0U1wa/DYQ4JXZXjvKyu5r h3PXpVyl+Y3XKVw4UBz7QmMEekSI/qO5UxEPQ/MeN6w7XXVh0roSeUBlj7artW7g3D5d DNUA== X-Forwarded-Encrypted: i=1; AJvYcCUUmyd7Nx8ucfwzW7qkyS6Hhan3FPZCq1HOHgzURgLfoHJ/9yn2aNBBrp0ogQVSAwXfKNqmGfF5ezYm79Jg@vger.kernel.org X-Gm-Message-State: AOJu0Yx34H1pvbzHh8lDIOW6Fnrs9unGVuNh1tgd0513GD0CouycRohl vxXP8d7M9zFNksH44nCG/S/b6aRbzUFKyo1RO7sETue8GRraOf0bNZUbAXBjYNA= X-Google-Smtp-Source: AGHT+IGNabSsCKonQoe8b+3leqigUS1MxYLou4wkGKabmMMQ9sy0zqR8b1138Alaw/M4bLorq8cnFg== X-Received: by 2002:a05:6870:e2d5:b0:288:2fb6:a385 with SMTP id 586e51a60fabf-295600dc985mr21855599fac.20.1731598101550; Thu, 14 Nov 2024 07:28:21 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.20 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:20 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 14/17] iomap: make buffered writes work with RWF_UNCACHED Date: Thu, 14 Nov 2024 08:25:18 -0700 Message-ID: <20241114152743.2381672-16-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Add iomap buffered write support for RWF_UNCACHED. If RWF_UNCACHED is set for a write, mark the folios being written as uncached. Then writeback completion will drop the pages. The write_iter handler simply kicks off writeback for the pages, and writeback completion will take care of the rest. This still needs the user of the iomap buffered write helpers to call iocb_uncached_write() upon successful issue of the writes. Signed-off-by: Jens Axboe --- fs/iomap/buffered-io.c | 15 +++++++++++++-- include/linux/iomap.h | 8 +++++++- 2 files changed, 20 insertions(+), 3 deletions(-) diff --git a/fs/iomap/buffered-io.c b/fs/iomap/buffered-io.c index ef0b68bccbb6..2f2a5db04a68 100644 --- a/fs/iomap/buffered-io.c +++ b/fs/iomap/buffered-io.c @@ -603,6 +603,8 @@ struct folio *iomap_get_folio(struct iomap_iter *iter, loff_t pos, size_t len) if (iter->flags & IOMAP_NOWAIT) fgp |= FGP_NOWAIT; + if (iter->flags & IOMAP_UNCACHED) + fgp |= FGP_UNCACHED; fgp |= fgf_set_order(len); return __filemap_get_folio(iter->inode->i_mapping, pos >> PAGE_SHIFT, @@ -1023,8 +1025,9 @@ ssize_t iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i, const struct iomap_ops *ops, void *private) { + struct address_space *mapping = iocb->ki_filp->f_mapping; struct iomap_iter iter = { - .inode = iocb->ki_filp->f_mapping->host, + .inode = mapping->host, .pos = iocb->ki_pos, .len = iov_iter_count(i), .flags = IOMAP_WRITE, @@ -1034,9 +1037,14 @@ iomap_file_buffered_write(struct kiocb *iocb, struct iov_iter *i, if (iocb->ki_flags & IOCB_NOWAIT) iter.flags |= IOMAP_NOWAIT; + if (iocb->ki_flags & IOCB_UNCACHED) + iter.flags |= IOMAP_UNCACHED; - while ((ret = iomap_iter(&iter, ops)) > 0) + while ((ret = iomap_iter(&iter, ops)) > 0) { + if (iocb->ki_flags & IOCB_UNCACHED) + iter.iomap.flags |= IOMAP_F_UNCACHED; iter.processed = iomap_write_iter(&iter, i); + } if (unlikely(iter.pos == iocb->ki_pos)) return ret; @@ -1770,6 +1778,9 @@ static int iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, size_t poff = offset_in_folio(folio, pos); int error; + if (folio_test_uncached(folio)) + wpc->iomap.flags |= IOMAP_F_UNCACHED; + if (!wpc->ioend || !iomap_can_add_to_ioend(wpc, pos)) { new_ioend: error = iomap_submit_ioend(wpc, 0); diff --git a/include/linux/iomap.h b/include/linux/iomap.h index f61407e3b121..0a88043676f2 100644 --- a/include/linux/iomap.h +++ b/include/linux/iomap.h @@ -74,9 +74,14 @@ struct vm_fault; * IOMAP_F_STALE indicates that the iomap is not valid any longer and the file * range it covers needs to be remapped by the high level before the operation * can proceed. + * + * IOMAP_F_UNCACHED is set to indicate that writes to the page cache (and + * hence writeback) will result in folios being evicted as soon as the + * updated bytes are written back to the storage. */ #define IOMAP_F_SIZE_CHANGED (1U << 8) #define IOMAP_F_STALE (1U << 9) +#define IOMAP_F_UNCACHED (1U << 10) /* * Flags from 0x1000 up are for file system specific usage: @@ -173,8 +178,9 @@ struct iomap_folio_ops { #define IOMAP_NOWAIT (1 << 5) /* do not block */ #define IOMAP_OVERWRITE_ONLY (1 << 6) /* only pure overwrites allowed */ #define IOMAP_UNSHARE (1 << 7) /* unshare_file_range */ +#define IOMAP_UNCACHED (1 << 8) /* uncached IO */ #ifdef CONFIG_FS_DAX -#define IOMAP_DAX (1 << 8) /* DAX mapping */ +#define IOMAP_DAX (1 << 9) /* DAX mapping */ #else #define IOMAP_DAX 0 #endif /* CONFIG_FS_DAX */ From patchwork Thu Nov 14 15:25:19 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875206 Received: from mail-oo1-f50.google.com (mail-oo1-f50.google.com [209.85.161.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 36E211B0F17 for ; Thu, 14 Nov 2024 15:28:23 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.50 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598106; cv=none; b=ihJqks8aazLkgsyPlzlX3dnK4F1PiR3gj3uy9ZoBhf9nVgdlDsdlU1GPY3++8Ew7v7nEThwuKgmmH7+ortmEGy3VNNc5A94FCyn+qhb2nCreBNdULPLDEA564yt6JAQj+H0fkRSacl0YMQM62W5NBSeOh15/O0RjdA8pEqP0HDM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598106; c=relaxed/simple; bh=JN6ziuaz3h3w93syhui0sSuROUWYBUO3x3xlg3f4CsA=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bW6uoHgfaEI+kxYYIz1XDT71fXNc+xnjb9XucYO9za0TLBwGonnc4Wd7IEx4+dL6lmef+XSriR4qA/+tgqYeQW4co14n5B1IXhLYtQPRNXeGoJb/auox09NKyZVROlBQ+omh2DQakS0g0qxz1uk16sQrpeINVOWjjdDZRAZRAOs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=WoVpXnI4; arc=none smtp.client-ip=209.85.161.50 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="WoVpXnI4" Received: by mail-oo1-f50.google.com with SMTP id 006d021491bc7-5eb9ee4f14cso355467eaf.1 for ; Thu, 14 Nov 2024 07:28:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598103; x=1732202903; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=LL88Qd2tMlsmBvcuEyGfQPoq7EBSjG4mmD1oB3CTmL4=; b=WoVpXnI4R1Ju0TIEnacr+++McpiDpZpH9Q7rsPYH9SpUn9vMZESA2hvlpzAFb9slK5 kHIWPHr2fDItNhMzgknehqNQK8tLhPA9JEvW6hM73LhaRRy6P1wbF/oxMPPvvikdQ/Zo 3sMPrcD5h4Ji5j8LgRvmBHgma4Qs8y09MMpZ9DnC2FgWelnCb80ECIqLDA7SbF6FwbPl 5mw+If/NSXb+vofSMxXaNJ078D5LfKDV2JciLksbXBrH3M8HL0sYiVrD35w7uu0bImjQ QiWj0Z+1VD54U37NAz28/FTSm3TmZ4IaXCvolffB+k3C6WtLklSP3aDYWTWP1lxSYnA9 eOgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598103; x=1732202903; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=LL88Qd2tMlsmBvcuEyGfQPoq7EBSjG4mmD1oB3CTmL4=; b=tNoNSO6rAkrd59IGxzd3qFVuy+IPStVFy6U7Mgupj4O6ygYk1/7c/RAQNlGD5S3Isg LLYNgOL4WnBT4q+gBB0ojjg461bU9eKqlytQMlddtrk4HZa3DKshQwf4C811yba4qS0M xHp2chJyxR4LJI3OTRS/4NtL9svFNoSLXgOZIORoYxvwb8o9ImHm+Pc6vxvx/e440BO1 +P80RrtDPzoAcRNHzF9e7eQP/43x/giQSHGTvsRyKuX2iy7jDpI5OFOg/l6A+F9dNghq fTZfd3Zv/ppjPMukaa3fHsPQKXgpgtNmCt7UZe+dLkoYBhXvzqze31p7zZ/ffxyR0UVy Zv4Q== X-Forwarded-Encrypted: i=1; AJvYcCXeUrdBG0UY9KgGS0N98FdRYvBvgFxoKGdpgLiOx9ZDDsIEyZ/7lI4Z1QW6GBZEaYRxIQRUOkHyZ+MRixii@vger.kernel.org X-Gm-Message-State: AOJu0YxJN1VQHwumrfgKkHkhQrlbRfVy1fxNy8MjmTE0Vid3LbKPFPww +G59j9QvL7b94BE6Ng5efBr/o8c3Xze+H64uZq0watk2FlPYJUP6wAHwlOH+TZ0= X-Google-Smtp-Source: AGHT+IGxBOidvnNn2woru7d3dY6RKyVHYMtr7QGUJGpSzdzZfL3L6KYNggyKAJszR8sQZicyWvWugQ== X-Received: by 2002:a4a:e90e:0:b0:5eb:88a5:20e8 with SMTP id 006d021491bc7-5ee9ecc2c38mr2597677eaf.1.1731598103136; Thu, 14 Nov 2024 07:28:23 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:22 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 15/17] xfs: punt uncached write completions to the completion wq Date: Thu, 14 Nov 2024 08:25:19 -0700 Message-ID: <20241114152743.2381672-17-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 They need non-irq context guaranteed, to be able to prune ranges from the page cache. Treat them like unwritten extents and punt them to the completion workqueue. Signed-off-by: Jens Axboe --- fs/xfs/xfs_aops.c | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/fs/xfs/xfs_aops.c b/fs/xfs/xfs_aops.c index 559a3a577097..c86fc2b8f344 100644 --- a/fs/xfs/xfs_aops.c +++ b/fs/xfs/xfs_aops.c @@ -416,9 +416,12 @@ xfs_prepare_ioend( memalloc_nofs_restore(nofs_flag); - /* send ioends that might require a transaction to the completion wq */ + /* + * Send ioends that might require a transaction or need blocking + * context to the completion wq + */ if (xfs_ioend_is_append(ioend) || ioend->io_type == IOMAP_UNWRITTEN || - (ioend->io_flags & IOMAP_F_SHARED)) + (ioend->io_flags & (IOMAP_F_SHARED|IOMAP_F_UNCACHED))) ioend->io_bio.bi_end_io = xfs_end_bio; return status; } From patchwork Thu Nov 14 15:25:20 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875207 Received: from mail-oo1-f54.google.com (mail-oo1-f54.google.com [209.85.161.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6483E1B2183 for ; Thu, 14 Nov 2024 15:28:25 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.54 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598107; cv=none; b=ppy7xX5FiODDfR/CGlpMLE03u4H8Mo5kifyT4qSjSEXtlPX1yYNyIWo428RvdgGfZRaUYktTzzP/PQU5usAMNwSMR4x68OGr8LHI9qdnXGf5uMA4bOXg+Bmsk8NJ/G0I6jI7Gz6AlqjPBENbh1TYr3vqykrVFadgDvq6mpePmR4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598107; c=relaxed/simple; bh=EWturTTFhXmkkEY1vgH0sCBOxeoOqbPcFCX1D34ye48=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=GkIeApWHlD8csUWchm5OOZfudAKzjvWLybzY8Nu+j6athPjUEl8iVvdUZB6wl1ekHAvVkW4y/wXCVcjUerVCba3P8Llsfl64xtd6Q4m9aud/PZlDM+qeehJx6sVaaoMANDOlUxoG2f6K0DtlIuFjy6Ij9+KvvY/G6HFlO08opZA= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=Vqv0OtCw; arc=none smtp.client-ip=209.85.161.54 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="Vqv0OtCw" Received: by mail-oo1-f54.google.com with SMTP id 006d021491bc7-5ee9dbf1b47so432194eaf.1 for ; Thu, 14 Nov 2024 07:28:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598104; x=1732202904; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=NyBHBKICHAilzWbXxQX4w3ArnCN9Xkn7LoXdkHxZ+NE=; b=Vqv0OtCwgRDqsZtuQe08cRI5AEU4os8/ta0n4eX/9ZzESqQr69CLPRaiNzfcALUAMb aCHuZLxROp6xGe2OFYherg9GVvtPc4ZPytYRlb7ZAU1YXLo73tMCG8+XXwbnEdRC2apn F7FVPaW6JyU6pWRUOovA9Wsor71N2l3XfSmtTGH9Wlaov6WWbD8BrbVgtJutLcEn+kS8 iffYTdAD3JeLbC6yYgcbX+1+CWmxard7f0dv66gi42wzXTfeawuVkxuMUFVygJRVBlWy WrBfOK2BNxRfRDtbZujF1cDTeAzQWfWRinS1A8GeBXJY6qE0+I6Sx52yGTaII0XUT6rW n7uQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598104; x=1732202904; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=NyBHBKICHAilzWbXxQX4w3ArnCN9Xkn7LoXdkHxZ+NE=; b=MFpLwurnGugr2JZEpH1+00oaerKNyOmuqxlFlvarbhJo+0fYp8EHlTu5YskYB1RZIf yZVpKO6rGpzrcWHU/fr0+gLr0AXOL+iCl/YTtDZdBHQbk9NHppwIqHWhPbxkVQvHqAjD Vzfhmb2PR/8JuFLkdvCx0FeAtmNpfFa+9Lei5Up1EDqqjmVdHDSs+k+vKEA29vgTmSEa jD9t5kF2zxaKCIf7+liF5S1GZnZN6cCYmZfQmz9eY6pTFuCzHpRlLwPx+e/1AHVy8sxX RlUVDLA7R1k7oLGmd9FKr2/2qBVH5Qd+TKZmcAKU9mj5y7V25G75ymevy0X/KbzIsRif 61+w== X-Forwarded-Encrypted: i=1; AJvYcCWqAclPJeWx9B29Wmu+BNXqvez5M6oUvjMh6IZ0Xss070LsedM5E/lafZJXTk/6R0gFUy3EvY4tiPJKAsmX@vger.kernel.org X-Gm-Message-State: AOJu0YwjjFNGdhYocVT6uQ3wYsHBSl16GtEnn1X1x7qPv//lTbc56qIL RgwOWx6ZZoLTZf9PcsmK44UUaFK/nu61gBwNnfD1WC21T0hdSLimrst/L1SLHAfo4F6gzFo8PSb vj2M= X-Google-Smtp-Source: AGHT+IGTccgjkWiSk1+iFGbOjrOkQSSEn+5BvJmTbBLfatp4glFbzMa7kfcy1vSIijdABdXFBUl4NQ== X-Received: by 2002:a05:6830:7101:b0:718:8ce4:6912 with SMTP id 46e09a7af769-71a6af419f8mr3092006a34.14.1731598104601; Thu, 14 Nov 2024 07:28:24 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.23 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:23 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 16/17] xfs: flag as supporting FOP_UNCACHED Date: Thu, 14 Nov 2024 08:25:20 -0700 Message-ID: <20241114152743.2381672-18-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Read side was already fully supported, and with the write side appropriately punted to the worker queue, all that's needed now is setting FOP_UNCACHED in the file_operations structure to enable full support for read and write uncached IO. Signed-off-by: Jens Axboe --- fs/xfs/xfs_file.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/fs/xfs/xfs_file.c b/fs/xfs/xfs_file.c index b19916b11fd5..4fe593896bc5 100644 --- a/fs/xfs/xfs_file.c +++ b/fs/xfs/xfs_file.c @@ -1595,7 +1595,8 @@ const struct file_operations xfs_file_operations = { .fadvise = xfs_file_fadvise, .remap_file_range = xfs_file_remap_range, .fop_flags = FOP_MMAP_SYNC | FOP_BUFFER_RASYNC | - FOP_BUFFER_WASYNC | FOP_DIO_PARALLEL_WRITE, + FOP_BUFFER_WASYNC | FOP_DIO_PARALLEL_WRITE | + FOP_UNCACHED, }; const struct file_operations xfs_dir_file_operations = { From patchwork Thu Nov 14 15:25:21 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jens Axboe X-Patchwork-Id: 13875208 Received: from mail-oo1-f49.google.com (mail-oo1-f49.google.com [209.85.161.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id A7E291B393E for ; Thu, 14 Nov 2024 15:28:27 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.161.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598109; cv=none; b=tGHcxnQzxFFhNqSIRmgaCxCfKA1fOte85o1gS8iV/UFAloKiohI6g+STssm6xf9ZB+06EiSF45aaWRTS3Da1kb1IgyTNREFBj9srcC1i4LgNgJCeJNRBrKYilF/V5IAviKtzRiM/IL/RPdclINHOQtfsDlNPzikJGcyc/ky0fjM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1731598109; c=relaxed/simple; bh=KvrrsZd4o42w6FFEWFQ1mvoiCpBhCV2tMq6FGu3bD6I=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=ZSqokaDkCNLdEjx9bpwy/581be/NUzosVQAfWe5LubyFrP+sc+Xj/V5S5I+ZAFkaoMIxf+dJ1dO9C1dEzc6vDPenSr6VEO78jL9iZTocoumBphRyv72MdaWVQXT57SYbOy2VAka4z0Sdal8+CUJ12nfWLOlG03vxNPvj3Ji/wME= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk; spf=pass smtp.mailfrom=kernel.dk; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b=ryPp7Zo/; arc=none smtp.client-ip=209.85.161.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=kernel.dk Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=kernel.dk Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=kernel-dk.20230601.gappssmtp.com header.i=@kernel-dk.20230601.gappssmtp.com header.b="ryPp7Zo/" Received: by mail-oo1-f49.google.com with SMTP id 006d021491bc7-5ebc0dbc566so315556eaf.3 for ; Thu, 14 Nov 2024 07:28:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kernel-dk.20230601.gappssmtp.com; s=20230601; t=1731598107; x=1732202907; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jsUz6pEpY+ox9az2F5q2xnny+koVz5fK77xTMpol5Tk=; b=ryPp7Zo/FFxNM8nzscrCmwbmpbBbx3HBf2MkZTBwejmXhORSfWY2POIJp/oJbCIr82 XZcbAoorLW7+awt8LjNlVcKUFt3jjTlIhgnXQ1d9LrNPTFv7v6j34vob3JjxyObeceUx jB7EjJE0ChVaIsXdSotebhjSYyNyCxhcHhlU9e3bgaBw0bzfjZ/AHs0aSRIcvsI31WV6 laM6ovcE4vCNzNS+kLK4+HI8UnYZN55+tSd085le9TNnKfZKjVgVuUJ7iLZyt89BvmXd 4SwEMBpWo0POPcqWo/OjRLDn9rCM4YU3NZHJBj/zjYbUHsKDAgZqw1NmyHyPPpBbgaZp x8jg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731598107; x=1732202907; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jsUz6pEpY+ox9az2F5q2xnny+koVz5fK77xTMpol5Tk=; b=sXZAk5WRZh6oKqudzQJHThko8+slTas0a0UFJrjItZURg70sr3UvPKGkuNE6rABB2a o0vnwTEoxA5MsN4i9ucG8I3Yxq9TrRGy7WgH173WWi81EciqbEd2zPs0gkvxtWMFImNa LoalL0gDHI24QNfsHcvTJwcUm9Qkt0s2Rox8NBdqB56LRJyJgGcxWR6Td2DMbcTQLSkW XVRaMDt8vhbfxVGbxcG4JRHIcCHwblvQFjI7GDRenKwSwvyjZotyERAfnLG3+84Thz3V dlricpERhsdmThnVEk9mCeEDGnqUdANbtvldgWMPFpmUgdtV5of3aLXRefkiMjy+RRcR T3DA== X-Forwarded-Encrypted: i=1; AJvYcCXBp7Buwsd4aK3IjbNF/6pCE67vMVd0nIInY299XbsFPYqo+8azQm4zoR2wjpPg750keC0BnqenZrfRMStE@vger.kernel.org X-Gm-Message-State: AOJu0YyPMu4i/GseARlQMhzTQeHFgrI0snpfkYU+UeTK2415n34oQ5Lj ucmdzTxmSBoCoqjFO0tg4olYHwUkFgbq36WeV1LyKJH0RRwUu2Itms40IWjI+8c= X-Google-Smtp-Source: AGHT+IFc1nzkbPZXQe4aKtw+LtP2VNQBWJFtVjU1qLyGyhTsHecVf/ilzi9Wv0164LJAdz5N+SGBoQ== X-Received: by 2002:a05:6820:1b08:b0:5ee:a5b:d172 with SMTP id 006d021491bc7-5ee57c530cdmr17735794eaf.5.1731598106027; Thu, 14 Nov 2024 07:28:26 -0800 (PST) Received: from localhost.localdomain ([96.43.243.2]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-5eea026eb41sm368250eaf.39.2024.11.14.07.28.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 14 Nov 2024 07:28:25 -0800 (PST) From: Jens Axboe To: linux-mm@kvack.org, linux-fsdevel@vger.kernel.org Cc: hannes@cmpxchg.org, clm@meta.com, linux-kernel@vger.kernel.org, willy@infradead.org, kirill@shutemov.name, linux-btrfs@vger.kernel.org, linux-ext4@vger.kernel.org, linux-xfs@vger.kernel.org, bfoster@redhat.com, Jens Axboe Subject: [PATCH 17/17] btrfs: add support for uncached writes Date: Thu, 14 Nov 2024 08:25:21 -0700 Message-ID: <20241114152743.2381672-19-axboe@kernel.dk> X-Mailer: git-send-email 2.45.2 In-Reply-To: <20241114152743.2381672-2-axboe@kernel.dk> References: <20241114152743.2381672-2-axboe@kernel.dk> Precedence: bulk X-Mailing-List: linux-fsdevel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 The read side is already covered as btrfs uses the generic filemap helpers. For writes, just pass in FGP_UNCACHED if uncached IO is being done, then the folios created should be marked appropriately. For IO completion, ensure that writing back folios that are uncached gets punted to one of the btrfs workers, as task context is needed for that. Add an 'uncached_io' member to struct btrfs_bio to manage that. With that, add FOP_UNCACHED to the btrfs file_operations fop_flags structure, enabling use of RWF_UNCACHED. Signed-off-by: Jens Axboe --- fs/btrfs/bio.c | 4 ++-- fs/btrfs/bio.h | 2 ++ fs/btrfs/extent_io.c | 8 +++++++- fs/btrfs/file.c | 9 ++++++--- 4 files changed, 17 insertions(+), 6 deletions(-) diff --git a/fs/btrfs/bio.c b/fs/btrfs/bio.c index 7e0f9600b80c..253e1a656934 100644 --- a/fs/btrfs/bio.c +++ b/fs/btrfs/bio.c @@ -334,7 +334,7 @@ static void btrfs_end_bio_work(struct work_struct *work) struct btrfs_bio *bbio = container_of(work, struct btrfs_bio, end_io_work); /* Metadata reads are checked and repaired by the submitter. */ - if (is_data_bbio(bbio)) + if (bio_op(&bbio->bio) == REQ_OP_READ && is_data_bbio(bbio)) btrfs_check_read_bio(bbio, bbio->bio.bi_private); else btrfs_bio_end_io(bbio, bbio->bio.bi_status); @@ -351,7 +351,7 @@ static void btrfs_simple_end_io(struct bio *bio) if (bio->bi_status) btrfs_log_dev_io_error(bio, dev); - if (bio_op(bio) == REQ_OP_READ) { + if (bio_op(bio) == REQ_OP_READ || bbio->uncached_io) { INIT_WORK(&bbio->end_io_work, btrfs_end_bio_work); queue_work(btrfs_end_io_wq(fs_info, bio), &bbio->end_io_work); } else { diff --git a/fs/btrfs/bio.h b/fs/btrfs/bio.h index e2fe16074ad6..39b98326c98f 100644 --- a/fs/btrfs/bio.h +++ b/fs/btrfs/bio.h @@ -82,6 +82,8 @@ struct btrfs_bio { /* Save the first error status of split bio. */ blk_status_t status; + bool uncached_io; + /* * This member must come last, bio_alloc_bioset will allocate enough * bytes for entire btrfs_bio but relies on bio being last. diff --git a/fs/btrfs/extent_io.c b/fs/btrfs/extent_io.c index 872cca54cc6c..b97b21178ed7 100644 --- a/fs/btrfs/extent_io.c +++ b/fs/btrfs/extent_io.c @@ -760,8 +760,11 @@ static void submit_extent_folio(struct btrfs_bio_ctrl *bio_ctrl, ASSERT(bio_ctrl->end_io_func); if (bio_ctrl->bbio && - !btrfs_bio_is_contig(bio_ctrl, folio, disk_bytenr, pg_offset)) + !btrfs_bio_is_contig(bio_ctrl, folio, disk_bytenr, pg_offset)) { + if (folio_test_uncached(folio)) + bio_ctrl->bbio->uncached_io = true; submit_one_bio(bio_ctrl); + } do { u32 len = size; @@ -779,6 +782,9 @@ static void submit_extent_folio(struct btrfs_bio_ctrl *bio_ctrl, len = bio_ctrl->len_to_oe_boundary; } + if (folio_test_uncached(folio)) + bio_ctrl->bbio->uncached_io = true; + if (!bio_add_folio(&bio_ctrl->bbio->bio, folio, len, pg_offset)) { /* bio full: move on to a new one */ submit_one_bio(bio_ctrl); diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 4fb521d91b06..cfee783f4c4d 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -919,7 +919,7 @@ static gfp_t get_prepare_gfp_flags(struct inode *inode, bool nowait) static noinline int prepare_pages(struct inode *inode, struct page **pages, size_t num_pages, loff_t pos, size_t write_bytes, bool force_uptodate, - bool nowait) + bool nowait, bool uncached) { int i; unsigned long index = pos >> PAGE_SHIFT; @@ -928,6 +928,8 @@ static noinline int prepare_pages(struct inode *inode, struct page **pages, int ret = 0; int faili; + if (uncached) + fgp_flags |= FGP_UNCACHED; for (i = 0; i < num_pages; i++) { again: pages[i] = pagecache_get_page(inode->i_mapping, index + i, @@ -1323,7 +1325,8 @@ ssize_t btrfs_buffered_write(struct kiocb *iocb, struct iov_iter *i) * contents of pages from loop to loop */ ret = prepare_pages(inode, pages, num_pages, - pos, write_bytes, force_page_uptodate, false); + pos, write_bytes, force_page_uptodate, + false, iocb->ki_flags & IOCB_UNCACHED); if (ret) { btrfs_delalloc_release_extents(BTRFS_I(inode), reserve_bytes); @@ -3802,7 +3805,7 @@ const struct file_operations btrfs_file_operations = { .compat_ioctl = btrfs_compat_ioctl, #endif .remap_file_range = btrfs_remap_file_range, - .fop_flags = FOP_BUFFER_RASYNC | FOP_BUFFER_WASYNC, + .fop_flags = FOP_BUFFER_RASYNC | FOP_BUFFER_WASYNC | FOP_UNCACHED, }; int btrfs_fdatawrite_range(struct btrfs_inode *inode, loff_t start, loff_t end)