From patchwork Tue Nov 12 17:55:15 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Josef Bacik X-Patchwork-Id: 13872726 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 03AE3D42BBE for ; Tue, 12 Nov 2024 17:56:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 2EB1B6B0085; Tue, 12 Nov 2024 12:56:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 29B556B009C; Tue, 12 Nov 2024 12:56:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 13C656B00F8; Tue, 12 Nov 2024 12:56:31 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id E9E5F6B0085 for ; Tue, 12 Nov 2024 12:56:30 -0500 (EST) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 5B093C045A for ; Tue, 12 Nov 2024 17:56:30 +0000 (UTC) X-FDA: 82778194800.17.A2428C5 Received: from mail-yw1-f175.google.com (mail-yw1-f175.google.com [209.85.128.175]) by imf16.hostedemail.com (Postfix) with ESMTP id E0E84180010 for ; Tue, 12 Nov 2024 17:55:46 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=toxicpanda-com.20230601.gappssmtp.com header.s=20230601 header.b=047+RbFm; spf=none (imf16.hostedemail.com: domain of josef@toxicpanda.com has no SPF policy when checking 209.85.128.175) smtp.mailfrom=josef@toxicpanda.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1731434134; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=g48wmOGZabKsmMEfPaYhGe4dtOd9ozHYTrWY8XD5r2k=; b=2cXQRPFa2efz5ERjeclu2GRQC/0AcUAjGiVWejAIrSQrhOSXNmYhS8tlrcNjjtQGcqk+li O4VQLWrPBoYOY85FXmMM7EJp7+kooA5ZGsbS4gme67IAedPZglokpzHbQoHdrbOhkwqSNa 0BYhC6K7AK0N6oFBxmx9xI5RmTaGGyE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1731434134; a=rsa-sha256; cv=none; b=ku3fsrdxeSMIwBJwAuIO9I4NUuyh1+5ARjpMXzw6og93I9oTaUicjddSwI8T6m6QuKStHh KO2b5vYfnZt9jX2Pemk4Vpeheoby5Gn3xnHBNqXNf6Db7RsoorKqyhWxB4PRgcRQSyaozp xMfxh1GnbQWxFioAhGkbd9QQ8wj+N7c= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=toxicpanda-com.20230601.gappssmtp.com header.s=20230601 header.b=047+RbFm; spf=none (imf16.hostedemail.com: domain of josef@toxicpanda.com has no SPF policy when checking 209.85.128.175) smtp.mailfrom=josef@toxicpanda.com; dmarc=none Received: by mail-yw1-f175.google.com with SMTP id 00721157ae682-6e38ebcc0abso62787507b3.2 for ; Tue, 12 Nov 2024 09:56:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20230601.gappssmtp.com; s=20230601; t=1731434187; x=1732038987; darn=kvack.org; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:from:to:cc:subject:date:message-id:reply-to; bh=g48wmOGZabKsmMEfPaYhGe4dtOd9ozHYTrWY8XD5r2k=; b=047+RbFmehf8C+k1Dm8qdX2WH6jPfE+Ntec48lrp0WYwo8evzMmniboDimDTwfAWdb gJZ5hd9H9qcQwfFBI90OaD11krYD8uIyhe7Nve+PeJSP0NxdMAGEY8flvkrBQsw49qYf PNhx463vAlKn0r4I54mozUKCFTSAxtplxFFG0GQsfToXLSBMY8jWMPKMaiHGGCHt/Hxv kBwNcEfGSfcNHEaLQaGVe+oa3DBGfb/qZ/xhvKTGfREKEtZ83mIP22+jY6Dg4dGayzVt Sm1F6uV9E5GQbVsbR4O/4rJFMoWppK+TF57sA9YeLH27rBKA2ivylkVYNMmBrgF4Mqs3 g+NQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1731434187; x=1732038987; h=content-transfer-encoding:mime-version:message-id:date:subject:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=g48wmOGZabKsmMEfPaYhGe4dtOd9ozHYTrWY8XD5r2k=; b=HRtvWwg54ZR5IXZ62hevOlrbIc6Qlo+IADbL+3/nMGOB5x55lMDYIWx4ItL6aD3l7w byEdX5dOPuWwP1RM6ZRHWs48FyKOi5wfKrWc//JUIlPdZu+RcBfWiuYz2Hn99oaTNAOR ErjKlbYecxUuqfgkOh/tc3ELw8JuC6g2HnMSYAnRUcgb5IX1UYuZU+Sh8nr91gfapWm8 vjPXp76QpwXGS97nHVCYXIAKiGOTsXbvMQ0C7yhNsLTKyu40x+ddzfTj2CRPSp1Fn3gi rY0em43268zU1+AGuOWEL12mmim82z2pI0xQ7QjAkuDgeBlm1VMURgNaXfHkN1s0hndS 0vZw== X-Forwarded-Encrypted: i=1; AJvYcCXVu3ZQPI/ucoJkGOykJbOivma2REzJCdhYUiriI1Ri6U3j5cI5otUaVledjyoTlwx8CR7pJITnbw==@kvack.org X-Gm-Message-State: AOJu0YygY/y+l00iYgmYn35LXAaixh1d+rzqdVVmdxjYd529kSwXikR7 GOHL3+v4bSCSrizlscV3lTVhaXrzMzQSqKKjqMZEtAkjWdIvo2qs00t4UYVPs8Y= X-Google-Smtp-Source: AGHT+IGENyk2mAziIYhZQg41cfzA0HPYUllNcDo1wflc5oTm5yw6vvf+phT1j0QUEo4365Asvd+k2g== X-Received: by 2002:a05:690c:6f92:b0:6ea:8556:1cd6 with SMTP id 00721157ae682-6eaddf84222mr180166387b3.30.1731434187543; Tue, 12 Nov 2024 09:56:27 -0800 (PST) Received: from localhost (syn-076-182-020-124.res.spectrum.com. [76.182.20.124]) by smtp.gmail.com with ESMTPSA id 00721157ae682-6eaceb7b34fsm26786047b3.115.2024.11.12.09.56.26 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 12 Nov 2024 09:56:26 -0800 (PST) From: Josef Bacik To: kernel-team@fb.com, linux-fsdevel@vger.kernel.org, jack@suse.cz, amir73il@gmail.com, brauner@kernel.org, torvalds@linux-foundation.org, linux-xfs@vger.kernel.org, linux-btrfs@vger.kernel.org, linux-mm@kvack.org, linux-ext4@vger.kernel.org Subject: [PATCH v7 00/18] fanotify: add pre-content hooks Date: Tue, 12 Nov 2024 12:55:15 -0500 Message-ID: X-Mailer: git-send-email 2.43.0 MIME-Version: 1.0 X-Stat-Signature: pq5d78sok7sejfdjcoj7e8rqu395id64 X-Rspam-User: X-Rspamd-Queue-Id: E0E84180010 X-Rspamd-Server: rspam02 X-HE-Tag: 1731434146-355458 X-HE-Meta: U2FsdGVkX18jbBPhlQbHbjNcttB9t1Rj9f+XJp1uQ3wHDCz+JNOIF0bQfQjAFP0YCqk16cWQaoElQIiSDAsLMhIQ1tq5OWvpRa+Y6ne6XKeA1dW2t3sSRHl7wtEQhXNqrUC0WfKerJ9yrtSa1rsiBoHAg5lAKIjra7OtMttYbZYqcYCEkfsRhjbt8k/Zo0RZJC7pzwqMwKX7b93K0gamgI/0EmKt59q69jpIspiQk4K99eRQBnE4yDH1HZF9dZb9uVl+Xs8fLnYsk875OFn7NLtq5jX5G73IXbWF+OTdrfnk6SzLXY9S6yGTBkB85hDgl1erqPJ5PbFpzQ/jARzxIjXqaW2tW1fFM0nVm8LF7AXmEbE8U80Jumwn5EMqsCS9lfmcJBwt3alCX5sLvIARjF056ZHGBYEEiizVizu+XsJp0C4Qjoi5fOv8sLbvnrg0ZJh228qJRxhdgNxDlRMpSh1udTyJpsIgeZ+VDG/FA6vBxFP13AbjExzMD4y+oqh7mDPkQ/mVUXMWZFbhVY1u3ynesWPsnshpYz6Bcf+tTZ8hUOupzp1J+LRHiAjcvp7MuVQeKIGK7QEGFRpqVndEhyUtyUuzjcc8IO58WzdkYoPbVQAgZc8cUeszWrZPuJPRDtzjL2QobWWtuwUHo4OtcbVfBagefN/nXZg+4PUM5bRXGI/gdiuJG3DIZN0i5QKwPPwDZ60t0oYBag+HRLYDs9FdZSFRzbdYVkBeLouVjHAWEU3lsC8hfpFVbokhOYdzgxQ3U1zJnWP+wU2curVRHm53Mx9FluBf98SFUoVlLfhMEAY6vDwkTmtBKP4QPHSBm6fbEaNMr3gsQII+4Y7MNac8r+OkmLpUdzZgm8QrHRjWIipSbGJI6b1FWn8ZRc81gt8Xo4b86FfMl5ORDCoFMeOGAvHrkU3p5/QNse8AEy9d6Y0xleG+n7wc8TcLAXgitECVFEZcmdQlQ0O6qgD 0Wc1DRkx ncbcmgvnsxSIpwEui45KQ6ZsmvXui1iIeF0GNGkJkJEq7/Z+gielhBtvEIghC5avVo+vzpf7CMPIqn0I7YhTL0T0KG6WlQHUL8Y7H4oAJdy/vRK7QYc8WgYGEvtNFw/5Eyilui48awShQ2sl3O2HLD8Tctb31Wk8V1TQNnexMSHNGZcnxEc9fkR8leRPx/bc9s1J2CY6aRG4D25gSZoOGFZiFGENmYJ1ERUp0WarAidC2OHozmSgyN+Zk4oSJCQObBM6FEoNMwpw9jOGJ++fzgn/M/QBg1c65pcQUAhI0ptW+77YPUHRa5UvtViSGyNBvUF0ObVgNRLoYntDTEOCFGk0BUvrT+Atr1lR4rLZ26IzqWsjyHepnkV4uktsSTcWESp81PgmJ8bONdnhNX0iAAXIsovox14LFTJNc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: v6: https://lore.kernel.org/linux-fsdevel/cover.1731355931.git.josef@toxicpanda.com/ v5: https://lore.kernel.org/linux-fsdevel/cover.1725481503.git.josef@toxicpanda.com/ v4: https://lore.kernel.org/linux-fsdevel/cover.1723670362.git.josef@toxicpanda.com/ v3: https://lore.kernel.org/linux-fsdevel/cover.1723228772.git.josef@toxicpanda.com/ v2: https://lore.kernel.org/linux-fsdevel/cover.1723144881.git.josef@toxicpanda.com/ v1: https://lore.kernel.org/linux-fsdevel/cover.1721931241.git.josef@toxicpanda.com/ v6->v7: - As per Linus's suggestion, Amir added the file flag FMODE_NOTIFY_PERM that will be set at open time if the file has permission related watches (this is the original malware style permission watches and the new precontent watches). All of the VFS hooks and the page fault hooks use this flag to determine if they should generate a notification to allow for a much cheaper check in the common case. v5->v6: - Linus had problems with this and rejected Jan's PR (https://lore.kernel.org/linux-fsdevel/20240923110348.tbwihs42dxxltabc@quack3/), so I'm respinning this series to address his concerns. Hopefully this is more acceptable. - Change the page fault hooks to happen only in the case where we have to add a page, not where there exists pages already. - Amir added a hook to truncate. - We made the flag per SB instead of per fstype, Amir wanted this because of some potential issues with other file system specific work he's doing. - Dropped the bcachefs patch, there were some concerns that we were doing something wrong, and it's not a huge deal to not have this feature for now. - Unfortunately the xfs write fault path still has to do the page fault hook before we know if we have a page or not, this is because of the locking that's done before we get to the part where we know if we have a page already or not, so that's the path that is still the same from last iteration. - I've re-validated this series with btrfs, xfs, and ext4 to make sure I didn't break anything. v4->v5: - Cleaned up the various "I'll fix it on commit" notes that Jan made since I had to respin the series anyway. - Renamed the filemap pagefault helper for fsnotify per Christians suggestion. - Added a FS_ALLOW_HSM flag per Jan's comments, based on Amir's rough sketch. - Added a patch to disable btrfs defrag on pre-content watched files. - Added a patch to turn on FS_ALLOW_HSM for all the file systems that I tested. - Added two fstests (which will be posted separately) to validate everything, re-validated the series with btrfs, xfs, ext4, and bcachefs to make sure I didn't break anything. v3->v4: - Trying to send a final verson Friday at 5pm before you go on vacation is a recipe for silly mistakes, fixed the xfs handling yet again, per Christoph's review. - Reworked the file system helper so it's handling of fpin was a little less silly, per Chinner's suggestion. - Updated the return values to not or in VM_FAULT_RETRY, as we have a comment in filemap_fault that says if VM_FAULT_ERROR is set we won't have VM_FAULT_RETRY set. v2->v3: - Fix the pagefault path to do MAY_ACCESS instead, updated the perm handler to emit PRE_ACCESS in this case, so we can avoid the extraneous perm event as per Amir's suggestion. - Reworked the exported helper so the per-filesystem changes are much smaller, per Amir's suggestion. - Fixed the screwup for DAX writes per Chinner's suggestion. - Added Christian's reviewed-by's where appropriate. v1->v2: - reworked the page fault logic based on Jan's suggestion and turned it into a helper. - Added 3 patches per-fs where we need to call the fsnotify helper from their ->fault handlers. - Disabled readahead in the case that there's a pre-content watch in place. - Disabled huge faults when there's a pre-content watch in place (entirely because it's untested, theoretically it should be straightforward to do). - Updated the command numbers. - Addressed the random spelling/grammer mistakes that Jan pointed out. - Addressed the other random nits from Jan. --- Original email --- Hello, These are the patches for the bare bones pre-content fanotify support. The majority of this work is Amir's, my contribution to this has solely been around adding the page fault hooks, testing and validating everything. I'm sending it because Amir is traveling a bunch, and I touched it last so I'm going to take all the hate and he can take all the credit. There is a PoC that I've been using to validate this work, you can find the git repo here https://github.com/josefbacik/remote-fetch This consists of 3 different tools. 1. populate. This just creates all the stub files in the directory from the source directory. Just run ./populate ~/linux ~/hsm-linux and it'll recursively create all of the stub files and directories. 2. remote-fetch. This is the actual PoC, you just point it at the source and destination directory and then you can do whatever. ./remote-fetch ~/linux ~/hsm-linux. 3. mmap-validate. This was to validate the pagefault thing, this is likely what will be turned into the selftest with remote-fetch. It creates a file and then you can validate the file matches the right pattern with both normal reads and mmap. Normally I do something like ./mmap-validate create ~/src/foo ./populate ~/src ~/dst ./rmeote-fetch ~/src ~/dst ./mmap-validate validate ~/dst/foo I did a bunch of testing, I also got some performance numbers. I copied a kernel tree, and then did remote-fetch, and then make -j4 Normal real 9m49.709s user 28m11.372s sys 4m57.304s HSM real 10m6.454s user 29m10.517s sys 5m2.617s So ~17 seconds more to build with HSM. I then did a make mrproper on both trees to see the size [root@fedora ~]# du -hs /src/linux 1.6G /src/linux [root@fedora ~]# du -hs dst 125M dst This mirrors the sort of savings we've seen in production. Meta has had these patches (minus the page fault patch) deployed in production for almost a year with our own utility for doing on-demand package fetching. The savings from this has been pretty significant. The page-fault hooks are necessary for the last thing we need, which is on-demand range fetching of executables. Some of our binaries are several gigs large, having the ability to remote fetch them on demand is a huge win for us not only with space savings, but with startup time of containers. There will be tests for this going into LTP once we're satisfied with the patches and they're on their way upstream. Thanks, Josef Amir Goldstein (11): fsnotify: opt-in for permission events at file_open_perm() time fanotify: don't skip extra event info if no info_mode is set fanotify: rename a misnamed constant fanotify: reserve event bit of deprecated FAN_DIR_MODIFY fsnotify: introduce pre-content permission events fsnotify: pass optional file access range in pre-content event fsnotify: generate pre-content permission event on open fsnotify: generate pre-content permission event on truncate fanotify: introduce FAN_PRE_ACCESS permission event fanotify: report file range info with pre-content events fanotify: allow to set errno in FAN_DENY permission response Josef Bacik (7): fanotify: add a helper to check for pre content events fanotify: disable readahead if we have pre-content watches mm: don't allow huge faults for files with pre content watches fsnotify: generate pre-content permission event on page fault xfs: add pre-content fsnotify hook for write faults btrfs: disable defrag on pre-content watched files fs: enable pre-content events on supported file systems fs/btrfs/ioctl.c | 9 +++ fs/btrfs/super.c | 2 +- fs/ext4/super.c | 3 + fs/namei.c | 10 ++- fs/notify/fanotify/fanotify.c | 33 ++++++-- fs/notify/fanotify/fanotify.h | 15 ++++ fs/notify/fanotify/fanotify_user.c | 120 +++++++++++++++++++++++------ fs/notify/fsnotify.c | 14 +++- fs/open.c | 31 +++++--- fs/xfs/xfs_file.c | 4 + fs/xfs/xfs_super.c | 2 +- include/linux/fanotify.h | 19 +++-- include/linux/fs.h | 4 +- include/linux/fsnotify.h | 120 ++++++++++++++++++++++++----- include/linux/fsnotify_backend.h | 71 ++++++++++++++++- include/linux/mm.h | 1 + include/uapi/linux/fanotify.h | 18 +++++ mm/filemap.c | 90 ++++++++++++++++++++++ mm/memory.c | 22 ++++++ mm/readahead.c | 13 ++++ security/selinux/hooks.c | 3 +- 21 files changed, 532 insertions(+), 72 deletions(-)