| Message ID | cover.1550136164.git.osandov@fb.com (mailing list archive) |
|---|---|
| Headers | show |
| Series | Allow setting file birth time with utimensat() | expand |
On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote: > From: Omar Sandoval <osandov@fb.com> > > Hi, > > Since statx was added in 4.11, userspace has had an interface for > reading btime (file creation time), but no way to set it. This RFC patch > series adds support for changing btime with utimensat(). Patch 1 adds > the VFS infrastructure, patch 2 adds the support to utimensat() with a > new flag, and the rest of the patches add filesystem support; I excluded > CIFS for now because I don't have a CIFS setup to test it on. > > Updating btime is useful for at least a couple of use cases: > > - Backup/restore programs (my motivation for this feature is btrfs send) > - File servers which interoperate with operating systems that allow > updating file creation time, including Mac OS [1] and Windows [2] So you're adding an interface that allows users to change the create time of files without needing any privileges? Inode create time is forensic metadata in XFS - information we use for sequence of event and inode lifetime analysis during examination of broken filesystem images and systems that have been broken into. Just because it's exposed to userspace via statx(), it doesn't mean that it is information that users should be allowed to change. i.e. allowing users to be able to change the create time on files makes it completely useless for the purpose it was added to XFS for... And allowing root to change the create time doesn't really help, because once you've broken into a system, this makes it really easy to cover tracks (e.g. we can't find files that were created and unlinked during the break in window anymore) and lay false trails.... Cheers, Dave.
On Fri, Feb 15, 2019 at 09:06:26AM +1100, Dave Chinner wrote: > On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote: > > From: Omar Sandoval <osandov@fb.com> > > > > Hi, > > > > Since statx was added in 4.11, userspace has had an interface for > > reading btime (file creation time), but no way to set it. This RFC patch > > series adds support for changing btime with utimensat(). Patch 1 adds > > the VFS infrastructure, patch 2 adds the support to utimensat() with a > > new flag, and the rest of the patches add filesystem support; I excluded > > CIFS for now because I don't have a CIFS setup to test it on. > > > > Updating btime is useful for at least a couple of use cases: > > > > - Backup/restore programs (my motivation for this feature is btrfs send) > > - File servers which interoperate with operating systems that allow > > updating file creation time, including Mac OS [1] and Windows [2] > > So you're adding an interface that allows users to change the create > time of files without needing any privileges? I think it'd be reasonable to make this a privileged operation. I didn't for this initial submission for a couple of reasons: 1. The precedent on Mac OS and Windows is that this isn't a privileged operation. 2. I knew there would be different opinions on this either way I went. > Inode create time is forensic metadata in XFS - information we use > for sequence of event and inode lifetime analysis during examination > of broken filesystem images and systems that have been broken into. > Just because it's exposed to userspace via statx(), it doesn't mean > that it is information that users should be allowed to change. i.e. > allowing users to be able to change the create time on files makes > it completely useless for the purpose it was added to XFS for... > > And allowing root to change the create time doesn't really help, > because once you've broken into a system, this makes it really easy > to cover tracks If the threat model is that the attacker has root, then they can overwrite the timestamp on disk anyways, no? > (e.g. we can't find files that were created and > unlinked during the break in window anymore) and lay false > trails.... Fair point, although there's still ctime during the break-in window, which I assume you'd be looking for anyways since files modified during the break-in window are also of interest. I see a few options, none of which are particularly nice: 1. Filesystems like XFS could choose not to support setting btime even if they support reading it. 2. XFS could add a second, writeable btime which is used for statx/utimes when available (it would fit in di_pad2...). 3. We could add a btime_writable sysctl/mount option/mkfs option. Thanks!
On Thu, Feb 14, 2019 at 03:14:29PM -0800, Omar Sandoval wrote: > On Fri, Feb 15, 2019 at 09:06:26AM +1100, Dave Chinner wrote: > > On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote: > > > From: Omar Sandoval <osandov@fb.com> > > > > > > Hi, > > > > > > Since statx was added in 4.11, userspace has had an interface for > > > reading btime (file creation time), but no way to set it. This RFC patch > > > series adds support for changing btime with utimensat(). Patch 1 adds > > > the VFS infrastructure, patch 2 adds the support to utimensat() with a > > > new flag, and the rest of the patches add filesystem support; I excluded > > > CIFS for now because I don't have a CIFS setup to test it on. > > > > > > Updating btime is useful for at least a couple of use cases: > > > > > > - Backup/restore programs (my motivation for this feature is btrfs send) > > > - File servers which interoperate with operating systems that allow > > > updating file creation time, including Mac OS [1] and Windows [2] > > > > So you're adding an interface that allows users to change the create > > time of files without needing any privileges? > > I think it'd be reasonable to make this a privileged operation. I didn't > for this initial submission for a couple of reasons: > > 1. The precedent on Mac OS and Windows is that this isn't a privileged > operation. Don't really care about them. Interop file servers that support these operations on other OSs will need to be storing this info in xattrs because they have to work on filesystems that don't support btime. > 2. I knew there would be different opinions on this either way I went. Yup. > > Inode create time is forensic metadata in XFS - information we use > > for sequence of event and inode lifetime analysis during examination > > of broken filesystem images and systems that have been broken into. > > Just because it's exposed to userspace via statx(), it doesn't mean > > that it is information that users should be allowed to change. i.e. > > allowing users to be able to change the create time on files makes > > it completely useless for the purpose it was added to XFS for... > > > > And allowing root to change the create time doesn't really help, > > because once you've broken into a system, this makes it really easy > > to cover tracks > > If the threat model is that the attacker has root, then they can > overwrite the timestamp on disk anyways, no? Modifying the block devicee under an active filesystem is fraught with danger, and there's no guarantee it will work if the metadata being modified is still active in memory. Corrupting the filesystem is a sure way to get noticed.... > > (e.g. we can't find files that were created and > > unlinked during the break in window anymore) and lay false > > trails.... > > Fair point, although there's still ctime during the break-in window, Unless you're smart enough to know how to trigger S_NOCMTIME or FMODE_NOCMTIME.... > which I assume you'd be looking for anyways since files modified during > the break-in window are also of interest. ... and then that also can't be guaranteed. :/ > I see a few options, none of which are particularly nice: > > 1. Filesystems like XFS could choose not to support setting btime even > if they support reading it. > 2. XFS could add a second, writeable btime which is used for > statx/utimes when available (it would fit in di_pad2...). > 3. We could add a btime_writable sysctl/mount option/mkfs option. 4. create time remains a read-only field, and btrfs grows its own special interface to twiddle it in btrfs-recv if it really is necessary. I'm still not convinced that even backup/restore should be doing this, because there's so much other metadata that is unique even on restored files that it doesn't really make any sense to me to lie about it being created in the past.... Cheers, Dave.
Hi, On 2/14/19 11:00 AM, Omar Sandoval wrote: > From: Omar Sandoval <osandov@fb.com> > > Since statx was added in 4.11, userspace has had an interface for > reading btime (file creation time), but no way to set it. This RFC patch > series adds support for changing btime with utimensat(). Patch 1 adds > the VFS infrastructure, patch 2 adds the support to utimensat() with a > new flag, and the rest of the patches add filesystem support; I excluded > CIFS for now because I don't have a CIFS setup to test it on. > > Updating btime is useful for at least a couple of use cases: > > - Backup/restore programs (my motivation for this feature is btrfs send) Can you give an example of such usefulness? What's the thing you run into that you can't do without having this? You guys are having a technical discussion about 'implementation or the interface' in this thread, while I'm wondering what I'm missing as btrfs send/receive user by not having this. I never needed it in my use cases. There's two levels of use case hidden in the above line. So, I don't mean why btrfs send/receive is useful (it is, for quick efficient replication of changes) but, what's an important usecase for btime on top of that? > - File servers which interoperate with operating systems that allow > updating file creation time, including Mac OS [1] and Windows [2] > > I've also included a man page patch, xfs_io support, and an xfstest. > > Thoughts on the implementation or the interface? > > Thanks! > > 1: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/setattrlist.2.html > 2: https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-setfiletime > > Omar Sandoval (6): > fs: add btime to struct iattr > fs: add AT_UTIME_BTIME for utimensat() > Btrfs: add support for setting btime > ext4: add support for setting btime > f2fs: add support for setting btime > xfs: add support for setting btime > > fs/attr.c | 6 +++ > fs/btrfs/inode.c | 2 + > fs/btrfs/super.c | 4 +- > fs/ext4/inode.c | 15 +++++- > fs/ext4/super.c | 2 +- > fs/f2fs/file.c | 19 ++++++-- > fs/f2fs/super.c | 2 +- > fs/utimes.c | 86 +++++++++++++++++++++------------- > fs/xfs/libxfs/xfs_format.h | 2 +- > fs/xfs/libxfs/xfs_log_format.h | 2 +- > fs/xfs/xfs_iops.c | 11 ++++- > fs/xfs/xfs_super.c | 2 +- > include/linux/fs.h | 4 ++ > include/uapi/linux/fcntl.h | 2 + > 14 files changed, 111 insertions(+), 48 deletions(-) >
On Fri, Feb 15, 2019 at 01:57:39AM +0000, Hans van Kranenburg wrote: > Hi, > > On 2/14/19 11:00 AM, Omar Sandoval wrote: > > From: Omar Sandoval <osandov@fb.com> > > > > Since statx was added in 4.11, userspace has had an interface for > > reading btime (file creation time), but no way to set it. This RFC patch > > series adds support for changing btime with utimensat(). Patch 1 adds > > the VFS infrastructure, patch 2 adds the support to utimensat() with a > > new flag, and the rest of the patches add filesystem support; I excluded > > CIFS for now because I don't have a CIFS setup to test it on. > > > > Updating btime is useful for at least a couple of use cases: > > > > - Backup/restore programs (my motivation for this feature is btrfs send) > > Can you give an example of such usefulness? What's the thing you run > into that you can't do without having this? That boils down to what's useful about having the file creation time, and it's really just another tidbit of information which you may or may not care about. Maybe you have a document that you've been editing for awhile, and you want to know when you started working on it. Or, you want to know when a user created some directory that they keep adding files to. If the file creation time is useful to you, then you likely want it preserved if you have to restore from backups. If I had to restore from backups yesterday and I'm trying to figure out when I started that document, I don't care that I restored that file yesterday, I want the real creation date. If you've never wondered when a file was created, then I'm sure you won't care whether btrfs send/receive preserves it :)
On Fri, Feb 15, 2019 at 11:16:57AM +1100, Dave Chinner wrote: > On Thu, Feb 14, 2019 at 03:14:29PM -0800, Omar Sandoval wrote: > > On Fri, Feb 15, 2019 at 09:06:26AM +1100, Dave Chinner wrote: > > > On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote: > > > > From: Omar Sandoval <osandov@fb.com> > > > > > > > > Hi, > > > > > > > > Since statx was added in 4.11, userspace has had an interface for > > > > reading btime (file creation time), but no way to set it. This RFC patch > > > > series adds support for changing btime with utimensat(). Patch 1 adds > > > > the VFS infrastructure, patch 2 adds the support to utimensat() with a > > > > new flag, and the rest of the patches add filesystem support; I excluded > > > > CIFS for now because I don't have a CIFS setup to test it on. > > > > > > > > Updating btime is useful for at least a couple of use cases: > > > > > > > > - Backup/restore programs (my motivation for this feature is btrfs send) > > > > - File servers which interoperate with operating systems that allow > > > > updating file creation time, including Mac OS [1] and Windows [2] > > > > > > So you're adding an interface that allows users to change the create > > > time of files without needing any privileges? > > > > I think it'd be reasonable to make this a privileged operation. I didn't > > for this initial submission for a couple of reasons: > > > > 1. The precedent on Mac OS and Windows is that this isn't a privileged > > operation. > > Don't really care about them. Interop file servers that support these > operations on other OSs will need to be storing this info in xattrs > because they have to work on filesystems that don't support btime. > > > 2. I knew there would be different opinions on this either way I went. > > Yup. > > > > Inode create time is forensic metadata in XFS - information we use > > > for sequence of event and inode lifetime analysis during examination > > > of broken filesystem images and systems that have been broken into. > > > Just because it's exposed to userspace via statx(), it doesn't mean > > > that it is information that users should be allowed to change. i.e. > > > allowing users to be able to change the create time on files makes > > > it completely useless for the purpose it was added to XFS for... > > > > > > And allowing root to change the create time doesn't really help, > > > because once you've broken into a system, this makes it really easy > > > to cover tracks > > > > If the threat model is that the attacker has root, then they can > > overwrite the timestamp on disk anyways, no? > > Modifying the block devicee under an active filesystem is fraught > with danger, and there's no guarantee it will work if the metadata > being modified is still active in memory. Corrupting the filesystem > is a sure way to get noticed.... > > > > (e.g. we can't find files that were created and > > > unlinked during the break in window anymore) and lay false > > > trails.... > > > > Fair point, although there's still ctime during the break-in window, > > Unless you're smart enough to know how to trigger S_NOCMTIME or > FMODE_NOCMTIME.... > > > which I assume you'd be looking for anyways since files modified during > > the break-in window are also of interest. > > ... and then that also can't be guaranteed. :/ > > > I see a few options, none of which are particularly nice: > > > > 1. Filesystems like XFS could choose not to support setting btime even > > if they support reading it. > > 2. XFS could add a second, writeable btime which is used for > > statx/utimes when available (it would fit in di_pad2...). > > 3. We could add a btime_writable sysctl/mount option/mkfs option. > > 4. create time remains a read-only field, and btrfs grows its own > special interface to twiddle it in btrfs-recv if it really is > necessary. I'm curious to hear what the ext4/f2fs/CIFS developers think. If no one else wants btime to be mutable, then I might as well make it Btrfs-specific. That is, assuming we reach consensus on the Btrfs side that btrfs receive should set btime. > I'm still not convinced that even backup/restore should be doing this, > because there's so much other metadata that is unique even on > restored files that it doesn't really make any sense to me to lie > about it being created in the past.... I suppose it depends on how you interpret btime: if it's strictly filesystem metadata, then it makes sense that it should be immutable; if it's metadata for the user's own purposes, then we should allow setting it.
On Thu, 14 Feb 2019 22:59:47 -0800, Omar Sandoval wrote: > On Fri, Feb 15, 2019 at 11:16:57AM +1100, Dave Chinner wrote: > > On Thu, Feb 14, 2019 at 03:14:29PM -0800, Omar Sandoval wrote: > > > On Fri, Feb 15, 2019 at 09:06:26AM +1100, Dave Chinner wrote: ... > > > > Inode create time is forensic metadata in XFS - information we use > > > > for sequence of event and inode lifetime analysis during examination > > > > of broken filesystem images and systems that have been broken into. > > > > Just because it's exposed to userspace via statx(), it doesn't mean > > > > that it is information that users should be allowed to change. i.e. > > > > allowing users to be able to change the create time on files makes > > > > it completely useless for the purpose it was added to XFS for... > > > > > > > > And allowing root to change the create time doesn't really help, > > > > because once you've broken into a system, this makes it really easy > > > > to cover tracks > > > > > > If the threat model is that the attacker has root, then they can > > > overwrite the timestamp on disk anyways, no? > > > > Modifying the block devicee under an active filesystem is fraught > > with danger, and there's no guarantee it will work if the metadata > > being modified is still active in memory. Corrupting the filesystem > > is a sure way to get noticed.... > > > > > > (e.g. we can't find files that were created and > > > > unlinked during the break in window anymore) and lay false > > > > trails.... > > > > > > Fair point, although there's still ctime during the break-in window, > > > > Unless you're smart enough to know how to trigger S_NOCMTIME or > > FMODE_NOCMTIME.... > > > > > which I assume you'd be looking for anyways since files modified during > > > the break-in window are also of interest. I'm not sure I follow the forensics use-case for immutable btime. I'd expect dm-verity or selinux/apparmor audits to do a better job for those worried about this kind of attack. > > ... and then that also can't be guaranteed. :/ > > > > > I see a few options, none of which are particularly nice: > > > > > > 1. Filesystems like XFS could choose not to support setting btime even > > > if they support reading it. > > > 2. XFS could add a second, writeable btime which is used for > > > statx/utimes when available (it would fit in di_pad2...). > > > 3. We could add a btime_writable sysctl/mount option/mkfs option. > > > > 4. create time remains a read-only field, and btrfs grows its own > > special interface to twiddle it in btrfs-recv if it really is > > necessary. > > I'm curious to hear what the ext4/f2fs/CIFS developers think. If no one > else wants btime to be mutable, then I might as well make it > Btrfs-specific. That is, assuming we reach consensus on the Btrfs side > that btrfs receive should set btime. Samba currently uses a user.DOSATTRIB xattr for tracking creation time. IMO a mutable btime accessible via statx would be useful for cross-protocol interoperability. Cheers, David
On 2/15/19 6:39 AM, Omar Sandoval wrote: > On Fri, Feb 15, 2019 at 01:57:39AM +0000, Hans van Kranenburg wrote: >> Hi, >> >> On 2/14/19 11:00 AM, Omar Sandoval wrote: >>> From: Omar Sandoval <osandov@fb.com> >>> >>> Since statx was added in 4.11, userspace has had an interface for >>> reading btime (file creation time), but no way to set it. This RFC patch >>> series adds support for changing btime with utimensat(). Patch 1 adds >>> the VFS infrastructure, patch 2 adds the support to utimensat() with a >>> new flag, and the rest of the patches add filesystem support; I excluded >>> CIFS for now because I don't have a CIFS setup to test it on. >>> >>> Updating btime is useful for at least a couple of use cases: >>> >>> - Backup/restore programs (my motivation for this feature is btrfs send) >> >> Can you give an example of such usefulness? What's the thing you run >> into that you can't do without having this? > > That boils down to what's useful about having the file creation time, > and it's really just another tidbit of information which you may or may > not care about. Maybe you have a document that you've been editing for > awhile, and you want to know when you started working on it. Or, you > want to know when a user created some directory that they keep adding > files to. > > If the file creation time is useful to you, then you likely want it > preserved if you have to restore from backups. If I had to restore from > backups yesterday and I'm trying to figure out when I started that > document, I don't care that I restored that file yesterday, I want the > real creation date. > > If you've never wondered when a file was created, then I'm sure you > won't care whether btrfs send/receive preserves it :) Thanks for the elaborate answer. I was just curious if this was to make send "more feature complete" or if there was something else special to it, and it seems to be the first one. ;] And for myself, I mostly treat btrfs send/receive like a improved replacement for rsync and usually don't care that much about btime indeed. Thanks,
On Feb 14, 2019, at 11:59 PM, Omar Sandoval <osandov@osandov.com> wrote: > > On Fri, Feb 15, 2019 at 11:16:57AM +1100, Dave Chinner wrote: >> On Thu, Feb 14, 2019 at 03:14:29PM -0800, Omar Sandoval wrote: >>> I see a few options, none of which are particularly nice: >>> >>> 1. Filesystems like XFS could choose not to support setting btime even >>> if they support reading it. >>> 2. XFS could add a second, writeable btime which is used for >>> statx/utimes when available (it would fit in di_pad2...). >>> 3. We could add a btime_writable sysctl/mount option/mkfs option. >> >> 4. create time remains a read-only field, and btrfs grows its own >> special interface to twiddle it in btrfs-recv if it really is >> necessary. > > I'm curious to hear what the ext4/f2fs/CIFS developers think. If no one > else wants btime to be mutable, then I might as well make it > Btrfs-specific. That is, assuming we reach consensus on the Btrfs side > that btrfs receive should set btime. > >> I'm still not convinced that even backup/restore should be doing this, >> because there's so much other metadata that is unique even on >> restored files that it doesn't really make any sense to me to lie >> about it being created in the past.... > > I suppose it depends on how you interpret btime: if it's strictly > filesystem metadata, then it makes sense that it should be immutable; if > it's metadata for the user's own purposes, then we should allow setting > it. My personal preference is that crtime/btime be read-only information that tells when the file itself was created in this filesystem, not some added metadata that is managed by userspace. There have been many times when I've needed to know when a file was _actually_ created in the filesystem, not what mtime/ctime report. While it may be a bit of a stretch to call this "forensic evidence", making it hard to change from except via total root compromise by a skilled hacker is very useful. If this were to go in (which I'm not in favour of), then there would need to be a CONFIG and/or runtime knob to turn it off (or better to only turn it on), similar to how FIPS and other security options can only go in one direction. Cheers, Andreas
On 15/02/19 00:06, Dave Chinner wrote: > On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote: >> From: Omar Sandoval <osandov@fb.com> >> >> Hi, >> >> Since statx was added in 4.11, userspace has had an interface for >> reading btime (file creation time), but no way to set it. This RFC patch >> series adds support for changing btime with utimensat(). Patch 1 adds >> the VFS infrastructure, patch 2 adds the support to utimensat() with a >> new flag, and the rest of the patches add filesystem support; I excluded >> CIFS for now because I don't have a CIFS setup to test it on. >> >> Updating btime is useful for at least a couple of use cases: >> [1] >> - Backup/restore programs (my motivation for this feature is btrfs send) >> - File servers which interoperate with operating systems that allow >> updating file creation time, including Mac OS [1] and Windows [2] > > So you're adding an interface that allows users to change the create > time of files without needing any privileges? > [2] > Inode create time is forensic metadata in XFS - information we use > for sequence of event and inode lifetime analysis during examination > of broken filesystem images and systems that have been broken into. > Just because it's exposed to userspace via statx(), it doesn't mean > that it is information that users should be allowed to change. i.e. > allowing users to be able to change the create time on files makes > it completely useless for the purpose it was added to XFS for... > <snap> I think the difference in opinion here is that there are two totally different BTIme out in the world. For two somewhat opposite motivations and it seems they both try to be crammed into the same on disk space. One - Author creation time This is a Windows originated creature and later MAC (and all vendors who make a living by serving cifs (hint see my email address)) This is a tag carried globally on the globe denoting the time of the original creator of the file. copy, download, backup-restore and so on preserve it from the very first original creation. This creature is a user oriented information. That needs to be carefully orchestrated by all parties Two - Local creation time This is an immutable local FS information that helps in debugging and FS-checking / recovery of data. It is an information that kind of denotes the order of creation of files on a local FS. So it looks like both sides are correct trying to preserve their own guy? XFS invented [2] I'd let it be. If you need [1] on XFS better push for a well defined standardized xattr and be in peace. BTRFS should decide which one of [2] or [1] it has space for in the inode and commit to it. Documenting well what it is. That said putting my Netapp hat. I would love to see an easy API for Author-creation-time BTime type of value. That is accessed uniformly by user-mode and/or Network file servers (NFS/CIFS). And would love to see a generic implementation of that interface that puts it into a standardized xattr if the FS in question does not have a native support for it [1]. So I love these patches. And would want to see this through. But let us understand each other? Thanks Boaz
On Sun, Feb 17, 2019 at 06:35:25PM +0200, Boaz Harrosh wrote: > On 15/02/19 00:06, Dave Chinner wrote: > > So you're adding an interface that allows users to change the create > > time of files without needing any privileges? > > Inode create time is forensic metadata in XFS - information we use > > for sequence of event and inode lifetime analysis during examination > > of broken filesystem images and systems that have been broken into. > I think the difference in opinion here is that there are two totally > different BTIme out in the world. For two somewhat opposite motivations > and it seems they both try to be crammed into the same on disk space. > > One - Author creation time > Two - Local creation time > So it looks like both sides are correct trying to preserve their own guy? I'd say that [2] is too easily gameable to be worth the effort. You can just change it on the disk. That right now it'd take some skill to find the right place to edit doesn't matter -- a tool to update the btime against your wishes would need to be written just once. Unlike btrfs, XFS doesn't even have a chain of checksums all the way to the root. On the other hand, [1] has a lot of uses. It can also be preserved in backups and version control (svnt and git-restore-mtime could be easily extended). I'd thus go with [2] -- any uses for [1] are better delegated to filesystem specific tools. Meow!
On Sun, Feb 17, 2019 at 9:55 AM Adam Borowski <kilobyte@angband.pl> wrote: > > On Sun, Feb 17, 2019 at 06:35:25PM +0200, Boaz Harrosh wrote: > > On 15/02/19 00:06, Dave Chinner wrote: > > > So you're adding an interface that allows users to change the create > > > time of files without needing any privileges? > > > > Inode create time is forensic metadata in XFS - information we use > > > for sequence of event and inode lifetime analysis during examination > > > of broken filesystem images and systems that have been broken into. > > > I think the difference in opinion here is that there are two totally > > different BTIme out in the world. For two somewhat opposite motivations > > and it seems they both try to be crammed into the same on disk space. > > > > One - Author creation time > > Two - Local creation time > > > So it looks like both sides are correct trying to preserve their own guy? > > I'd say that [2] is too easily gameable to be worth the effort. You can > just change it on the disk. That right now it'd take some skill to find the > right place to edit doesn't matter -- a tool to update the btime against > your wishes would need to be written just once. Unlike btrfs, XFS doesn't > even have a chain of checksums all the way to the root. > > On the other hand, [1] has a lot of uses. It can also be preserved in > backups and version control (svnt and git-restore-mtime could be easily > extended). > > I'd thus go with [2] -- any uses for [1] are better delegated to filesystem > specific tools. > I started out in the Windows world, and I found it quite handy to right-click a file and see when it was created. When I started using Linux, I saw things like this: Access: 2019-02-16 22:19:32.024284060 -0800 Modify: 2016-02-02 19:26:47.901766778 -0800 Change: 2016-02-02 19:26:47.907766649 -0800 and my mind boggled a bit. Modify makes sense. Change? What's that? Why do I care? Adding "birth" makes sense, and I think that filesystem-agnostic backup tools *should* be able to write it. So I'm highly in favor of this patch. If XFS wants to disallow writing the birth time, fine, but I think that behavior should be overridable.
On Sat, Feb 16, 2019 at 06:57:45PM -0700, Andreas Dilger wrote: > While it may be a bit of a stretch to call this "forensic evidence", making We do forensic analysis of corrupt filesystems looking for evidence of what went wrong, not just looking for evidence of what happened on systems that have been broken into. > it hard to change from except via total root compromise by a skilled hacker > is very useful. *nod*. > If this were to go in (which I'm not in favour of), then there would need to > be a CONFIG and/or runtime knob to turn it off (or better to only turn it on), > similar to how FIPS and other security options can only go in one direction. The problem here is that "inode birth time" is being conflated with "user document creation time". These two things are very different. i.e. One is filesystem internal information and is not related to when the original copy of the data in the file was created, the other is user specified metadata that is related to the file data contents and needs to travel with the data, not the filesystem. IMO, trying to make one on-disk field hold two different types of information defeats one or the other purpose, and nobody knows which one the field stores for any given file. I'd suggest that "authored date" should be a generic system xattr so most filesystems support it, not just those that have a birth time field on disk. Sure, modify it through utimesat() and expose it through statx() (as authored time, not birth time), but store it a system xattr rather than an internal filesystem metadata field that requires was never intended to be user modifiable. Cheers, Dave.
On Sun, Feb 17, 2019 at 12:40:09PM -0800, Andy Lutomirski wrote: > So I'm highly in favor of this patch. If XFS wants to disallow > writing the birth time, fine, but I think that behavior should be > overridable. Please, no. We need to have consistent behaviour between at least Linux local filesystems. Not "Chris thinks this is a good idea, while Dave and Ted think its a bad idea, so btrfs supports it and XFS and ext4 disallow it".
On Mon, Feb 18, 2019 at 08:04:47PM -0800, Matthew Wilcox wrote: > On Sun, Feb 17, 2019 at 12:40:09PM -0800, Andy Lutomirski wrote: > > So I'm highly in favor of this patch. If XFS wants to disallow > > writing the birth time, fine, but I think that behavior should be > > overridable. > > Please, no. We need to have consistent behaviour between at least > Linux local filesystems. Not "Chris thinks this is a good idea, > while Dave and Ted think its a bad idea, so btrfs supports it and > XFS and ext4 disallow it". And, quite frankly, this is the entire reason xattrs exist. i.e. so that generic file attributes can be stored persitently without each individual having to support them in their on-disk format. I wish people would stop trying to implement stuff like this in filesystem code and instead added it to the VFS and stored it in VFS defined system xattrs so that it is common across all filesystems. It also means that backup applications can preserve them during file copies without really even being aware of their meaning, simply by copying all the xattrs on the file... Cheers, Dave.
On Feb 17, 2019, at 8:35 AM, Boaz Harrosh <openosd@gmail.com> wrote: > > On 15/02/19 00:06, Dave Chinner wrote: >> On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote: >>> From: Omar Sandoval <osandov@fb.com> >>> >>> Hi, >>> >>> Since statx was added in 4.11, userspace has had an interface for >>> reading btime (file creation time), but no way to set it. This RFC patch >>> series adds support for changing btime with utimensat(). Patch 1 adds >>> the VFS infrastructure, patch 2 adds the support to utimensat() with a >>> new flag, and the rest of the patches add filesystem support; I excluded >>> CIFS for now because I don't have a CIFS setup to test it on. >>> >>> Updating btime is useful for at least a couple of use cases: >>> > [1] >>> - Backup/restore programs (my motivation for this feature is btrfs send) >>> - File servers which interoperate with operating systems that allow >>> updating file creation time, including Mac OS [1] and Windows [2] >> >> So you're adding an interface that allows users to change the create >> time of files without needing any privileges? >> > [2] >> Inode create time is forensic metadata in XFS - information we use >> for sequence of event and inode lifetime analysis during examination >> of broken filesystem images and systems that have been broken into. >> Just because it's exposed to userspace via statx(), it doesn't mean >> that it is information that users should be allowed to change. i.e. >> allowing users to be able to change the create time on files makes >> it completely useless for the purpose it was added to XFS for... >> > <snap> > > I think the difference in opinion here is that there are two totally > different BTIme out in the world. For two somewhat opposite motivations > and it seems they both try to be crammed into the same on disk space. > > One - Author creation time > This is a Windows originated creature and later MAC (and all vendors who > make a living by serving cifs (hint see my email address)) > > This is a tag carried globally on the globe denoting the time of the > original creator of the file. copy, download, backup-restore and so > on preserve it from the very first original creation. > This creature is a user oriented information. That needs to be carefully > orchestrated by all parties. One of the issues with user-supplied "creation time" is that it is vague. If Vim (or many other programs) is writing out an existing file, it will actually create a new file and rename it over the old file, so even though the newly-written file is "created" at write time, the content is older. Unless the creation time is actually stored internal to the file itself, rather than as filesystem metadata, it is unlikely to be kept across filesystems, between systems, etc. Conversely, there is already such metadata in existing file formats, e.g. .jpg (Exif.Image.DateTime), .png (tEXt Creation Time), .docx (Date Completed), etc. that sticks with the file regardless of the underlying storage medium. Cheers, Andreas
On Thu, Feb 14, 2019 at 02:00:07AM -0800, Omar Sandoval wrote: > From: Omar Sandoval <osandov@fb.com> > Since statx was added in 4.11, userspace has had an interface for > reading btime (file creation time), but no way to set it. This RFC patch > series adds support for changing btime with utimensat(). Patch 1 adds > the VFS infrastructure, patch 2 adds the support to utimensat() with a > new flag, and the rest of the patches add filesystem support; I excluded > CIFS for now because I don't have a CIFS setup to test it on. > > Updating btime is useful for at least a couple of use cases: > > - Backup/restore programs (my motivation for this feature is btrfs send) > - File servers which interoperate with operating systems that allow > updating file creation time, including Mac OS [1] and Windows [2] I don't have anything new to add to what has been said in the thread. The creation time is property of the filesystem and if user wants to track additional metadata, then as external attributes.
On Tue, Feb 19, 2019 at 09:18:20AM +1100, Dave Chinner wrote: > On Sat, Feb 16, 2019 at 06:57:45PM -0700, Andreas Dilger wrote: > > While it may be a bit of a stretch to call this "forensic evidence", making > > We do forensic analysis of corrupt filesystems looking for evidence > of what went wrong, not just looking for evidence of what happened > on systems that have been broken into. > > > it hard to change from except via total root compromise by a skilled hacker > > is very useful. > > *nod*. > > > If this were to go in (which I'm not in favour of), then there would need to > > be a CONFIG and/or runtime knob to turn it off (or better to only turn it on), > > similar to how FIPS and other security options can only go in one direction. > > The problem here is that "inode birth time" is being conflated with > "user document creation time". These two things are very different. > > i.e. One is filesystem internal information and is not related to > when the original copy of the data in the file was created, the > other is user specified metadata that is related to the file data > contents and needs to travel with the data, not the filesystem. > > IMO, trying to make one on-disk field hold two different types of > information defeats one or the other purpose, and nobody knows which > one the field stores for any given file. > > I'd suggest that "authored date" should be a generic system xattr so > most filesystems support it, not just those that have a birth time > field on disk. Sure, modify it through utimesat() and expose it > through statx() (as authored time, not birth time), but store it a > system xattr rather than an internal filesystem metadata field that > requires was never intended to be user modifiable. It seems that this is the general consensus, so I'll look into implementing this functionality as an xattr.
> On Feb 22, 2019, at 11:00 AM, Omar Sandoval <osandov@osandov.com> wrote: > > On Tue, Feb 19, 2019 at 09:18:20AM +1100, Dave Chinner wrote: >> On Sat, Feb 16, 2019 at 06:57:45PM -0700, Andreas Dilger wrote: >>> While it may be a bit of a stretch to call this "forensic evidence", >> >> We do forensic analysis of corrupt filesystems looking for evidence >> of what went wrong, not just looking for evidence of what happened >> on systems that have been broken into. >> >>> making it hard to change from except via total root compromise by a >>> skilled hacker is very useful. >> >> *nod*. >> >>> If this were to go in (which I'm not in favour of), then there would >>> need to be a CONFIG and/or runtime knob to turn it off (or better to >>> only turn it on), similar to how FIPS and other security options can >>> only go in one direction. >> >> The problem here is that "inode birth time" is being conflated with >> "user document creation time". These two things are very different. >> >> i.e. One is filesystem internal information and is not related to >> when the original copy of the data in the file was created, the >> other is user specified metadata that is related to the file data >> contents and needs to travel with the data, not the filesystem. >> >> IMO, trying to make one on-disk field hold two different types of >> information defeats one or the other purpose, and nobody knows which >> one the field stores for any given file. >> >> I'd suggest that "authored date" should be a generic system xattr so >> most filesystems support it, not just those that have a birth time >> field on disk. Sure, modify it through utimesat() and expose it >> through statx() (as authored time, not birth time), but store it a >> system xattr rather than an internal filesystem metadata field that >> requires was never intended to be user modifiable. > > It seems that this is the general consensus, so I'll look into > implementing this functionality as an xattr. I would recommend to look at how Samba is storing these attributes today, and do the same thing, maybe add support into GNU coreutils to handle this transparently. Cheers, Andreas
From: Omar Sandoval <osandov@fb.com> Hi, Since statx was added in 4.11, userspace has had an interface for reading btime (file creation time), but no way to set it. This RFC patch series adds support for changing btime with utimensat(). Patch 1 adds the VFS infrastructure, patch 2 adds the support to utimensat() with a new flag, and the rest of the patches add filesystem support; I excluded CIFS for now because I don't have a CIFS setup to test it on. Updating btime is useful for at least a couple of use cases: - Backup/restore programs (my motivation for this feature is btrfs send) - File servers which interoperate with operating systems that allow updating file creation time, including Mac OS [1] and Windows [2] I've also included a man page patch, xfs_io support, and an xfstest. Thoughts on the implementation or the interface? Thanks! 1: https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/setattrlist.2.html 2: https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-setfiletime Omar Sandoval (6): fs: add btime to struct iattr fs: add AT_UTIME_BTIME for utimensat() Btrfs: add support for setting btime ext4: add support for setting btime f2fs: add support for setting btime xfs: add support for setting btime fs/attr.c | 6 +++ fs/btrfs/inode.c | 2 + fs/btrfs/super.c | 4 +- fs/ext4/inode.c | 15 +++++- fs/ext4/super.c | 2 +- fs/f2fs/file.c | 19 ++++++-- fs/f2fs/super.c | 2 +- fs/utimes.c | 86 +++++++++++++++++++++------------- fs/xfs/libxfs/xfs_format.h | 2 +- fs/xfs/libxfs/xfs_log_format.h | 2 +- fs/xfs/xfs_iops.c | 11 ++++- fs/xfs/xfs_super.c | 2 +- include/linux/fs.h | 4 ++ include/uapi/linux/fcntl.h | 2 + 14 files changed, 111 insertions(+), 48 deletions(-)