| Message ID | 20200806202358.2265705-1-martin.agren@gmail.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Documentation: mark `--object-format=sha256` as experimental | expand |
On 2020-08-06 at 20:23:58, Martin Ågren wrote: > After eff45daab8 ("repository: enable SHA-256 support by default", > 2020-07-29), vanilla builds of Git enable the user to run, e.g., > > git init --object-format=sha256 > > and hack away. This can be a good way to gain experience with the > SHA-256 world, e.g., to find bugs that > > GIT_TEST_DEFAULT_HASH=sha256 make test > > doesn't spot. > > But it really is a separate world: Such SHA-256 repos will live entirely > separate from the (by now fairly large) set of SHA-1 repos. Interacting > across the border is possible in principle, e.g., through "diff + apply" > (or "format-patch + am"), but even that has its limitations: Applying a > SHA-256 diff in a SHA-1 repo works in the simple case, but if you need > to resort to `-3`, you're out of luck. > > Similarly, "push + pull" should work, but you really will be operating > mostly offset from the rest of the world. That might be ok by the time > you initialize your repository, and it might be ok for several months > after that, but there might come a day when you're starting to regret > your use of `git init --object-format=sha256` and have dug yourself into > a fairly deep hole. I do agree that they don't interoperate right now, and that we'd like it to in the future. But there are definitely people who can use SHA-256 support for new projects without problems. I'm aware of certain government agencies who very much do not want to use SHA-1 at all (and at some point will be legally prohibited from doing so), and they will be completely fine with the status quo. Some of those same organizations are unhappy about prohibited algorithms even being linked into the binaries they use. These folks can use a suitably new version of Git everywhere and not care about the lack of backwards compatibility. I am, of course, in favor of abandoning SHA-1 as fast as practically possible, but I understand that backwards compatibility is obviously a concern. > Workflows aside, let's consider a more technical aspect. Pack index > files (pack-*.idx) exist in two flavours: v1 and v2. The hash transition > document foresees a v3, which we do not yet support (and for all we > know, the final v3 might end up different from the one sketched in the > hash transition document). > > When the test suite is run with SHA-1 as the default hash algo, it > creates and consumes v2 pack files. But with SHA-256, we use an > undocumented, hybrid format where the header looks like v2, but where > the payload is not only "not correct SHA1", but where even the data > sizes are different. The trailing checksum is different, meaning no-one > (except us!) should/would try to interpret this file as a proper v2 pack > index. > > We could certainly (re)define v2 to match our SHA-256 behavior, but we > do foresee v3 for a reason. And that would still just fix this specific > issue. And even when everything around SHA-256 is well-defined and we > have SHA-1--SHA-256 interoperability, there's a risk, at least > initially, that somewhere we'd be permeating buggy data that we'd then > feel responsible for and need to be able to handle for a long time to > come. These are valid index v1 and v2 files, just with a different hash algorithm. v3 is there for the point where we do interoperate and need to store hash values of multiple algorithms at once. There's little to no benefit to v3 if you don't need multiple algorithm support, other than the fact that they declare the algorithms in them. This is no different than saying that our commit or tree objects are in a different form; they are syntactically identical, just with a different hash algorithm. That's how everything is in the .git directory. > diff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt > new file mode 100644 > index 0000000000..4cb106f0d1 > --- /dev/null > +++ b/Documentation/object-format-disclaimer.txt > @@ -0,0 +1,6 @@ > +THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still > +in an early stage. A SHA-256 repository will in general not be able to > +share work with "regular" SHA-1 repositories. It should be assumed > +that, e.g., Git internal file formats in relation to SHA-256 > +repositories may change in backwards-incompatible ways. Only use > +`--object-format=sha256` for testing purposes. I'm fine with marking the functionality experimental for a few releases, since it is possible we have bugs people haven't found, and adding a note about interoperability after that point, since I think that's a fair and valuable issue. I think if we go a few releases without any major issues, we can change this to the following: Note that a SHA-256 repository cannot yet share work with "regular" SHA-1 repositories. Many tools do not yet understand SHA-256 repositories, so users may wish to take this into account when creating new repositories.
On Fri, 7 Aug 2020 at 01:08, brian m. carlson <sandals@crustytoothpaste.net> wrote: > > On 2020-08-06 at 20:23:58, Martin Ågren wrote: > > After eff45daab8 ("repository: enable SHA-256 support by default", > > 2020-07-29), vanilla builds of Git enable the user to run, e.g., > > > > git init --object-format=sha256 > > > > and hack away. [...] [...] > > Similarly, "push + pull" should work, but you really will be operating > > mostly offset from the rest of the world. That might be ok by the time > > you initialize your repository, and it might be ok for several months > > after that, but there might come a day when you're starting to regret > > your use of `git init --object-format=sha256` and have dug yourself into > > a fairly deep hole. > > I do agree that they don't interoperate right now, and that we'd like it > to in the future. But there are definitely people who can use SHA-256 > support for new projects without problems. I'm aware of certain > government agencies who very much do not want to use SHA-1 at all (and > at some point will be legally prohibited from doing so), and they will > be completely fine with the status quo. Some of those same > organizations are unhappy about prohibited algorithms even being linked > into the binaries they use. These folks can use a suitably new version > of Git everywhere and not care about the lack of backwards > compatibility. > > I am, of course, in favor of abandoning SHA-1 as fast as practically > possible, but I understand that backwards compatibility is obviously a > concern. Yeah, I'd prefer them to know that they are early adopters and that they should be prepared for a situation where there's some incompatibility across versions. I don't just mean "I can't read my old SHA-1 data any more", I mean "I used Git v2.29.0 to create a SHA-256 repo and now Git v3.15.0 won't read it". (Or v2.31.0?) I've followed the work on the commit graph functionality and file format mostly from the sidelines. It's been lots of good work with lots of good outcome, but there also seems to have been (of course) a few incompatibilities, bugs and "argh, if only we'd have done it like this from the beginning". I'd assume the effort -- and potential for bugs and "ooh, we should have done it that way" -- for SHA-256--SHA-1 interoperability to be larger than what's been put into the commit graph so far. > > Workflows aside, let's consider a more technical aspect. Pack index > > files (pack-*.idx) exist in two flavours: v1 and v2. [...] [...] > > We could certainly (re)define v2 to match our SHA-256 behavior, but we > > do foresee v3 for a reason. And that would still just fix this specific > > issue. And even when everything around SHA-256 is well-defined and we > > have SHA-1--SHA-256 interoperability, there's a risk, at least > > initially, that somewhere we'd be permeating buggy data that we'd then > > feel responsible for and need to be able to handle for a long time to > > come. > > These are valid index v1 and v2 files, just with a different hash > algorithm. I claim that they are not valid, precisely because they use a different hash algorithm. > v3 is there for the point where we do interoperate and need > to store hash values of multiple algorithms at once. There's little to > no benefit to v3 if you don't need multiple algorithm support, other > than the fact that they declare the algorithms in them. One additional benefit: they'd correspond to a specification. :-) > This is no different than saying that our commit or tree objects are in > a different form; they are syntactically identical, just with a > different hash algorithm. That's how everything is in the .git > directory. For objects, I could perhaps accept that the format outlined in the hash transition document is the specification. That document says that pack indices "use a new v3 format that supports multiple hash functions." It then goes on to draft such a format. (Maybe it's a specification, but until there exists at least one implementation, I'd rather see it as a draft.) No mention of v2 pack indices with SHA-256 data, neither in that document, nor anywhere else in our documentation that I could find. The "v2 but with SHA-256" packfile index format we're producing contains lots of 32 B SHA-256s instead of 20 B SHA-1s, ok, that much can be guessed in one try. The index file ends with a 32 B SHA-256, after referencing the 32 B packfile SHA-256. Ok, maybe that could also be guessed. If we're committed to maintaining that format, we should put it down in writing. And if we're not committed to it, we should make that clear. The hash transition document foresees a packfile index format v3. Notably, it uses a _20_ B SHA-256 checksum and references a _20_ B SHA-256 packfile SHA-256. In light of that, are we certain that the "v2 with SHA-256" format outlined above is not a maintenance burden? Or that if there is any kind of cost, that it's worth it? Or, for that matter, that guessing the details of "v2 but with SHA-256" is trivial? I fully respect the effort that has gone into making the test suite run with 32 B SHA-256 instead of 20 B SHA-1. But do we really intend to support for many years to come the new file formats that such a test run produces and consumes? Bundles v3, yeah I guess so. Thanks for making that move! Pack index "v2 but with SHA-256", maybe. At the very least, we should set down our feet consciously. > > +THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still > > +in an early stage. A SHA-256 repository will in general not be able to > > +share work with "regular" SHA-1 repositories. It should be assumed > > +that, e.g., Git internal file formats in relation to SHA-256 > > +repositories may change in backwards-incompatible ways. Only use > > +`--object-format=sha256` for testing purposes. > > I'm fine with marking the functionality experimental for a few releases, > since it is possible we have bugs people haven't found, and adding a > note about interoperability after that point, since I think that's a > fair and valuable issue. I think if we go a few releases without any > major issues, we can change this to the following: > > Note that a SHA-256 repository cannot yet share work with "regular" > SHA-1 repositories. Many tools do not yet understand SHA-256 > repositories, so users may wish to take this into account when > creating new repositories. With respect, I think that's too aggressive. By that time, we may conclude that, e.g., the "v2 pack indices with SHA-256" file handling is robust. But I'd be surprised if using `git init --object-format=sha256` in June 2021 won't cause *some* extra work for users or ourselves further down the line compared to using a regular SHA-1 `git init`. Pushing to a SHA-1 hosting service will become *possible* at some point, but maybe it won't be *efficient enough to be practical in the real world* until some time after that. All those other, *new* file formats outlined in the hash transition document won't exist at that time (at least not in master). Now would probably be a good time to update the hash transition documents, first of all to tick off what we've already done, and second, to reassess the rest. Quoting: The first user-visible change is the introduction of the objectFormat extension (without compatObjectFormat). This requires: - implementing the loose-object-idx - teaching fsck about this mode of operation - using the hash function API (vtable) when computing object names - signing objects and verifying signatures - rejecting attempts to fetch from or push to an incompatible repository I don't think we're there yet. Maybe, e.g., the new loose-object-idx isn't strictly needed, in which case this part of the plan could be updated. (Or maybe who wrote the above thought there'd be some value in knowing that *all* SHA-256 repos *always* have loose-object-idx tables to save us from some file-discovery dancing?) We do say elsewhere in the document that Alongside the packfile, a SHA-256 repository stores a bidirectional mapping between SHA-256 and SHA-1 object names. So at the time we do not seem to be producing correct, proper, as-specified (or at least as-drafted) SHA-256 repositories. Or maybe in 2030, we can stop insisting on such a mapping, because everyone uses SHA-256 anyway, so then maybe it shouldn't be mandatory now, either. "Signing objects" is a bit vague, but under "Signed Commits", I see: [...] This means commits can be signed 1. using SHA-1 only, as in existing signed commit objects 2. using both SHA-1 and SHA-256, by using both gpgsig-sha256 and gpgsig fields. 3. using only SHA-256, by only using the gpgsig-sha256 field. Right now, we can do either 1 or 3. Maybe that's enough. I do think there's a bug in git-replace where we'll only remove the last signature, but as we'll currently only create one signature, that's perhaps "ok". I still believe we should think hard before saying (even if we only say so by omission) that, e.g., file structures are known-good and will be supported for a long time to come. Martin
Martin Ågren <martin.agren@gmail.com> writes: >> I'm fine with marking the functionality experimental for a few releases, >> since it is possible we have bugs people haven't found, and adding a >> note about interoperability after that point, since I think that's a >> fair and valuable issue. I think if we go a few releases without any >> major issues, we can change this to the following: >> >> Note that a SHA-256 repository cannot yet share work with "regular" >> SHA-1 repositories. Many tools do not yet understand SHA-256 >> repositories, so users may wish to take this into account when >> creating new repositories. > > With respect, I think that's too aggressive. By that time, we may > conclude that, e.g., the "v2 pack indices with SHA-256" file handling is > robust. But I'd be surprised if using `git init --object-format=sha256` > in June 2021 won't cause *some* extra work for users or ourselves > further down the line compared to using a regular SHA-1 `git init`. > Pushing to a SHA-1 hosting service will become *possible* at some point, > but maybe it won't be *efficient enough to be practical in the real > world* until some time after that. IOW, you question "if we go a few releases without any major issues" part? I tend to agree that for a large change like this, a few releases may not be sufficiently long time for a feature that is marked as experimental in big flashing red letters to get exercised enough to get major issues noticed. > All those other, *new* file formats outlined in the hash > transition document won't exist at that time (at least not in master). > > Now would probably be a good time to update the hash transition > documents, first of all to tick off what we've already done, and second, > to reassess the rest. Yes, it is a good idea to stop and see where in the overall large picture we currently are. Thanks.
On Fri, 7 Aug 2020 at 22:50, Junio C Hamano <gitster@pobox.com> wrote: > > Martin Ågren <martin.agren@gmail.com> writes: > > >> I'm fine with marking the functionality experimental for a few releases, > >> since it is possible we have bugs people haven't found, and adding a > >> note about interoperability after that point, since I think that's a > >> fair and valuable issue. I think if we go a few releases without any > >> major issues, we can change this to the following: > >> > >> Note that a SHA-256 repository cannot yet share work with "regular" > >> SHA-1 repositories. Many tools do not yet understand SHA-256 > >> repositories, so users may wish to take this into account when > >> creating new repositories. > > > > With respect, I think that's too aggressive. By that time, we may > > conclude that, e.g., the "v2 pack indices with SHA-256" file handling is > > robust. But I'd be surprised if using `git init --object-format=sha256` > > in June 2021 won't cause *some* extra work for users or ourselves > > further down the line compared to using a regular SHA-1 `git init`. > > Pushing to a SHA-1 hosting service will become *possible* at some point, > > but maybe it won't be *efficient enough to be practical in the real > > world* until some time after that. > > IOW, you question "if we go a few releases without any major issues" > part? I tend to agree that for a large change like this, a few > releases may not be sufficiently long time for a feature that is > marked as experimental in big flashing red letters to get exercised > enough to get major issues noticed. Yeah, thanks for summarizing what I failed to express using so many words. I'm fully open to the idea that some people want to leave SHA-1 behind and that they can do it today, in some "local" sense. If those people are fully aware that they are guinea pigs, it might actually be ok for us to subject them to a few rounds of "oops, Git v2.32.0 produces data that v2.34.0 and later will barf on". Or at least it would be on our table whether we wanted to be that cavalier. Once SHA-256 repos as such are no longer experimental, I fear that we can only buy ourselves that leeway by introducing fiftyeleven different config flags for "please produce auxiliary files X even if you don't actually use them", "please do use X, and I'm fully expecting to trip on them if you decide to tweak them in backwards-incompatible ways", and so on. The alternative to buying such leeway might be to establish, pretty early on, a respectable set of things we support "for compatibility reasons". > > Now would probably be a good time to update the hash transition > > documents, first of all to tick off what we've already done, and second, > > to reassess the rest. > > Yes, it is a good idea to stop and see where in the overall large > picture we currently are. Martin
Martin Ågren <martin.agren@gmail.com> writes: >> IOW, you question "if we go a few releases without any major issues" >> part? I tend to agree that for a large change like this, a few >> releases may not be sufficiently long time for a feature that is >> marked as experimental in big flashing red letters to get exercised >> enough to get major issues noticed. > > Yeah, thanks for summarizing what I failed to express using so many > words. > > I'm fully open to the idea that some people want to leave SHA-1 behind > and that they can do it today, in some "local" sense. If those people > are fully aware that they are guinea pigs, it might actually be ok for > us to subject them to a few rounds of "oops, Git v2.32.0 produces data > that v2.34.0 and later will barf on". Or at least it would be on our > table whether we wanted to be that cavalier. > > Once SHA-256 repos as such are no longer experimental, I fear that we > can only buy ourselves that leeway by introducing fiftyeleven different > config flags for "please produce auxiliary files X even if you don't > actually use them", "please do use X, and I'm fully expecting to trip on > them if you decide to tweak them in backwards-incompatible ways", and so > on. The alternative to buying such leeway might be to establish, pretty > early on, a respectable set of things we support "for compatibility > reasons". OK, so can we resolve this one way or the other and move on? For now, I'd vote for applying this warning patch, but with or without such warning, it is more important to iron out those details we fear might have to change. Thanks.
On 2020-08-13 at 20:34:10, Junio C Hamano wrote: > Martin Ågren <martin.agren@gmail.com> writes: > > >> IOW, you question "if we go a few releases without any major issues" > >> part? I tend to agree that for a large change like this, a few > >> releases may not be sufficiently long time for a feature that is > >> marked as experimental in big flashing red letters to get exercised > >> enough to get major issues noticed. > > > > Yeah, thanks for summarizing what I failed to express using so many > > words. > > > > I'm fully open to the idea that some people want to leave SHA-1 behind > > and that they can do it today, in some "local" sense. If those people > > are fully aware that they are guinea pigs, it might actually be ok for > > us to subject them to a few rounds of "oops, Git v2.32.0 produces data > > that v2.34.0 and later will barf on". Or at least it would be on our > > table whether we wanted to be that cavalier. > > > > Once SHA-256 repos as such are no longer experimental, I fear that we > > can only buy ourselves that leeway by introducing fiftyeleven different > > config flags for "please produce auxiliary files X even if you don't > > actually use them", "please do use X, and I'm fully expecting to trip on > > them if you decide to tweak them in backwards-incompatible ways", and so > > on. The alternative to buying such leeway might be to establish, pretty > > early on, a respectable set of things we support "for compatibility > > reasons". > > OK, so can we resolve this one way or the other and move on? > > For now, I'd vote for applying this warning patch, but with or > without such warning, it is more important to iron out those details > we fear might have to change. I'm fine with applying this patch. As for changes, I don't think there's any changes we need to make for a stage 4 implementation. It works and it passes the testsuite. I've verified the block and gcrypt SHA-256 implementations produce identical results, which was my major worry. Other than the philosophical disagreement over whether index v1 and v2 should support SHA-256, I don't think there's any points of contention. When we add support for SHA-1 interoperability, then we'll need pack index v3, proper multi-pack index support, and the loose object index. Those are being written at the moment, and I think it's fine to add a tool to generate the necessary files once that code is in place (which, it's looking like, can just be a shell script that pipes every loose object to git hash-object and rebuilds the pack indexes). Support for interoperability is an additional extension, so we don't have any compatibility concerns and we can generate all of those files based on that option. I don't plan to enable support for extensions.compatObjectFormat without all of the required pieces in place. There won't be any incremental steps there.
diff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt index 9316d9a80b..139285dd0f 100644 --- a/Documentation/git-index-pack.txt +++ b/Documentation/git-index-pack.txt @@ -100,6 +100,8 @@ OPTIONS value is set or outside a repository. + This option cannot be used with --stdin. ++ +include::object-format-disclaimer.txt[] NOTES ----- diff --git a/Documentation/git-init.txt b/Documentation/git-init.txt index ddfe265da5..f35f70f13d 100644 --- a/Documentation/git-init.txt +++ b/Documentation/git-init.txt @@ -53,6 +53,8 @@ current working directory. Specify the given object format (hash algorithm) for the repository. The valid values are 'sha1' and (if enabled) 'sha256'. 'sha1' is the default. ++ +include::object-format-disclaimer.txt[] --template=<template_directory>:: diff --git a/Documentation/git-show-index.txt b/Documentation/git-show-index.txt index 39b1d8eaa1..e49318a5a0 100644 --- a/Documentation/git-show-index.txt +++ b/Documentation/git-show-index.txt @@ -44,6 +44,8 @@ OPTIONS valid values are 'sha1' and (if enabled) 'sha256'. The default is the algorithm for the current repository (set by `extensions.objectFormat`), or 'sha1' if no value is set or outside a repository.. ++ +include::object-format-disclaimer.txt[] GIT --- diff --git a/Documentation/git.txt b/Documentation/git.txt index 3e50065198..e0a2690ba2 100644 --- a/Documentation/git.txt +++ b/Documentation/git.txt @@ -497,7 +497,8 @@ double-quotes and respecting backslash escapes. E.g., the value If this variable is set, the default hash algorithm for new repositories will be set to this value. This value is currently ignored when cloning; the setting of the remote repository - is used instead. The default is "sha1". + is used instead. The default is "sha1". THIS VARIABLE IS + EXPERIMENTAL! See `--object-format` in linkgit:git-init[1]. Git Commits ~~~~~~~~~~~ diff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt new file mode 100644 index 0000000000..4cb106f0d1 --- /dev/null +++ b/Documentation/object-format-disclaimer.txt @@ -0,0 +1,6 @@ +THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still +in an early stage. A SHA-256 repository will in general not be able to +share work with "regular" SHA-1 repositories. It should be assumed +that, e.g., Git internal file formats in relation to SHA-256 +repositories may change in backwards-incompatible ways. Only use +`--object-format=sha256` for testing purposes.
After eff45daab8 ("repository: enable SHA-256 support by default", 2020-07-29), vanilla builds of Git enable the user to run, e.g., git init --object-format=sha256 and hack away. This can be a good way to gain experience with the SHA-256 world, e.g., to find bugs that GIT_TEST_DEFAULT_HASH=sha256 make test doesn't spot. But it really is a separate world: Such SHA-256 repos will live entirely separate from the (by now fairly large) set of SHA-1 repos. Interacting across the border is possible in principle, e.g., through "diff + apply" (or "format-patch + am"), but even that has its limitations: Applying a SHA-256 diff in a SHA-1 repo works in the simple case, but if you need to resort to `-3`, you're out of luck. Similarly, "push + pull" should work, but you really will be operating mostly offset from the rest of the world. That might be ok by the time you initialize your repository, and it might be ok for several months after that, but there might come a day when you're starting to regret your use of `git init --object-format=sha256` and have dug yourself into a fairly deep hole. Workflows aside, let's consider a more technical aspect. Pack index files (pack-*.idx) exist in two flavours: v1 and v2. The hash transition document foresees a v3, which we do not yet support (and for all we know, the final v3 might end up different from the one sketched in the hash transition document). When the test suite is run with SHA-1 as the default hash algo, it creates and consumes v2 pack files. But with SHA-256, we use an undocumented, hybrid format where the header looks like v2, but where the payload is not only "not correct SHA1", but where even the data sizes are different. The trailing checksum is different, meaning no-one (except us!) should/would try to interpret this file as a proper v2 pack index. We could certainly (re)define v2 to match our SHA-256 behavior, but we do foresee v3 for a reason. And that would still just fix this specific issue. And even when everything around SHA-256 is well-defined and we have SHA-1--SHA-256 interoperability, there's a risk, at least initially, that somewhere we'd be permeating buggy data that we'd then feel responsible for and need to be able to handle for a long time to come. In short: we need some time and leeway. Wherever `--object-format` is mentioned in our documentation, let's make it clear that using it with "sha256" is experimental. If we later need to explain why we can't handle data we generated back in 2020, we can always point to this paragraph we're adding here. By "include::"-ing a small blurb, we should be able to be consistent throughout the documentation and can eventually gradually tone down the severity of this text. One day, we might even use it to start phasing out `--object-format=sha1`, but let's not get ahead of ourselves... There's also `extensions.objectFormat`, but it's only mentioned three times. Twice where we're adding this new disclaimer and in the third spot we already have a "do not edit" warning. From there, interested readers should eventually find this new one that we're adding here. Because `GIT_DEFAULT_HASH` provides another entry point to this functionality, document the experimental nature of it too. Signed-off-by: Martin Ågren <martin.agren@gmail.com> --- Documentation/git-index-pack.txt | 2 ++ Documentation/git-init.txt | 2 ++ Documentation/git-show-index.txt | 2 ++ Documentation/git.txt | 3 ++- Documentation/object-format-disclaimer.txt | 6 ++++++ 5 files changed, 14 insertions(+), 1 deletion(-) create mode 100644 Documentation/object-format-disclaimer.txt