| Message ID | a3902dad424983a4f0dfcda68e0b8bf64a0b2113.1709240261.git.dirk@gouders.net (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Documentation/user-manual.txt: try to clarify on object hashes | expand |
Dirk Gouders <dirk@gouders.net> writes: > If someone spends the time to work through the documentation, the > subject "hashes" can lead to contradictions: > > The README of the initial commit states hashes are generated from > compressed data (which changed very soon), whereas > Documentation/user-manual.txt says they are generated from original > data. > > Don't give doubts a chance: clarify this and present a simple example > on how object hashes can be generated manually. I'd rather not to waste readers' attention to historical wart. > @@ -4095,6 +4095,39 @@ that is used to name the object is the hash of the original data > plus this header, so `sha1sum` 'file' does not match the object name > for 'file'. The paragraph above (part of it is hidden before the hunk) clearly states what the naming rules are. We hash the original and then compress. If I use an implementation of Git that drives the zlib at compression level 1, and if you clone from my repository with another implementation of Git whose zlib is driven at compression level 9, our .git/objects/01/2345...90 files may not be identical, but when uncompressed they should store the same contents, so "hash then compress" is the only sensible choice that is not affected by the compression to give stable names to objects. > +Starting with the initial commit, hashing was done on the compressed > +data and the file README of that commit explicitely states this: > + > +"The SHA1 hash is always the hash of the _compressed_ object, not the > +original one." > + > +This changed soon after that with commit > +d98b46f8d9a3 (Do SHA1 hash _before_ compression.). Unfortunately, the > +commit message doesn't provide the detailed reasoning. These three are about Git development history, which by itself may be of interest for some people, but the main target audience of the user-manual is probably different from them. They may be interested to learn how Git works, but it is only to feel that they understand how the "magic" things Git does, like "a cryptographic hash of contents is enough to uniquely identify the contents being tracked", works well to trust their precious contents [*]. Side note: https://lore.kernel.org/git/Pine.LNX.4.58.0504200144260.6467@ppc970.osdl.org/ explains the reason behind the change to those who did not find it obvious. FYI, another "breaking" change we did earlier in the history of the project was to update the sort order of paths in tree objects. We do not need to confuse readers by talking about the original and updated sort order. The only thing they need, when they want to get the feeling that they understand how things work, is the description of how things work in the version of Git they have ready access to. Historical mistakes we made, corrections we made and why, are certainly of interest but not for the target audience of this document. On the other hand, ... > +The following is a short example that demonstrates how hashes can be > +generated manually: > + > +Let's asume a small text file with the content "Hello git.\n" > +------------------------------------------------- > +$ cat > hello.txt <<EOF > +Hello git. > +EOF > +------------------------------------------------- > + > +We can now manually generate the hash `git` would use for this file: > + > +- The object we want the hash for is of type "blob" and its size is > + 11 bytes. > + > +- Prepend the object header to the file content and feed this to > + sha1sum(1): > + > +------------------------------------------------- > +$ printf "blob 11\0" | cat - hello.txt | sha1sum > +7217614ba6e5f4e7db2edaa2cdf5fb5ee4358b57 . > +------------------------------------------------- > + ... something like the above (modulo coding style) would be a useful addition to help those who want to convince themselves they understand how (some parts of) Git works under the hood, and I think it would be a welcome addition to some subset of such readers (the rest of the world may feel it is way too much detail, though). I would draw the line between this one and a similar description and demonstration of historical mistakes, which is not as relevant as how things work in the current system. In other words, to me, it is OK to dig a bit deep to show how the current scheme works but it is way too much to do the same for versions of the system that do not exist anymore. But others may draw the line differently and consider even the above a bit too much detail, which is a position I would also accept. Thanks.
Junio C Hamano <gitster@pobox.com> writes: > Dirk Gouders <dirk@gouders.net> writes: > >> If someone spends the time to work through the documentation, the >> subject "hashes" can lead to contradictions: >> >> The README of the initial commit states hashes are generated from >> compressed data (which changed very soon), whereas >> Documentation/user-manual.txt says they are generated from original >> data. >> >> Don't give doubts a chance: clarify this and present a simple example >> on how object hashes can be generated manually. > > I'd rather not to waste readers' attention to historical wart. Yes, but -- I should have mentioned it -- the document itself suggests to read the initial commit. But I don't mean to argue about that, perhaps I digged to deep into details. >> @@ -4095,6 +4095,39 @@ that is used to name the object is the hash of the original data >> plus this header, so `sha1sum` 'file' does not match the object name >> for 'file'. > > The paragraph above (part of it is hidden before the hunk) clearly > states what the naming rules are. We hash the original and then > compress. If I use an implementation of Git that drives the zlib at > compression level 1, and if you clone from my repository with > another implementation of Git whose zlib is driven at compression > level 9, our .git/objects/01/2345...90 files may not be identical, > but when uncompressed they should store the same contents, so "hash > then compress" is the only sensible choice that is not affected by > the compression to give stable names to objects. Thank your for that detail. >> +Starting with the initial commit, hashing was done on the compressed >> +data and the file README of that commit explicitely states this: >> + >> +"The SHA1 hash is always the hash of the _compressed_ object, not the >> +original one." >> + >> +This changed soon after that with commit >> +d98b46f8d9a3 (Do SHA1 hash _before_ compression.). Unfortunately, the >> +commit message doesn't provide the detailed reasoning. > > These three are about Git development history, which by itself may > be of interest for some people, but the main target audience of the > user-manual is probably different from them. They may be interested > to learn how Git works, but it is only to feel that they understand > how the "magic" things Git does, like "a cryptographic hash of > contents is enough to uniquely identify the contents being tracked", > works well to trust their precious contents [*]. > > Side note: > https://lore.kernel.org/git/Pine.LNX.4.58.0504200144260.6467@ppc970.osdl.org/ > explains the reason behind the change to those who did not find > it obvious. > > FYI, another "breaking" change we did earlier in the history of the > project was to update the sort order of paths in tree objects. We > do not need to confuse readers by talking about the original and > updated sort order. The only thing they need, when they want to get > the feeling that they understand how things work, is the description > of how things work in the version of Git they have ready access to. > Historical mistakes we made, corrections we made and why, are > certainly of interest but not for the target audience of this > document. Again thank you, very interesting reading. > On the other hand, ... > >> +The following is a short example that demonstrates how hashes can be >> +generated manually: >> + >> +Let's asume a small text file with the content "Hello git.\n" >> +------------------------------------------------- >> +$ cat > hello.txt <<EOF >> +Hello git. >> +EOF >> +------------------------------------------------- >> + >> +We can now manually generate the hash `git` would use for this file: >> + >> +- The object we want the hash for is of type "blob" and its size is >> + 11 bytes. >> + >> +- Prepend the object header to the file content and feed this to >> + sha1sum(1): >> + >> +------------------------------------------------- >> +$ printf "blob 11\0" | cat - hello.txt | sha1sum >> +7217614ba6e5f4e7db2edaa2cdf5fb5ee4358b57 . >> +------------------------------------------------- >> + > > ... something like the above (modulo coding style) would be a useful > addition to help those who want to convince themselves they > understand how (some parts of) Git works under the hood, and I think > it would be a welcome addition to some subset of such readers (the > rest of the world may feel it is way too much detail, though). > > I would draw the line between this one and a similar description and > demonstration of historical mistakes, which is not as relevant as > how things work in the current system. In other words, to me, it is > OK to dig a bit deep to show how the current scheme works but it is > way too much to do the same for versions of the system that do not > exist anymore. > > But others may draw the line differently and consider even the above > a bit too much detail, which is a position I would also accept. > > Thanks.
Dirk Gouders <dirk@gouders.net> writes: >> I'd rather not to waste readers' attention to historical wart. > > Yes, but -- I should have mentioned it -- the document itself suggests > to read the initial commit. Ahh, yes, we'd need to hedge that part. Good thinking. I am still not sure if the first hunk below is a good idea or it is too much detail. The second hunk may be worth doing. Thanks. Documentation/user-manual.txt | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git c/Documentation/user-manual.txt w/Documentation/user-manual.txt index 6433903491..1027055784 100644 --- c/Documentation/user-manual.txt +++ w/Documentation/user-manual.txt @@ -4093,7 +4093,8 @@ that not only specifies their type, but also provides size information about the data in the object. It's worth noting that the SHA-1 hash that is used to name the object is the hash of the original data plus this header, so `sha1sum` 'file' does not match the object name -for 'file'. +for 'file' (the earliest versions of Git hashed slightly differently +but the conclusion is still the same). As a result, the general consistency of an object can always be tested independently of the contents or the type of the object: all objects can @@ -4123,7 +4124,8 @@ $ git switch --detach e83c5163 ---------------------------------------------------- The initial revision lays the foundation for almost everything Git has -today, but is small enough to read in one sitting. +today (even though details may differ in a few places), but is small +enough to read in one sitting. Note that terminology has changed since that revision. For example, the README in that revision uses the word "changeset" to describe what we
Junio C Hamano <gitster@pobox.com> writes: > Dirk Gouders <dirk@gouders.net> writes: >> +The following is a short example that demonstrates how hashes can be >> +generated manually: >> + >> +Let's asume a small text file with the content "Hello git.\n" >> +------------------------------------------------- >> +$ cat > hello.txt <<EOF >> +Hello git. >> +EOF >> +------------------------------------------------- >> + >> +We can now manually generate the hash `git` would use for this file: >> + >> +- The object we want the hash for is of type "blob" and its size is >> + 11 bytes. >> + >> +- Prepend the object header to the file content and feed this to >> + sha1sum(1): >> + >> +------------------------------------------------- >> +$ printf "blob 11\0" | cat - hello.txt | sha1sum >> +7217614ba6e5f4e7db2edaa2cdf5fb5ee4358b57 . >> +------------------------------------------------- >> + > > ... something like the above (modulo coding style) would be a useful > addition to help those who want to convince themselves they > understand how (some parts of) Git works under the hood, and I think > it would be a welcome addition to some subset of such readers (the > rest of the world may feel it is way too much detail, though). May I ask what you meant by "modulo coding style", e.g. where I should look at to make the code of similar style? I would also add that git-hash-object(1) could be used to verify the result if you think that is OK. In addition to a suggestion in another mail, the commit would consist of substantial content you suggested and perhaps, you could tell me how to express this; would a Helped-By be correct? Dirk
Dirk Gouders <dirk@gouders.net> writes: > May I ask what you meant by "modulo coding style", e.g. where I should > look at to make the code of similar style? Documentation/CodingGuidelines would be a good start, but * A here-doc for a single liner is probably an overkill. Why not echo "Hello, world" >file In either way, in our codebase a redirection operator '>' (or '<') has one whitespace before it, and no whitespace after it before the file. * printf piped to "cat - file" whose output feeds another pipe looked unusual. More usual way novices write may be { printf ... ; cat file; } | sha1sum were the two things I noticed. > I would also add that git-hash-object(1) could be used to verify the > result if you think that is OK. git hash-object can be used to replace that whole thing ;-) > In addition to a suggestion in another mail, the commit would > consist of substantial content you suggested and perhaps, you could tell > me how to express this; would a Helped-By be correct? I think many may prefer to downcase B in By, but if it is "substantial", probably. I do not think I gave much in this discussion to become a substantial addition to the original, though.
Junio C Hamano <gitster@pobox.com> writes: > Dirk Gouders <dirk@gouders.net> writes: > >> May I ask what you meant by "modulo coding style", e.g. where I should >> look at to make the code of similar style? > > Documentation/CodingGuidelines would be a good start, but > > * A here-doc for a single liner is probably an overkill. Why not > > echo "Hello, world" >file > > In either way, in our codebase a redirection operator '>' (or > '<') has one whitespace before it, and no whitespace after it > before the file. > > * printf piped to "cat - file" whose output feeds another pipe > looked unusual. More usual way novices write may be > > { printf ... ; cat file; } | sha1sum > > were the two things I noticed. > >> I would also add that git-hash-object(1) could be used to verify the >> result if you think that is OK. > > git hash-object can be used to replace that whole thing ;-) > >> In addition to a suggestion in another mail, the commit would >> consist of substantial content you suggested and perhaps, you could tell >> me how to express this; would a Helped-By be correct? > > I think many may prefer to downcase B in By, but if it is > "substantial", probably. I do not think I gave much in this > discussion to become a substantial addition to the original, though. Thank you for the explanation (some of which I should have found by myself). I will send the prepared v2 when I solved my struggling with range-diffs; that concept is new to me and I have a slow brain -- if one at all. Dirk
diff --git a/Documentation/user-manual.txt b/Documentation/user-manual.txt index 6433903491..8dfb81e045 100644 --- a/Documentation/user-manual.txt +++ b/Documentation/user-manual.txt @@ -4095,6 +4095,39 @@ that is used to name the object is the hash of the original data plus this header, so `sha1sum` 'file' does not match the object name for 'file'. +Starting with the initial commit, hashing was done on the compressed +data and the file README of that commit explicitely states this: + +"The SHA1 hash is always the hash of the _compressed_ object, not the +original one." + +This changed soon after that with commit +d98b46f8d9a3 (Do SHA1 hash _before_ compression.). Unfortunately, the +commit message doesn't provide the detailed reasoning. + +The following is a short example that demonstrates how hashes can be +generated manually: + +Let's asume a small text file with the content "Hello git.\n" +------------------------------------------------- +$ cat > hello.txt <<EOF +Hello git. +EOF +------------------------------------------------- + +We can now manually generate the hash `git` would use for this file: + +- The object we want the hash for is of type "blob" and its size is + 11 bytes. + +- Prepend the object header to the file content and feed this to + sha1sum(1): + +------------------------------------------------- +$ printf "blob 11\0" | cat - hello.txt | sha1sum +7217614ba6e5f4e7db2edaa2cdf5fb5ee4358b57 . +------------------------------------------------- + As a result, the general consistency of an object can always be tested independently of the contents or the type of the object: all objects can be validated by verifying that (a) their hashes match the content of the
If someone spends the time to work through the documentation, the subject "hashes" can lead to contradictions: The README of the initial commit states hashes are generated from compressed data (which changed very soon), whereas Documentation/user-manual.txt says they are generated from original data. Don't give doubts a chance: clarify this and present a simple example on how object hashes can be generated manually. Signed-off-by: Dirk Gouders <dirk@gouders.net> --- Documentation/user-manual.txt | 33 +++++++++++++++++++++++++++++++++ 1 file changed, 33 insertions(+)