mbox series

[00/13] Support for arbitrary schemes in credentials

Message ID 20240324011301.1553072-1-sandals@crustytoothpaste.net (mailing list archive)
Headers show
Series Support for arbitrary schemes in credentials | expand

Message

brian m. carlson March 24, 2024, 1:12 a.m. UTC
Right now, HTTP authentication in Git is mostly limited to approaches
that require a username and password or are Kerberos (GSSAPI).  In
addition, we effectively require that libcurl (or, for other software,
such as Git LFS, using the credential helper, that HTTP library) knows
how to implement the authentication scheme.

However, this poses two sets of problems.  First, some sites, such as
Azure DevOps, want to use Bearer authentication, which we don't support.
This is implemented using `http.extraHeader`, which is not a secure way
to store credentials, since our credential helper protocol does not
support this functionality.

In addition, other tools using the credential helper protocol do not
support the variety of authentication mechanisms that Git does.
Specifically, making NTLM function in a useful way on Windows is
nontrivial and requires extensive integration and testing with C code,
and because of this difficulty and the fact that NTLM uses cryptography
known to be insecure since 1995, there is often little interest in
implementing this support outside of libcurl. However, it would be
helpful if people who want to use it can still use it.

This series introduces new functionality to the credential helper
protocol that allows helpers to produce credentials for arbitrary HTTP
authentication schemes using the `authtype` and `credential`[0] fields.
This allows a suitable credential helper to send Bearer credentials or
any other standard or custom authentication scheme.  (It may be able to
be extended to other functionality in the future, such as
git-send-email, to implement custom SASL functionality, and due care has
been taken to make the protocol adequately flexible for that purpose.)

In addition, the protocol is also expanded to include per-helper state
and multi-legged authentication (the former is effectively required for
the latter).  The per-helper state can be useful to help credential
helpers identify where the credential is stored, or any other
information necessary.  Because NTLM and Negotiate (Kerberos/wrapped
NTLM) require two rounds of authentication, the multi-legged
authentication support along with per-helper state allows the helper to
support these authentication methods without Git or other clients having
to be aware of how they work.  (This would also be useful for SASL, as
mentioned above.)

This series introduces a capability mechanism to announce this
functionality, which allows a helper to provide a username and password
on older versions of Git while supporting more advanced functionality on
newer versions.  (This is especially important on Azure DevOps, where
NTLM uses a username and password but Basic or Bearer can use a personal
access token.)  It is also designed such that extremely simple
credential helpers, such as the shell one-liner in the Git FAQ that
reads from the environment, don't accidentally claim to support
functionality they don't offer.

In addition, there is documentation for the expanded protocol, although
none of the built-in helpers have been updated (that will be a future
series for those for which it's possible).

My personal interest here is getting credentials out of config files
with `http.extraHeader` (which a future series will produce a warning
for) and also allowing Git LFS to support Digest and NTLM with a
suitable credential helper.  Git LFS used to support NTLM using custom
code (because the Go standard library does not), but it was found to be
broken in lots of ways on Windows, and nobody with a Windows system
wanted to fix it or support it, so we removed it.  However, there are
still some people who do want to use it, so allowing them to use a
custom credential helper they maintain themselves seems like the best
way forward.  Despite the advantages of this series for Azure DevOps, I
have no personal or professional stake in their product; my only
interest is the general one in whether their users can securely store
credentials.  I believe the changes here are of general advantage to the
Git userbase in a variety of ways such that the goal of this series
should be uncontroversial.

Feedback on any portion of this series is of course welcome.

[0] A name different from `password` was explicitly chosen to avoid
confusion from less capable protocol helpers so that they don't
accidentally send invalid data.  This does have the downside that
credential helpers must learn a new field to not log, but that should be
generally easy to fix in most cases.

brian m. carlson (13):
  credential: add an authtype field
  remote-curl: reset headers on new request
  http: use new headers for each object request
  credential: add a field for pre-encoded credentials
  credential: gate new fields on capability
  docs: indicate new credential protocol fields
  http: add support for authtype and credential
  credential: add an argument to keep state
  credential: enable state capability
  docs: set a limit on credential line length
  t5563: refactor for multi-stage authentication
  strvec: implement swapping two strvecs
  credential: add support for multistage credential rounds

 Documentation/git-credential.txt   |  59 +++++-
 builtin/credential-cache--daemon.c |   2 +-
 builtin/credential-store.c         |   2 +-
 builtin/credential.c               |   7 +-
 credential.c                       | 114 ++++++++++-
 credential.h                       |  69 ++++++-
 http.c                             | 128 +++++++-----
 http.h                             |   5 +
 imap-send.c                        |   2 +-
 remote-curl.c                      |  14 +-
 strvec.c                           |   7 +
 strvec.h                           |   5 +
 t/lib-httpd/nph-custom-auth.sh     |  17 +-
 t/t0300-credentials.sh             | 136 ++++++++++++-
 t/t5563-simple-http-auth.sh        | 308 +++++++++++++++++++++++++----
 15 files changed, 760 insertions(+), 115 deletions(-)

Comments

Junio C Hamano March 24, 2024, 2:24 a.m. UTC | #1
"brian m. carlson" <sandals@crustytoothpaste.net> writes:

> ... and because of this difficulty and the fact that NTLM uses cryptography
> known to be insecure since 1995, there is often little interest in
> implementing this support outside of libcurl. However, it would be
> helpful if people who want to use it can still use it.

This position was a bit surprising to me to come from you, but
perhaps I am mixing up my recollection of your past work on this
project with somebody else's?  I somehow expected to hear something
more like "if a less secure thing is cumbersome to implement, let it
be, as that is better for the world".  But I am OK to add less secure
thing as long as it is an opt-in "easy way out".

Everything else I read in the cover letter made sense to me.  I just
wanted to say that the above part was a bit surprising.

Thanks.
brian m. carlson March 24, 2024, 3:21 p.m. UTC | #2
On 2024-03-24 at 02:24:46, Junio C Hamano wrote:
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> 
> > ... and because of this difficulty and the fact that NTLM uses cryptography
> > known to be insecure since 1995, there is often little interest in
> > implementing this support outside of libcurl. However, it would be
> > helpful if people who want to use it can still use it.
> 
> This position was a bit surprising to me to come from you, but
> perhaps I am mixing up my recollection of your past work on this
> project with somebody else's?  I somehow expected to hear something
> more like "if a less secure thing is cumbersome to implement, let it
> be, as that is better for the world".  But I am OK to add less secure
> thing as long as it is an opt-in "easy way out".
> 
> Everything else I read in the cover letter made sense to me.  I just
> wanted to say that the above part was a bit surprising.

I do firmly feel that NTLM should go gentle into that good night.

However, we've seen a lot of user questions on Git LFS from users who
want to use NTLM or Digest, which we don't support and probably will not
for security and maintainability reasons, but which their corporate
environment imprudently requires. Thus, my goal is to make it _possible_
for people to implement this, but it to make it their responsibility (or
that of a suitable open source project) to do so instead of asking me to
maintain it.  That, I think, is a fair and equitable tradeoff for this
situation.

Also, right now, Windows users sometimes choose to use the older v2.13.3
version of Git LFS which, due to a Go standard library bug, has an
arbitrary code execution vulnerability, but which did support NTLM.
Thus, it would also be better for security to have people on a suitably
patched version of Git LFS with an external NTLM helper.

This series would also, in an approach that's better for security, allow
people to use better Kerberos mechanisms than what Windows supports
natively, or to use an AWS HMAC-v4-style request signing (using
HMAC-SHA-256) if they want to do so, both of which would be a win for
security.

I could have put all of this into the cover letter, but I felt it was
pretty long and didn't want to sell this as an advantage only for Git
LFS when I think it provides general benefit for Git users.  I know
the policy of the project is not to prioritize any one external
project, and I try to be sensitive to that here.

I don't think this constitutes a marked change in my historical
"let's-remove-all-the-obsolete-cryptography-and-obsolete-operating-systems"
approach, but I see how it might look that way at first glance.
Junio C Hamano March 24, 2024, 4:13 p.m. UTC | #3
"brian m. carlson" <sandals@crustytoothpaste.net> writes:

>> be, as that is better for the world".  But I am OK to add less secure
>> thing as long as it is an opt-in "easy way out".
> 
> ... Thus, my goal is to make it _possible_
> for people to implement this, but it to make it their responsibility (or
> that of a suitable open source project) to do so instead of asking me to
> maintain it.  That, I think, is a fair and equitable tradeoff for this
> situation.

Yup, I think we are in agreement.  I have no problem with giving
them a long rope ;-)
M Hickford March 30, 2024, 8 a.m. UTC | #4
On Sun, 24 Mar 2024 at 01:13, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> Right now, HTTP authentication in Git is mostly limited to approaches
> that require a username and password or are Kerberos (GSSAPI).  In
> addition, we effectively require that libcurl (or, for other software,
> such as Git LFS, using the credential helper, that HTTP library) knows
> how to implement the authentication scheme.
>
> However, this poses two sets of problems.  First, some sites, such as
> Azure DevOps, want to use Bearer authentication, which we don't support.

Git's lack of support for Bearer auth is also a frustration for OAuth
-- OAuth tokens are intended as Bearer tokens. As a workaround,
popular Git hosts support Basic auth with an OAuth token as password.
For certain hosts, this requires a magic username such as 'oauth2'
(GitLab) or 'x-token-auth' (Bitbucket). This is problematic if the
user has multiple accounts, or a different username stored in
credential.username or the remote URL.

> This is implemented using `http.extraHeader`, which is not a secure way
> to store credentials, since our credential helper protocol does not
> support this functionality.

Config key http.extraHeader is particularly unsuitable to hold
short-lived Bearer tokens. For example, GitLab's OAuth tokens expire
after 2 hours.

>
> In addition, other tools using the credential helper protocol do not
> support the variety of authentication mechanisms that Git does.
> Specifically, making NTLM function in a useful way on Windows is
> nontrivial and requires extensive integration and testing with C code,
> and because of this difficulty and the fact that NTLM uses cryptography
> known to be insecure since 1995, there is often little interest in
> implementing this support outside of libcurl. However, it would be
> helpful if people who want to use it can still use it.
>
> This series introduces new functionality to the credential helper
> protocol that allows helpers to produce credentials for arbitrary HTTP
> authentication schemes using the `authtype` and `credential`[0] fields.

Exciting! I drafted a patch to make git-credential-oauth generate
Bearer credentials
https://github.com/hickford/git-credential-oauth/pull/51 . Tested
together with your patch, it works for host bitbucket.org. Though of
course it isn't practical while storage helpers (such as
credential-cache, credential-wincred, credential-libsecret and
credential-osxkeychain) are unable to store the new attributes -- the
user has to complete the OAuth flow for every authentication.

> This allows a suitable credential helper to send Bearer credentials or
> any other standard or custom authentication scheme.  (It may be able to
> be extended to other functionality in the future, such as
> git-send-email, to implement custom SASL functionality, and due care has
> been taken to make the protocol adequately flexible for that purpose.)
>
> In addition, the protocol is also expanded to include per-helper state
> and multi-legged authentication (the former is effectively required for
> the latter).  The per-helper state can be useful to help credential
> helpers identify where the credential is stored, or any other
> information necessary.  Because NTLM and Negotiate (Kerberos/wrapped
> NTLM) require two rounds of authentication, the multi-legged
> authentication support along with per-helper state allows the helper to
> support these authentication methods without Git or other clients having
> to be aware of how they work.  (This would also be useful for SASL, as
> mentioned above.)
>
> This series introduces a capability mechanism to announce this
> functionality, which allows a helper to provide a username and password
> on older versions of Git while supporting more advanced functionality on
> newer versions.  (This is especially important on Azure DevOps, where
> NTLM uses a username and password but Basic or Bearer can use a personal
> access token.)  It is also designed such that extremely simple
> credential helpers, such as the shell one-liner in the Git FAQ that
> reads from the environment, don't accidentally claim to support
> functionality they don't offer.




>
> In addition, there is documentation for the expanded protocol, although
> none of the built-in helpers have been updated (that will be a future
> series for those for which it's possible).
>
> My personal interest here is getting credentials out of config files
> with `http.extraHeader` (which a future series will produce a warning
> for) and also allowing Git LFS to support Digest and NTLM with a
> suitable credential helper.  Git LFS used to support NTLM using custom
> code (because the Go standard library does not), but it was found to be
> broken in lots of ways on Windows, and nobody with a Windows system
> wanted to fix it or support it, so we removed it.  However, there are
> still some people who do want to use it, so allowing them to use a
> custom credential helper they maintain themselves seems like the best
> way forward.  Despite the advantages of this series for Azure DevOps, I
> have no personal or professional stake in their product; my only
> interest is the general one in whether their users can securely store
> credentials.  I believe the changes here are of general advantage to the
> Git userbase in a variety of ways such that the goal of this series
> should be uncontroversial.
>
> Feedback on any portion of this series is of course welcome.
>
> [0] A name different from `password` was explicitly chosen to avoid
> confusion from less capable protocol helpers so that they don't
> accidentally send invalid data.  This does have the downside that
> credential helpers must learn a new field to not log, but that should be
> generally easy to fix in most cases.
>
> brian m. carlson (13):
>   credential: add an authtype field
>   remote-curl: reset headers on new request
>   http: use new headers for each object request
>   credential: add a field for pre-encoded credentials
>   credential: gate new fields on capability
>   docs: indicate new credential protocol fields
>   http: add support for authtype and credential
>   credential: add an argument to keep state
>   credential: enable state capability
>   docs: set a limit on credential line length
>   t5563: refactor for multi-stage authentication
>   strvec: implement swapping two strvecs
>   credential: add support for multistage credential rounds
>
>  Documentation/git-credential.txt   |  59 +++++-
>  builtin/credential-cache--daemon.c |   2 +-
>  builtin/credential-store.c         |   2 +-
>  builtin/credential.c               |   7 +-
>  credential.c                       | 114 ++++++++++-
>  credential.h                       |  69 ++++++-
>  http.c                             | 128 +++++++-----
>  http.h                             |   5 +
>  imap-send.c                        |   2 +-
>  remote-curl.c                      |  14 +-
>  strvec.c                           |   7 +
>  strvec.h                           |   5 +
>  t/lib-httpd/nph-custom-auth.sh     |  17 +-
>  t/t0300-credentials.sh             | 136 ++++++++++++-
>  t/t5563-simple-http-auth.sh        | 308 +++++++++++++++++++++++++----
>  15 files changed, 760 insertions(+), 115 deletions(-)
>
M Hickford March 30, 2024, 8:16 a.m. UTC | #5
On Sun, 24 Mar 2024 at 01:13, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> Right now, HTTP authentication in Git is mostly limited to approaches
> that require a username and password or are Kerberos (GSSAPI).  In
> addition, we effectively require that libcurl (or, for other software,
> such as Git LFS, using the credential helper, that HTTP library) knows
> how to implement the authentication scheme.
>
> However, this poses two sets of problems.  First, some sites, such as
> Azure DevOps, want to use Bearer authentication, which we don't support.
> This is implemented using `http.extraHeader`, which is not a secure way
> to store credentials, since our credential helper protocol does not
> support this functionality.
>
> In addition, other tools using the credential helper protocol do not
> support the variety of authentication mechanisms that Git does.
> Specifically, making NTLM function in a useful way on Windows is
> nontrivial and requires extensive integration and testing with C code,
> and because of this difficulty and the fact that NTLM uses cryptography
> known to be insecure since 1995, there is often little interest in
> implementing this support outside of libcurl. However, it would be
> helpful if people who want to use it can still use it.
>
> This series introduces new functionality to the credential helper
> protocol that allows helpers to produce credentials for arbitrary HTTP
> authentication schemes using the `authtype` and `credential`[0] fields.
> This allows a suitable credential helper to send Bearer credentials or
> any other standard or custom authentication scheme.  (It may be able to
> be extended to other functionality in the future, such as
> git-send-email, to implement custom SASL functionality, and due care has
> been taken to make the protocol adequately flexible for that purpose.)
>
> In addition, the protocol is also expanded to include per-helper state
> and multi-legged authentication (the former is effectively required for
> the latter).  The per-helper state can be useful to help credential
> helpers identify where the credential is stored, or any other
> information necessary.  Because NTLM and Negotiate (Kerberos/wrapped
> NTLM) require two rounds of authentication, the multi-legged
> authentication support along with per-helper state allows the helper to
> support these authentication methods without Git or other clients having
> to be aware of how they work.  (This would also be useful for SASL, as
> mentioned above.)
>
> This series introduces a capability mechanism to announce this
> functionality, which allows a helper to provide a username and password
> on older versions of Git while supporting more advanced functionality on
> newer versions.  (This is especially important on Azure DevOps, where
> NTLM uses a username and password but Basic or Bearer can use a personal
> access token.)  It is also designed such that extremely simple
> credential helpers, such as the shell one-liner in the Git FAQ that
> reads from the environment, don't accidentally claim to support
> functionality they don't offer.
>
> In addition, there is documentation for the expanded protocol, although
> none of the built-in helpers have been updated (that will be a future
> series for those for which it's possible).

If you've time, I think it would be instructive to add (conditional)
tests to libcredential and support in at least one storage helper,
such as credential-cache.

This would clarify the behaviour in subtle cases involving expiry and
multiple helpers.

>
> My personal interest here is getting credentials out of config files
> with `http.extraHeader` (which a future series will produce a warning
> for) and also allowing Git LFS to support Digest and NTLM with a
> suitable credential helper.  Git LFS used to support NTLM using custom
> code (because the Go standard library does not), but it was found to be
> broken in lots of ways on Windows, and nobody with a Windows system
> wanted to fix it or support it, so we removed it.  However, there are
> still some people who do want to use it, so allowing them to use a
> custom credential helper they maintain themselves seems like the best
> way forward.  Despite the advantages of this series for Azure DevOps, I
> have no personal or professional stake in their product; my only
> interest is the general one in whether their users can securely store
> credentials.  I believe the changes here are of general advantage to the
> Git userbase in a variety of ways such that the goal of this series
> should be uncontroversial.
>
> Feedback on any portion of this series is of course welcome.
>
> [0] A name different from `password` was explicitly chosen to avoid
> confusion from less capable protocol helpers so that they don't
> accidentally send invalid data.  This does have the downside that
> credential helpers must learn a new field to not log, but that should be
> generally easy to fix in most cases.
>
> brian m. carlson (13):
>   credential: add an authtype field
>   remote-curl: reset headers on new request
>   http: use new headers for each object request
>   credential: add a field for pre-encoded credentials
>   credential: gate new fields on capability
>   docs: indicate new credential protocol fields
>   http: add support for authtype and credential
>   credential: add an argument to keep state
>   credential: enable state capability
>   docs: set a limit on credential line length
>   t5563: refactor for multi-stage authentication
>   strvec: implement swapping two strvecs
>   credential: add support for multistage credential rounds
>
>  Documentation/git-credential.txt   |  59 +++++-
>  builtin/credential-cache--daemon.c |   2 +-
>  builtin/credential-store.c         |   2 +-
>  builtin/credential.c               |   7 +-
>  credential.c                       | 114 ++++++++++-
>  credential.h                       |  69 ++++++-
>  http.c                             | 128 +++++++-----
>  http.h                             |   5 +
>  imap-send.c                        |   2 +-
>  remote-curl.c                      |  14 +-
>  strvec.c                           |   7 +
>  strvec.h                           |   5 +
>  t/lib-httpd/nph-custom-auth.sh     |  17 +-
>  t/t0300-credentials.sh             | 136 ++++++++++++-
>  t/t5563-simple-http-auth.sh        | 308 +++++++++++++++++++++++++----
>  15 files changed, 760 insertions(+), 115 deletions(-)
>
Calvin Wan April 2, 2024, 10:26 p.m. UTC | #6
Hi Brian,

While I personally do not know the specifics of how Git authentication
works at Google, I am passing this series along to the team that does
own Git authentication (adding Jackson to this reply).

"brian m. carlson" <sandals@crustytoothpaste.net> writes:
> Right now, HTTP authentication in Git is mostly limited to approaches
> that require a username and password or are Kerberos (GSSAPI).  In
> addition, we effectively require that libcurl (or, for other software,
> such as Git LFS, using the credential helper, that HTTP library) knows
> how to implement the authentication scheme.
> 
> However, this poses two sets of problems.  First, some sites, such as
> Azure DevOps, want to use Bearer authentication, which we don't support.
> This is implemented using `http.extraHeader`, which is not a secure way
> to store credentials, since our credential helper protocol does not
> support this functionality.

My first thought was if using `http.extraHeader` is insecure as you
claim and we use that internally, then how do we ensure that it is
secure? Or are you claiming that using `http.extraHeader` out of the box
without an additional security layer is insecure?

Thanks,
Calvin
brian m. carlson April 4, 2024, 1:01 a.m. UTC | #7
On 2024-04-02 at 22:26:19, Calvin Wan wrote:
> Hi Brian,
> 
> While I personally do not know the specifics of how Git authentication
> works at Google, I am passing this series along to the team that does
> own Git authentication (adding Jackson to this reply).
> 
> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
> > Right now, HTTP authentication in Git is mostly limited to approaches
> > that require a username and password or are Kerberos (GSSAPI).  In
> > addition, we effectively require that libcurl (or, for other software,
> > such as Git LFS, using the credential helper, that HTTP library) knows
> > how to implement the authentication scheme.
> > 
> > However, this poses two sets of problems.  First, some sites, such as
> > Azure DevOps, want to use Bearer authentication, which we don't support.
> > This is implemented using `http.extraHeader`, which is not a secure way
> > to store credentials, since our credential helper protocol does not
> > support this functionality.
> 
> My first thought was if using `http.extraHeader` is insecure as you
> claim and we use that internally, then how do we ensure that it is
> secure? Or are you claiming that using `http.extraHeader` out of the box
> without an additional security layer is insecure?

Storing plaintext credentials on disk is just not a good idea, and it's
not a secure way to store them.  This is why `.netrc` is a less than
great idea, but for Git, it's also possible to have a shared repository
where information in `.git/config` can leak, and sometimes people also
just expose Git repositories accidentally over HTTP (say, on websites)
and leak all of their config.  Sometimes people put `http.extraheader`
in `~/.gitconfig` and then check it into their public dotfiles, and then
push it to GitHub, for example.

It's a little less of a problem if it's a personal laptop and nobody
else uses it, but it's still a lot easier to accidentally expose an
arbitrary file or for an attack to exfiltrate an existing file (just
through a bug in existing software) than it is to necessarily execute
the arbitrary code necessary to read the data out of the system
credential store.

`http.extraheader` for Authorization headers usually necessitates that
the data is either stored in the config file or passed on the command
line, and that's why it's insecure.  Certainly, you could configure it
to read only from the environment using `--config-env` or you could
configure your system to store the data in the config only with a
single, highly restricted service account and it might be okay.

The kind of usage of `http.extraheader` that's likely to be fine is just
passing an extra header that some broken proxy needs to be satisfied,
like setting a specific language or faking a header that the proxy needs
to think Git's a web browser (since, of course, if it's not Internet
Explorer, it's insecure).  As long as you're not storing credentials or
secrets in `http.extraheader`, I have no objections.

I don't know what you're using it for at Google, but of course if it is
Authorization headers, then I'm hoping this series will help you avoid
needing to do that.
Jackson Toeniskoetter April 8, 2024, 6:42 p.m. UTC | #8
Just to clarify, we at Google do not use extraheader to pass along
credentials. Instead we use the .gitcookies file[1], which iiuc gets
read by the git process directly. I'm not a security expert but I
imagine the risk surface of extraheader and .gitcookies is similar.

Reading your post on
https://lore.kernel.org/git/20240324011301.1553072-1-sandals@crustytoothpaste.net/,
it's unclear to me why the credential helper protocol needs to be
updated. If the goal is to support Bearer tokens, can that not just be
implemented using extraheader? It seems like the goal of getting
credentials out of the config file can be accomplished without
updating the credential helper protocol. So is the bigger goal to
support more robust and modern auth schemes which require multiple
steps? That would be useful to Google; multi-step auth would probably
be a more elegant way for us to stop using .gitcookies than other
solutions we were considering.

[1] This is actually only for external contributors. Google employees
have a more robust authentication mechanism. Concerns have been raised
internally about our usage of gitcookies, but it hasn't been made a
priority to address because a leaked credential would not allow an
attacker to commit bad code, only read it or initiate a code review.
M Hickford April 11, 2024, 7 a.m. UTC | #9
On Mon, 8 Apr 2024 at 19:43, Jackson Toeniskoetter <jackdt@google.com> wrote:
>
> Just to clarify, we at Google do not use extraheader to pass along
> credentials. Instead we use the .gitcookies file[1], which iiuc gets
> read by the git process directly. I'm not a security expert but I
> imagine the risk surface of extraheader and .gitcookies is similar.

Hi Jackson. To clarify, are you describing hosts *.googlesource.com
such as https://go.googlesource.com/? It always confused me that the
'generate password' feature gives you something different to a
password.

https://www.googlesource.com/new-password
https://gerrit-review.googlesource.com/Documentation/user-upload.html

Have you tried OAuth credential helper git-credential-oauth
https://github.com/hickford/git-credential-oauth? It can authenticate
to *.googlesource.com without setup. OAuth has security advantages
over .gitcookies because the OAuth access tokens expire after 2 hours
and the OAuth refresh tokens are single use. These credentials are
stored in the user's choice of secure storage such as
git-credential-cache, git-credential-libsecret, git-credential-wincred
or git-credential-osxkeychain.

I encourage you to try it out. You'll need a local web browser because
Google's OAuth configuration currently forbids device authorization
grant for the relevant scope
https://issues.gerritcodereview.com/issues/300279941

>
> Reading your post on
> https://lore.kernel.org/git/20240324011301.1553072-1-sandals@crustytoothpaste.net/,
> it's unclear to me why the credential helper protocol needs to be
> updated. If the goal is to support Bearer tokens, can that not just be
> implemented using extraheader? It seems like the goal of getting
> credentials out of the config file can be accomplished without
> updating the credential helper protocol. So is the bigger goal to
> support more robust and modern auth schemes which require multiple
> steps? That would be useful to Google; multi-step auth would probably
> be a more elegant way for us to stop using .gitcookies than other
> solutions we were considering.
>
> [1] This is actually only for external contributors. Google employees
> have a more robust authentication mechanism. Concerns have been raised
> internally about our usage of gitcookies, but it hasn't been made a
> priority to address because a leaked credential would not allow an
> attacker to commit bad code, only read it or initiate a code review.
M Hickford April 11, 2024, 7 a.m. UTC | #10
On Sun, 24 Mar 2024 at 01:13, brian m. carlson
<sandals@crustytoothpaste.net> wrote:
>
> Right now, HTTP authentication in Git is mostly limited to approaches
> that require a username and password or are Kerberos (GSSAPI).  In
> addition, we effectively require that libcurl (or, for other software,
> such as Git LFS, using the credential helper, that HTTP library) knows
> how to implement the authentication scheme.
>
> However, this poses two sets of problems.  First, some sites, such as
> Azure DevOps, want to use Bearer authentication, which we don't support.
> This is implemented using `http.extraHeader`, which is not a secure way
> to store credentials, since our credential helper protocol does not
> support this functionality.
>
> In addition, other tools using the credential helper protocol do not
> support the variety of authentication mechanisms that Git does.
> Specifically, making NTLM function in a useful way on Windows is
> nontrivial and requires extensive integration and testing with C code,
> and because of this difficulty and the fact that NTLM uses cryptography
> known to be insecure since 1995, there is often little interest in
> implementing this support outside of libcurl. However, it would be
> helpful if people who want to use it can still use it.
>
> This series introduces new functionality to the credential helper
> protocol that allows helpers to produce credentials for arbitrary HTTP
> authentication schemes using the `authtype` and `credential`[0] fields.
> This allows a suitable credential helper to send Bearer credentials or
> any other standard or custom authentication scheme.  (It may be able to
> be extended to other functionality in the future, such as
> git-send-email, to implement custom SASL functionality, and due care has
> been taken to make the protocol adequately flexible for that purpose.)
>
> In addition, the protocol is also expanded to include per-helper state
> and multi-legged authentication (the former is effectively required for
> the latter).  The per-helper state can be useful to help credential
> helpers identify where the credential is stored, or any other
> information necessary.  Because NTLM and Negotiate (Kerberos/wrapped
> NTLM) require two rounds of authentication, the multi-legged
> authentication support along with per-helper state allows the helper to
> support these authentication methods without Git or other clients having
> to be aware of how they work.  (This would also be useful for SASL, as
> mentioned above.)
>
> This series introduces a capability mechanism to announce this
> functionality, which allows a helper to provide a username and password
> on older versions of Git while supporting more advanced functionality on
> newer versions.  (This is especially important on Azure DevOps, where
> NTLM uses a username and password but Basic or Bearer can use a personal
> access token.)  It is also designed such that extremely simple
> credential helpers, such as the shell one-liner in the Git FAQ that
> reads from the environment, don't accidentally claim to support
> functionality they don't offer.
>
> In addition, there is documentation for the expanded protocol, although
> none of the built-in helpers have been updated (that will be a future
> series for those for which it's possible).
>
> My personal interest here is getting credentials out of config files
> with `http.extraHeader` (which a future series will produce a warning
> for) and also allowing Git LFS to support Digest and NTLM with a
> suitable credential helper.  Git LFS used to support NTLM using custom
> code (because the Go standard library does not), but it was found to be
> broken in lots of ways on Windows, and nobody with a Windows system
> wanted to fix it or support it, so we removed it.  However, there are
> still some people who do want to use it, so allowing them to use a
> custom credential helper they maintain themselves seems like the best
> way forward.  Despite the advantages of this series for Azure DevOps, I
> have no personal or professional stake in their product; my only
> interest is the general one in whether their users can securely store
> credentials.  I believe the changes here are of general advantage to the
> Git userbase in a variety of ways such that the goal of this series
> should be uncontroversial.

Do you happen to know any public Git remotes that support Digest or
NTLM authentication?



>
> Feedback on any portion of this series is of course welcome.
>
> [0] A name different from `password` was explicitly chosen to avoid
> confusion from less capable protocol helpers so that they don't
> accidentally send invalid data.  This does have the downside that
> credential helpers must learn a new field to not log, but that should be
> generally easy to fix in most cases.
>
> brian m. carlson (13):
>   credential: add an authtype field
>   remote-curl: reset headers on new request
>   http: use new headers for each object request
>   credential: add a field for pre-encoded credentials
>   credential: gate new fields on capability
>   docs: indicate new credential protocol fields
>   http: add support for authtype and credential
>   credential: add an argument to keep state
>   credential: enable state capability
>   docs: set a limit on credential line length
>   t5563: refactor for multi-stage authentication
>   strvec: implement swapping two strvecs
>   credential: add support for multistage credential rounds
>
>  Documentation/git-credential.txt   |  59 +++++-
>  builtin/credential-cache--daemon.c |   2 +-
>  builtin/credential-store.c         |   2 +-
>  builtin/credential.c               |   7 +-
>  credential.c                       | 114 ++++++++++-
>  credential.h                       |  69 ++++++-
>  http.c                             | 128 +++++++-----
>  http.h                             |   5 +
>  imap-send.c                        |   2 +-
>  remote-curl.c                      |  14 +-
>  strvec.c                           |   7 +
>  strvec.h                           |   5 +
>  t/lib-httpd/nph-custom-auth.sh     |  17 +-
>  t/t0300-credentials.sh             | 136 ++++++++++++-
>  t/t5563-simple-http-auth.sh        | 308 +++++++++++++++++++++++++----
>  15 files changed, 760 insertions(+), 115 deletions(-)
>
brian m. carlson April 12, 2024, 12:09 a.m. UTC | #11
On 2024-04-08 at 18:42:56, Jackson Toeniskoetter wrote:
> Just to clarify, we at Google do not use extraheader to pass along
> credentials. Instead we use the .gitcookies file[1], which iiuc gets
> read by the git process directly. I'm not a security expert but I
> imagine the risk surface of extraheader and .gitcookies is similar.
> 
> Reading your post on
> https://lore.kernel.org/git/20240324011301.1553072-1-sandals@crustytoothpaste.net/,
> it's unclear to me why the credential helper protocol needs to be
> updated. If the goal is to support Bearer tokens, can that not just be
> implemented using extraheader? It seems like the goal of getting
> credentials out of the config file can be accomplished without
> updating the credential helper protocol. So is the bigger goal to
> support more robust and modern auth schemes which require multiple
> steps? That would be useful to Google; multi-step auth would probably
> be a more elegant way for us to stop using .gitcookies than other
> solutions we were considering.

Bearer authentication certainly can be implemented using
http.extraheader and the config file is also not a secure way to store
credentials, which is why the credential helper protocol is being
updated, since then people will be able to store the credentials in a
password manager or other secure store.  Other HTTP schemes will also be
supported as long as they don't require headers other than Authorization
and the credential helper can implement them on behalf of Git.
brian m. carlson April 12, 2024, 12:13 a.m. UTC | #12
On 2024-04-11 at 07:00:00, M Hickford wrote:
> Do you happen to know any public Git remotes that support Digest or
> NTLM authentication?

Azure DevOps supports NTLM in some cases.  I don't use it, so I can't
say in what situations you can use it, but I know from the Git LFS issue
tracker it's a common configuration.

Note that if you just want a test environment, you can use Apache with
Digest authentication.  However, it's limited to MD5, which should no
longer be used for any reason, so you'll probably want to try a
different implementation.