| Message ID | 3aa095dc824ea7f659e6c11aa2c21be6cadbe985.1637158762.git.gitgitgadget@gmail.com (mailing list archive) |
|---|---|
| State | Superseded |
| Headers | show |
| Series | Upstreaming the Scalar command | expand |
On 11/17/2021 9:19 AM, Johannes Schindelin via GitGitGadget wrote: > From: Johannes Schindelin <johannes.schindelin@gmx.de> > > The Scalar command will be contributed incrementally, over a bunch of > patch series. Let's document what Scalar is about, and then describe the > patch series that are planned. > > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> > --- > contrib/scalar/README.md | 71 ++++++++++++++++++++++++++++++++++++++++ > 1 file changed, 71 insertions(+) > create mode 100644 contrib/scalar/README.md > > diff --git a/contrib/scalar/README.md b/contrib/scalar/README.md > new file mode 100644 > index 00000000000..7898a683ba5 > --- /dev/null > +++ b/contrib/scalar/README.md > @@ -0,0 +1,71 @@ > +# Scalar - an opinionated repository management tool > + > +Scalar is an add-on to Git, helping Git scale to very large repositories and > +worktrees. I would rephrase this as "Scalar is an add-on to Git that helps users take advantage of advanced performance features in Git." Git scales just fine, only it helps to enable some features that are off by default. > Originally implemented in C# using .NET Core, based on the learnings > +from the VFS for Git project, most of the techniques developed by the Scalar > +project have been integrated into core Git already: > + > +* partial clone, > +* commit graphs, > +* multi-pack index, > +* sparse checkout (cone mode), > +* scheduled background maintenance, > +* etc > + > +This directory contains the remaining parts of Scalar that are not (yet) in > +core Git. > + > +## Roadmap > + > +The idea is to populate this directory via incremental patch series and > +eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The > +current plan involves the following patch series: > + > +- `scalar-the-beginning`: The initial patch series which sets up > + `contrib/scalar/` and populates it with a minimal `scalar` command that > + demonstrates the fundamental ideas. > + > +- `scalar-c-and-C`: The `scalar` command learns about two options that can be > + specified before the command, `-c <key>=<value>` and `-C <directory>`. > + > +- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand. > + > +- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar > + init` and in `scalar clone`, for an enormous performance boost when working > + in large worktrees. This patch series necessarily depends on Jeff Hostetler's > + FSMonitor patch series to be integrated into Git. You say 'scalar init' but do you mean 'scalar register'? > +- `scalar-gentler-config-locking`: Scalar enlistments are registered in the > + user's Git config. This usually does not represent any problem because it is > + rare for a user to register an enlistment. However, in Scalar's functional > + tests, Scalar enlistments are created galore, and in parallel, which can lead > + to lock contention. This patch series works around that problem by re-trying > + to lock the config file in a gentle fashion. > + > +- `scalar-extra-docs`: Add some extensive documentation that has been written > + in the original Scalar project (all subject to discussion, of course). > + > +- `optionally-install-scalar`: Now that Scalar is feature (and documentation) > + complete and is verified in CI builds, let's offer to install it. > + > +- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to > + `gitk-git/` and to `git-gui/`, making it a top-level command. This final one is where we can make the final call about where Scalar should exist in the tree and how optional it should be. This would also move the Scalar man pages into Documentation/, along with possibly the docs from 'scalar-extra-docs', and the tests into t/. The benefit of leaving this until the end is that we can see the entirety of Scalar before making a final call. > +The following two patch series exist, but there is no plan to integrate them > +into Git's source tree: > + > +- `scalar-with-gvfs`: The primary purpose of this patch series is to support > + existing Scalar users whose repositories are hosted in Azure Repos (which > + does not support Git's partial clones, but supports its predecessor, the GVFS > + protocol, which is used by Scalar to emulate the partial clone). > + > + Since the GVFS protocol will never be supported by core Git, this patch > + series will remain in Microsoft's fork of Git. > + > +- `run-scalar-functional-tests`: The Scalar project developed a quite > + comprehensive set of integration tests (or, "Functional Tests"). They are the > + sole remaining part of the original C#-based Scalar project, and this patch > + adds a GitHub workflow that runs them all. > + > + Since the tests partially depend on features that are only provided in the > + `scalar-with-gvfs` patch series, this patch cannot be upstreamed. These topics (in some form or another) exist on microsoft/git and are available via GPL, so we don't intend to say "we are withholding these patches" but instead are saying "We don't think the Git community is interested in these patches." There are some interesting ideas there, but the implementation is too specific to Azure Repos to be of much help in general. These still exist mainly because the GVFS protocol is what Azure Repos has instead of partial clone. We are focused instead on improving partial clone. Thanks, -Stolee
Hi Stolee, On Wed, 17 Nov 2021, Derrick Stolee wrote: > On 11/17/2021 9:19 AM, Johannes Schindelin via GitGitGadget wrote: > > From: Johannes Schindelin <johannes.schindelin@gmx.de> > > > > The Scalar command will be contributed incrementally, over a bunch of > > patch series. Let's document what Scalar is about, and then describe the > > patch series that are planned. > > > > Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de> > > --- > > contrib/scalar/README.md | 71 ++++++++++++++++++++++++++++++++++++++++ > > 1 file changed, 71 insertions(+) > > create mode 100644 contrib/scalar/README.md > > > > diff --git a/contrib/scalar/README.md b/contrib/scalar/README.md > > new file mode 100644 > > index 00000000000..7898a683ba5 > > --- /dev/null > > +++ b/contrib/scalar/README.md > > @@ -0,0 +1,71 @@ > > +# Scalar - an opinionated repository management tool > > + > > +Scalar is an add-on to Git, helping Git scale to very large repositories and > > +worktrees. > > I would rephrase this as "Scalar is an add-on to Git that helps users take > advantage of advanced performance features in Git." Good idea! > Git scales just fine, only it helps to enable some features that are off > by default. > > > Originally implemented in C# using .NET Core, based on the learnings > > +from the VFS for Git project, most of the techniques developed by the Scalar > > +project have been integrated into core Git already: > > + > > +* partial clone, > > +* commit graphs, > > +* multi-pack index, > > +* sparse checkout (cone mode), > > +* scheduled background maintenance, > > +* etc > > + > > +This directory contains the remaining parts of Scalar that are not (yet) in > > +core Git. > > + > > +## Roadmap > > + > > +The idea is to populate this directory via incremental patch series and > > +eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The > > +current plan involves the following patch series: > > + > > +- `scalar-the-beginning`: The initial patch series which sets up > > + `contrib/scalar/` and populates it with a minimal `scalar` command that > > + demonstrates the fundamental ideas. > > + > > +- `scalar-c-and-C`: The `scalar` command learns about two options that can be > > + specified before the command, `-c <key>=<value>` and `-C <directory>`. > > + > > +- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand. > > + > > +- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar > > + init` and in `scalar clone`, for an enormous performance boost when working > > + in large worktrees. This patch series necessarily depends on Jeff Hostetler's > > + FSMonitor patch series to be integrated into Git. > > You say 'scalar init' but do you mean 'scalar register'? D'oh. Yes, I meant `scalar register`. I was thinking of `git init` too much when I wrote that. > > +- `scalar-gentler-config-locking`: Scalar enlistments are registered in the > > + user's Git config. This usually does not represent any problem because it is > > + rare for a user to register an enlistment. However, in Scalar's functional > > + tests, Scalar enlistments are created galore, and in parallel, which can lead > > + to lock contention. This patch series works around that problem by re-trying > > + to lock the config file in a gentle fashion. > > + > > +- `scalar-extra-docs`: Add some extensive documentation that has been written > > + in the original Scalar project (all subject to discussion, of course). > > + > > +- `optionally-install-scalar`: Now that Scalar is feature (and documentation) > > + complete and is verified in CI builds, let's offer to install it. > > + > > +- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to > > + `gitk-git/` and to `git-gui/`, making it a top-level command. > > This final one is where we can make the final call about where Scalar should > exist in the tree and how optional it should be. This would also move the > Scalar man pages into Documentation/, along with possibly the docs from > 'scalar-extra-docs', and the tests into t/. The benefit of leaving this until > the end is that we can see the entirety of Scalar before making a final call. Yes. It allows the current patch series to focus on the core functionality of Scalar. The `move-scalar-to-toplevel` patch series will present an excellent opportunity to discuss the merits and complications of accepting Scalar as a top-level command into Git, without having to delay the current patch series any further. > > +The following two patch series exist, but there is no plan to integrate them > > +into Git's source tree: > > + > > +- `scalar-with-gvfs`: The primary purpose of this patch series is to support > > + existing Scalar users whose repositories are hosted in Azure Repos (which > > + does not support Git's partial clones, but supports its predecessor, the GVFS > > + protocol, which is used by Scalar to emulate the partial clone). > > + > > + Since the GVFS protocol will never be supported by core Git, this patch > > + series will remain in Microsoft's fork of Git. > > + > > +- `run-scalar-functional-tests`: The Scalar project developed a quite > > + comprehensive set of integration tests (or, "Functional Tests"). They are the > > + sole remaining part of the original C#-based Scalar project, and this patch > > + adds a GitHub workflow that runs them all. > > + > > + Since the tests partially depend on features that are only provided in the > > + `scalar-with-gvfs` patch series, this patch cannot be upstreamed. > > These topics (in some form or another) exist on microsoft/git and are available > via GPL, so we don't intend to say "we are withholding these patches" but instead > are saying "We don't think the Git community is interested in these patches." > There are some interesting ideas there, but the implementation is too specific to > Azure Repos to be of much help in general. These still exist mainly because the > GVFS protocol is what Azure Repos has instead of partial clone. We are focused > instead on improving partial clone. Good point. What I wrote does not fully reflect that. I have changed it. Do you see anything in the remainder of the patch series that still needs to be improved? Thanks, Dscho
diff --git a/contrib/scalar/README.md b/contrib/scalar/README.md new file mode 100644 index 00000000000..7898a683ba5 --- /dev/null +++ b/contrib/scalar/README.md @@ -0,0 +1,71 @@ +# Scalar - an opinionated repository management tool + +Scalar is an add-on to Git, helping Git scale to very large repositories and +worktrees. Originally implemented in C# using .NET Core, based on the learnings +from the VFS for Git project, most of the techniques developed by the Scalar +project have been integrated into core Git already: + +* partial clone, +* commit graphs, +* multi-pack index, +* sparse checkout (cone mode), +* scheduled background maintenance, +* etc + +This directory contains the remaining parts of Scalar that are not (yet) in +core Git. + +## Roadmap + +The idea is to populate this directory via incremental patch series and +eventually move to a top-level directory next to `gitk-git/` and to `git-gui/`. The +current plan involves the following patch series: + +- `scalar-the-beginning`: The initial patch series which sets up + `contrib/scalar/` and populates it with a minimal `scalar` command that + demonstrates the fundamental ideas. + +- `scalar-c-and-C`: The `scalar` command learns about two options that can be + specified before the command, `-c <key>=<value>` and `-C <directory>`. + +- `scalar-diagnose`: The `scalar` command is taught the `diagnose` subcommand. + +- `scalar-and-builtin-fsmonitor`: The built-in FSMonitor is enabled in `scalar + init` and in `scalar clone`, for an enormous performance boost when working + in large worktrees. This patch series necessarily depends on Jeff Hostetler's + FSMonitor patch series to be integrated into Git. + +- `scalar-gentler-config-locking`: Scalar enlistments are registered in the + user's Git config. This usually does not represent any problem because it is + rare for a user to register an enlistment. However, in Scalar's functional + tests, Scalar enlistments are created galore, and in parallel, which can lead + to lock contention. This patch series works around that problem by re-trying + to lock the config file in a gentle fashion. + +- `scalar-extra-docs`: Add some extensive documentation that has been written + in the original Scalar project (all subject to discussion, of course). + +- `optionally-install-scalar`: Now that Scalar is feature (and documentation) + complete and is verified in CI builds, let's offer to install it. + +- `move-scalar-to-toplevel`: Now that Scalar is complete, let's move it next to + `gitk-git/` and to `git-gui/`, making it a top-level command. + +The following two patch series exist, but there is no plan to integrate them +into Git's source tree: + +- `scalar-with-gvfs`: The primary purpose of this patch series is to support + existing Scalar users whose repositories are hosted in Azure Repos (which + does not support Git's partial clones, but supports its predecessor, the GVFS + protocol, which is used by Scalar to emulate the partial clone). + + Since the GVFS protocol will never be supported by core Git, this patch + series will remain in Microsoft's fork of Git. + +- `run-scalar-functional-tests`: The Scalar project developed a quite + comprehensive set of integration tests (or, "Functional Tests"). They are the + sole remaining part of the original C#-based Scalar project, and this patch + adds a GitHub workflow that runs them all. + + Since the tests partially depend on features that are only provided in the + `scalar-with-gvfs` patch series, this patch cannot be upstreamed.