[v5,0/8] Introducing remote ODBs

Message

Christian Couder Sept. 25, 2018, 11:53 a.m. UTC

This path series is a follow up from the patch series called "odb
remote" that I sent earlier this year, which was itself a follow up
from previous series. See the links section for more information.

Goal
~~~~

This series is about introducing a remote ODB mechanism and showing
that this mechanism makes it is possible to:

  - have more than one promisor remote
  - specify different parameters for each promisor remote
  - make it possible later to have other kind of promisor/odb remotes

This also restores the distributed nature of Git which was kind of
broken for promisor remotes.

Explanations
~~~~~~~~~~~~

The extensions.partialclone config option limits the partial clone and
promisor features to only one remote.

That config option also prevents having other kind of promisor/partial
clone/odb remotes. By other kind I mean remotes that would not
necessarily be git repos, but that could store objects (that's where
ODB, for Object DataBase, comes from) and could provide those objects
to Git through a helper (or driver) script or program.

If we want more than one promisor remote, we also need to be able to
specify different parameters for each promisor remote. For example now
core.partialclonefilter is used to specify some filters for the
promisor remote, but how can we nicely specify different partial clone
filters if we have more than one promisor remote?

With the changes in this patch series core.partialclonefilter is
replaced with odb.<remote odb name>.partialclonefilter, so that
parameters for a remote odb are properly grouped together in the
section where the remote odb is defined.

So an added benefit is that the "remote.<remote name>.*" config name
space is not overloaded with more config variables.

Discussion
~~~~~~~~~~

I am not sure that it is ok to completely replace the
"extensions.partialclone" config option. Even if it is fully replaced,
no "extensions.remoteodb" is implemented in these patches, as maybe
the "extensions.partialclone" name could be kept even if the
underlying mechanism is the remote odb mechanism. 

I think that the remote odb mechanism is much more extensible, so I
think using "extensions.partialclone" to specify a promisor remote
should be at least deprecated.

Changes compared to V4 of this patch series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The main change is that the interface of remote_odb_get_direct() in
patch 3/8 is changed, so that it can fetch more than one object. This
remove the needs for remote_odb_get_many_direct(), so the patch that
introduced this function (4/9 in V4) has been removed.

High level overview of this patch series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  - Patch 1/8:

This makes functions in fetch-object.c return an error code, which is
necessary to later tell that they failed and try another remote odb
when there is more than one. This could also just be seen as a fix to
these functions.

  - Patch 2/8:

This introduces the minimum infrastructure for remote odbs.

  - Patches 3/8:

This patch implements remote_odb_get_direct() using fetch_objects()
from "fetch-object.c". Compared to V4, the interface of many functions
now uses oids instead of sha1s.

  - Patch 4/8:

This implement remote_odb_reinit() which will be needed to reparse the
remote odb configuration.

  - Patches 5/8 and 6/8:

These patches integrate the remote odb mechanism into the
promisor/narrow clone code. The "extensions.partialClone" config
option is replaced by "odb.<name>.promisorRemote" and
"core.partialCloneFilter" is replaced by
"odb.<name>.partialCloneFilter". (See the Discussion section below
about this.)

  - Patch 7/8:

This adds a test case that shows that now more than one promisor
remote can be used.

  - Patch 8/8:

This starts documenting the remote odb mechanism.

Links
~~~~~

This patch series on GitHub:

V5: https://github.com/chriscool/git/commits/remote-odb
V4: https://github.com/chriscool/git/commits/remote-odb5
V3: https://github.com/chriscool/git/commits/remote-odb3
V2: https://github.com/chriscool/git/commits/remote-odb2
V1: https://github.com/chriscool/git/commits/remote-odb1

Discussions related to previous versions:

V4: https://public-inbox.org/git/[e-mail]/
V3: https://public-inbox.org/git/[e-mail]/
V2: https://public-inbox.org/git/[e-mail]/
V1: https://public-inbox.org/git/[e-mail]/

Previous "odb remote" series:

https://public-inbox.org/git/[e-mail]/
https://github.com/chriscool/git/commits/odb-remote

Version 1 and 2 of the "Promisor remotes and external ODB support" series:

https://public-inbox.org/git/[e-mail]/
https://public-inbox.org/git/[e-mail]/

Version 1 and 2 of the "Promisor remotes and external ODB support" series on GitHub:

https://github.com/chriscool/git/commits/gl-small-promisor-external-odb12
https://github.com/chriscool/git/commits/gl-small-promisor-external-odb71


Christian Couder (8):
  fetch-object: make functions return an error code
  Add initial remote odb support
  remote-odb: implement remote_odb_get_direct()
  remote-odb: add remote_odb_reinit()
  Use remote_odb_get_direct() and has_remote_odb()
  Use odb.origin.partialclonefilter instead of core.partialclonefilter
  t0410: test fetching from many promisor remotes
  Documentation/config: add odb.<name>.promisorRemote

 Documentation/config.txt      |   5 ++
 Makefile                      |   2 +
 builtin/cat-file.c            |   5 +-
 builtin/fetch.c               |  13 ++--
 builtin/gc.c                  |   3 +-
 builtin/repack.c              |   3 +-
 cache.h                       |   2 -
 connected.c                   |   3 +-
 environment.c                 |   1 -
 fetch-object.c                |  13 ++--
 fetch-object.h                |   4 +-
 list-objects-filter-options.c |  51 ++++++++-------
 list-objects-filter-options.h |   3 +-
 odb-helper.c                  |  31 +++++++++
 odb-helper.h                  |  23 +++++++
 packfile.c                    |   3 +-
 remote-odb.c                  | 120 ++++++++++++++++++++++++++++++++++
 remote-odb.h                  |   9 +++
 setup.c                       |   7 +-
 sha1-file.c                   |  14 ++--
 t/t0410-partial-clone.sh      |  62 ++++++++++++------
 t/t5500-fetch-pack.sh         |   4 +-
 t/t5601-clone.sh              |   2 +-
 t/t5616-partial-clone.sh      |   4 +-
 t/t5702-protocol-v2.sh        |   2 +-
 unpack-trees.c                |   6 +-
 26 files changed, 309 insertions(+), 86 deletions(-)
 create mode 100644 odb-helper.c
 create mode 100644 odb-helper.h
 create mode 100644 remote-odb.c
 create mode 100644 remote-odb.h