[v1,0/8] Many promisor remotes

Message

Christian Couder Dec. 11, 2018, 5:27 a.m. UTC

This path series is a follow up from the "remote odb" patch series
that I sent earlier this year, which were a follow up from previous
series. See the links section for more information.

The goal of this patch series is to make it possible to have and to
fetch missing objects from multiple remotes instead of only one.

For now the fetch order is the order of the remotes in the config.

I selected the name "Promisor remote" over "Partial clone remote"
because it is shorter and because it is not only about cloning.

The existing extensions.partialclone is respected, but it is not
written in the config when a partial clone or fetch is made. Instead
remote.<name>.promisor is set to "true". This may create a
compatibility issue, but it makes it possible to start using many
promisor remotes by just cloning and fetching from different remotes
with partial clone filters. The compatibility issue could be resolved
in a future iteration by just setting extensions.partialclone instead
of remote.<name>.promisor the first time a promisor remote is used.

In general I have tried to change as few things as possible.

Yeah, this is missing documentation for now.

High level overview of this patch series
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  - Patch 1/8:

This makes functions in fetch-object.c return an error code, which is
necessary to later tell that they failed and try another promisor
remote when there is more than one. This could also just be seen as a
fix to these functions.

  - Patch 2/8:

This introduces the minimum infrastructure for promisor remotes.

  - Patch 3/8, 4/8 and 5/8:

This add a few missing bits in the promisor remote infrastructure that
will be needed in the following patches.

  - Patch 6/8:

This replaces the previous interface to use only one promisor remote
defined in extensions.partialclone by the new interface created by the
previous patches.

  - Patch 7/8:

This replaces the way a partial clone filter was handled by a new way
based on the previous patches that support more than one partial clone
filter.

  - Patch 8/8:

This adds a test case that shows that now more than one promisor
remote can be used.

Links
~~~~~

This new patch series is a follow up from the discussions related to
the remote odb V4 patch series:

https://public-inbox.org/git/[e-mail]/

Especially in:

https://public-inbox.org/git/CAP8UFD3nrhjANwNDqTwx5ZtnZNcnbAFqUN=u=[e-mail]/

I said that I would like to work on things in the following order:

  1) Teaching partial clone to attempt to fetch missing objects from
multiple remotes instead of only one using the order in the config.

  2) Simplifying the protocol for fetching missing objects so that it
can be satisfied by a lighter weight object storage system than a full
Git server.

  3) Making it possible to explicitly define an order in which the
remotes are accessed.

  4) Making the criteria for what objects can be missing more
aggressive, so that I can "git add" a large file and work with it
using Git without even having a second copy of that object in my local
object store.

And this patch series is about the 1).

This patch series on GitHub:

https://github.com/chriscool/git/commits/many-promisor-remotes

The previous remote odb patch series on GitHub:

V5: https://github.com/chriscool/git/commits/remote-odb
V4: https://github.com/chriscool/git/commits/remote-odb5
V3: https://github.com/chriscool/git/commits/remote-odb3
V2: https://github.com/chriscool/git/commits/remote-odb2
V1: https://github.com/chriscool/git/commits/remote-odb1

Discussions related to previous versions of the odb patch series:

V4: https://public-inbox.org/git/[e-mail]/
V3: https://public-inbox.org/git/[e-mail]/
V2: https://public-inbox.org/git/[e-mail]/
V1: https://public-inbox.org/git/[e-mail]/



Christian Couder (8):
  fetch-object: make functions return an error code
  Add initial support for many promisor remotes
  promisor-remote: implement promisors_get_direct()
  promisor-remote: add promisor_remote_reinit()
  promisor-remote: use repository_format_partial_clone
  Use promisors_get_direct() and has_promisor_remote()
  promisor-remote: parse remote.*.partialclonefilter
  t0410: test fetching from many promisor remotes

 Makefile                      |   1 +
 builtin/cat-file.c            |   5 +-
 builtin/fetch.c               |  13 +--
 builtin/gc.c                  |   3 +-
 builtin/repack.c              |   3 +-
 cache-tree.c                  |   3 +-
 connected.c                   |   3 +-
 fetch-object.c                |  13 +--
 fetch-object.h                |   4 +-
 list-objects-filter-options.c |  51 ++++++------
 list-objects-filter-options.h |   3 +-
 packfile.c                    |   3 +-
 promisor-remote.c             | 147 ++++++++++++++++++++++++++++++++++
 promisor-remote.h             |  22 +++++
 sha1-file.c                   |  14 ++--
 t/t0410-partial-clone.sh      |  26 +++++-
 t/t5601-clone.sh              |   3 +-
 t/t5616-partial-clone.sh      |   4 +-
 unpack-trees.c                |   6 +-
 19 files changed, 269 insertions(+), 58 deletions(-)
 create mode 100644 promisor-remote.c
 create mode 100644 promisor-remote.h