[0/5] fetch: more optimizations for mirror fetches

Message

Patrick Steinhardt Feb. 23, 2022, 12:35 p.m. UTC

Hi,

this is another patch series with the aim to speed up mirror fetches. It
applies on top of e6ebfd0e8c (The sixth batch, 2022-02-18) with
3824153b23 (Merge branch 'ps/fetch-atomic' into next, 2022-02-18) merged
into it to fix a conflict.

The patch series is structured as follows:

    - Patch 1: Another conversion to use the commit-graph, this time to
      look up "want" and "want-ref" lines in git-upload-pack(1).

    - Patch 2: This skips useless commit lookups when we don't write
      FETCH_HEAD. These were only used to compute the correct order in
      which we append to FETCH_HEAD.

    - Patch 3-5: This optimizes the way we look up symbolic refs in the
      files backend. Currently we always search the packed-refs file,
      which cannot ever contain symbolic references.

In total, this patch series brings down the time it takes to replicate
one of our biggest repos in numbers of references from 75 seconds to 60
seconds.

Patrick

Patrick Steinhardt (5):
  upload-pack: look up "want" lines via commit-graph
  fetch: avoid lookup of commits when not appending to FETCH_HEAD
  refs: add ability for backends to special-case reading of symbolic
    refs
  remote: read symbolic refs via `refs_read_symbolic_ref()`
  refs/files-backend: optimize reading of symbolic refs

 builtin/fetch.c       | 42 +++++++++++++++++++++++++++---------------
 builtin/remote.c      |  8 +++++---
 refs.c                | 17 +++++++++++++++++
 refs.h                |  3 +++
 refs/debug.c          |  1 +
 refs/files-backend.c  | 33 ++++++++++++++++++++++++++++-----
 refs/packed-backend.c |  1 +
 refs/refs-internal.h  | 16 ++++++++++++++++
 remote.c              | 14 +++++++-------
 upload-pack.c         | 20 +++++++++++++++++---
 10 files changed, 122 insertions(+), 33 deletions(-)