mbox series

[00/16] reftable: overhaul the API to expose access to blocks

Message ID 20250331-pks-reftable-polishing-v1-0-ebed5247434c@pks.im (mailing list archive)
Headers show
Series reftable: overhaul the API to expose access to blocks | expand

Message

Patrick Steinhardt March 31, 2025, 8:41 a.m. UTC 
Hi,

this patch series is a bigger overhaul of the reftable API. This
overhaul has two main motivations:

  - The reftable library is now standalone and can be used by code bases
    other than Git, like libgit2. This patch series thus renames a
    couple of subsystems to have more intuitive names before we gain any
    new users.

  - Some of the data of reftables isn't accessible at all via public
    interfaces. Most importantly, it is not possible to access
    individual blocks in a table. While users shouldn't need that access
    most of the time, an upcoming usecase that Git itself has is to
    implement consistency checks for the reftable backend. Here we'll
    want to read through blocks and their respective records one by one
    to ensure that they are sane and then iterate through records
    contained in these blocks.

The patch series is structured as follows:

  - Patch 1 is a trivial global refactoring to fix formatting of the
    license headers. They have been annoying me for far too long.

  - Patches 2 to 8 consolidate and rename a couple of data structures:

      - `reftable_reader` becomes `reftable_table`, as it is used to
        access an individual table.

      - `reftable_block` becomes `reftable_block_data`, as it is only a
        simple container for the underlying bytes.

      - `reftable_block_reader` becomes `reftable_block`, as it is used
        to access an individual block.

    Overall, the data structures are now called after what they provide
    access to compared to the rather generic previous names. This is
    also in line with other data structures like `reftable_merged_table`
    and `reftable_stack`.

  - Patches 9 to 13 refactor the block interface so that it can expose a
    generic `reftable_iterator`, granting generic access to all of its
    contained records.

  - Patches 14 to 16 refactor the table interface to expose a new
    iterator over its contained blocks.

  - Patch 17 refactors `reftable_table_print_blocks` to be implemented
    on top of these new iterators. This allows us to move it out of the
    library codebase into the test helper.

The series is built on Git v2.49.0 with ps/reftable-sans-compat-util at
8f6a2dbe340 (Makefile: skip reftable library for Coccinelle, 2025-02-18)
merged into it.

Thanks!

Patrick

---
Patrick Steinhardt (16):
      reftable: fix formatting of the license header
      reftable/reader: rename data structure to "table"
      reftable/blocksource: consolidate code into a single file
      reftable/block: simplify how we track restart points
      reftable/table: move reading block into block reader
      reftable/block: rename `block` to `block_data`
      reftable/block: rename `block_reader` to `reftable_block`
      git-zlib: use `struct z_stream_s` instead of typedef
      reftable/block: create public interface for reading blocks
      reftable/block: store block pointer in the block iterator
      reftable/block: make block iterators reseekable
      reftable/block: expose a generic iterator over reftable records
      reftable/table: add `reftable_table` to the public interface
      reftable/table: introduce iterator for table blocks
      reftable/constants: make block types part of the public interface
      reftable/table: move printing logic into test helper

 .../howto/recover-corrupted-object-harder.adoc     |   4 +-
 Makefile                                           |   4 +-
 compat/zlib-compat.h                               |   4 +-
 git-zlib.h                                         |   2 +-
 meson.build                                        |   2 +-
 reftable/basics.c                                  |  12 +-
 reftable/basics.h                                  |  19 +-
 reftable/block.c                                   | 284 ++++++++-----
 reftable/block.h                                   |  85 ++--
 reftable/blocksource.c                             |  67 ++-
 reftable/blocksource.h                             |  39 +-
 reftable/constants.h                               |  18 +-
 reftable/error.c                                   |  12 +-
 reftable/iter.c                                    |  36 +-
 reftable/iter.h                                    |  18 +-
 reftable/merged.c                                  |  42 +-
 reftable/merged.h                                  |  16 +-
 reftable/pq.c                                      |  12 +-
 reftable/pq.h                                      |  12 +-
 reftable/reader.h                                  |  67 ---
 reftable/record.c                                  |  52 +--
 reftable/record.h                                  |  12 +-
 reftable/reftable-basics.h                         |  10 +-
 reftable/reftable-block.h                          |  74 ++++
 reftable/reftable-blocksource.h                    |  29 +-
 reftable/reftable-constants.h                      |  18 +
 reftable/reftable-error.h                          |  12 +-
 reftable/reftable-iterator.h                       |  12 +-
 reftable/reftable-merged.h                         |  18 +-
 reftable/reftable-reader.h                         |  72 ----
 reftable/reftable-record.h                         |  12 +-
 reftable/reftable-stack.h                          |  12 +-
 reftable/reftable-table.h                          | 115 +++++
 reftable/reftable-writer.h                         |  12 +-
 reftable/stack.c                                   | 188 ++++-----
 reftable/stack.h                                   |  16 +-
 reftable/system.h                                  |  12 +-
 reftable/{reader.c => table.c}                     | 463 +++++++++------------
 reftable/table.h                                   |  29 ++
 reftable/tree.c                                    |  12 +-
 reftable/tree.h                                    |  12 +-
 reftable/writer.c                                  |  34 +-
 reftable/writer.h                                  |  12 +-
 t/helper/test-reftable.c                           |  81 +++-
 t/meson.build                                      |   2 +-
 t/t0613-reftable-write-options.sh                  |   9 +
 t/unit-tests/t-reftable-block.c                    | 218 +++++++---
 t/unit-tests/t-reftable-merged.c                   |  86 ++--
 t/unit-tests/t-reftable-pq.c                       |  10 +-
 t/unit-tests/t-reftable-reader.c                   |  96 -----
 t/unit-tests/t-reftable-readwrite.c                | 106 ++---
 t/unit-tests/t-reftable-record.c                   |  40 +-
 t/unit-tests/t-reftable-stack.c                    |  66 +--
 t/unit-tests/t-reftable-table.c                    | 205 +++++++++
 54 files changed, 1638 insertions(+), 1274 deletions(-)


---
base-commit: bc705c20b9a88abaff0f379bab6d545f012656af
change-id: 20241210-pks-reftable-polishing-332ce318cdea