From patchwork Mon Oct 28 20:34:52 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13854112 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C2CC61E04B8 for ; Mon, 28 Oct 2024 20:35:18 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147722; cv=none; b=fLwHqO1y2LLzIgB3R1e+VRtdmKToiLnoNaUfT6AxGVDLSjDhGg/zFPPnYIj9f+VsKaGRos+QHJ3cno7Mlg4Ljj7Pj3b3bSIJSr5/VgvStcpNc5WiuICAxRDXJYa5J/m/QhWJCARvf7Nkfffor2vT3KuQJfn4OGkoHiI/8DA69T0= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147722; c=relaxed/simple; bh=D8wqdm4BLbtyhYoTPlFPcB9KHfCUgoe2JolJ7Bn7cvw=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=FeCZIngaY3YRzIIhbNod3ybjGSDiwbuyu2xb4RnNKaa2Ybh9rX1FEMz9QJECqSu9TN7OCJBli+4c8WRlHV16UnQBzqk8UaYQ4Rhn2QkbH5bShz3CXdDEDH43skYoIRvwOfz/owViDv2WCOS2Dg+jOhUbWEffINX8SWCti357hlg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=a3fogR77; arc=none smtp.client-ip=209.85.160.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="a3fogR77" Received: by mail-qt1-f179.google.com with SMTP id d75a77b69052e-460d1145cd8so39057391cf.3 for ; Mon, 28 Oct 2024 13:35:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730147717; x=1730752517; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=jenA9nffZLeofPCg+6OUxAoi32evtcfhSrI7JvAoz/s=; b=a3fogR77rpfSkfywIUX3lJbpM/lmS3bj0Ue4by+P4XDOGsXdPtIVlDqYtTu2Qo24OX dtb6GiR23GvMcgx2/W8XZmINDHxExkSyJak2muD7hZ2Jjjr8AOfx1Qi/cpWnKAIpoiNe ZQyhWmfOnDFOlzwgpsNuk4R6fEpMxaJXAw/GmO35tliHvvN8FAtV1I87DKnR3t1tSEpt rgp6DPW1SjCLCoF2bnw2sN/NpQ26y+/0X8iCrHaLbodQLjBjj3a+rMZLrG1mLepfrQ5Z BEIRC/iMBC8IGBl7TDROi+1UtoHLu0ZYu3Ltp8/NarjO7ZjUCTXIM/eYTjvyT34Gult4 Ibiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730147717; x=1730752517; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=jenA9nffZLeofPCg+6OUxAoi32evtcfhSrI7JvAoz/s=; b=Tzmy/+YbFrQixTQ0ZQdzFw3bST2j/OCNqpE4vUGnMZMjt9Lz6MFKbYfUcWF//4Ot8U R+fdJh1lbWKXW7zmmGe9dF1T+XECCTS7IiImkF7M7U2jY8f3W6LgmnzYMrr6FGiMRsHe HsUSZx+++xFg9WWyQSaHmnAhIIH6MXNlXgJhwu3dNKeLZecpi0ZJoaxPBxd3nb3/hQQ9 fGjX+ti94qgfRKy8XKYtAYX+LfY6nB+Fr0PDbzjdKc9CfPlBM5a8jkCs80PRTMH+MyXu 9MdQ8Ll6CVMBcNzIwPyQGcSGkiVAbJN3X+FOHepi32pwOSLaVjU1pNyFstPpZYCLbeQ4 C8xQ== X-Gm-Message-State: AOJu0YyuNWFWZtCBFDe6+sc21bZ509eSL43WdUn1M9BNOcyiz5oB3lfn LzEn6FJ0Jc8kcGhJyWhyqvgmruih8PmonpT1oUQlcOEwgUhSlnXVIg9kXFrM X-Google-Smtp-Source: AGHT+IExNeTfTfjBJR+eV1BwlIvfniWOXlJnMidkFfNIOd3pdak/0cF/eBWI49GzyWIO3nwRECq7fA== X-Received: by 2002:ac8:7d13:0:b0:461:17e6:2651 with SMTP id d75a77b69052e-4613bfc96c0mr132474321cf.8.1730147717437; Mon, 28 Oct 2024 13:35:17 -0700 (PDT) Received: from localhost.localdomain ([184.147.230.152]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4613228c099sm38215791cf.46.2024.10.28.13.35.16 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 28 Oct 2024 13:35:16 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: calvinwan@google.com, jonathantanmy@google.com, chriscool@tuxfamily.org, eric.peijian@gmail.com, karthik.188@gmail.com, toon@iotcl.com, jltobler@gmail.com Subject: [PATCH v5 1/6] fetch-pack: refactor packet writing Date: Mon, 28 Oct 2024 16:34:52 -0400 Message-ID: <20241028203457.19715-2-eric.peijian@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241028203457.19715-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> <20241028203457.19715-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Calvin Wan Refactor write_fetch_command_and_capabilities() to be a more general purpose function write_command_and_capabilities(), so that it can be used by both fetch and future commands. Here "command" means the "operations" supported by Git’s wire protocol https://git-scm.com/docs/protocol-v2. An example would be a git's subcommand, such as git-fetch(1); or an operation supported by the server side such as "object-info" implemented in "a2ba162cda (object-info: support for retrieving object info, 2021-04-20)". The new write_command_and_capabilities() function is also moved to connect.c, so that it becomes accessible to other commands. Helped-by: Jonathan Tan Helped-by: Christian Couder Signed-off-by: Calvin Wan Signed-off-by: Eric Ju --- connect.c | 34 ++++++++++++++++++++++++++++++++++ connect.h | 4 ++++ fetch-pack.c | 36 ++---------------------------------- 3 files changed, 40 insertions(+), 34 deletions(-) diff --git a/connect.c b/connect.c index 58f53d8dcb..bb4e4eec44 100644 --- a/connect.c +++ b/connect.c @@ -688,6 +688,40 @@ int server_supports(const char *feature) return !!server_feature_value(feature, NULL); } +void write_command_and_capabilities(struct strbuf *req_buf, const char *command, + const struct string_list *server_options) +{ + const char *hash_name; + int advertise_sid; + + git_config_get_bool("transfer.advertisesid", &advertise_sid); + + ensure_server_supports_v2(command); + packet_buf_write(req_buf, "command=%s", command); + if (server_supports_v2("agent")) + packet_buf_write(req_buf, "agent=%s", git_user_agent_sanitized()); + if (advertise_sid && server_supports_v2("session-id")) + packet_buf_write(req_buf, "session-id=%s", trace2_session_id()); + if (server_options && server_options->nr) { + ensure_server_supports_v2("server-option"); + for (int i = 0; i < server_options->nr; i++) + packet_buf_write(req_buf, "server-option=%s", + server_options->items[i].string); + } + + if (server_feature_v2("object-format", &hash_name)) { + const int hash_algo = hash_algo_by_name(hash_name); + if (hash_algo_by_ptr(the_hash_algo) != hash_algo) + die(_("mismatched algorithms: client %s; server %s"), + the_hash_algo->name, hash_name); + packet_buf_write(req_buf, "object-format=%s", the_hash_algo->name); + } else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1) { + die(_("the server does not support algorithm '%s'"), + the_hash_algo->name); + } + packet_buf_delim(req_buf); +} + enum protocol { PROTO_LOCAL = 1, PROTO_FILE, diff --git a/connect.h b/connect.h index 1645126c17..2ed009066e 100644 --- a/connect.h +++ b/connect.h @@ -1,6 +1,7 @@ #ifndef CONNECT_H #define CONNECT_H +#include "string-list.h" #include "protocol.h" #define CONNECT_VERBOSE (1u << 0) @@ -30,4 +31,7 @@ void check_stateless_delimiter(int stateless_rpc, struct packet_reader *reader, const char *error); +void write_command_and_capabilities(struct strbuf *req_buf, const char *command, + const struct string_list *server_options); + #endif diff --git a/fetch-pack.c b/fetch-pack.c index f752da93a8..533fb76f95 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1314,38 +1314,6 @@ static int add_haves(struct fetch_negotiator *negotiator, return haves_added; } -static void write_fetch_command_and_capabilities(struct strbuf *req_buf, - const struct string_list *server_options) -{ - const char *hash_name; - - ensure_server_supports_v2("fetch"); - packet_buf_write(req_buf, "command=fetch"); - if (server_supports_v2("agent")) - packet_buf_write(req_buf, "agent=%s", git_user_agent_sanitized()); - if (advertise_sid && server_supports_v2("session-id")) - packet_buf_write(req_buf, "session-id=%s", trace2_session_id()); - if (server_options && server_options->nr) { - int i; - ensure_server_supports_v2("server-option"); - for (i = 0; i < server_options->nr; i++) - packet_buf_write(req_buf, "server-option=%s", - server_options->items[i].string); - } - - if (server_feature_v2("object-format", &hash_name)) { - int hash_algo = hash_algo_by_name(hash_name); - if (hash_algo_by_ptr(the_hash_algo) != hash_algo) - die(_("mismatched algorithms: client %s; server %s"), - the_hash_algo->name, hash_name); - packet_buf_write(req_buf, "object-format=%s", the_hash_algo->name); - } else if (hash_algo_by_ptr(the_hash_algo) != GIT_HASH_SHA1) { - die(_("the server does not support algorithm '%s'"), - the_hash_algo->name); - } - packet_buf_delim(req_buf); -} - static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, struct fetch_pack_args *args, const struct ref *wants, struct oidset *common, @@ -1356,7 +1324,7 @@ static int send_fetch_request(struct fetch_negotiator *negotiator, int fd_out, int done_sent = 0; struct strbuf req_buf = STRBUF_INIT; - write_fetch_command_and_capabilities(&req_buf, args->server_options); + write_command_and_capabilities(&req_buf, "fetch", args->server_options); if (args->use_thin_pack) packet_buf_write(&req_buf, "thin-pack"); @@ -2174,7 +2142,7 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips, the_repository, "%d", negotiation_round); strbuf_reset(&req_buf); - write_fetch_command_and_capabilities(&req_buf, server_options); + write_command_and_capabilities(&req_buf, "fetch", server_options); packet_buf_write(&req_buf, "wait-for-done"); From patchwork Mon Oct 28 20:34:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13854113 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B70021E00A0 for ; Mon, 28 Oct 2024 20:35:19 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147722; cv=none; b=uRNFh2WRArBdQgXgyfXTL56hsdDgd/sIgU5rOOHGhKBlIbG8GCgBsqDWyUoSoyrx0RlCTZCGcSN4EkWG41Z4gAsDNP3SZK+PInbtr8pO4DBxRWKbBTd228wrNP+1rjm/vpOUl9gyHhu9SvOpx1+G9do2WFuLfsdCHgWamauSmSM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147722; c=relaxed/simple; bh=DMMQh7V1oDuDvCYzO0NDaRD3Dz4Evr6CC+tuGinKoK4=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=W5KeZ2/yPeRIM5Obxx3zbjmVmV99VRr9of/gXQT0xEh7bH0qs4opQURL/0x4hH/sl4FcrvMoOSATMeTenX2qb2st1x9T79hkkH//oPm1flN1biO9QMW234R5A7GIh1DOw346PW1jH6RixQTDZaqgBS/BEn5qW99ZdE0DFQ7LaJU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=JfcZyIPT; arc=none smtp.client-ip=209.85.160.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="JfcZyIPT" Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-460b16d4534so22883371cf.3 for ; Mon, 28 Oct 2024 13:35:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730147718; x=1730752518; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=zhK6fj0H2pwBr0NBtbbGdg3r2mhQKP8KxlbU/cQf5Iw=; b=JfcZyIPTrG6w8ksM3B+85vcz5yFatDkbgTvfZr9QlGx5nQCzXbOg56vJHhd2SYBivt 2ffnj8SHCEH56E2bne10KyGCs8jML4gaXUdZt8LaGqXU5DMlBjNauITKk6Vy0tbf6/as xv8KQbCBo5nvrhIb7B0cIw5Lvp+BBdFBhU7E4uN84COxPEHxs4WSRtZjxRTJqzgrYqwU 7+2+J2+nM+PSdZCFTpJ5IPlGPIQ4yCvhOfY6yn0Zn3M6BEdl5O6XQMrKSFnWAKt0WrSW NjmMQGEiAj831+8FCMTPLl9wPQK0zHXsYKaypOA9/g0g0gM+UuZP+x1u7RaGUpogRUx1 PrbQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730147718; x=1730752518; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=zhK6fj0H2pwBr0NBtbbGdg3r2mhQKP8KxlbU/cQf5Iw=; b=O+j/jbWxgIpDmTiY2gqRg8GEFvhCOmV3sezpVLNUU0M8bKMnxFz7Ynsm8hEBPyVyS+ kEE7HLfmCCDBwlYDfhfwZCEknuVbP3SveeshHcvTuB/gq9yU4zdmZK5RZtVcbP2GV3s1 k2uGOR/ptKx+CLymrmeKclLw5TwboOMU53YgWeDY0KRmpKRwe7HcINom8x0Ke9AcOTuY 46Xbcgrw12iqQcnxUXuXHhq63p1NMVnyOuTdiN0if9O4G8V/3P/t+JKQ4vFzvlhVPsQG APUX4UlpQXgmWce6q3btapU6hg+menAOiZAvrf/4NxEm4vjAh4EsM1WDUflJgt4YahPy 9aQA== X-Gm-Message-State: AOJu0YzDNHJ2LEc9VD6UNOQ+EG9lPnRspfhoTqECiGndyVfzPI9/8YgZ Lji8LDK9FbvIjCuO9m4n4r23IHT4qGNpQjr2gNmQAtpAQ4/N/PIrs7TxiB6A X-Google-Smtp-Source: AGHT+IGSCVbJzNfX2r+UATdtwddkdDz8TmDF7Uy1XmAQFfmlHhWUQG9jDT0AqmOKeTz3QBVwVKl2kg== X-Received: by 2002:a05:622a:14b:b0:461:169e:d2f1 with SMTP id d75a77b69052e-4613c1b9ecfmr125402051cf.60.1730147718440; Mon, 28 Oct 2024 13:35:18 -0700 (PDT) Received: from localhost.localdomain ([184.147.230.152]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4613228c099sm38215791cf.46.2024.10.28.13.35.17 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 28 Oct 2024 13:35:18 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: calvinwan@google.com, jonathantanmy@google.com, chriscool@tuxfamily.org, eric.peijian@gmail.com, karthik.188@gmail.com, toon@iotcl.com, jltobler@gmail.com Subject: [PATCH v5 2/6] fetch-pack: move fetch initialization Date: Mon, 28 Oct 2024 16:34:53 -0400 Message-ID: <20241028203457.19715-3-eric.peijian@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241028203457.19715-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> <20241028203457.19715-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Calvin Wan There are some variables initialized at the start of the do_fetch_pack_v2() state machine. Currently, they are initialized in FETCH_CHECK_LOCAL, which is the initial state set at the beginning of the function. However, a subsequent patch will allow for another initial state, while still requiring these initialized variables. Move the initialization to be before the state machine, so that they are set regardless of the initial state. Note that there is no change in behavior, because we're moving code from the beginning of the first state to just before the execution of the state machine. Helped-by: Jonathan Tan Helped-by: Christian Couder Signed-off-by: Calvin Wan Signed-off-by: Eric Ju --- fetch-pack.c | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index 533fb76f95..afffbcaafc 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1645,18 +1645,18 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, reader.me = "fetch-pack"; } + /* v2 supports these by default */ + allow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1; + use_sideband = 2; + if (args->depth > 0 || args->deepen_since || args->deepen_not) + args->deepen = 1; + while (state != FETCH_DONE) { switch (state) { case FETCH_CHECK_LOCAL: sort_ref_list(&ref, ref_compare_name); QSORT(sought, nr_sought, cmp_ref_by_name); - /* v2 supports these by default */ - allow_unadvertised_object_request |= ALLOW_REACHABLE_SHA1; - use_sideband = 2; - if (args->depth > 0 || args->deepen_since || args->deepen_not) - args->deepen = 1; - /* Filter 'ref' by 'sought' and those that aren't local */ mark_complete_and_common_ref(negotiator, args, &ref); filter_refs(args, &ref, sought, nr_sought); From patchwork Mon Oct 28 20:34:54 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13854114 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C87141EE00F for ; Mon, 28 Oct 2024 20:35:20 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147723; cv=none; b=haswbg2Qg1UyUfK/2xhfyVY8rmyN5r0u4D8MTjCs0NsW+AhK3bgHwceiwsv3T3M1MfM0egGFCzi3usRRJs2iux5IYlm3SMk1+QFhokRL1jlVEVUZJAeQ2TwCBkkraWzUgFt91rMzYbfpysSDj+XH7n41cOe+Jzrlajo+8r5cag4= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147723; c=relaxed/simple; bh=yjyz+Vz840lW2No3gACSbRZE3MV5qia5BFfczHhmR5Y=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=C1EvTDhiTVIp4or+1p2SO8hIT49qZc8Gv/TM7+5F3+qSm0lzUoUUy8c0J84uEcbswb2CrcsnDJkFg83Pc6TqRAColdmtdNRZBE6LqMlw/trraLJ99y3n0GuehUI/7Fq84WCMdMPimYqNbWOckhP+jtOBOOrFKvydxx4IDYwal40= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=KUidnzfh; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="KUidnzfh" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-46094b68e30so35525311cf.0 for ; Mon, 28 Oct 2024 13:35:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730147719; x=1730752519; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=0cpGIf3Q2e9lMZFbZTS9wWeKcb2Sr6a33QDERJUPy1w=; b=KUidnzfhzdrG5Sgz+NnHie/IrMvFrRm1lboY7OylZgKaO+OUFzRBy+eh9fh8KnnQPq 9Xn1JtP7sEWVZpnpye7mDNm1hLRzcbpfdI6dwbJt3silA2xnrSM/zl7Gz10FvyaLiVHV jdwxL6cVFrV8Hz8O9BCrPINL9+lVmL4zchPLNJxz69nui1duRwPw/5m6AygZoFjwNs3N g1GrgRL6Ios0tvuxP632O4QlL4Gm8Urgj11EnyTK0rT4Oph4/nq07lfmrtJxidXa/mdI 2gt3/JEu70XZjbrFX0fEuKXBC5B5YPEZxoj9tbCZliRPFIP0EXySYdgD1tVBhGQlQ7aS FQEQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730147719; x=1730752519; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=0cpGIf3Q2e9lMZFbZTS9wWeKcb2Sr6a33QDERJUPy1w=; b=LhkxIc5+yQ7tUsshK5lQBHbuFba7W4oeVEPcUkwxYih9V9TLbVgXBJDpeFKunTiRbY J5F4pUxja/alAD1hMqkUKb+dK9n7r19C6CDjf+4TjVHXEZivCpwKagwwqWvfgEDjsUS/ zhqTG6g6hld9ibzXtjEY1igMaDYmfg0gGBBZw6WkDT5zucS82VyBIZNZBpLqFXWmd++T 8a+X7zOmodyFP5Y2GHHb2FSE5ASlaaBEa1KZitwD5zfKguPHmeq3DCMw4KdNKP6NQlyM SCyXUmSbgNbdkVJ+vi9fjfApCyaPUG+QzQ0NqFXTTXOCn3OVNabvIkANGpEmt/QFhk8B J7jA== X-Gm-Message-State: AOJu0YzyHmbfotb62eW5KiiwbjnLS853p2jnO0xvflePVqgkdVJ8Xiaq o/hZGOikmiEJ0l/dQ8yDRJtSjCZHNsHtW8G9iDpBb6c9I9k1UyzVWJEh2ilq X-Google-Smtp-Source: AGHT+IELs9/Cl4VdSR328OBSZAXBdvmFSjElVJHlaSLTLyOJ+SKDJWRqCRm8rDGyIaB8B/x9+OuWMg== X-Received: by 2002:a05:622a:c5:b0:460:933a:f6c with SMTP id d75a77b69052e-4613c0303bfmr163867311cf.26.1730147719575; Mon, 28 Oct 2024 13:35:19 -0700 (PDT) Received: from localhost.localdomain ([184.147.230.152]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4613228c099sm38215791cf.46.2024.10.28.13.35.18 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 28 Oct 2024 13:35:19 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: calvinwan@google.com, jonathantanmy@google.com, chriscool@tuxfamily.org, eric.peijian@gmail.com, karthik.188@gmail.com, toon@iotcl.com, jltobler@gmail.com Subject: [PATCH v5 3/6] serve: advertise object-info feature Date: Mon, 28 Oct 2024 16:34:54 -0400 Message-ID: <20241028203457.19715-4-eric.peijian@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241028203457.19715-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> <20241028203457.19715-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Calvin Wan In order for a client to know what object-info components a server can provide, advertise supported object-info features. This will allow a client to decide whether to query the server for object-info or fetch as a fallback. Helped-by: Jonathan Tan Helped-by: Christian Couder Signed-off-by: Calvin Wan Signed-off-by: Eric Ju --- serve.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/serve.c b/serve.c index d674764a25..c3d8098642 100644 --- a/serve.c +++ b/serve.c @@ -70,7 +70,7 @@ static void session_id_receive(struct repository *r UNUSED, trace2_data_string("transfer", NULL, "client-sid", client_sid); } -static int object_info_advertise(struct repository *r, struct strbuf *value UNUSED) +static int object_info_advertise(struct repository *r, struct strbuf *value) { if (advertise_object_info == -1 && repo_config_get_bool(r, "transfer.advertiseobjectinfo", @@ -78,6 +78,8 @@ static int object_info_advertise(struct repository *r, struct strbuf *value UNUS /* disabled by default */ advertise_object_info = 0; } + if (value && advertise_object_info) + strbuf_addstr(value, "size"); return advertise_object_info; } From patchwork Mon Oct 28 20:34:55 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13854116 Received: from mail-yb1-f179.google.com (mail-yb1-f179.google.com [209.85.219.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id EB9F21EE015 for ; Mon, 28 Oct 2024 20:35:21 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.179 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147724; cv=none; b=apAF5YmSPVrWdNd1FrYPkxuUii7WRWvoczamnfspLU47w2ev2yAqJs7v69gAsmAjXEa9jrQQ0T3hWUIsbxZd8QGzA3WUNhQln8mrQaUejrA34385c0XTnN70AHdSXvuGl1fxSuFh3FkiQCVnaGP7hVOJ1dNrARlunj+KqO9shcs= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147724; c=relaxed/simple; bh=FjiwOm43HDgJp+aZZCdq2tmPOutPb4IS1QreCIXYJu0=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version:Content-Type; b=R10GnvFnjuKgGnciAA74e7+1Tvte++pw6Oq1rQuRR84xHn7FKxX4KexDm1UrNwVrqsiUU5KrbfMX4jj3pFZ6pBHeIq5pl0Govj6WZr1Ht8zhxQPghNutKkGpWIY/TZJf5nX8xc+UwCj2ply/s41IFRbgcjFRK8qjwxzM6zQcG2c= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=j7jFoDBm; arc=none smtp.client-ip=209.85.219.179 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="j7jFoDBm" Received: by mail-yb1-f179.google.com with SMTP id 3f1490d57ef6-e0875f1e9edso4779404276.1 for ; Mon, 28 Oct 2024 13:35:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730147721; x=1730752521; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=bLTESXYO5Q2VPO2t1ZblnTkpPaTGYqk0UFlBrBdMu3I=; b=j7jFoDBmGPR5KLzJhqjLmvjNexuVSeOiv0iajY0/NkUrE7qb2Akqmda50CjzALNKk+ hwfGOsMDp+6LzNkgV6Bj0RKr/YQplM7n64IsJPA7gq5oABO/f/mNlKFIz9NSu5pgViVK IaUZNNIKTfUw8rpbBNxsw7OB8A4Ss9cxiNMnkAVlzUXW0PzPUedLb5rgyCHFUHWw84PB J7dXrWU8faSIVkngif6y2DVlhSnU3bynNXuKwu39KI5WDuo35uaJ5MlzVWIIW0T6qWEC 96wAM+5/A9FvFU8SbvM/1H+w83f/C3fmPBvp3otj1GL1jsY03loC/kV5H3/uZBjnaOS0 xpfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730147721; x=1730752521; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=bLTESXYO5Q2VPO2t1ZblnTkpPaTGYqk0UFlBrBdMu3I=; b=ZQJ9W9dKu/8zxO62MSQs0LZDs/4tyN+44o7ez4cIIf+7gKrIk+/8ckD6H+ga/DV+Tg y38td4eqGB3E9m2cVpFmPk1kcCmbBihVaTLmOFQav0psadvnMRTUwV3yBsKpTdpo2rs8 9VigvkKRhGgTZcuBF59giasR54SwzLAe+a22/SILX+QWfRGm9oBU9JLjSIDTpX8rzYeM ZenI5I3rbdq1mkr4UFUYlKStaA+X+i/9YWDTlJyxkDRnhLVNtdgGD2ZDCDFVGFVCdU+n CfxhQ+rDJ23x+g6I5ZEwrqTIr1gR1XeOT5cf6GB42mnLGMQ1Z4CSxuUljkqjGO34+oMe Y/zw== X-Gm-Message-State: AOJu0YwA+jiu9MqxbpRcJFNve4ZDUJ7HNDgYUvXonYxX6OqZ6aZVuORq 43Lwo2F0comtCjw+xEdxsOjz6nDcpHyT0RhZXIPU7nAZguRlKnibsiBqaIQ5 X-Google-Smtp-Source: AGHT+IHCwTNEyqDwQTQEiQB66z7oJpsTjFomIZ5U6Bdi54bKNKIMsa4W6x8ZvYZZDY7qH10nDnTjzQ== X-Received: by 2002:a05:6902:2b84:b0:e2b:ce99:909b with SMTP id 3f1490d57ef6-e3087c36061mr6647746276.57.1730147720615; Mon, 28 Oct 2024 13:35:20 -0700 (PDT) Received: from localhost.localdomain ([184.147.230.152]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4613228c099sm38215791cf.46.2024.10.28.13.35.19 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 28 Oct 2024 13:35:20 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: calvinwan@google.com, jonathantanmy@google.com, chriscool@tuxfamily.org, eric.peijian@gmail.com, karthik.188@gmail.com, toon@iotcl.com, jltobler@gmail.com Subject: [PATCH v5 4/6] transport: add client support for object-info Date: Mon, 28 Oct 2024 16:34:55 -0400 Message-ID: <20241028203457.19715-5-eric.peijian@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241028203457.19715-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> <20241028203457.19715-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 From: Calvin Wan Sometimes it is useful to get information about an object without having to download it completely. The server logic has already been implemented in “a2ba162cda (object-info: support for retrieving object info, 2021-04-20)”. Add client functions to communicate with the server. The client currently supports requesting a list of object ids with feature 'size' from a v2 server. If a server does not advertise the feature, then the client falls back to making the request through 'fetch'. Helped-by: Jonathan Tan Helped-by: Christian Couder Signed-off-by: Calvin Wan Signed-off-by: Eric Ju --- Makefile | 1 + fetch-object-info.c | 95 +++++++++++++++++++++++++++++++++++++++++++++ fetch-object-info.h | 18 +++++++++ fetch-pack.c | 3 ++ fetch-pack.h | 2 + transport-helper.c | 11 +++++- transport.c | 80 ++++++++++++++++++++++++++++++++++++-- transport.h | 11 ++++++ 8 files changed, 216 insertions(+), 5 deletions(-) create mode 100644 fetch-object-info.c create mode 100644 fetch-object-info.h diff --git a/Makefile b/Makefile index 8d8cc6ab90..3969ddcaa8 100644 --- a/Makefile +++ b/Makefile @@ -1024,6 +1024,7 @@ LIB_OBJS += ewah/ewah_rlw.o LIB_OBJS += exec-cmd.o LIB_OBJS += fetch-negotiator.o LIB_OBJS += fetch-pack.o +LIB_OBJS += fetch-object-info.o LIB_OBJS += fmt-merge-msg.o LIB_OBJS += fsck.o LIB_OBJS += fsmonitor.o diff --git a/fetch-object-info.c b/fetch-object-info.c new file mode 100644 index 0000000000..4d7c2d265f --- /dev/null +++ b/fetch-object-info.c @@ -0,0 +1,95 @@ +#include "git-compat-util.h" +#include "gettext.h" +#include "hex.h" +#include "pkt-line.h" +#include "connect.h" +#include "oid-array.h" +#include "object-store-ll.h" +#include "fetch-object-info.h" +#include "string-list.h" + +/** + * send_object_info_request sends git-cat-file object-info command and its + * arguments into the request buffer. + */ +static void send_object_info_request(const int fd_out, struct object_info_args *args) +{ + struct strbuf req_buf = STRBUF_INIT; + + write_command_and_capabilities(&req_buf, "object-info", args->server_options); + + if (unsorted_string_list_has_string(args->object_info_options, "size")) + packet_buf_write(&req_buf, "size"); + + if (args->oids) { + for (size_t i = 0; i < args->oids->nr; i++) + packet_buf_write(&req_buf, "oid %s", oid_to_hex(&args->oids->oid[i])); + } + + packet_buf_flush(&req_buf); + if (write_in_full(fd_out, req_buf.buf, req_buf.len) < 0) + die_errno(_("unable to write request to remote")); + + strbuf_release(&req_buf); +} + +/** + * fetch_object_info sends git-cat-file object-info command into the request buf + * and read the results from packets. + */ +int fetch_object_info(const enum protocol_version version, struct object_info_args *args, + struct packet_reader *reader, struct object_info *object_info_data, + const int stateless_rpc, const int fd_out) +{ + int size_index = -1; + + switch (version) { + case protocol_v2: + if (!server_supports_v2("object-info")) + return -1; + if (unsorted_string_list_has_string(args->object_info_options, "size") + && !server_supports_feature("object-info", "size", 0)) + return -1; + send_object_info_request(fd_out, args); + break; + case protocol_v1: + case protocol_v0: + die(_("wrong protocol version. expected v2")); + case protocol_unknown_version: + BUG("unknown protocol version"); + } + + for (size_t i = 0; i < args->object_info_options->nr; i++) { + if (packet_reader_read(reader) != PACKET_READ_NORMAL) { + check_stateless_delimiter(stateless_rpc, reader, "stateless delimiter expected"); + return -1; + } + if (unsorted_string_list_has_string(args->object_info_options, reader->line)) { + if (!strcmp(reader->line, "size")) { + size_index = i; + for (size_t j = 0; j < args->oids->nr; j++) + object_info_data[j].sizep = xcalloc(1, sizeof(long)); + } + continue; + } + return -1; + } + + for (size_t i = 0; packet_reader_read(reader) == PACKET_READ_NORMAL && i < args->oids->nr; i++){ + struct string_list object_info_values = STRING_LIST_INIT_DUP; + + string_list_split(&object_info_values, reader->line, ' ', -1); + if (0 <= size_index) { + if (!strcmp(object_info_values.items[1 + size_index].string, "")) + die("object-info: not our ref %s", + object_info_values.items[0].string); + + *object_info_data[i].sizep = strtoul(object_info_values.items[1 + size_index].string, NULL, 10); + } + + string_list_clear(&object_info_values, 0); + } + check_stateless_delimiter(stateless_rpc, reader, "stateless delimiter expected"); + + return 0; +} diff --git a/fetch-object-info.h b/fetch-object-info.h new file mode 100644 index 0000000000..b1e545532f --- /dev/null +++ b/fetch-object-info.h @@ -0,0 +1,18 @@ +#ifndef FETCH_OBJECT_INFO_H +#define FETCH_OBJECT_INFO_H + +#include "pkt-line.h" +#include "protocol.h" +#include "object-store-ll.h" + +struct object_info_args { + struct string_list *object_info_options; + const struct string_list *server_options; + struct oid_array *oids; +}; + +int fetch_object_info(enum protocol_version version, struct object_info_args *args, + struct packet_reader *reader, struct object_info *object_info_data, + int stateless_rpc, int fd_out); + +#endif /* FETCH_OBJECT_INFO_H */ diff --git a/fetch-pack.c b/fetch-pack.c index afffbcaafc..8b4143d752 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1651,6 +1651,9 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, if (args->depth > 0 || args->deepen_since || args->deepen_not) args->deepen = 1; + if (args->object_info) + state = FETCH_SEND_REQUEST; + while (state != FETCH_DONE) { switch (state) { case FETCH_CHECK_LOCAL: diff --git a/fetch-pack.h b/fetch-pack.h index b5c579cdae..cf7cedf161 100644 --- a/fetch-pack.h +++ b/fetch-pack.h @@ -16,6 +16,7 @@ struct fetch_pack_args { const struct string_list *deepen_not; struct list_objects_filter_options filter_options; const struct string_list *server_options; + struct object_info *object_info_data; /* * If not NULL, during packfile negotiation, fetch-pack will send "have" @@ -42,6 +43,7 @@ struct fetch_pack_args { unsigned reject_shallow_remote:1; unsigned deepen:1; unsigned refetch:1; + unsigned object_info:1; /* * Indicate that the remote of this request is a promisor remote. The diff --git a/transport-helper.c b/transport-helper.c index 013ec79dc9..334b35174e 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -709,8 +709,8 @@ static int fetch_refs(struct transport *transport, /* * If we reach here, then the server, the client, and/or the transport - * helper does not support protocol v2. --negotiate-only requires - * protocol v2. + * helper does not support protocol v2. --negotiate-only and cat-file + * remote-object-info require protocol v2. */ if (data->transport_options.acked_commits) { warning(_("--negotiate-only requires protocol v2")); @@ -726,6 +726,13 @@ static int fetch_refs(struct transport *transport, free_refs(dummy); } + /* fail the command explicitly to avoid further commands input. */ + if (transport->smart_options->object_info) + die(_("remote-object-info requires protocol v2")); + + if (!data->get_refs_list_called) + get_refs_list_using_list(transport, 0); + count = 0; for (i = 0; i < nr_heads; i++) if (!(to_fetch[i]->status & REF_STATUS_UPTODATE)) diff --git a/transport.c b/transport.c index 47fda6a773..41e157d73d 100644 --- a/transport.c +++ b/transport.c @@ -9,6 +9,7 @@ #include "hook.h" #include "pkt-line.h" #include "fetch-pack.h" +#include "fetch-object-info.h" #include "remote.h" #include "connect.h" #include "send-pack.h" @@ -418,6 +419,7 @@ static int fetch_refs_via_pack(struct transport *transport, struct ref *refs = NULL; struct fetch_pack_args args; struct ref *refs_tmp = NULL, **to_fetch_dup = NULL; + struct ref *object_info_refs = NULL; memset(&args, 0, sizeof(args)); args.uploadpack = data->options.uploadpack; @@ -444,11 +446,71 @@ static int fetch_refs_via_pack(struct transport *transport, args.server_options = transport->server_options; args.negotiation_tips = data->options.negotiation_tips; args.reject_shallow_remote = transport->smart_options->reject_shallow; + args.object_info = transport->smart_options->object_info; + + if (transport->smart_options + && transport->smart_options->object_info + && transport->smart_options->object_info_oids->nr > 0) { + struct ref *ref_itr = object_info_refs = alloc_ref(""); + struct packet_reader reader; + struct object_info_args obj_info_args = { 0 }; + + obj_info_args.server_options = transport->server_options; + obj_info_args.object_info_options = transport->smart_options->object_info_options; + obj_info_args.oids = transport->smart_options->object_info_oids; + + connect_setup(transport, 0); + packet_reader_init(&reader, data->fd[0], NULL, 0, + PACKET_READ_CHOMP_NEWLINE | + PACKET_READ_GENTLE_ON_EOF | + PACKET_READ_DIE_ON_ERR_PACKET); + + data->version = discover_version(&reader); + transport->hash_algo = reader.hash_algo; + + if (!fetch_object_info(data->version, &obj_info_args, &reader, + data->options.object_info_data, transport->stateless_rpc, + data->fd[1])) { + /* + * If the code reaches here, fetch_object_info is successful and + * remote object info are retrieved from packets (i.e. without + * downloading the objects). + */ + goto cleanup; + } - if (!data->finished_handshake) { - int i; + /* + * If the code reaches here, it means we can't retrieve object info from + * packets, and we will fallback to downland the pack files. + * We set quiet and no_progress to be true, so that the internal call to + * fetch-pack is less verbose. + */ + args.object_info_data = data->options.object_info_data; + args.quiet = 1; + args.no_progress = 1; + + /* + * Allocate memory for object info data according to oids. + * The actual results will be retrieved later from the downloaded + * pack files. + */ + for (size_t i = 0; i < transport->smart_options->object_info_oids->nr; i++) { + ref_itr->old_oid = transport->smart_options->object_info_oids->oid[i]; + ref_itr->exact_oid = 1; + if (i == transport->smart_options->object_info_oids->nr - 1) + /* last element, no need to allocate to next */ + ref_itr->next = NULL; + else + ref_itr->next = alloc_ref(""); + + ref_itr = ref_itr->next; + } + + transport->remote_refs = object_info_refs; + + } else if (!data->finished_handshake) { int must_list_refs = 0; - for (i = 0; i < nr_heads; i++) { + for (int i = 0; i < nr_heads; i++) { if (!to_fetch[i]->exact_oid) { must_list_refs = 1; break; @@ -494,6 +556,17 @@ static int fetch_refs_via_pack(struct transport *transport, &transport->pack_lockfiles, data->version); data->finished_handshake = 0; + + /* Retrieve object info data from the downloaded pack files */ + if (args.object_info) { + struct ref *ref_cpy_reader = object_info_refs; + for (int i = 0; ref_cpy_reader; i++) { + oid_object_info_extended(the_repository, &ref_cpy_reader->old_oid, + &args.object_info_data[i], OBJECT_INFO_LOOKUP_REPLACE); + ref_cpy_reader = ref_cpy_reader->next; + } + } + data->options.self_contained_and_connected = args.self_contained_and_connected; data->options.connectivity_checked = args.connectivity_checked; @@ -504,6 +577,7 @@ static int fetch_refs_via_pack(struct transport *transport, ret = -1; cleanup: + free_refs(object_info_refs); close(data->fd[0]); if (data->fd[1] >= 0) close(data->fd[1]); diff --git a/transport.h b/transport.h index 44100fa9b7..42b8ee1251 100644 --- a/transport.h +++ b/transport.h @@ -5,6 +5,7 @@ #include "remote.h" #include "list-objects-filter-options.h" #include "string-list.h" +#include "object-store.h" struct git_transport_options { unsigned thin : 1; @@ -30,6 +31,12 @@ struct git_transport_options { */ unsigned connectivity_checked:1; + /* + * Transport will attempt to pull only object-info. Fallbacks + * to pulling entire object if object-info is not supported. + */ + unsigned object_info : 1; + int depth; const char *deepen_since; const struct string_list *deepen_not; @@ -53,6 +60,10 @@ struct git_transport_options { * common commits to this oidset instead of fetching any packfiles. */ struct oidset *acked_commits; + + struct oid_array *object_info_oids; + struct object_info *object_info_data; + struct string_list *object_info_options; }; enum transport_family { From patchwork Mon Oct 28 20:34:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13854115 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id AE6A71EE02A for ; Mon, 28 Oct 2024 20:35:22 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147724; cv=none; b=tfd7PoMgU224iB7PbkzN/x1NB+d9S6aRTmQNRMmAlyDMm/E1JZ0WmOYPDrG+Zak+vD8K4LhDWHKGiNO44wFvHDhqRhBnZiCWnZ5PcHzeLFVgudVzg8DdoOpBPD5OMgahMDYD1tnaSgHsLjHJ7MmRdNg2rZYdUlbNnmNPYyjAp30= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147724; c=relaxed/simple; bh=FMOM9jhFBpxYkrmdNgtvOsXn4xzPCnMWTBVYg03X7m8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=bZl4cXXG1S2xNqVljTJf0L2FAgt1A7JzA9tSZFl6Y1F2fsedJxg54Kj6PrDMhqigdf6o62io4vyjc264W4LTSVLX0Y5ZLbKyqYyPh2bVYN6eApFd/qLJjxU8sWi4VnLF0GTcdVy/IPSOybEkH8Tc5pt++4xjs4fQ51EXS4n0oXw= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=dY4E2zS/; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dY4E2zS/" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-460c2418e37so37421751cf.0 for ; Mon, 28 Oct 2024 13:35:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730147721; x=1730752521; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7bXXKwzFSC0TkD8Nt5yO4VGdeeX7EPsBU1Pyy8CMfhc=; b=dY4E2zS/5EwBS0qXMgq01xNmmg1isrUJNBXVFYD9Bt5Q42yhWgQcftrfYwqVDu/ziz 8z1r6vGv87SXGil+FjWUGz3TxZZoOR8l5pHF/6leTOQ+CneMK1VLGuDVWk3C8Yzb3iRU 8hKUDMrt99/XiJ170xj7rloDlyzRjVZO/QUt6LnpXW64rlHYXHeiBxcg/0oI06Bb+C1j 62qUC57qwJKYSPyN+QKosdTAcQUcGROmAnWxM+0d93UOnGCuIXt+vrzXIoNA9bgqwwB1 Xmi3E8xSYgxacYtq9r0XU+QwC+m6lSJ3q258N5XvARogktNa4/Jhn7svwkFD80M3UI4w 1Vbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730147721; x=1730752521; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7bXXKwzFSC0TkD8Nt5yO4VGdeeX7EPsBU1Pyy8CMfhc=; b=bLnIl/ZYVYSsOUAcZOh3TB4eTknS8QYCAE5J3fBAORZ+cZieFYQtvDezyKjoKiU+VJ Yxypr4Z/1ywirhNlyz8vD5qzJ7CpdmrwVBGJoCazzg7p3FLQrErPibqzrdZz8fcVtH3d /fFSqAxK8JNiH4fouduQs62+m54NChSvHV5/0F8ntMBnKxf/loEMK9/wnnfz8Xq9D3Wb ZmWUnLdeodRPY9KHQOdRVnA73YfP5jdzb13sCNmQREEYNncX2hmSkIuvmH5EIAL1FuPK YM8sw+5lYeMTksfvMTeFo75BwGjN8DLkUZx3GRAg7zfGkNabhCpPpzkdn701XtWaNCgy nw9Q== X-Gm-Message-State: AOJu0YxYBhugySpQv13IK8NVMtyAzi2sw1kDO/viQg5SLxX8bQOQc80w wmXutTS41wkuukwZM6Pw0vjChPHnvQXO4CKx4QjTjT+oFw4EAVp96l5OnaZD X-Google-Smtp-Source: AGHT+IGF5psGdRIG6wLW/pK4hvomWhXlY8RmPKHYlc2edfBbzkFa0MiDXXSPVwBiqU97+5hXWEkBJA== X-Received: by 2002:a05:622a:1181:b0:458:2622:808a with SMTP id d75a77b69052e-4613c00d122mr179171481cf.20.1730147721431; Mon, 28 Oct 2024 13:35:21 -0700 (PDT) Received: from localhost.localdomain ([184.147.230.152]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4613228c099sm38215791cf.46.2024.10.28.13.35.20 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 28 Oct 2024 13:35:21 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: calvinwan@google.com, jonathantanmy@google.com, chriscool@tuxfamily.org, eric.peijian@gmail.com, karthik.188@gmail.com, toon@iotcl.com, jltobler@gmail.com Subject: [PATCH v5 5/6] cat-file: add declaration of variable i inside its for loop Date: Mon, 28 Oct 2024 16:34:56 -0400 Message-ID: <20241028203457.19715-6-eric.peijian@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241028203457.19715-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> <20241028203457.19715-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Some code declares variable i and only uses it in a for loop, not in any other logic outside the loop. Change the declaration of i to be inside the for loop for readability. Helped-by: Christian Couder Signed-off-by: Eric Ju --- builtin/cat-file.c | 11 +++-------- 1 file changed, 3 insertions(+), 8 deletions(-) diff --git a/builtin/cat-file.c b/builtin/cat-file.c index bfdfb51c7c..5db55fabc4 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -673,12 +673,10 @@ static void dispatch_calls(struct batch_options *opt, struct queued_cmd *cmd, int nr) { - int i; - if (!opt->buffer_output) die(_("flush is only for --buffer mode")); - for (i = 0; i < nr; i++) + for (size_t i = 0; i < nr; i++) cmd[i].fn(opt, cmd[i].line, output, data); fflush(stdout); @@ -686,9 +684,7 @@ static void dispatch_calls(struct batch_options *opt, static void free_cmds(struct queued_cmd *cmd, size_t *nr) { - size_t i; - - for (i = 0; i < *nr; i++) + for (size_t i = 0; i < *nr; i++) FREE_AND_NULL(cmd[i].line); *nr = 0; @@ -714,7 +710,6 @@ static void batch_objects_command(struct batch_options *opt, size_t alloc = 0, nr = 0; while (strbuf_getdelim_strip_crlf(&input, stdin, opt->input_delim) != EOF) { - int i; const struct parse_cmd *cmd = NULL; const char *p = NULL, *cmd_end; struct queued_cmd call = {0}; @@ -724,7 +719,7 @@ static void batch_objects_command(struct batch_options *opt, if (isspace(*input.buf)) die(_("whitespace before command: '%s'"), input.buf); - for (i = 0; i < ARRAY_SIZE(commands); i++) { + for (size_t i = 0; i < ARRAY_SIZE(commands); i++) { if (!skip_prefix(input.buf, commands[i].name, &cmd_end)) continue; From patchwork Mon Oct 28 20:34:57 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Eric Ju X-Patchwork-Id: 13854117 Received: from mail-qt1-f178.google.com (mail-qt1-f178.google.com [209.85.160.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4EAE81E1C36 for ; Mon, 28 Oct 2024 20:35:24 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.178 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147729; cv=none; b=PQRtpTVNlT4pn3LJDBIGFE+9sPaKEOtNEc/+Bt4+LSlV3Xj4vOReX8/kbH0/UQTrEZhSHym9eQTBFd7QrNlMRRKLbKhc1iSq65ZZ/5XQJqNyuCAzQZpV30QRyHhW/5nuSuAhcvUtpwloY5RgJqipZGigZ8+39I7X69HuLaVlAyY= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1730147729; c=relaxed/simple; bh=5ldaz5uVBhNuMyDYZW1kJxbet3pXDpGwgh48xvdAWa8=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=oe6wfwJSVp3bQOAj0VIs/d/IKGPTsSvt/PWlropc/IMsmlw900DxYap6q4EzQkRKV8gHbGFY5aKLm0mjUY7W70k8Y6XEb3oIcw9yDIGb8yc+W5VihONU0A2fZK9N5jFQbHti/AmgYR/6//pSE1L+fz9Hv8DdXUtaz3N1UHI5N0w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=flLNiNgB; arc=none smtp.client-ip=209.85.160.178 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="flLNiNgB" Received: by mail-qt1-f178.google.com with SMTP id d75a77b69052e-46089a6849bso30639901cf.3 for ; Mon, 28 Oct 2024 13:35:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1730147723; x=1730752523; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=MHfdZFyM1xjaJi4Fj+13YJ/Nvhct0a66cbQSozNyJiU=; b=flLNiNgBUgd/9b04YiuMyQYdJdFOWoyFIQGh12G5vq0nQNzsWHbkXb8DsjwSwUnTsb XmrWTChMgoYQrrMme+jQ///VdvemeO8OKtxL0/b3db0rYJVPAtby93Pvwcn+IbrK6wZW /1ATDBz/S48aJ5TmAgCcDPlmSGRAmV8YfEQI6xtzzSZQpnDjmtKagFGrf+bdEdeoWCZh FGwEDxeIcuxjQ+uNspGP5DNM0nd2hGXoMh6bckoq6ZG4r3XD3li9K8LQMTeI7Z4mhqAp lcNxJtktMJJv5YgB4unqm+qBnHjzhlJjyPoBdsF52v2JlLRBZG/puW8Bw10KNvpwRBqd KBQQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1730147723; x=1730752523; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MHfdZFyM1xjaJi4Fj+13YJ/Nvhct0a66cbQSozNyJiU=; b=quBIufGxyFNtLruTKvAPJ48JO0tUvzddlIF+Kj7zpE+5Rgt2grWk9rnlB4m9e3QgvD xJBZ63FqG1Np4row52phu/dhp0kDN0jXCgncaHUZVMbNMm9sxZa+pQadW1+hUCZFWngy IxGdwE2IU625yhSpZXsYqq976hul9vSN9Xjnh3ukDf0Yqon9d0HgTTHEfEqSEVbPH9iy cDpMtYYBhR0byy2evXYGOIPR/z+niXsGozTxf9ZlOErIczUmaEUchpPBnqjoXDhjYEj1 SvXQu0+v6L8OeFuPcwxrVKx5PBGw5hnQ3g9KsOTMb09BJpGIxVBWyfz7rdx4c3CXZeRX Rk3w== X-Gm-Message-State: AOJu0Yyof25ETaud7Xm2CTtK2yDxvKblnbeIMdkvHxWvkUHfVXggRnSu wLRI8rKHAEbfNu08c6yJpi939gO/nKdHrsxw7e4wlMoHhTQx4yfE8pez1g3/ X-Google-Smtp-Source: AGHT+IFelNV9zhpC8ik4N8TZ//yOypl0FPfrekCITsUkEZfdNKKapIYr9GnWjM1xTh2SuQ0yuGHi9Q== X-Received: by 2002:ac8:7fc1:0:b0:460:e89d:897e with SMTP id d75a77b69052e-4613c0153fbmr119760161cf.18.1730147722624; Mon, 28 Oct 2024 13:35:22 -0700 (PDT) Received: from localhost.localdomain ([184.147.230.152]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-4613228c099sm38215791cf.46.2024.10.28.13.35.21 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Mon, 28 Oct 2024 13:35:22 -0700 (PDT) From: Eric Ju To: git@vger.kernel.org Cc: calvinwan@google.com, jonathantanmy@google.com, chriscool@tuxfamily.org, eric.peijian@gmail.com, karthik.188@gmail.com, toon@iotcl.com, jltobler@gmail.com Subject: [PATCH v5 6/6] cat-file: add remote-object-info to batch-command Date: Mon, 28 Oct 2024 16:34:57 -0400 Message-ID: <20241028203457.19715-7-eric.peijian@gmail.com> X-Mailer: git-send-email 2.47.0 In-Reply-To: <20241028203457.19715-1-eric.peijian@gmail.com> References: <20240628190503.67389-1-eric.peijian@gmail.com> <20241028203457.19715-1-eric.peijian@gmail.com> Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Since the `info` command in cat-file --batch-command prints object info for a given object, it is natural to add another command in cat-file --batch-command to print object info for a given object from a remote. Add `remote-object-info` to cat-file --batch-command. While `info` takes object ids one at a time, this creates overhead when making requests to a server so `remote-object-info` instead can take multiple object ids at once. cat-file --batch-command is generally implemented in the following manner: - Receive and parse input from user - Call respective function attached to command - Get object info, print object info In --buffer mode, this changes to: - Receive and parse input from user - Store respective function attached to command in a queue - After flush, loop through commands in queue - Call respective function attached to command - Get object info, print object info Notice how the getting and printing of object info is accomplished one at a time. As described above, this creates a problem for making requests to a server. Therefore, `remote-object-info` is implemented in the following manner: - Receive and parse input from user If command is `remote-object-info`: - Get object info from remote - Loop through and print each object info Else: - Call respective function attached to command - Parse input, get object info, print object info And finally for --buffer mode `remote-object-info`: - Receive and parse input from user - Store respective function attached to command in a queue - After flush, loop through commands in queue: If command is `remote-object-info`: - Get object info from remote - Loop through and print each object info Else: - Call respective function attached to command - Get object info, print object info To summarize, `remote-object-info` gets object info from the remote and then loop through the object info passed in, printing the info. In order for remote-object-info to avoid remote communication overhead in the non-buffer mode, the objects are passed in as such: remote-object-info ... rather than remote-object-info remote-object-info ... remote-object-info Helped-by: Jonathan Tan Helped-by: Christian Couder Signed-off-by: Calvin Wan Signed-off-by: Eric Ju --- Documentation/git-cat-file.txt | 24 +- builtin/cat-file.c | 108 +++- object-file.c | 11 + object-store-ll.h | 3 + t/lib-cat-file.sh | 16 + t/t1006-cat-file.sh | 13 +- t/t1017-cat-file-remote-object-info.sh | 739 +++++++++++++++++++++++++ 7 files changed, 897 insertions(+), 17 deletions(-) create mode 100644 t/lib-cat-file.sh create mode 100755 t/t1017-cat-file-remote-object-info.sh diff --git a/Documentation/git-cat-file.txt b/Documentation/git-cat-file.txt index d5890ae368..f2be00b599 100644 --- a/Documentation/git-cat-file.txt +++ b/Documentation/git-cat-file.txt @@ -149,6 +149,13 @@ info :: Print object info for object reference ``. This corresponds to the output of `--batch-check`. +remote-object-info ...:: + Print object info for object references `` at specified without + downloading objects from remote. If the object-info capability is not + supported by the server, the objects will be downloaded instead. + Error when no object references are provided. + This command may be combined with `--buffer`. + flush:: Used with `--buffer` to execute all preceding commands that were issued since the beginning or since the last flush was issued. When `--buffer` @@ -290,7 +297,8 @@ newline. The available atoms are: The full hex representation of the object name. `objecttype`:: - The type of the object (the same as `cat-file -t` reports). + The type of the object (the same as `cat-file -t` reports). See + `CAVEATS` below. Not supported by `remote-object-info`. `objectsize`:: The size, in bytes, of the object (the same as `cat-file -s` @@ -298,13 +306,14 @@ newline. The available atoms are: `objectsize:disk`:: The size, in bytes, that the object takes up on disk. See the - note about on-disk sizes in the `CAVEATS` section below. + note about on-disk sizes in the `CAVEATS` section below. Not + supported by `remote-object-info`. `deltabase`:: If the object is stored as a delta on-disk, this expands to the full hex representation of the delta base object name. Otherwise, expands to the null OID (all zeroes). See `CAVEATS` - below. + below. Not supported by `remote-object-info`. `rest`:: If this atom is used in the output string, input lines are split @@ -314,7 +323,10 @@ newline. The available atoms are: line) are output in place of the `%(rest)` atom. If no format is specified, the default format is `%(objectname) -%(objecttype) %(objectsize)`. +%(objecttype) %(objectsize)`, except for `remote-object-info` commands which use +`%(objectname) %(objectsize)` for now because "%(objecttype)" is not supported yet. +WARNING: When "%(objecttype)" is supported, the default format WILL be unified, so +DO NOT RELY on the current the default format to stay the same!!! If `--batch` is specified, or if `--batch-command` is used with the `contents` command, the object information is followed by the object contents (consisting @@ -396,6 +408,10 @@ scripting purposes. CAVEATS ------- +Note that since %(objecttype), %(objectsize:disk) and %(deltabase) are +currently not supported by the `remote-object-info` command, we will error +and exit when they are in the format string. + Note that the sizes of objects on disk are reported accurately, but care should be taken in drawing conclusions about which refs or objects are responsible for disk usage. The size of a packed non-delta object may be diff --git a/builtin/cat-file.c b/builtin/cat-file.c index 5db55fabc4..714c182f39 100644 --- a/builtin/cat-file.c +++ b/builtin/cat-file.c @@ -24,6 +24,9 @@ #include "promisor-remote.h" #include "mailmap.h" #include "write-or-die.h" +#include "alias.h" +#include "remote.h" +#include "transport.h" enum batch_mode { BATCH_MODE_CONTENTS, @@ -42,9 +45,12 @@ struct batch_options { char input_delim; char output_delim; const char *format; + int use_remote_info; }; static const char *force_path; +static struct object_info *remote_object_info; +static struct oid_array object_info_oids = OID_ARRAY_INIT; static struct string_list mailmap = STRING_LIST_INIT_NODUP; static int use_mailmap; @@ -528,7 +534,7 @@ static void batch_one_object(const char *obj_name, enum get_oid_result result; result = get_oid_with_context(the_repository, obj_name, - flags, &data->oid, &ctx); + flags, &data->oid, &ctx); if (result != FOUND) { switch (result) { case MISSING_OBJECT: @@ -576,6 +582,59 @@ static void batch_one_object(const char *obj_name, object_context_release(&ctx); } +static int get_remote_info(struct batch_options *opt, int argc, const char **argv) +{ + int retval = 0; + struct remote *remote = NULL; + struct object_id oid; + struct string_list object_info_options = STRING_LIST_INIT_NODUP; + static struct transport *gtransport; + + /* + * Change the format to "%(objectname) %(objectsize)" when + * remote-object-info command is used. Once we start supporting objecttype + * the default format should change to DEFAULT_FORMAT + */ + if (!opt->format) + opt->format = "%(objectname) %(objectsize)"; + + remote = remote_get(argv[0]); + if (!remote) + die(_("must supply valid remote when using remote-object-info")); + + oid_array_clear(&object_info_oids); + for (size_t i = 1; i < argc; i++) { + if (get_oid_hex(argv[i], &oid)) + die(_("Not a valid object name %s"), argv[i]); + oid_array_append(&object_info_oids, &oid); + } + + gtransport = transport_get(remote, NULL); + if (gtransport->smart_options) { + CALLOC_ARRAY(remote_object_info, object_info_oids.nr); + gtransport->smart_options->object_info = 1; + gtransport->smart_options->object_info_oids = &object_info_oids; + + /* 'objectsize' is the only option currently supported */ + if (!strstr(opt->format, "%(objectsize)")) + die(_("%s is currently not supported with remote-object-info"), opt->format); + + string_list_append(&object_info_options, "size"); + + if (object_info_options.nr > 0) { + gtransport->smart_options->object_info_options = &object_info_options; + gtransport->smart_options->object_info_data = remote_object_info; + retval = transport_fetch_refs(gtransport, NULL); + } + } else { + retval = -1; + } + + string_list_clear(&object_info_options, 0); + transport_disconnect(gtransport); + return retval; +} + struct object_cb_data { struct batch_options *opt; struct expand_data *expand; @@ -667,6 +726,52 @@ static void parse_cmd_info(struct batch_options *opt, batch_one_object(line, output, opt, data); } +static void parse_cmd_remote_object_info(struct batch_options *opt, + const char *line, + struct strbuf *output, + struct expand_data *data) +{ + int count; + const char **argv; + + char *line_to_split = xstrdup_or_null(line); + count = split_cmdline(line_to_split, &argv); + if (get_remote_info(opt, count, argv)) + goto cleanup; + + opt->use_remote_info = 1; + data->skip_object_info = 1; + for (size_t i = 0; i < object_info_oids.nr; i++) { + + data->oid = object_info_oids.oid[i]; + + if (remote_object_info[i].sizep) { + data->size = *remote_object_info[i].sizep; + } else { + /* + * When reaching here, it means remote-object-info can't retrieve + * information from server without downloading them, and the objects + * have been fetched to client already. + * Print the information using the logic for local objects. + */ + data->skip_object_info = 0; + } + + opt->batch_mode = BATCH_MODE_INFO; + batch_object_write(argv[i+1], output, opt, data, NULL, 0); + + } + opt->use_remote_info = 0; + data->skip_object_info = 0; + +cleanup: + for (size_t i = 0; i < object_info_oids.nr; i++) + free_object_info_contents(&remote_object_info[i]); + free(line_to_split); + free(argv); + free(remote_object_info); +} + static void dispatch_calls(struct batch_options *opt, struct strbuf *output, struct expand_data *data, @@ -698,6 +803,7 @@ static const struct parse_cmd { } commands[] = { { "contents", parse_cmd_contents, 1}, { "info", parse_cmd_info, 1}, + { "remote-object-info", parse_cmd_remote_object_info, 1}, { "flush", NULL, 0}, }; diff --git a/object-file.c b/object-file.c index b1a3463852..181cde98e1 100644 --- a/object-file.c +++ b/object-file.c @@ -3132,3 +3132,14 @@ int read_loose_object(const char *path, munmap(map, mapsize); return ret; } + +void free_object_info_contents(struct object_info *object_info) +{ + if (!object_info) + return; + free(object_info->typep); + free(object_info->sizep); + free(object_info->disk_sizep); + free(object_info->delta_base_oid); + free(object_info->type_name); +} diff --git a/object-store-ll.h b/object-store-ll.h index 53b8e693b1..611e2ca708 100644 --- a/object-store-ll.h +++ b/object-store-ll.h @@ -548,4 +548,7 @@ int for_each_object_in_pack(struct packed_git *p, int for_each_packed_object(each_packed_object_fn, void *, enum for_each_object_flags flags); +/* Free pointers inside of object_info, but not object_info itself */ +void free_object_info_contents(struct object_info *object_info); + #endif /* OBJECT_STORE_LL_H */ diff --git a/t/lib-cat-file.sh b/t/lib-cat-file.sh new file mode 100644 index 0000000000..9fb20be308 --- /dev/null +++ b/t/lib-cat-file.sh @@ -0,0 +1,16 @@ +# Library of git-cat-file related tests. + +# Print a string without a trailing newline +echo_without_newline () { + printf '%s' "$*" +} + +# Print a string without newlines and replaces them with a NULL character (\0). +echo_without_newline_nul () { + echo_without_newline "$@" | tr '\n' '\0' +} + +# Calculate the length of a string removing any leading spaces. +strlen () { + echo_without_newline "$1" | wc -c | sed -e 's/^ *//' +} diff --git a/t/t1006-cat-file.sh b/t/t1006-cat-file.sh index d36cd7c086..d8a851c427 100755 --- a/t/t1006-cat-file.sh +++ b/t/t1006-cat-file.sh @@ -4,6 +4,7 @@ test_description='git cat-file' TEST_PASSES_SANITIZE_LEAK=true . ./test-lib.sh +. "$TEST_DIRECTORY"/lib-cat-file.sh test_cmdmode_usage () { test_expect_code 129 "$@" 2>err && @@ -99,18 +100,6 @@ do ' done -echo_without_newline () { - printf '%s' "$*" -} - -echo_without_newline_nul () { - echo_without_newline "$@" | tr '\n' '\0' -} - -strlen () { - echo_without_newline "$1" | wc -c | sed -e 's/^ *//' -} - run_tests () { type=$1 oid=$2 diff --git a/t/t1017-cat-file-remote-object-info.sh b/t/t1017-cat-file-remote-object-info.sh new file mode 100755 index 0000000000..f4bff07311 --- /dev/null +++ b/t/t1017-cat-file-remote-object-info.sh @@ -0,0 +1,739 @@ +#!/bin/sh + +test_description='git cat-file --batch-command with remote-object-info command' + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh +. "$TEST_DIRECTORY"/lib-cat-file.sh + +hello_content="Hello World" +hello_size=$(strlen "$hello_content") +hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin) + +# This is how we get 13: +# 13 = + + + , where +# file mode is 100644, which is 6 characters; +# file name is hello, which is 5 characters +# a space is 1 character and a null is 1 character +tree_size=$(($(test_oid rawsz) + 13)) + +commit_message="Initial commit" + +# This is how we get 137: +# 137 = + + + +# + + +# + + +# + +# +# An easier way to calculate is: 1. use `git cat-file commit | wc -c`, +# to get 177, 2. then deduct 40 hex characters to get 137 +commit_size=$(($(test_oid hexsz) + 137)) + +tag_header_without_oid="type blob +tag hellotag +tagger $GIT_COMMITTER_NAME <$GIT_COMMITTER_EMAIL>" +tag_header_without_timestamp="object $hello_oid +$tag_header_without_oid" +tag_description="This is a tag" +tag_content="$tag_header_without_timestamp 0 +0000 + +$tag_description" + +tag_oid=$(echo_without_newline "$tag_content" | git hash-object -t tag --stdin -w) +tag_size=$(strlen "$tag_content") + +set_transport_variables () { + hello_oid=$(echo_without_newline "$hello_content" | git hash-object --stdin) + tree_oid=$(git -C "$1" write-tree) + commit_oid=$(echo_without_newline "$commit_message" | git -C "$1" commit-tree $tree_oid) + tag_oid=$(echo_without_newline "$tag_content" | git -C "$1" hash-object -t tag --stdin -w) + tag_size=$(strlen "$tag_content") +} + +# This section tests --batch-command with remote-object-info command +# Since "%(objecttype)" is currently not supported by the command remote-object-info , +# the filters are set to "%(objectname) %(objectsize)" in some test cases. + +# Test --batch-command remote-object-info with 'git://' transport with +# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability +. "$TEST_DIRECTORY"/lib-git-daemon.sh +start_git_daemon --export-all --enable=receive-pack +daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent + +test_expect_success 'create repo to be served by git-daemon' ' + git init "$daemon_parent" && + echo_without_newline "$hello_content" > $daemon_parent/hello && + git -C "$daemon_parent" update-index --add hello && + git -C "$daemon_parent" config transfer.advertiseobjectinfo true && + git clone "$GIT_DAEMON_URL/parent" -n "$daemon_parent/daemon_client_empty" +' + +test_expect_success 'batch-command remote-object-info git://' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + # These results prove remote-object-info can get object info from the remote + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results prove remote-object-info did not download objects from the remote + echo "$hello_oid missing" >>expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid + remote-object-info "$GIT_DAEMON_URL/parent" $tree_oid + remote-object-info "$GIT_DAEMON_URL/parent" $commit_oid + remote-object-info "$GIT_DAEMON_URL/parent" $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info git:// multiple sha1 per line' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + # These results prove remote-object-info can get object info from the remote + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results prove remote-object-info did not download objects from the remote + echo "$hello_oid missing" >>expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid $tree_oid $commit_oid $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info git:// default filter' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + GIT_TRACE_PACKET=1 git cat-file --batch-command >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid $tree_oid + remote-object-info "$GIT_DAEMON_URL/parent" $commit_oid $tag_oid + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command --buffer remote-object-info git://' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + # These results prove remote-object-info can get object info from the remote + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results prove remote-object-info did not download objects from the remote + echo "$hello_oid missing" >>expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" --buffer >actual <<-EOF && + remote-object-info "$GIT_DAEMON_URL/parent" $hello_oid $tree_oid + remote-object-info "$GIT_DAEMON_URL/parent" $commit_oid $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + flush + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command -Z remote-object-info git:// default filter' ' + ( + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + printf "%s\0" "$hello_oid $hello_size" >expect && + printf "%s\0" "$tree_oid $tree_size" >>expect && + printf "%s\0" "$commit_oid $commit_size" >>expect && + printf "%s\0" "$tag_oid $tag_size" >>expect && + + printf "%s\0" "$hello_oid missing" >>expect && + printf "%s\0" "$tree_oid missing" >>expect && + printf "%s\0" "$commit_oid missing" >>expect && + printf "%s\0" "$tag_oid missing" >>expect && + + batch_input="remote-object-info $GIT_DAEMON_URL/parent $hello_oid $tree_oid +remote-object-info $GIT_DAEMON_URL/parent $commit_oid $tag_oid +info $hello_oid +info $tree_oid +info $commit_oid +info $tag_oid +" && + echo_without_newline_nul "$batch_input" >commands_null_delimited && + + git cat-file --batch-command -Z < commands_null_delimited >actual && + test_cmp expect actual + ) +' + +# Test --batch-command remote-object-info with 'git://' and +# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability + +test_expect_success 'remote-object-info fallback git://: fetch objects to client' ' + ( + git -C "$daemon_parent" config transfer.advertiseobjectinfo false && + set_transport_variables "$daemon_parent" && + cd "$daemon_parent/daemon_client_empty" && + + # Prove object is not on the client + echo "$hello_oid missing" >expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + # These results prove remote-object-info can retrieve object info + echo "$hello_oid $hello_size" >>expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results are for the info command + # They prove objects are downloaded + echo "$hello_oid $hello_size" >>expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + remote-object-info $GIT_DAEMON_URL/parent $hello_oid $tree_oid $commit_oid $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + EOF + + # revert server state back + git -C "$daemon_parent" config transfer.advertiseobjectinfo true && + + test_cmp expect actual + ) +' + +stop_git_daemon + +# Test --batch-command remote-object-info with 'file://' transport with +# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability +# shellcheck disable=SC2016 +test_expect_success 'create repo to be served by file:// transport' ' + git init server && + git -C server config protocol.version 2 && + git -C server config transfer.advertiseobjectinfo true && + echo_without_newline "$hello_content" > server/hello && + git -C server update-index --add hello && + git clone -n "file://$(pwd)/server" file_client_empty +' + +test_expect_success 'batch-command remote-object-info file://' ' + ( + set_transport_variables "server" && + server_path="$(pwd)/server" && + cd file_client_empty && + + # These results prove remote-object-info can get object info from the remote + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results prove remote-object-info did not download objects from the remote + echo "$hello_oid missing" >>expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "file://${server_path}" $hello_oid + remote-object-info "file://${server_path}" $tree_oid + remote-object-info "file://${server_path}" $commit_oid + remote-object-info "file://${server_path}" $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info file:// multiple sha1 per line' ' + ( + set_transport_variables "server" && + server_path="$(pwd)/server" && + cd file_client_empty && + + # These results prove remote-object-info can get object info from the remote + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results prove remote-object-info did not download objects from the remote + echo "$hello_oid missing" >>expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "file://${server_path}" $hello_oid $tree_oid $commit_oid $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command --buffer remote-object-info file://' ' + ( + set_transport_variables "server" && + server_path="$(pwd)/server" && + cd file_client_empty && + + # These results prove remote-object-info can get object info from the remote + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results prove remote-object-info did not download objects from the remote + echo "$hello_oid missing" >>expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" --buffer >actual <<-EOF && + remote-object-info "file://${server_path}" $hello_oid $tree_oid + remote-object-info "file://${server_path}" $commit_oid $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + flush + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info file:// default filter' ' + ( + set_transport_variables "server" && + server_path="$(pwd)/server" && + cd file_client_empty && + + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + git cat-file --batch-command >actual <<-EOF && + remote-object-info "file://${server_path}" $hello_oid $tree_oid + remote-object-info "file://${server_path}" $commit_oid $tag_oid + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command -Z remote-object-info file:// default filter' ' + ( + set_transport_variables "server" && + server_path="$(pwd)/server" && + cd file_client_empty && + + printf "%s\0" "$hello_oid $hello_size" >expect && + printf "%s\0" "$tree_oid $tree_size" >>expect && + printf "%s\0" "$commit_oid $commit_size" >>expect && + printf "%s\0" "$tag_oid $tag_size" >>expect && + + printf "%s\0" "$hello_oid missing" >>expect && + printf "%s\0" "$tree_oid missing" >>expect && + printf "%s\0" "$commit_oid missing" >>expect && + printf "%s\0" "$tag_oid missing" >>expect && + + batch_input="remote-object-info \"file://${server_path}\" $hello_oid $tree_oid +remote-object-info \"file://${server_path}\" $commit_oid $tag_oid +info $hello_oid +info $tree_oid +info $commit_oid +info $tag_oid +" && + echo_without_newline_nul "$batch_input" >commands_null_delimited && + + git cat-file --batch-command -Z < commands_null_delimited >actual && + test_cmp expect actual + ) +' + +# Test --batch-command remote-object-info with 'file://' and +# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability + +test_expect_success 'remote-object-info fallback file://: fetch objects to client' ' + ( + set_transport_variables "server" && + server_path="$(pwd)/server" && + git -C "${server_path}" config transfer.advertiseobjectinfo false && + cd file_client_empty && + + # Prove object is not on the client + echo "$hello_oid missing" >expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + # These results prove remote-object-info can retrieve object info + echo "$hello_oid $hello_size" >>expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results are for the info command + # They prove objects are downloaded + echo "$hello_oid $hello_size" >>expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + remote-object-info "file://${server_path}" $hello_oid $tree_oid $commit_oid $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + EOF + + # revert server state back + git -C "${server_path}" config transfer.advertiseobjectinfo true && + test_cmp expect actual + ) +' + +# Test --batch-command remote-object-info with 'http://' transport with +# transfer.advertiseobjectinfo set to true, i.e. server has object-info capability + +. "$TEST_DIRECTORY"/lib-httpd.sh +start_httpd + +test_expect_success 'create repo to be served by http:// transport' ' + git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" config http.receivepack true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" config transfer.advertiseobjectinfo true && + echo_without_newline "$hello_content" > $HTTPD_DOCUMENT_ROOT_PATH/http_parent/hello && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" update-index --add hello && + git clone "$HTTPD_URL/smart/http_parent" -n "$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty" +' + +test_expect_success 'batch-command remote-object-info http://' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty" && + + # These results prove remote-object-info can get object info from the remote + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results prove remote-object-info did not download objects from the remote + echo "$hello_oid missing" >>expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_oid + remote-object-info "$HTTPD_URL/smart/http_parent" $tree_oid + remote-object-info "$HTTPD_URL/smart/http_parent" $commit_oid + remote-object-info "$HTTPD_URL/smart/http_parent" $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info http:// one line' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty" && + + # These results prove remote-object-info can get object info from the remote + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results prove remote-object-info did not download objects from the remote + echo "$hello_oid missing" >>expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" >actual <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_oid $tree_oid $commit_oid $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command --buffer remote-object-info http://' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty" && + + # These results prove remote-object-info can get object info from the remote + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results prove remote-object-info did not download objects from the remote + echo "$hello_oid missing" >>expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + git cat-file --batch-command="%(objectname) %(objectsize)" --buffer >actual <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_oid $tree_oid + remote-object-info "$HTTPD_URL/smart/http_parent" $commit_oid $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + flush + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command remote-object-info http:// default filter' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty" && + + echo "$hello_oid $hello_size" >expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + git cat-file --batch-command >actual <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_oid $tree_oid + remote-object-info "$HTTPD_URL/smart/http_parent" $commit_oid $tag_oid + EOF + test_cmp expect actual + ) +' + +test_expect_success 'batch-command -Z remote-object-info http:// default filter' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty" && + + printf "%s\0" "$hello_oid $hello_size" >expect && + printf "%s\0" "$tree_oid $tree_size" >>expect && + printf "%s\0" "$commit_oid $commit_size" >>expect && + printf "%s\0" "$tag_oid $tag_size" >>expect && + + batch_input="remote-object-info $HTTPD_URL/smart/http_parent $hello_oid $tree_oid +remote-object-info $HTTPD_URL/smart/http_parent $commit_oid $tag_oid +" && + echo_without_newline_nul "$batch_input" >commands_null_delimited && + + git cat-file --batch-command -Z < commands_null_delimited >actual && + test_cmp expect actual + ) +' + +test_expect_success 'remote-object-info fails on unspported filter option (objectsize:disk)' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + test_must_fail git cat-file --batch-command="%(objectsize:disk)" 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_oid + EOF + test_grep "%(objectsize:disk) is currently not supported with remote-object-info" err + ) +' + +test_expect_success 'remote-object-info fails on unspported filter option (deltabase)' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + test_must_fail git cat-file --batch-command="%(deltabase)" 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_oid + EOF + test_grep "%(deltabase) is currently not supported with remote-object-info" err + ) +' + +test_expect_success 'remote-object-info fails on server with legacy protocol' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + test_must_fail git -c protocol.version=0 cat-file --batch-command="%(objectname) %(objectsize)" 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_oid + EOF + test_grep "remote-object-info requires protocol v2" err + ) +' + +test_expect_success 'remote-object-info fails on server with legacy protocol fallback' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + + test_must_fail git -c protocol.version=0 cat-file --batch-command 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_oid + EOF + test_grep "remote-object-info requires protocol v2" err + ) +' + +test_expect_success 'remote-object-info fails on malformed OID' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + malformed_object_id="this_id_is_not_valid" && + + test_must_fail git cat-file --batch-command="%(objectname) %(objectsize)" 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $malformed_object_id + EOF + test_grep "Not a valid object name '$malformed_object_id'" err + ) +' + +test_expect_success 'remote-object-info fails on malformed OID fallback' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + malformed_object_id="this_id_is_not_valid" && + + test_must_fail git cat-file --batch-command 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $malformed_object_id + EOF + test_grep "Not a valid object name '$malformed_object_id'" err + ) +' + +test_expect_success 'remote-object-info fails on missing OID' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + git clone "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" missing_oid_repo && + test_commit -C missing_oid_repo message1 c.txt && + cd missing_oid_repo && + + object_id=$(git rev-parse message1:c.txt) && + test_must_fail git cat-file --batch-command="%(objectname) %(objectsize)" 2>err <<-EOF && + remote-object-info "$HTTPD_URL/smart/http_parent" $object_id + EOF + test_grep "object-info: not our ref $object_id" err + ) +' + +# Test --batch-command remote-object-info with 'http://' transport and +# transfer.advertiseobjectinfo set to false, i.e. server does not have object-info capability + +test_expect_success 'remote-object-info fallback http://: fetch objects to client' ' + ( + set_transport_variables "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" config transfer.advertiseobjectinfo false && + cd "$HTTPD_DOCUMENT_ROOT_PATH/http_client_empty" && + + # Prove object is not on the client + echo "$hello_oid missing" >expect && + echo "$tree_oid missing" >>expect && + echo "$commit_oid missing" >>expect && + echo "$tag_oid missing" >>expect && + + # These results prove remote-object-info can retrieve object info + echo "$hello_oid $hello_size" >>expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + # These results are for the info command + # They prove objects are downloaded + echo "$hello_oid $hello_size" >>expect && + echo "$tree_oid $tree_size" >>expect && + echo "$commit_oid $commit_size" >>expect && + echo "$tag_oid $tag_size" >>expect && + + git cat-file --batch-command >actual <<-EOF && + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + remote-object-info "$HTTPD_URL/smart/http_parent" $hello_oid $tree_oid $commit_oid $tag_oid + info $hello_oid + info $tree_oid + info $commit_oid + info $tag_oid + EOF + + # revert server state back + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" config transfer.advertiseobjectinfo true && + + test_cmp expect actual + ) +' + +# DO NOT add non-httpd-specific tests here, because the last part of this +# test script is only executed when httpd is available and enabled. + +test_done