From patchwork Wed Nov 16 19:51:34 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13045687 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3140C4332F for ; Wed, 16 Nov 2022 19:51:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234142AbiKPTvy (ORCPT ); Wed, 16 Nov 2022 14:51:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55112 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232377AbiKPTvt (ORCPT ); Wed, 16 Nov 2022 14:51:49 -0500 Received: from mail-wr1-x436.google.com (mail-wr1-x436.google.com [IPv6:2a00:1450:4864:20::436]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 44DCA45A02 for ; Wed, 16 Nov 2022 11:51:47 -0800 (PST) Received: by mail-wr1-x436.google.com with SMTP id k8so31736538wrh.1 for ; Wed, 16 Nov 2022 11:51:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=/vixHV679y33VgTXhSIXK+evywACQByl7SjY6sOsa8o=; b=Az/PSKOzQ/JVDew5anLFnIc11RYog9aXAcFnseFURYp71RQ3RyUkzMJ8B9Fy5ucH7D YbJdwJ44M7CthMu21OQffMD3sYqtgdWlB4dS/TLM/WnmUYE8bhAvSW1vZdRoRHZSI/Ko LZ1uW6zuD+6oGIDFkLFeBZb9kBk/FE7BEPVGl41gLA0Q0243e3ZCabKKLQSnRCweXuCH m6ImJZ5BJkPlgQxWfV2V/J61SyOxnXw2PQevtGMpleMvyGHuNVp8u+fB9FwU9ZlIt4nP nleR0tbru4Q5o8zNlIVUPqACwqtZh+laVkvkyWqAyuZAp5yc96rk7dZFVhiv4Tputnfo /qFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=/vixHV679y33VgTXhSIXK+evywACQByl7SjY6sOsa8o=; b=hmg78CQwDuS8HqK8eB2q1rahTSUOTaroiq65OdalUiVMXHA4JELD0Fk8s5QZHZ5Ohq 4iek6pduy6J/TbuNoP8HNZ9VuuTvv2zykO7yR7xmqszoWx1nWAwdUOYW3tyEHyiSr3hF Fczhly8UgPH2BI6sXT5b7oFYOTB01lm+Ux6a3pxp1IXQ0YybW6QJz/E0NU7mnRqjIn8S F9KXyC267bTYURyYd3F+nF7OpJp2l1VzsNIQ5/qqhTrfAeCt4NIS3L/xwQS1oTYiBvtR Nd7w+BFEgo/+9kCnMZucyB6fKAF3NQ4QHuyc0HTw3V7wsYu8qYaBqcoj6Nokn26guSHs 9lnA== X-Gm-Message-State: ANoB5plFqb3U4CEY7kS8tPyutvKhBEpbFNtIcgqreFwKsx1czx7HosnO AOx1gRltBUzRa12yP76/iWE3bBJpryI= X-Google-Smtp-Source: AA0mqf7YXcaAtJNuRn7jHcjTNQ6A+nlux7ndQ5XWmRxYmgMBkA2htte+TRp+BN2XCgNTkxdl53SQUw== X-Received: by 2002:a05:6000:61a:b0:241:6e0a:bfe6 with SMTP id bn26-20020a056000061a00b002416e0abfe6mr14736218wrb.34.1668628305457; Wed, 16 Nov 2022 11:51:45 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f19-20020a05600c4e9300b003cfd4a50d5asm3383296wmq.34.2022.11.16.11.51.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Nov 2022 11:51:45 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 16 Nov 2022 19:51:34 +0000 Subject: [PATCH v2 1/9] protocol v2: add server-side "bundle-uri" skeleton MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Add a skeleton server-side implementation of a new "bundle-uri" command to protocol v2. This will allow conforming clients to optionally seed their initial clones or incremental fetches from URLs containing "*.bundle" files created with "git bundle create". This change only performs the basic boilerplate of advertising a new protocol v2 capability. The new 'bundle-uri' capability allows a client to request a list of bundles. Right now, the server only returns a flush packet, which corresponds to an empty advertisement. The bundle.* config namespace describes which key-value pairs will be communicated across this interface in future updates. The critical bit right now is that the new boolean uploadPack.adverstiseBundleURIs config value signals whether or not this capability should be advertised at all. An earlier version of this patch [1] used a different transfer format than the "key=value" pairs in the current implementation. The change was made to unify the protocol v2 command with the bundle lists provided by independent bundle servers. Further, the standard allows for the server to advertise a URI that contains a bundle list. This allows users automatically discovering bundle providers that are loosely associated with the origin server, but without the origin server knowing exactly which bundles are currently available. [1] https://lore.kernel.org/git/RFC-patch-v2-01.13-2fc87ce092b-20220311T155841Z-avarab@gmail.com/ The very-deep headings needed to be modified to stop at level 4 due to documentation build issues. These were not recognized in earlier builds since the file was previously in the Documentation/technical/ directory and was built in a different way. With its current location, the heavily-nested details were causing build issues and they are now replaced with a bulletted list of details. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- Documentation/gitprotocol-v2.txt | 201 +++++++++++++++++++++++++++++++ bundle-uri.c | 36 ++++++ bundle-uri.h | 7 ++ serve.c | 6 + t/t5701-git-serve.sh | 40 +++++- 5 files changed, 289 insertions(+), 1 deletion(-) diff --git a/Documentation/gitprotocol-v2.txt b/Documentation/gitprotocol-v2.txt index 59bf41cefb9..10bd2d40cec 100644 --- a/Documentation/gitprotocol-v2.txt +++ b/Documentation/gitprotocol-v2.txt @@ -578,6 +578,207 @@ and associated requested information, each separated by a single space. obj-info = obj-id SP obj-size +bundle-uri +~~~~~~~~~~ + +If the 'bundle-uri' capability is advertised, the server supports the +`bundle-uri' command. + +The capability is currently advertised with no value (i.e. not +"bundle-uri=somevalue"), a value may be added in the future for +supporting command-wide extensions. Clients MUST ignore any unknown +capability values and proceed with the 'bundle-uri` dialog they +support. + +The 'bundle-uri' command is intended to be issued before `fetch` to +get URIs to bundle files (see linkgit:git-bundle[1]) to "seed" and +inform the subsequent `fetch` command. + +The client CAN issue `bundle-uri` before or after any other valid +command. To be useful to clients it's expected that it'll be issued +after an `ls-refs` and before `fetch`, but CAN be issued at any time +in the dialog. + +DISCUSSION of bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^^ + +The intent of the feature is optimize for server resource consumption +in the common case by changing the common case of fetching a very +large PACK during linkgit:git-clone[1] into a smaller incremental +fetch. + +It also allows servers to achieve better caching in combination with +an `uploadpack.packObjectsHook` (see linkgit:git-config[1]). + +By having new clones or fetches be a more predictable and common +negotiation against the tips of recently produces *.bundle file(s). +Servers might even pre-generate the results of such negotiations for +the `uploadpack.packObjectsHook` as new pushes come in. + +One way that servers could take advantage of these bundles is that the +server would anticipate that fresh clones will download a known bundle, +followed by catching up to the current state of the repository using ref +tips found in that bundle (or bundles). + +PROTOCOL for bundle-uri +^^^^^^^^^^^^^^^^^^^^^^^ + +A `bundle-uri` request takes no arguments, and as noted above does not +currently advertise a capability value. Both may be added in the +future. + +When the client issues a `command=bundle-uri` request, the response is a +list of key-value pairs provided as packet lines with value +`=`. Each `` should be interpreted as a config key from +the `bundle.*` namespace to construct a list of bundles. These keys are +grouped by a `bundle..` subsection, where each key corresponding to a +given `` contributes attributes to the bundle defined by that ``. +See linkgit:git-config[1] for the specific details of these keys and how +the Git client will interpret their values. + +Clients MUST parse the line according to the above format, lines that do +not conform to the format SHOULD be discarded. The user MAY be warned in +such a case. + +bundle-uri CLIENT AND SERVER EXPECTATIONS +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +URI CONTENTS:: +The content at the advertised URIs MUST be one of two types. ++ +The advertised URI may contain a bundle file that `git bundle verify` +would accept. I.e. they MUST contain one or more reference tips for +use by the client, MUST indicate prerequisites (in any) with standard +"-" prefixes, and MUST indicate their "object-format", if +applicable. ++ +The advertised URI may alternatively contain a plaintext file that `git +config --list` would accept (with the `--file` option). The key-value +pairs in this list are in the `bundle.*` namespace (see +linkgit:git-config[1]). + +bundle-uri CLIENT ERROR RECOVERY:: +A client MUST above all gracefully degrade on errors, whether that +error is because of bad missing/data in the bundle URI(s), because +that client is too dumb to e.g. understand and fully parse out bundle +headers and their prerequisite relationships, or something else. ++ +Server operators should feel confident in turning on "bundle-uri" and +not worry if e.g. their CDN goes down that clones or fetches will run +into hard failures. Even if the server bundle bundle(s) are +incomplete, or bad in some way the client should still end up with a +functioning repository, just as if it had chosen not to use this +protocol extension. ++ +All subsequent discussion on client and server interaction MUST keep +this in mind. + +bundle-uri SERVER TO CLIENT:: +The ordering of the returned bundle uris is not significant. Clients +MUST parse their headers to discover their contained OIDS and +prerequisites. A client MUST consider the content of the bundle(s) +themselves and their header as the ultimate source of truth. ++ +A server MAY even return bundle(s) that don't have any direct +relationship to the repository being cloned (either through accident, +or intentional "clever" configuration), and expect a client to sort +out what data they'd like from the bundle(s), if any. + +bundle-uri CLIENT TO SERVER:: +The client SHOULD provide reference tips found in the bundle header(s) +as 'have' lines in any subsequent `fetch` request. A client MAY also +ignore the bundle(s) entirely if doing so is deemed worse for some +reason, e.g. if the bundles can't be downloaded, it doesn't like the +tips it finds etc. + +WHEN ADVERTISED BUNDLE(S) REQUIRE NO FURTHER NEGOTIATION:: +If after issuing `bundle-uri` and `ls-refs`, and getting the header(s) +of the bundle(s) the client finds that the ref tips it wants can be +retrieved entirely from advertised bundle(s), the client MAY disconnect +from the Git server. The results of such a 'clone' or 'fetch' should be +indistinguishable from the state attained without using bundle-uri. + +EARLY CLIENT DISCONNECTIONS AND ERROR RECOVERY:: +A client MAY perform an early disconnect while still downloading the +bundle(s) (having streamed and parsed their headers). In such a case +the client MUST gracefully recover from any errors related to +finishing the download and validation of the bundle(s). ++ +I.e. a client might need to re-connect and issue a 'fetch' command, +and possibly fall back to not making use of 'bundle-uri' at all. ++ +This "MAY" behavior is specified as such (and not a "SHOULD") on the +assumption that a server advertising bundle uris is more likely than +not to be serving up a relatively large repository, and to be pointing +to URIs that have a good chance of being in working order. A client +MAY e.g. look at the payload size of the bundles as a heuristic to see +if an early disconnect is worth it, should falling back on a full +"fetch" dialog be necessary. + +WHEN ADVERTISED BUNDLE(S) REQUIRE FURTHER NEGOTIATION:: +A client SHOULD commence a negotiation of a PACK from the server via +the "fetch" command using the OID tips found in advertised bundles, +even if's still in the process of downloading those bundle(s). ++ +This allows for aggressive early disconnects from any interactive +server dialog. The client blindly trusts that the advertised OID tips +are relevant, and issues them as 'have' lines, it then requests any +tips it would like (usually from the "ls-refs" advertisement) via +'want' lines. The server will then compute a (hopefully small) PACK +with the expected difference between the tips from the bundle(s) and +the data requested. ++ +The only connection the client then needs to keep active is to the +concurrently downloading static bundle(s), when those and the +incremental PACK are retrieved they should be inflated and +validated. Any errors at this point should be gracefully recovered +from, see above. + +bundle-uri PROTOCOL FEATURES +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The client constructs a bundle list from the `=` pairs +provided by the server. These pairs are part of the `bundle.*` namespace +as documented in linkgit:git-config[1]. In this section, we discuss some +of these keys and describe the actions the client will do in response to +this information. + +In particular, the `bundle.version` key specifies an integer value. The +only accepted value at the moment is `1`, but if the client sees an +unexpected value here then the client MUST ignore the bundle list. + +As long as `bundle.version` is understood, all other unknown keys MAY be +ignored by the client. The server will guarantee compatibility with older +clients, though newer clients may be better able to use the extra keys to +minimize downloads. + +Any backwards-incompatible addition of pre-URI key-value will be +guarded by a new `bundle.version` value or values in 'bundle-uri' +capability advertisement itself, and/or by new future `bundle-uri` +request arguments. + +Some example key-value pairs that are not currently implemented but could +be implemented in the future include: + + * Add a "hash=" or "size=" advertise the expected hash or + size of the bundle file. + + * Advertise that one or more bundle files are the same (to e.g. have + clients round-robin or otherwise choose one of N possible files). + + * A "oid=" shortcut and "prerequisite=" shortcut. For + expressing the common case of a bundle with one tip and no + prerequisites, or one tip and one prerequisite. ++ +This would allow for optimizing the common case of servers who'd like +to provide one "big bundle" containing only their "main" branch, +and/or incremental updates thereof. ++ +A client receiving such a a response MAY assume that they can skip +retrieving the header from a bundle at the indicated URI, and thus +save themselves and the server(s) the request(s) needed to inspect the +headers of that bundle or bundles. + GIT --- Part of the linkgit:git[1] suite diff --git a/bundle-uri.c b/bundle-uri.c index 79a914f961b..32022595964 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -563,6 +563,42 @@ cleanup: return result; } +/** + * API for serve.c. + */ + +int bundle_uri_advertise(struct repository *r, struct strbuf *value) +{ + static int advertise_bundle_uri = -1; + + if (advertise_bundle_uri != -1) + goto cached; + + advertise_bundle_uri = 0; + git_config_get_maybe_bool("uploadpack.advertisebundleuris", &advertise_bundle_uri); + +cached: + return advertise_bundle_uri; +} + +int bundle_uri_command(struct repository *r, + struct packet_reader *request) +{ + struct packet_writer writer; + packet_writer_init(&writer, 1); + + while (packet_reader_read(request) == PACKET_READ_NORMAL) + die(_("bundle-uri: unexpected argument: '%s'"), request->line); + if (request->status != PACKET_READ_FLUSH) + die(_("bundle-uri: expected flush after arguments")); + + /* TODO: Implement the communication */ + + packet_writer_flush(&writer); + + return 0; +} + /** * General API for {transport,connect}.c etc. */ diff --git a/bundle-uri.h b/bundle-uri.h index 4dbc269823c..357111ecce8 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -4,6 +4,7 @@ #include "hashmap.h" #include "strbuf.h" +struct packet_reader; struct repository; struct string_list; @@ -92,6 +93,12 @@ int bundle_uri_parse_config_format(const char *uri, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * API for serve.c. + */ +int bundle_uri_advertise(struct repository *r, struct strbuf *value); +int bundle_uri_command(struct repository *r, struct packet_reader *request); + /** * General API for {transport,connect}.c etc. */ diff --git a/serve.c b/serve.c index 733347f602a..cbf4a143cfe 100644 --- a/serve.c +++ b/serve.c @@ -7,6 +7,7 @@ #include "protocol-caps.h" #include "serve.h" #include "upload-pack.h" +#include "bundle-uri.h" static int advertise_sid = -1; static int client_hash_algo = GIT_HASH_SHA1; @@ -135,6 +136,11 @@ static struct protocol_capability capabilities[] = { .advertise = always_advertise, .command = cap_object_info, }, + { + .name = "bundle-uri", + .advertise = bundle_uri_advertise, + .command = bundle_uri_command, + }, }; void protocol_v2_advertise_capabilities(void) diff --git a/t/t5701-git-serve.sh b/t/t5701-git-serve.sh index 1896f671cb3..f21e5e9d33d 100755 --- a/t/t5701-git-serve.sh +++ b/t/t5701-git-serve.sh @@ -13,7 +13,7 @@ test_expect_success 'test capability advertisement' ' wrong_algo sha1:sha256 wrong_algo sha256:sha1 EOF - cat >expect <<-EOF && + cat >expect.base <<-EOF && version 2 agent=git/$(git version | cut -d" " -f3) ls-refs=unborn @@ -21,8 +21,11 @@ test_expect_success 'test capability advertisement' ' server-option object-format=$(test_oid algo) object-info + EOF + cat >expect.trailer <<-EOF && 0000 EOF + cat expect.base expect.trailer >expect && GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ --advertise-capabilities >out && @@ -342,4 +345,39 @@ test_expect_success 'basics of object-info' ' test_cmp expect actual ' +test_expect_success 'test capability advertisement with uploadpack.advertiseBundleURIs' ' + test_config uploadpack.advertiseBundleURIs true && + + cat >expect.extra <<-EOF && + bundle-uri + EOF + cat expect.base \ + expect.extra \ + expect.trailer >expect && + + GIT_TEST_SIDEBAND_ALL=0 test-tool serve-v2 \ + --advertise-capabilities >out && + test-tool pkt-line unpack actual && + test_cmp expect actual +' + +test_expect_success 'basics of bundle-uri: dies if not enabled' ' + test-tool pkt-line pack >in <<-EOF && + command=bundle-uri + 0000 + EOF + + cat >err.expect <<-\EOF && + fatal: invalid command '"'"'bundle-uri'"'"' + EOF + + cat >expect <<-\EOF && + ERR serve: invalid command '"'"'bundle-uri'"'"' + EOF + + test_must_fail test-tool serve-v2 --stateless-rpc out 2>err.actual && + test_cmp err.expect err.actual && + test_must_be_empty out +' + test_done From patchwork Wed Nov 16 19:51:35 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13045689 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 134AAC4332F for ; Wed, 16 Nov 2022 19:52:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234206AbiKPTv5 (ORCPT ); Wed, 16 Nov 2022 14:51:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55116 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232851AbiKPTvt (ORCPT ); Wed, 16 Nov 2022 14:51:49 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 0CE1F45EE0 for ; Wed, 16 Nov 2022 11:51:48 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id i186-20020a1c3bc3000000b003cfe29a5733so2489119wma.3 for ; Wed, 16 Nov 2022 11:51:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=3UI49Ux3Lhj27XScB9qQIdWE3hSbDUtOzn9Zu6tf32w=; b=hMNgp4cgOgeEk1TpRO68gSj3eanQzuMVbm+xDaDC4OViIfMPkqT43Y7C7yaSfeQigk 6dTvh8Iy7dikdbuFvBRwXwyGaR6qOYEsZJ8rmafpn4f+eFUjRf5zjq9U5Dw02VIXL0Rl B+aIbM6WYDRcWIoKxBKxRd12w5xGpVYYjBcNjy4MjaTCVnlV3C8q4ZPQSKzJH1MxwElD /WI9VWJS5J2BdempI5zpkiVTjduM47S36HFb/TbKPaMwfwf3v2AUZGpNEbtVL+cpAqqX VTtkoUaO8kIrBA6+wDDdJP7uQlVA3fEYxz/qteQbNvqDW/FBJcB9l8NSRG3/kt/AoF74 qblQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3UI49Ux3Lhj27XScB9qQIdWE3hSbDUtOzn9Zu6tf32w=; b=FLzjXSL+U/paMtC+G0tgTaXoyj8EYzmzfPOBna5MpVwo3mJHkZDS2zBgZ5IizDDcVL r+etIB9rn6yAVib9fUnfw/StKQfwPsUp1VpeqZn/bYdafHPAp2t/kKtGLf6A1WN9oEqf XYeWCS6D9fX6Qo/x6vM17WIUasulfpGtzg8dw15ZScWsUES5ivcB6mZMxdnlrMhZ2u2/ Ye6BMW1EPKTKojTZZqzKaYcSpentjb+vyOoPqdxggXwDkAn5CmzsQYI6mt04xKFLEad7 GdTJXdt8PJJZv3iOfk9n8vzIgmH1+woCh9VGZD+9mILsJ3N/TzsWCIKBRwjZ3vlBzQ2r +puA== X-Gm-Message-State: ANoB5pkwHM3PuQdlI4PK2VWWzBMyUtzhTE329XUO7IW4CzWaMwFtOgII F0tf8+nRm7FX4b0BdWWPqKyzO6cHVaM= X-Google-Smtp-Source: AA0mqf7wYr//D3gY7PUK0j3CgRMhlXvsoEv7hpkTp2CR+HuCU0c7Cwl0HCDMsvSe3nc/K3ucS2645g== X-Received: by 2002:a05:600c:3ba8:b0:3cf:6182:a458 with SMTP id n40-20020a05600c3ba800b003cf6182a458mr3200202wms.70.1668628306319; Wed, 16 Nov 2022 11:51:46 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x4-20020a1c7c04000000b003cf7055c014sm2991035wmc.1.2022.11.16.11.51.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Nov 2022 11:51:45 -0800 (PST) Message-Id: <0d85aef965d62a684b25f388a946abd3c17809fc.1668628303.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 16 Nov 2022 19:51:35 +0000 Subject: [PATCH v2 2/9] bundle-uri client: add minimal NOOP client MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Set up all the needed client parts of the "bundle-uri" protocol extension, without actually doing anything with the bundle URIs. I.e. if the server says it supports "bundle-uri" we'll issue a command=bundle-uri after command=ls-refs when we're cloning. We'll parse the returned output using the code already tested for in t5750-bundle-uri-parse.sh. What we aren't doing is actually acting on that data, i.e. downloading the bundle(s) before we get to doing the command=fetch, and adjusting our negotiation dialog appropriately. I'll do that in subsequent commits. There's a question of what level of encapsulation we should use here, I've opted to use connect.h in clone.c, but we could also e.g. make transport_get_remote_refs() invoke this, i.e. make it implicitly get the bundle-uri list for later steps. This approach means that we don't "support" this in "git fetch" for now. I'm starting with the case of initial clones, although as noted in preceding commits to the protocol documentation nothing about this approach precludes getting bundles on incremental fetches. For the t5732-protocol-v2-bundle-uri-http.sh it's not easy to set environment variables for git-upload-pack (it's started by Apache), so let's skip the test under T5730_HTTP, and add unused T5730_{FILE,GIT} prerequisites for consistency and future use. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- builtin/clone.c | 7 ++ bundle-uri.c | 4 + connect.c | 47 ++++++++ remote.h | 5 + t/lib-t5730-protocol-v2-bundle-uri.sh | 148 +++++++++++++++++++++++++ t/t5730-protocol-v2-bundle-uri-file.sh | 36 ++++++ t/t5731-protocol-v2-bundle-uri-git.sh | 17 +++ t/t5732-protocol-v2-bundle-uri-http.sh | 17 +++ transport-helper.c | 13 +++ transport-internal.h | 7 ++ transport.c | 55 +++++++++ transport.h | 19 ++++ 12 files changed, 375 insertions(+) create mode 100644 t/lib-t5730-protocol-v2-bundle-uri.sh create mode 100755 t/t5730-protocol-v2-bundle-uri-file.sh create mode 100755 t/t5731-protocol-v2-bundle-uri-git.sh create mode 100755 t/t5732-protocol-v2-bundle-uri-http.sh diff --git a/builtin/clone.c b/builtin/clone.c index 547d6464b3c..edf98295af2 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -27,6 +27,7 @@ #include "iterator.h" #include "sigchain.h" #include "branch.h" +#include "connect.h" #include "remote.h" #include "run-command.h" #include "connected.h" @@ -1266,6 +1267,12 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport); + if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/bundle-uri.c b/bundle-uri.c index 32022595964..2201b604b11 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -571,6 +571,10 @@ int bundle_uri_advertise(struct repository *r, struct strbuf *value) { static int advertise_bundle_uri = -1; + if (value && + git_env_bool("GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE", 0)) + strbuf_addstr(value, "test-unknown-capability-value"); + if (advertise_bundle_uri != -1) goto cached; diff --git a/connect.c b/connect.c index 5ea53deda23..d39effb7492 100644 --- a/connect.c +++ b/connect.c @@ -15,6 +15,7 @@ #include "version.h" #include "protocol.h" #include "alias.h" +#include "bundle-uri.h" static char *server_capabilities_v1; static struct strvec server_capabilities_v2 = STRVEC_INIT; @@ -491,6 +492,52 @@ static void send_capabilities(int fd_out, struct packet_reader *reader) } } +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct bundle_list *bundles, int stateless_rpc) +{ + int line_nr = 1; + + /* Assert bundle-uri support */ + server_supports_v2("bundle-uri", 1); + + /* (Re-)send capabilities */ + send_capabilities(fd_out, reader); + + /* Send command */ + packet_write_fmt(fd_out, "command=bundle-uri\n"); + packet_delim(fd_out); + + /* Send options */ + if (git_env_bool("GIT_TEST_PROTOCOL_BAD_BUNDLE_URI", 0)) + packet_write_fmt(fd_out, "test-bad-client\n"); + packet_flush(fd_out); + + /* Process response from server */ + while (packet_reader_read(reader) == PACKET_READ_NORMAL) { + const char *line = reader->line; + line_nr++; + + if (!bundle_uri_parse_line(bundles, line)) + continue; + + return error(_("error on bundle-uri response line %d: %s"), + line_nr, line); + } + + if (reader->status != PACKET_READ_FLUSH) + return error(_("expected flush after bundle-uri listing")); + + /* + * Might die(), but obscure enough that that's OK, e.g. in + * serve.c we'll call BUG() on its equivalent (the + * PACKET_READ_RESPONSE_END check). + */ + check_stateless_delimiter(stateless_rpc, reader, + _("expected response end packet after ref listing")); + + return 0; +} + struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, struct ref **list, int for_push, struct transport_ls_refs_options *transport_options, diff --git a/remote.h b/remote.h index 1c4621b414b..1ebbe42792e 100644 --- a/remote.h +++ b/remote.h @@ -234,6 +234,11 @@ struct ref **get_remote_refs(int fd_out, struct packet_reader *reader, const struct string_list *server_options, int stateless_rpc); +/* Used for protocol v2 in order to retrieve refs from a remote */ +struct bundle_list; +int get_remote_bundle_uri(int fd_out, struct packet_reader *reader, + struct bundle_list *bundles, int stateless_rpc); + int resolve_remote_symref(struct ref *ref, struct ref *list); /* diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh new file mode 100644 index 00000000000..27294e9c976 --- /dev/null +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh @@ -0,0 +1,148 @@ +# Included from t573*-protocol-v2-bundle-uri-*.sh + +T5730_PARENT= +T5730_URI= +T5730_BUNDLE_URI= +case "$T5730_PROTOCOL" in +file) + T5730_PARENT=file_parent + T5730_URI="file://$PWD/file_parent" + T5730_BUNDLE_URI="$T5730_URI/fake.bdl" + test_set_prereq T5730_FILE + ;; +git) + . "$TEST_DIRECTORY"/lib-git-daemon.sh + start_git_daemon --export-all --enable=receive-pack + T5730_PARENT="$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent" + T5730_URI="$GIT_DAEMON_URL/parent" + T5730_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq T5730_GIT + ;; +http) + . "$TEST_DIRECTORY"/lib-httpd.sh + start_httpd + T5730_PARENT="$HTTPD_DOCUMENT_ROOT_PATH/http_parent" + T5730_URI="$HTTPD_URL/smart/http_parent" + T5730_BUNDLE_URI="https://example.com/fake.bdl" + test_set_prereq T5730_HTTP + ;; +*) + BUG "Need to pass valid T5730_PROTOCOL (was $T5730_PROTOCOL)" + ;; +esac + +test_expect_success "setup protocol v2 $T5730_PROTOCOL:// tests" ' + git init "$T5730_PARENT" && + test_commit -C "$T5730_PARENT" one && + git -C "$T5730_PARENT" config uploadpack.advertiseBundleURIs true +' + +# Poor man's URI escaping. Good enough for the test suite whose trash +# directory has a space in it. See 93c3fcbe4d4 (git-svn: attempt to +# mimic SVN 1.7 URL canonicalization, 2012-07-28) for prior art. +test_uri_escape() { + sed 's/ /%20/g' +} + +case "$T5730_PROTOCOL" in +http) + test_expect_success "setup config for $T5730_PROTOCOL:// tests" ' + git -C "$T5730_PARENT" config http.receivepack true + ' + ;; +*) + ;; +esac +T5730_BUNDLE_URI_ESCAPED=$(echo "$T5730_BUNDLE_URI" | test_uri_escape) + +test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: no bundle-uri" ' + test_when_finished "rm -f log" && + test_when_finished "git -C \"$T5730_PARENT\" config uploadpack.advertiseBundleURIs true" && + git -C "$T5730_PARENT" config uploadpack.advertiseBundleURIs false && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + ! grep bundle-uri log +' + +test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: have bundle-uri" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + uploadpack.bundleURI "$T5730_BUNDLE_URI_ESCAPED" && + + GIT_TRACE_PACKET="$PWD/log" \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + # Server advertised bundle-uri capability + grep bundle-uri log +' + +test_expect_success !T5730_HTTP "bad client with $T5730_PROTOCOL:// using protocol v2" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" uploadpack.bundleURI \ + "$T5730_BUNDLE_URI_ESCAPED" && + + cat >err.expect <<-\EOF && + Cloning into '"'"'child'"'"'... + EOF + case "$T5730_PROTOCOL" in + file) + cat >fatal-bundle-uri.expect <<-\EOF + fatal: bundle-uri: unexpected argument: '"'"'test-bad-client'"'"' + EOF + ;; + *) + cat >fatal.expect <<-\EOF + fatal: read error: Connection reset by peer + EOF + ;; + esac && + + test_when_finished "rm -rf child" && + test_must_fail ok=sigpipe env \ + GIT_TRACE_PACKET="$PWD/log" \ + GIT_TEST_PROTOCOL_BAD_BUNDLE_URI=true \ + git -c protocol.version=2 \ + clone "$T5730_URI" child \ + >out 2>err && + test_must_be_empty out && + + grep -v -e ^fatal: -e ^error: err >err.actual && + test_cmp err.expect err.actual && + + case "$T5730_PROTOCOL" in + file) + # Due to general race conditions with client/server replies we + # may or may not get "fatal: the remote end hung up + # expectedly" here + grep "^fatal: bundle-uri:" err >fatal-bundle-uri.actual && + test_cmp fatal-bundle-uri.expect fatal-bundle-uri.actual + ;; + *) + grep "^fatal:" err >fatal.actual && + # Due to the same race conditions this might be + # "fatal: read error: Connection reset by peer", "fatal: the remote end + # hung up unexpectedly" etc. + cat fatal.actual && + test_file_not_empty fatal.actual + ;; + esac && + + grep "clone> test-bad-client$" log >sent-bad-request && + test_file_not_empty sent-bad-request +' diff --git a/t/t5730-protocol-v2-bundle-uri-file.sh b/t/t5730-protocol-v2-bundle-uri-file.sh new file mode 100755 index 00000000000..89203d3a23c --- /dev/null +++ b/t/t5730-protocol-v2-bundle-uri-file.sh @@ -0,0 +1,36 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'file://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'file://' transport +# +T5730_PROTOCOL=file +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_expect_success "unknown capability value with $T5730_PROTOCOL:// using protocol v2" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + uploadpack.bundleURI "$T5730_BUNDLE_URI_ESCAPED" && + + GIT_TRACE_PACKET="$PWD/log" \ + GIT_TEST_BUNDLE_URI_UNKNOWN_CAPABILITY_VALUE=true \ + git \ + -c protocol.version=2 \ + ls-remote --symref "$T5730_URI" \ + >actual 2>err && + + # Server responded using protocol v2 + grep "< version 2" log && + + grep "> bundle-uri=test-unknown-capability-value" log +' + +test_done diff --git a/t/t5731-protocol-v2-bundle-uri-git.sh b/t/t5731-protocol-v2-bundle-uri-git.sh new file mode 100755 index 00000000000..282847b311f --- /dev/null +++ b/t/t5731-protocol-v2-bundle-uri-git.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +T5730_PROTOCOL=git +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_done diff --git a/t/t5732-protocol-v2-bundle-uri-http.sh b/t/t5732-protocol-v2-bundle-uri-http.sh new file mode 100755 index 00000000000..fcc1cf3faef --- /dev/null +++ b/t/t5732-protocol-v2-bundle-uri-http.sh @@ -0,0 +1,17 @@ +#!/bin/sh + +test_description="Test bundle-uri with protocol v2 and 'git://' transport" + +TEST_NO_CREATE_REPO=1 + +GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main +export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME + +. ./test-lib.sh + +# Test protocol v2 with 'git://' transport +# +T5730_PROTOCOL=http +. "$TEST_DIRECTORY"/lib-t5730-protocol-v2-bundle-uri.sh + +test_done diff --git a/transport-helper.c b/transport-helper.c index e95267a4ab5..3ea7c2bb5ad 100644 --- a/transport-helper.c +++ b/transport-helper.c @@ -1267,9 +1267,22 @@ static struct ref *get_refs_list_using_list(struct transport *transport, return ret; } +static int get_bundle_uri(struct transport *transport) +{ + get_helper(transport); + + if (process_connect(transport, 0)) { + do_take_over(transport); + return transport->vtable->get_bundle_uri(transport); + } + + return -1; +} + static struct transport_vtable vtable = { .set_option = set_helper_option, .get_refs_list = get_refs_list, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs, .push_refs = push_refs, .connect = connect_helper, diff --git a/transport-internal.h b/transport-internal.h index c4ca0b733ac..90ea749e5cf 100644 --- a/transport-internal.h +++ b/transport-internal.h @@ -26,6 +26,13 @@ struct transport_vtable { struct ref *(*get_refs_list)(struct transport *transport, int for_push, struct transport_ls_refs_options *transport_options); + /** + * Populates the remote side's bundle-uri under protocol v2, + * if the "bundle-uri" capability was advertised. Returns 0 if + * OK, negative values on error. + */ + int (*get_bundle_uri)(struct transport *transport); + /** * Fetch the objects for the given refs. Note that this gets * an array, and should ignore the list structure. diff --git a/transport.c b/transport.c index e7b97194c10..a020adc1f56 100644 --- a/transport.c +++ b/transport.c @@ -22,6 +22,7 @@ #include "protocol.h" #include "object-store.h" #include "color.h" +#include "bundle-uri.h" static int transport_use_color = -1; static char transport_colors[][COLOR_MAXLEN] = { @@ -359,6 +360,25 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus return handshake(transport, for_push, options, 1); } +static int get_bundle_uri(struct transport *transport) +{ + struct git_transport_data *data = transport->data; + struct packet_reader reader; + int stateless_rpc = transport->stateless_rpc; + + if (!transport->bundles) { + CALLOC_ARRAY(transport->bundles, 1); + init_bundle_list(transport->bundles); + } + + packet_reader_init(&reader, data->fd[0], NULL, 0, + PACKET_READ_CHOMP_NEWLINE | + PACKET_READ_GENTLE_ON_EOF); + + return get_remote_bundle_uri(data->fd[1], &reader, + transport->bundles, stateless_rpc); +} + static int fetch_refs_via_pack(struct transport *transport, int nr_heads, struct ref **to_fetch) { @@ -902,6 +922,7 @@ static int disconnect_git(struct transport *transport) static struct transport_vtable taken_over_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .disconnect = disconnect_git @@ -1054,6 +1075,7 @@ static struct transport_vtable bundle_vtable = { static struct transport_vtable builtin_smart_vtable = { .get_refs_list = get_refs_via_connect, + .get_bundle_uri = get_bundle_uri, .fetch_refs = fetch_refs_via_pack, .push_refs = git_transport_push, .connect = connect_git, @@ -1068,6 +1090,9 @@ struct transport *transport_get(struct remote *remote, const char *url) ret->progress = isatty(2); string_list_init_dup(&ret->pack_lockfiles); + CALLOC_ARRAY(ret->bundles, 1); + init_bundle_list(ret->bundles); + if (!remote) BUG("No remote provided to transport_get()"); @@ -1482,6 +1507,34 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) return rc; } +int transport_get_remote_bundle_uri(struct transport *transport) +{ + const struct transport_vtable *vtable = transport->vtable; + + /* Check config only once. */ + if (transport->got_remote_bundle_uri++) + return 0; + + /* + * "Support" protocol v0 and v2 without bundle-uri support by + * silently degrading to a NOOP. + */ + if (!server_supports_v2("bundle-uri", 0)) + return 0; + + /* + * This is intentionally below the transport.injectBundleURI, + * we want to be able to inject into protocol v0, or into the + * dialog of a server who doesn't support this. + */ + if (!vtable->get_bundle_uri) + return error(_("bundle-uri operation not supported by protocol")); + + if (vtable->get_bundle_uri(transport) < 0) + return error(_("could not retrieve server-advertised bundle-uri list")); + return 0; +} + void transport_unlock_pack(struct transport *transport, unsigned int flags) { int in_signal_handler = !!(flags & TRANSPORT_UNLOCK_PACK_IN_SIGNAL_HANDLER); @@ -1512,6 +1565,8 @@ int transport_disconnect(struct transport *transport) ret = transport->vtable->disconnect(transport); if (transport->got_remote_refs) free_refs((void *)transport->remote_refs); + clear_bundle_list(transport->bundles); + free(transport->bundles); free(transport); return ret; } diff --git a/transport.h b/transport.h index b5bf7b3e704..85150f504fb 100644 --- a/transport.h +++ b/transport.h @@ -62,6 +62,7 @@ enum transport_family { TRANSPORT_FAMILY_IPV6 }; +struct bundle_list; struct transport { const struct transport_vtable *vtable; @@ -76,6 +77,18 @@ struct transport { */ unsigned got_remote_refs : 1; + /** + * Indicates whether we already called get_bundle_uri_list(); set by + * transport.c::transport_get_remote_bundle_uri(). + */ + unsigned got_remote_bundle_uri : 1; + + /* + * The results of "command=bundle-uri", if both sides support + * the "bundle-uri" capability. + */ + struct bundle_list *bundles; + /* * Transports that call take-over destroys the data specific to * the transport type while doing so, and cannot be reused. @@ -281,6 +294,12 @@ void transport_ls_refs_options_release(struct transport_ls_refs_options *opts); const struct ref *transport_get_remote_refs(struct transport *transport, struct transport_ls_refs_options *transport_options); +/** + * Retrieve bundle URI(s) from a remote. Populates "struct + * transport"'s "bundle_uri" and "got_remote_bundle_uri". + */ +int transport_get_remote_bundle_uri(struct transport *transport); + /* * Fetch the hash algorithm used by a remote. * From patchwork Wed Nov 16 19:51:36 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13045688 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E79EC433FE for ; Wed, 16 Nov 2022 19:52:10 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234226AbiKPTwC (ORCPT ); Wed, 16 Nov 2022 14:52:02 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234094AbiKPTvu (ORCPT ); Wed, 16 Nov 2022 14:51:50 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id F260F43AF8 for ; Wed, 16 Nov 2022 11:51:48 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id w14so31697842wru.8 for ; Wed, 16 Nov 2022 11:51:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=MUdRbpUXZ/e0h09r4Q7akeFJHmNHo+hT1vq/WO9qf70=; b=LmgsYgKfaEk/VUV/FcXHJTska3SybDmUwVUE5Qg/dU2r0+COxtZjaWzQ9t7S39bp8M D6NYkoNzag7B2XLw1Xg4WFiVePVsgaMRpS5AyTNdc2pbwzw6R1qn4C4DbECjANw7Q38X 6ZiAKhnPY8Zo0A88RQB6UXQqC01fyhFv6PjrSN0o7vVfA7w8wdaQTfK4/fS8YuroS/1M RmlTdf9sFuP8SHfZtvQKabeNiS245699rMB2cHmpyCCKiAtwnH7zU9HBNkRXy51Pdgwe jcTeGHnlf6MvIL6r43LHI+SaaTRWPx9Z03sJT88ZsmFRKQxcNC3W7gd/jCH1Ki8giF9G 1Dow== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=MUdRbpUXZ/e0h09r4Q7akeFJHmNHo+hT1vq/WO9qf70=; b=XoSvR0sd98zLASl0QrIKVBgefCthNxPk+SIYwdncfDLTHuMuxnsDz7/blMSHuXhk1V m+SuUj2tkTB9EqIbLbqhZPDbPAamLOjjklwfMd1oQm6DWB+xJslDRtcgPnnX7JwTBhFj j6aFYZrEMB3vAcHbvp8rFQ4glQlflKn3Wx23VvZ01bv2nncP+JGH+LcRU1i1xiu8eRYO Wsez8NbcK2DK+jlPs3PfaI1Lc/LeRexoY38fT8uUfMJufEMkHWKh28R2AIWJYaALR6CG JS/XbZrptYLLWaF3oEpzzwX1mrXOnpUqG1k7NqOnZUy5/dFcHm/Oa35yt3PRwVsV1Scu MeDg== X-Gm-Message-State: ANoB5pkf8ScwxHKnaNl0o0ywK5A+uIM+fiBmv4pzdQM4Uo7UXbW+JFlX 0v0KFts3DeEJ+AQCAEpG1RqJ7ikQctI= X-Google-Smtp-Source: AA0mqf43NYOCm86ZpjFjOa/pR2huvMosCenC3TRYTSPExg55MLlVZpM+c5zrmaTD8u73MWuVMJ3ywQ== X-Received: by 2002:adf:e2d1:0:b0:236:8638:121a with SMTP id d17-20020adfe2d1000000b002368638121amr13918513wrj.188.1668628307225; Wed, 16 Nov 2022 11:51:47 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id o4-20020a1c7504000000b003c64c186206sm2879087wmc.16.2022.11.16.11.51.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Nov 2022 11:51:46 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 16 Nov 2022 19:51:36 +0000 Subject: [PATCH v2 3/9] bundle-uri client: add helper for testing server MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= Add a 'test-tool bundle-uri ls-remote' command. This is a thin wrapper for issuing protocol v2 "bundle-uri" commands to a server, and to the parsing routines in bundle-uri.c. Since in the "git clone" case we'll have already done the handshake(), but not here, introduce a "got_advertisement" state along with "got_remote_heads". It seems to me that the "got_remote_heads" is badly named in the first place, and the whole logic of eagerly getting ls-refs on handshake() or not could be refactored somewhat, but let's not do that now, and instead just add another self-documenting state variable. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- builtin/clone.c | 2 +- t/helper/test-bundle-uri.c | 46 +++++++++++++++++++ t/lib-t5730-protocol-v2-bundle-uri.sh | 63 ++++++++++++++++++++++----- transport.c | 43 ++++++++++++++---- transport.h | 6 ++- 5 files changed, 139 insertions(+), 21 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index edf98295af2..22b1e506452 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1271,7 +1271,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix) * Populate transport->got_remote_bundle_uri and * transport->bundle_uri. We might get nothing. */ - transport_get_remote_bundle_uri(transport); + transport_get_remote_bundle_uri(transport, 1); if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index 25afd393428..ffb975b7b4f 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -3,6 +3,10 @@ #include "bundle-uri.h" #include "strbuf.h" #include "string-list.h" +#include "transport.h" +#include "ref-filter.h" +#include "remote.h" +#include "refs.h" enum input_mode { KEY_VALUE_PAIRS, @@ -68,6 +72,46 @@ usage: usage_with_options(usage, options); } +static int cmd_ls_remote(int argc, const char **argv) +{ + const char *uploadpack = NULL; + struct string_list server_options = STRING_LIST_INIT_DUP; + const char *dest; + struct remote *remote; + struct transport *transport; + int status = 0; + + dest = argc > 1 ? argv[1] : NULL; + + remote = remote_get(dest); + if (!remote) { + if (dest) + die(_("bad repository '%s'"), dest); + die(_("no remote configured to get bundle URIs from")); + } + if (!remote->url_nr) + die(_("remote '%s' has no configured URL"), dest); + + transport = transport_get(remote, NULL); + if (uploadpack) + transport_set_option(transport, TRANS_OPT_UPLOADPACK, uploadpack); + if (server_options.nr) + transport->server_options = &server_options; + + if (transport_get_remote_bundle_uri(transport, 0) < 0) { + error(_("could not get the bundle-uri list")); + status = 1; + goto cleanup; + } + + print_bundle_list(stdout, transport->bundles); + +cleanup: + if (transport_disconnect(transport)) + return 1; + return status; +} + int cmd__bundle_uri(int argc, const char **argv) { const char *usage[] = { @@ -88,6 +132,8 @@ int cmd__bundle_uri(int argc, const char **argv) return cmd__bundle_uri_parse(argc - 1, argv + 1, KEY_VALUE_PAIRS); if (!strcmp(argv[1], "parse-config")) return cmd__bundle_uri_parse(argc - 1, argv + 1, CONFIG_FILE); + if (!strcmp(argv[1], "ls-remote")) + return cmd_ls_remote(argc - 1, argv + 1); error("there is no test-tool bundle-uri tool '%s'", argv[1]); usage: diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh index 27294e9c976..c327544641b 100644 --- a/t/lib-t5730-protocol-v2-bundle-uri.sh +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh @@ -34,7 +34,9 @@ esac test_expect_success "setup protocol v2 $T5730_PROTOCOL:// tests" ' git init "$T5730_PARENT" && test_commit -C "$T5730_PARENT" one && - git -C "$T5730_PARENT" config uploadpack.advertiseBundleURIs true + git -C "$T5730_PARENT" config uploadpack.advertiseBundleURIs true && + git -C "$T5730_PARENT" config bundle.version 1 && + git -C "$T5730_PARENT" config bundle.mode all ' # Poor man's URI escaping. Good enough for the test suite whose trash @@ -61,9 +63,8 @@ test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: no bundl git -C "$T5730_PARENT" config uploadpack.advertiseBundleURIs false && GIT_TRACE_PACKET="$PWD/log" \ - git \ - -c protocol.version=2 \ - ls-remote --symref "$T5730_URI" \ + test-tool bundle-uri \ + ls-remote "$T5730_URI" \ >actual 2>err && # Server responded using protocol v2 @@ -76,12 +77,11 @@ test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: have bun test_when_finished "rm -f log" && test_config -C "$T5730_PARENT" \ - uploadpack.bundleURI "$T5730_BUNDLE_URI_ESCAPED" && + bundle.only.uri "$T5730_BUNDLE_URI_ESCAPED" && GIT_TRACE_PACKET="$PWD/log" \ - git \ - -c protocol.version=2 \ - ls-remote --symref "$T5730_URI" \ + test-tool bundle-uri \ + ls-remote "$T5730_URI" \ >actual 2>err && # Server responded using protocol v2 @@ -94,8 +94,8 @@ test_expect_success "connect with $T5730_PROTOCOL:// using protocol v2: have bun test_expect_success !T5730_HTTP "bad client with $T5730_PROTOCOL:// using protocol v2" ' test_when_finished "rm -f log" && - test_config -C "$T5730_PARENT" uploadpack.bundleURI \ - "$T5730_BUNDLE_URI_ESCAPED" && + test_config -C "$T5730_PARENT" \ + bundle.only.uri "$T5730_BUNDLE_URI_ESCAPED" && cat >err.expect <<-\EOF && Cloning into '"'"'child'"'"'... @@ -146,3 +146,46 @@ test_expect_success !T5730_HTTP "bad client with $T5730_PROTOCOL:// using protoc grep "clone> test-bad-client$" log >sent-bad-request && test_file_not_empty sent-bad-request ' + +test_expect_success "ls-remote with $T5730_PROTOCOL:// using protocol v2" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + bundle.only.uri "$T5730_BUNDLE_URI_ESCAPED" && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + EOF + GIT_TRACE_PACKET="$PWD/log" \ + test-tool bundle-uri \ + ls-remote \ + "$T5730_URI" \ + >actual && + test_cmp_config_output expect actual +' + +test_expect_success "ls-remote with $T5730_PROTOCOL:// using protocol v2 and extra data" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + bundle.only.uri "$T5730_BUNDLE_URI_ESCAPED" && + + # Extra data should be ignored + test_config -C "$T5730_PARENT" bundle.only.extra bogus && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + EOF + GIT_TRACE_PACKET="$PWD/log" \ + test-tool bundle-uri \ + ls-remote \ + "$T5730_URI" \ + >actual && + test_cmp_config_output expect actual +' diff --git a/transport.c b/transport.c index a020adc1f56..86460f5be28 100644 --- a/transport.c +++ b/transport.c @@ -198,6 +198,7 @@ struct git_transport_data { struct git_transport_options options; struct child_process *conn; int fd[2]; + unsigned got_advertisement : 1; unsigned got_remote_heads : 1; enum protocol_version version; struct oid_array extra_have; @@ -346,6 +347,7 @@ static struct ref *handshake(struct transport *transport, int for_push, BUG("unknown protocol version"); } data->got_remote_heads = 1; + data->got_advertisement = 1; transport->hash_algo = reader.hash_algo; if (reader.line_peeked) @@ -371,6 +373,33 @@ static int get_bundle_uri(struct transport *transport) init_bundle_list(transport->bundles); } + if (!data->got_advertisement) { + struct ref *refs; + struct git_transport_data *data = transport->data; + enum protocol_version version; + + refs = handshake(transport, 0, NULL, 0); + version = data->version; + + switch (version) { + case protocol_v2: + assert(!refs); + break; + case protocol_v0: + case protocol_v1: + case protocol_unknown_version: + assert(refs); + break; + } + } + + /* + * "Support" protocol v0 and v2 without bundle-uri support by + * silently degrading to a NOOP. + */ + if (!server_supports_v2("bundle-uri", 0)) + return 0; + packet_reader_init(&reader, data->fd[0], NULL, 0, PACKET_READ_CHOMP_NEWLINE | PACKET_READ_GENTLE_ON_EOF); @@ -1507,7 +1536,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) return rc; } -int transport_get_remote_bundle_uri(struct transport *transport) +int transport_get_remote_bundle_uri(struct transport *transport, int quiet) { const struct transport_vtable *vtable = transport->vtable; @@ -1515,20 +1544,16 @@ int transport_get_remote_bundle_uri(struct transport *transport) if (transport->got_remote_bundle_uri++) return 0; - /* - * "Support" protocol v0 and v2 without bundle-uri support by - * silently degrading to a NOOP. - */ - if (!server_supports_v2("bundle-uri", 0)) - return 0; - /* * This is intentionally below the transport.injectBundleURI, * we want to be able to inject into protocol v0, or into the * dialog of a server who doesn't support this. */ - if (!vtable->get_bundle_uri) + if (!vtable->get_bundle_uri) { + if (quiet) + return -1; return error(_("bundle-uri operation not supported by protocol")); + } if (vtable->get_bundle_uri(transport) < 0) return error(_("could not retrieve server-advertised bundle-uri list")); diff --git a/transport.h b/transport.h index 85150f504fb..dd0115b83bf 100644 --- a/transport.h +++ b/transport.h @@ -297,8 +297,12 @@ const struct ref *transport_get_remote_refs(struct transport *transport, /** * Retrieve bundle URI(s) from a remote. Populates "struct * transport"'s "bundle_uri" and "got_remote_bundle_uri". + * + * With `quiet=1` it will not complain if the serve doesn't support + * the protocol, but only if we discover the server uses it, and + * encounter issues then. */ -int transport_get_remote_bundle_uri(struct transport *transport); +int transport_get_remote_bundle_uri(struct transport *transport, int quiet); /* * Fetch the hash algorithm used by a remote. From patchwork Wed Nov 16 19:51:37 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13045694 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 64310C4332F for ; Wed, 16 Nov 2022 19:52:21 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234583AbiKPTwU (ORCPT ); Wed, 16 Nov 2022 14:52:20 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55124 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234095AbiKPTvu (ORCPT ); Wed, 16 Nov 2022 14:51:50 -0500 Received: from mail-wr1-x429.google.com (mail-wr1-x429.google.com [IPv6:2a00:1450:4864:20::429]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9E08E4875A for ; Wed, 16 Nov 2022 11:51:49 -0800 (PST) Received: by mail-wr1-x429.google.com with SMTP id h9so31746260wrt.0 for ; Wed, 16 Nov 2022 11:51:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=tC4MklC4EotCdkubgLTySzDpazds/cBLtZSjsLhluuw=; b=oEAqBQ3Gan0hPFcwrCrhmjCw0tuB/KiyK9/zv/lg0ypY8BNLsJm/DyULwKmBzspTzj 9P/9o15ZbW4HWdZyinYANJ4iGx/0hdsYaksa0r4hXvA2yfMH+duqQUVhS5h8lq6cIZCa bdnb2B9GN1Smcw1bIJ4ygt23GRYEemtL5lCYaA4kgWwTRei8CKIFKs+w39SQCvxPmT9K esKcDQdv1wFrWr93rk60y1DxolgDcU3Zc0x1jCoG0uZOwU8LS71F06kNna5OhZJ0iCs4 9VQAHt2UWIujkRHRuK0p7EQi3pCpBfHjl3QfTI59/Yj5bT0ne3Ed+O+u3CC1aokO+pPu PIFg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tC4MklC4EotCdkubgLTySzDpazds/cBLtZSjsLhluuw=; b=mm9LTxIh9k5vwQYTA8vtiZhuSePdm3p8ds0utVxYI789Fy8RI7wzY2PDFqVrlQGNKG Wc2BjiUXXs1R5VCvDyurRxW2tFoh1foIBaXMpnSov2iMkH11aXF50oAP5lvIqWMLWkId LQzeHOjTSiHxBL/gWgE8tJCvF23I1ep03vb6FnBLCtovsbfZ69s+DEuHWtTgI7BnNJUj 7SbqBlwcA86x5fV+SkW2L3zBkz66qR5hp54MC/3PFJ+5o6f/eQ48Uxn4qB05J6PoiJ9g cEnRH+36u2MqsPoSpuZg/bmFh03Z+itVj7JeoZvltgSSE/+Jtd6nbsGkj5A1Hm/Ri5RL lI/A== X-Gm-Message-State: ANoB5plF0YBDT9j3DNqVQzOD07lAaM2CjcD0gfUwZ0HkRDDTMKKT+jcI GKMhlxbb3NnycwxhAfgmxF5WI2Tcv3g= X-Google-Smtp-Source: AA0mqf5KhkbIATYSn1ARzuSkQiTI0cdkUNGD5aa8PqerJh4ZDAsiJrHwnGaMKvwNym5Ed759CJ1T6g== X-Received: by 2002:a5d:5914:0:b0:236:6a5b:5610 with SMTP id v20-20020a5d5914000000b002366a5b5610mr14756797wrd.195.1668628308065; Wed, 16 Nov 2022 11:51:48 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k18-20020a05600c1c9200b003b47b80cec3sm3509053wms.42.2022.11.16.11.51.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Nov 2022 11:51:47 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 16 Nov 2022 19:51:37 +0000 Subject: [PATCH v2 4/9] bundle-uri: serve bundle.* keys from config Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Implement the "bundle-uri" protocol v2 capability by populating the key=value packet lines from the local Git config. The list of bundles is provided from the keys beginning with "bundle.". In the future, we may want to filter this list to be more specific to the exact known keys that the server intends to share, but for flexibility at the moment we will assume that the config values are well-formed. Signed-off-by: Derrick Stolee --- bundle-uri.c | 16 +++++++++++- t/lib-t5730-protocol-v2-bundle-uri.sh | 35 +++++++++++++++++++++++++++ 2 files changed, 50 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 2201b604b11..3469f1aaa98 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -585,6 +585,16 @@ cached: return advertise_bundle_uri; } +static int config_to_packet_line(const char *key, const char *value, void *data) +{ + struct packet_reader *writer = data; + + if (!strncmp(key, "bundle.", 7)) + packet_write_fmt(writer->fd, "%s=%s", key, value); + + return 0; +} + int bundle_uri_command(struct repository *r, struct packet_reader *request) { @@ -596,7 +606,11 @@ int bundle_uri_command(struct repository *r, if (request->status != PACKET_READ_FLUSH) die(_("bundle-uri: expected flush after arguments")); - /* TODO: Implement the communication */ + /* + * Read all "bundle.*" config lines to the client as key=value + * packet lines. + */ + git_config(config_to_packet_line, &writer); packet_writer_flush(&writer); diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh index c327544641b..000fcc5e20b 100644 --- a/t/lib-t5730-protocol-v2-bundle-uri.sh +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh @@ -158,6 +158,8 @@ test_expect_success "ls-remote with $T5730_PROTOCOL:// using protocol v2" ' [bundle] version = 1 mode = all + [bundle "only"] + uri = $T5730_BUNDLE_URI_ESCAPED EOF GIT_TRACE_PACKET="$PWD/log" \ test-tool bundle-uri \ @@ -181,6 +183,39 @@ test_expect_success "ls-remote with $T5730_PROTOCOL:// using protocol v2 and ext [bundle] version = 1 mode = all + [bundle "only"] + uri = $T5730_BUNDLE_URI_ESCAPED + EOF + GIT_TRACE_PACKET="$PWD/log" \ + test-tool bundle-uri \ + ls-remote \ + "$T5730_URI" \ + >actual && + test_cmp_config_output expect actual +' + + +test_expect_success "ls-remote with $T5730_PROTOCOL:// using protocol v2 with list" ' + test_when_finished "rm -f log" && + + test_config -C "$T5730_PARENT" \ + bundle.bundle1.uri "$T5730_BUNDLE_URI_ESCAPED-1.bdl" && + test_config -C "$T5730_PARENT" \ + bundle.bundle2.uri "$T5730_BUNDLE_URI_ESCAPED-2.bdl" && + test_config -C "$T5730_PARENT" \ + bundle.bundle3.uri "$T5730_BUNDLE_URI_ESCAPED-3.bdl" && + + # All data about bundle URIs + cat >expect <<-EOF && + [bundle] + version = 1 + mode = all + [bundle "bundle1"] + uri = $T5730_BUNDLE_URI_ESCAPED-1.bdl + [bundle "bundle2"] + uri = $T5730_BUNDLE_URI_ESCAPED-2.bdl + [bundle "bundle3"] + uri = $T5730_BUNDLE_URI_ESCAPED-3.bdl EOF GIT_TRACE_PACKET="$PWD/log" \ test-tool bundle-uri \ From patchwork Wed Nov 16 19:51:38 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= X-Patchwork-Id: 13045691 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 033BFC433FE for ; Wed, 16 Nov 2022 19:52:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234278AbiKPTwO (ORCPT ); Wed, 16 Nov 2022 14:52:14 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55142 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234118AbiKPTvv (ORCPT ); Wed, 16 Nov 2022 14:51:51 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B54445A02 for ; Wed, 16 Nov 2022 11:51:50 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id c3-20020a1c3503000000b003bd21e3dd7aso2511272wma.1 for ; Wed, 16 Nov 2022 11:51:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=tLooz02+SsovFOZcXwMGlFj9MxU6sRX+bGtaH84Oqg8=; b=NWu4c95hTtpch0AVcBh0BxDnu9geOIHT8h9XzymMBHDxjoOtNWiWmFW3lL8rhA/IkH +TuJTy2CVEGg4y7mCS3ai181t9L7/XwDZEsk60Eitxy82GMjhzJX5K0KI4TgQhpfz5hI OMkvzOrkrtwk6lAtlNez98lvph3JhdpoS8qWamt2YyGYUdGlbkUYwkMm1W+c6/vVdP3q frabDJfcLyR/nqKPI7ONumGeaSEPvCMBp/lzi4wvYDoHaUKY2eWOBDF+Xt1CqxDwvTRk int0po+RgB3FJc9R+lAjlfsVCTriG3iT8vIVmky8XWOCnzEiBYd2KQvaBJp87X+QuYcc /IHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:fcc:content-transfer-encoding:mime-version:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=tLooz02+SsovFOZcXwMGlFj9MxU6sRX+bGtaH84Oqg8=; b=6HdwyFDZgEiCNkcRvcdeZF0JPFRFJ8F6+DM6GcgeV9XHWMuf5fIl8NCGiaV1cux8Yd QzJ7M3VoXnRvaBAyPyTSErjL+lB+VfGj7QKQ1PTBFYjNR9eouMdc+a0yXuASPAoFf5vf YeOsdlHaW30THTvzxpUwPGPtuhAbN6Bl53Rm4ow/+MLzdjQdEPW3CzgZHThmkke9mM/R aJP97uJjmhcYvgSqOobh0Ihd63LVFZhzzmfOyCaHdGDG1lVTpl+AsZ3F3bqjv7Hn0qW0 z0xuLBN1FaPDoVrH2rA1UTr4iBuBX4D0VdGET+ekcPtZfTkdLweg/up/eGfXYNlPY0zy ODNQ== X-Gm-Message-State: ANoB5pmAy8rZuOoHiVq3gnKNlozllu3JKgG+yISSvbdmqC4eZY9rei5I 2ttbJiITd/q+d+8K/eT2KNede/SlAgg= X-Google-Smtp-Source: AA0mqf4PdQi7luWSTnUXxyW0olQrrvvSsISe6LaiYSPtPD/WWDheLcmp/bbfSuydRgEjpxojfRebwA== X-Received: by 2002:a1c:7504:0:b0:3cf:6b10:ca8d with SMTP id o4-20020a1c7504000000b003cf6b10ca8dmr3316695wmc.44.1668628308834; Wed, 16 Nov 2022 11:51:48 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 7-20020a1c1907000000b003cfbe1da539sm3130191wmz.36.2022.11.16.11.51.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Nov 2022 11:51:48 -0800 (PST) Message-Id: <933974689312bbb130236c81550ee3467f295a43.1668628303.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 16 Nov 2022 19:51:38 +0000 Subject: [PATCH v2 5/9] bundle-uri client: add boolean transfer.bundleURI setting MIME-Version: 1.0 Fcc: Sent To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , =?utf-8?b?w4Z2YXIgQXJuZmrDtnI=?= =?utf-8?b?w7AgQmphcm1hc29u?= Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= From: =?UTF-8?q?=C3=86var=20Arnfj=C3=B6r=C3=B0=20Bjarmason?= The yet-to-be introduced client support for bundle-uri will always fall back on a full clone, but we'd still like to be able to ignore a server's bundle-uri advertisement entirely. The new transfer.bundleURI config option defaults to 'false', but a user can set it to 'true' to enable checking for bundle URIs from the origin Git server using protocol v2. To enable this setting by default in the correct tests, add a GIT_TEST_BUNDLE_URI environment variable. Co-authored-by: Derrick Stolee Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Derrick Stolee --- Documentation/config/transfer.txt | 6 ++++++ t/lib-t5730-protocol-v2-bundle-uri.sh | 3 +++ transport.c | 10 +++++++--- 3 files changed, 16 insertions(+), 3 deletions(-) diff --git a/Documentation/config/transfer.txt b/Documentation/config/transfer.txt index 264812cca4d..c3ac767d1e4 100644 --- a/Documentation/config/transfer.txt +++ b/Documentation/config/transfer.txt @@ -115,3 +115,9 @@ transfer.unpackLimit:: transfer.advertiseSID:: Boolean. When true, client and server processes will advertise their unique session IDs to their remote counterpart. Defaults to false. + +transfer.bundleURI:: + When `true`, local `git clone` commands will request bundle + information from the remote server (if advertised) and download + bundles before continuing the clone through the Git protocol. + Defaults to `false`. diff --git a/t/lib-t5730-protocol-v2-bundle-uri.sh b/t/lib-t5730-protocol-v2-bundle-uri.sh index 000fcc5e20b..872bc39ad1b 100644 --- a/t/lib-t5730-protocol-v2-bundle-uri.sh +++ b/t/lib-t5730-protocol-v2-bundle-uri.sh @@ -1,5 +1,8 @@ # Included from t573*-protocol-v2-bundle-uri-*.sh +GIT_TEST_BUNDLE_URI=1 +export GIT_TEST_BUNDLE_URI + T5730_PARENT= T5730_URI= T5730_BUNDLE_URI= diff --git a/transport.c b/transport.c index 86460f5be28..b33180226ae 100644 --- a/transport.c +++ b/transport.c @@ -1538,6 +1538,7 @@ int transport_fetch_refs(struct transport *transport, struct ref *refs) int transport_get_remote_bundle_uri(struct transport *transport, int quiet) { + int value = 0; const struct transport_vtable *vtable = transport->vtable; /* Check config only once. */ @@ -1545,10 +1546,13 @@ int transport_get_remote_bundle_uri(struct transport *transport, int quiet) return 0; /* - * This is intentionally below the transport.injectBundleURI, - * we want to be able to inject into protocol v0, or into the - * dialog of a server who doesn't support this. + * Don't use bundle-uri at all, if configured not to. Only proceed + * if GIT_TEST_BUNDLE_URI=1 or transfer.bundleURI=true. */ + if (!git_env_bool("GIT_TEST_BUNDLE_URI", 0) && + (git_config_get_bool("transfer.bundleuri", &value) || !value)) + return 0; + if (!vtable->get_bundle_uri) { if (quiet) return -1; From patchwork Wed Nov 16 19:51:39 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13045690 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2A38C43217 for ; Wed, 16 Nov 2022 19:52:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234173AbiKPTwK (ORCPT ); Wed, 16 Nov 2022 14:52:10 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229446AbiKPTvv (ORCPT ); Wed, 16 Nov 2022 14:51:51 -0500 Received: from mail-wm1-x330.google.com (mail-wm1-x330.google.com [IPv6:2a00:1450:4864:20::330]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 39CDD4732B for ; Wed, 16 Nov 2022 11:51:51 -0800 (PST) Received: by mail-wm1-x330.google.com with SMTP id o30so12685744wms.2 for ; Wed, 16 Nov 2022 11:51:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=yED6eliENaJBDbhgcKLiK4M2LH9Jm+dGgAJNGYKvQeY=; b=TP3i8LHiBJMcZTVPGp0PPhMEjnl4tkW9u1xrr6pOde+7LQeJReaJpb+pEkDt8G935s 5V+azr5I5HJ9u+kbImQZLXXRwVdilCgS4J0N24h201G0F68Gc/+g4SUKyH5NIsh+K4Pr tK3W2uEsVEa4v8ZJXGmz/3p+c6V+srYtbXQEszMzrqX+WRQ1FxH/SPWvVkND6g0oAJ04 IZsI7qW2PTd5iEl8BUwCONp40rpxizZa8RV9yF5JVMb2sZ76xJBhqdNIg5GMwZUkO/BK eBCTtTnOynWP6KnURad0AJpsS7wm6ke1ORabspMCxe9YfHux4eOK0Ry2Zm0zNxnMgWUT Q71A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=yED6eliENaJBDbhgcKLiK4M2LH9Jm+dGgAJNGYKvQeY=; b=nf7whTPLkdhMQ5ITv8uZ9rD42Xga/PoTli7aXu68iEHpdBQy6ridQwa/KxCZNuXxrb ELv0t/lfg4Rw88oxNGW2zTNTwUnfUlbIInoXJIzm4trPsLnxvlQ6l9D/J8C0gfQuU4yh NU/sTDxlWm55NuLrlEdlP56fIySs7EzXFBadK1X2ITMcaGYo9H6wJASl65PHqhvP2/JO /4FRzrKAOtInOJ94KMUcfNQHuJc8DanoSlDJf1VBXWjDpPyhUVSCZj+nW6z+ZxBl+aBS XIIjIqRBoD8Q2tvvLqWHKzFzOVkj9YYtu35nQZBULWDiGC2e7/BuaWSIJxlwTJlEOO+P pzLQ== X-Gm-Message-State: ANoB5pk1SZB4l2Moxx2i1tPhZfg1FCgpXwUjujgZWvg3TJThsqCDrIvv iLQlU+lDi5+3lDZSHeljMKbCgzVmkm4= X-Google-Smtp-Source: AA0mqf7sQJMftLpUwJpkWjIreCwOACgkYTEBDI5g+nfZw6A2MNnA5aTakfya9z6TLf8XzsKHwJvwcw== X-Received: by 2002:a05:600c:4a2a:b0:3b4:c00d:2329 with SMTP id c42-20020a05600c4a2a00b003b4c00d2329mr3114241wmp.124.1668628309487; Wed, 16 Nov 2022 11:51:49 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id g10-20020a05600c4eca00b003c6b874a0dfsm4316682wmq.14.2022.11.16.11.51.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Nov 2022 11:51:49 -0800 (PST) Message-Id: <7d86852c01513ee131bf993302416f4c881a0bc6.1668628303.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 16 Nov 2022 19:51:39 +0000 Subject: [PATCH v2 6/9] strbuf: introduce strbuf_strip_file_from_path() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The strbuf_parent_directory() method was added as a static method in contrib/scalar by d0feac4e8c0 (scalar: 'register' sets recommended config and starts maintenance, 2021-12-03) and then removed in 65f6a9eb0b9 (scalar: constrain enlistment search, 2022-08-18), but now there is a need for a similar method in the bundle URI feature. Re-add the method, this time in strbuf.c, but with a new name: strbuf_strip_file_from_path(). The method requirements are slightly modified to allow a trailing slash, in which case nothing is done, which makes the name change valuable. The return value is the number of bytes removed. Signed-off-by: Derrick Stolee --- strbuf.c | 9 +++++++++ strbuf.h | 12 ++++++++++++ 2 files changed, 21 insertions(+) diff --git a/strbuf.c b/strbuf.c index 0890b1405c5..8d1e2e8bb61 100644 --- a/strbuf.c +++ b/strbuf.c @@ -1200,3 +1200,12 @@ int strbuf_edit_interactively(struct strbuf *buffer, const char *path, free(path2); return res; } + +size_t strbuf_strip_file_from_path(struct strbuf *buf) +{ + size_t len = buf->len; + size_t offset = offset_1st_component(buf->buf); + char *path_sep = find_last_dir_sep(buf->buf + offset); + strbuf_setlen(buf, path_sep ? path_sep - buf->buf + 1 : offset); + return len - buf->len; +} diff --git a/strbuf.h b/strbuf.h index 76965a17d44..4822b713786 100644 --- a/strbuf.h +++ b/strbuf.h @@ -664,6 +664,18 @@ int launch_sequence_editor(const char *path, struct strbuf *buffer, int strbuf_edit_interactively(struct strbuf *buffer, const char *path, const char *const *env); +/* + * Remove the filename from the provided path string. If the path + * contains a trailing separator, then the path is considered a directory + * and nothing is modified. Returns the number of characters removed from + * the path. + * + * Examples: + * - "/path/to/file" -> "/path/to/" (returns: 4) + * - "/path/to/dir/" -> "/path/to/dir/" (returns: 0) + */ +size_t strbuf_strip_file_from_path(struct strbuf *buf); + void strbuf_add_lines(struct strbuf *sb, const char *prefix, const char *buf, From patchwork Wed Nov 16 19:51:40 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13045693 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 66A7BC433FE for ; Wed, 16 Nov 2022 19:52:20 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234437AbiKPTwT (ORCPT ); Wed, 16 Nov 2022 14:52:19 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55140 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234117AbiKPTvv (ORCPT ); Wed, 16 Nov 2022 14:51:51 -0500 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id E42744876C for ; Wed, 16 Nov 2022 11:51:50 -0800 (PST) Received: by mail-wr1-x435.google.com with SMTP id w14so31698060wru.8 for ; Wed, 16 Nov 2022 11:51:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=+ikiRzeF6zm3KWpzMsAsHvlZOU4f/T44FQ8of3kKb7I=; b=OHkPMBUT0atw3R4+9kCzp9mnZQJqFGT7xjVKbZzT3+AylCsBY/IiZKpAejCl2UQvNS k9Lh6625mr5XJG11fR2aA9BOzq2eyYTPdbobf2H9vBBdRx3V/MAPY5mdQ9XtPAltvAxt 92xWT2dA2HXZRkBU3XgJFh9qgUJm8BXD785JooOaubAbrC01OklT7rXIZPEQ9VmSu5fk HGNqnp/Twyax33Bxw5FxOeGvDdBz1Ot/slOF93oOKTIvWY8brtoCLqk92d5fca4QQSB2 XdD3HLVCYCOBZJReG2bRPSr+IPuR3SbOJngjQzTCsWbvLZQt2mN6LHZezYRc0gvaEv+7 heFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+ikiRzeF6zm3KWpzMsAsHvlZOU4f/T44FQ8of3kKb7I=; b=WZDr7SWGjkdnDkrnpd3F8rceCgdb+YXZ4argE1uNm6xoQJqSEFmBrx+d+gnnbsqfob B1rd5sR8tpzREl6m0OIlFcM9TVFqWqI/8Wj5MOgHIUHw7b95YS2FpxyhT/SNCjvtfATD u950s+5tFfDMKVJOrd19mn4hsoilonxL/8uOauHDh2SmOteRVrYAoeG1RfKtRj8eKMMQ msJPSBAcNWZWdDK2E2edmC6k2AVMmcua8sYhgQMWGUHEmQx+Y8iTpWybz8FgLIDTDUtU olJeS6kEPMFvLRtx6t6sJKV4IUOG+6v8E2tXS78bSUQ/uQZdM7Oslh7hWpmI3T6gKI7u /WyA== X-Gm-Message-State: ANoB5pnnz/EXkyEnuHbl95OED57Ch+DVzXSiW7evfJhE3LMTQD3Dmycw B7Cv6SCXjT5PzfBVtCYaTmm8NSF+adc= X-Google-Smtp-Source: AA0mqf6symV2khvUUofwd3b8qlAqK9XT1cBipvhW7aUVbfBubrTh6yVf4aRqYbalAAk2a+u+rX0ULA== X-Received: by 2002:a5d:58c3:0:b0:238:8896:7876 with SMTP id o3-20020a5d58c3000000b0023888967876mr14840395wrf.645.1668628310362; Wed, 16 Nov 2022 11:51:50 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h12-20020a05600c350c00b003c6bd91caa5sm3442569wmq.17.2022.11.16.11.51.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Nov 2022 11:51:50 -0800 (PST) Message-Id: <186e112d821d9a42ffbc3c8b46e24b2b4bb3dfe8.1668628303.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Wed, 16 Nov 2022 19:51:40 +0000 Subject: [PATCH v2 7/9] bundle-uri: allow relative URLs in bundle lists Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Bundle providers may want to distribute that data across multiple CDNs. This might require a change in the base URI, all the way to the domain name. If all bundles require an absolute URI in their 'uri' value, then every push to a CDN would require altering the table of contents to match the expected domain and exact location within it. Allow a bundle list to specify a relative URI for the bundles. This allows easier distribution of bundle data. Signed-off-by: Derrick Stolee --- bundle-uri.c | 16 ++++++++++- bundle-uri.h | 9 +++++++ t/helper/test-bundle-uri.c | 2 ++ t/t5750-bundle-uri-parse.sh | 54 +++++++++++++++++++++++++++++++++++++ transport.c | 3 +++ 5 files changed, 83 insertions(+), 1 deletion(-) diff --git a/bundle-uri.c b/bundle-uri.c index 3469f1aaa98..ab91bb32e9b 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -7,6 +7,7 @@ #include "hashmap.h" #include "pkt-line.h" #include "config.h" +#include "remote.h" static int compare_bundles(const void *hashmap_cmp_fn_data, const struct hashmap_entry *he1, @@ -49,6 +50,7 @@ void clear_bundle_list(struct bundle_list *list) for_all_bundles_in_list(list, clear_remote_bundle_info, NULL); hashmap_clear_and_free(&list->bundles, struct remote_bundle_info, ent); + free(list->baseURI); } int for_all_bundles_in_list(struct bundle_list *list, @@ -163,7 +165,7 @@ static int bundle_list_update(const char *key, const char *value, if (!strcmp(subkey, "uri")) { if (bundle->uri) return -1; - bundle->uri = xstrdup(value); + bundle->uri = relative_url(list->baseURI, value, NULL); return 0; } @@ -190,6 +192,18 @@ int bundle_uri_parse_config_format(const char *uri, .error_action = CONFIG_ERROR_ERROR, }; + if (!list->baseURI) { + struct strbuf baseURI = STRBUF_INIT; + strbuf_addstr(&baseURI, uri); + + /* + * If the URI does not end with a trailing slash, then + * remove the filename portion of the path. This is + * important for relative URIs. + */ + strbuf_strip_file_from_path(&baseURI); + list->baseURI = strbuf_detach(&baseURI, NULL); + } result = git_config_from_file_with_options(config_to_bundle_list, filename, list, &opts); diff --git a/bundle-uri.h b/bundle-uri.h index 357111ecce8..7905e56732c 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -61,6 +61,15 @@ struct bundle_list { int version; enum bundle_list_mode mode; struct hashmap bundles; + + /** + * The baseURI of a bundle_list is used as the base for any + * relative URIs advertised by the bundle list at that location. + * + * When the list is generated from a Git server, then use that + * server's location. + */ + char *baseURI; }; void init_bundle_list(struct bundle_list *list); diff --git a/t/helper/test-bundle-uri.c b/t/helper/test-bundle-uri.c index ffb975b7b4f..5aa0b494ce3 100644 --- a/t/helper/test-bundle-uri.c +++ b/t/helper/test-bundle-uri.c @@ -40,6 +40,8 @@ static int cmd__bundle_uri_parse(int argc, const char **argv, enum input_mode mo init_bundle_list(&list); + list.baseURI = xstrdup(""); + switch (mode) { case KEY_VALUE_PAIRS: if (argc != 1) diff --git a/t/t5750-bundle-uri-parse.sh b/t/t5750-bundle-uri-parse.sh index c2fe3f9c5a5..ed5262a8d2b 100755 --- a/t/t5750-bundle-uri-parse.sh +++ b/t/t5750-bundle-uri-parse.sh @@ -30,6 +30,30 @@ test_expect_success 'bundle_uri_parse_line() just URIs' ' test_cmp_config_output expect actual ' +test_expect_success 'bundle_uri_parse_line(): relative URIs' ' + cat >in <<-\EOF && + bundle.one.uri=bundle.bdl + bundle.two.uri=../bundle.bdl + bundle.three.uri=sub/dir/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /sub/dir/bundle.bdl + EOF + + test-tool bundle-uri parse-key-values in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + test_expect_success 'bundle_uri_parse_line() parsing edge cases: empty key or value' ' cat >in <<-\EOF && =bogus-value @@ -136,6 +160,36 @@ test_expect_success 'parse config format: just URIs' ' test_cmp_config_output expect actual ' +test_expect_success 'parse config format: relative URIs' ' + cat >in <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = bundle.bdl + [bundle "two"] + uri = ../bundle.bdl + [bundle "three"] + uri = sub/dir/bundle.bdl + EOF + + cat >expect <<-\EOF && + [bundle] + version = 1 + mode = all + [bundle "one"] + uri = /bundle.bdl + [bundle "two"] + uri = bundle.bdl + [bundle "three"] + uri = /sub/dir/bundle.bdl + EOF + + test-tool bundle-uri parse-config in >actual 2>err && + test_must_be_empty err && + test_cmp_config_output expect actual +' + test_expect_success 'parse config format edge cases: empty key or value' ' cat >in1 <<-\EOF && = bogus-value diff --git a/transport.c b/transport.c index b33180226ae..2c4ff0c2023 100644 --- a/transport.c +++ b/transport.c @@ -1553,6 +1553,9 @@ int transport_get_remote_bundle_uri(struct transport *transport, int quiet) (git_config_get_bool("transfer.bundleuri", &value) || !value)) return 0; + if (!transport->bundles->baseURI) + transport->bundles->baseURI = xstrdup(transport->url); + if (!vtable->get_bundle_uri) { if (quiet) return -1; From patchwork Wed Nov 16 19:51:41 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13045692 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4CD2CC4332F for ; Wed, 16 Nov 2022 19:52:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234217AbiKPTwP (ORCPT ); Wed, 16 Nov 2022 14:52:15 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55122 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234129AbiKPTvy (ORCPT ); Wed, 16 Nov 2022 14:51:54 -0500 Received: from mail-wr1-x431.google.com (mail-wr1-x431.google.com [IPv6:2a00:1450:4864:20::431]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id AF2AE56578 for ; Wed, 16 Nov 2022 11:51:52 -0800 (PST) Received: by mail-wr1-x431.google.com with SMTP id v1so31681192wrt.11 for ; Wed, 16 Nov 2022 11:51:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=FX+Qi4khe7Z9E7YyYDM1gBhRQumkHyVsNI42uohCUiQ=; b=maPEH+9Owxya2QFjHdzgfVDq11vPZkIAcNw3ED4XJcIlmwZqqL/csQ0mU/eMubhpcY tETtD/PS2Ua+0KQPFayx7Um6PPP45aH0+FFPgX/wzggFuPkSIywSaFGSLg/ZEGzgBKe9 6ePh1uDJ4ZVFf1SnPH4/n2OHS1Sa2R1S+rj9sWom65FU9T44XjzFky+nHSK61NBNLjPH Eii8JHZhJHg5wgPDwlzA7ucHHrxGFRr5XoHz45yWdBJzaoK/hrgxoV74+DcOYLocvLI6 4FEAdz3onYoIz1J7AS39OHjDCCx29SULiR+zQX0YLcSq6zukatE9VIvvBhG2/rQh33NF zDCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=FX+Qi4khe7Z9E7YyYDM1gBhRQumkHyVsNI42uohCUiQ=; b=e/xAkQ5PAvYTESXAaqWmUpIx1RqlEvEMDDP+mBnMNGDMKWSP6UYMB3HomslMCR13MH G0bkpYkZL5iTdfZZCn2SoHW0Ki0aN9U5uJSETfUx2o6xY3adDQzPgk2QqT60OOcSVR/K o4yvNgjvBB4/duHEd3j856JtjLYjezi4rwBL/p+kt193sc4leGA1YlMAkkUW9qFXj7TR Bb1RPvhar5dwig0jXXdInstKROBrMvv0dkn+zHRjtdWIDXefl3vH6QPsifNoRsgLAu0c 9I7qSMh1iKN26LIf4IZv+XxIkH/MRopjNKZkiBxv28xP0Gqh1KWkjCkcgLPh+2eAtCCY x0fQ== X-Gm-Message-State: ANoB5pl07o8EuSn1CybaO/7u6JibS5wU53pXNxj3zxiK0dnoa9vTd748 StbG3smF+5Tsth5nKruwxSwW0Mf/fjk= X-Google-Smtp-Source: AA0mqf70SiEljeo4JZ/5NQc4MxJmeMhHyVdueB5TVaNIYZW6wxcT0gK3DqYOW0wqvCROjX25S23kyA== X-Received: by 2002:adf:fec8:0:b0:228:d897:228 with SMTP id q8-20020adffec8000000b00228d8970228mr14667200wrs.539.1668628311031; Wed, 16 Nov 2022 11:51:51 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id f23-20020a1c6a17000000b003c65c9a36dfsm2840645wmc.48.2022.11.16.11.51.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Nov 2022 11:51:50 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 16 Nov 2022 19:51:41 +0000 Subject: [PATCH v2 8/9] bundle-uri: download bundles from an advertised list Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The logic in fetch_bundle_uri() is useful for the --bundle-uri option of 'git clone', but is not helpful when the clone operation discovers a list of URIs from the bundle-uri protocol v2 command. To actually download and unbundle the advertised bundles, we need a different mechanism. Create the new fetch_bundle_list() method which is very similar to fetch_bundle_uri() except that it relies on download_bundle_list() instead of fetch_bundle_uri_internal(). The download_bundle_list() method will recursively call fetch_bundle_uri_internal() if any of the advertised URIs serve a bundle list instead of a bundle. This will also follow the bundle.list.mode setting from the input list: "any" will download only one such URI while "all" will download data from all of the URIs. In an identical way to fetch_bundle_uri(), the bundles are unbundled after all of the bundle lists have been expanded and all necessary URIs. Signed-off-by: Derrick Stolee --- bundle-uri.c | 21 +++++++++++++++++++++ bundle-uri.h | 11 +++++++++++ 2 files changed, 32 insertions(+) diff --git a/bundle-uri.c b/bundle-uri.c index ab91bb32e9b..5914d220c43 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -577,6 +577,27 @@ cleanup: return result; } +int fetch_bundle_list(struct repository *r, const char *uri, struct bundle_list *list) +{ + int result; + struct bundle_list global_list; + + init_bundle_list(&global_list); + + /* If a bundle is added to this global list, then it is required. */ + global_list.mode = BUNDLE_MODE_ALL; + + if ((result = download_bundle_list(r, list, &global_list, 0))) + goto cleanup; + + result = unbundle_all_bundles(r, &global_list); + +cleanup: + for_all_bundles_in_list(&global_list, unlink_bundle, NULL); + clear_bundle_list(&global_list); + return result; +} + /** * API for serve.c. */ diff --git a/bundle-uri.h b/bundle-uri.h index 7905e56732c..a75b68d2f5a 100644 --- a/bundle-uri.h +++ b/bundle-uri.h @@ -102,6 +102,17 @@ int bundle_uri_parse_config_format(const char *uri, */ int fetch_bundle_uri(struct repository *r, const char *uri); +/** + * Given a bundle list that was already advertised (likely by the + * bundle-uri protocol v2 verb) at the given uri, fetch and unbundle the + * bundles according to the bundle strategy of that list. + * + * Returns non-zero if no bundle information is found at the given 'uri'. + */ +int fetch_bundle_list(struct repository *r, + const char *uri, + struct bundle_list *list); + /** * API for serve.c. */ From patchwork Wed Nov 16 19:51:42 2022 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 13045695 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C3042C433FE for ; Wed, 16 Nov 2022 19:52:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234691AbiKPTwW (ORCPT ); Wed, 16 Nov 2022 14:52:22 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55216 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S234160AbiKPTvy (ORCPT ); Wed, 16 Nov 2022 14:51:54 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 869185986F for ; Wed, 16 Nov 2022 11:51:53 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id m7-20020a05600c090700b003cf8a105d9eso2262413wmp.5 for ; Wed, 16 Nov 2022 11:51:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=DehlC/Za5CApBqVxDQr2OvqcFiarvPgr2QlY3Yv5sQI=; b=hk9jWIxih9UXPsC50p0E9YpgnOW53TNbgH/5x55oXkrTUok6/GhFipqtIuxw7cWO/P sQdOIZGxJQzeQ6J3f0CD62kX+8uL3tlf/x3FuHx6ThTtPTkWHU/k+bjGTRrRoYhxcpJf 9XLwCieMyPaBUiaO9Y7BQH52TDBspzAsqK9CSDRnKdM1qamiDvaWf+ohKgtIBQsxTUUy fxEjc1JUdDYRxBI7O7BYzG6njaxjUS9Aljc0eNTcKE8bHCkaYRUmqPXjw2Gubq1pS2Wq 7tqdF5mKGEz6YkndmR5nt9p/NCoz5esvkAQheceWhuKBLs+ZxqHVaw1yrrgK85pd7wA0 GTsg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=DehlC/Za5CApBqVxDQr2OvqcFiarvPgr2QlY3Yv5sQI=; b=puEe9eV7tesL2tcNG09OOm8qV6K78KSKG2Yk5XlvcM3OtaVoof/ISCJ4BwDeoAj9NX KsvUSL3OTA2A5CSaL4mY5HYwxYGcv/BAy8g1vaxY/4Buwxcb6e+oHrA1Dz/lZSvp8HSg O9TlgFsWsqOwvGeSwLkMIjciio5aFXjFbpPMMbmKukJjOK+x/Kb5u99I+MVXVXux2h8f R0LXPJ9vEmXyKrJfHmDzAzkIteapj9euT0D3R2tK4L3LaNQSjMoL2EQE/mQlh7puEnTU 0g8e0SyJm/SPtCITc7c0d7IbgUgb4jBebn55F6Vd2MzZ0aoIPXTASc5vfp+kC5NcASoR JNug== X-Gm-Message-State: ANoB5pky8DmJoRqW6wpwtxyEQo4drONTqPVAqJCj9vPuua1QMfIF6iIg hDVKkEtzC8v3LpQMuuGYNIdIzrLeQMo= X-Google-Smtp-Source: AA0mqf4HpLa/4f3LD4BbXjSbD3piEt315rn3ubWu2txlyMrjuj44o8QNsRJCKteqd4ONaxDOyabGdg== X-Received: by 2002:a05:600c:1e85:b0:3cf:84be:aaf9 with SMTP id be5-20020a05600c1e8500b003cf84beaaf9mr3125551wmb.111.1668628311832; Wed, 16 Nov 2022 11:51:51 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id a16-20020a056000051000b002365730eae8sm15872505wrf.55.2022.11.16.11.51.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 16 Nov 2022 11:51:51 -0800 (PST) Message-Id: In-Reply-To: References: Date: Wed, 16 Nov 2022 19:51:42 +0000 Subject: [PATCH v2 9/9] clone: unbundle the advertised bundles Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, me@ttaylorr.com, newren@gmail.com, avarab@gmail.com, mjcheetham@outlook.com, steadmon@google.com, chooglen@google.com, jonathantanmy@google.com, dyroneteng@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee A previous change introduced the transport methods to acquire a bundle list from the 'bundle-uri' protocol v2 command, when advertised _and_ when the client has chosen to enable the feature. Teach Git to download and unbundle the data advertised by those bundles during 'git clone'. Also, since the --bundle-uri option exists, we do not want to mix the advertised bundles with the user-specified bundles. Signed-off-by: Derrick Stolee --- builtin/clone.c | 26 +++++++++++++++++---- t/t5601-clone.sh | 59 ++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+), 5 deletions(-) diff --git a/builtin/clone.c b/builtin/clone.c index 22b1e506452..09f10477ed6 100644 --- a/builtin/clone.c +++ b/builtin/clone.c @@ -1267,11 +1267,27 @@ int cmd_clone(int argc, const char **argv, const char *prefix) if (refs) mapped_refs = wanted_peer_refs(refs, &remote->fetch); - /* - * Populate transport->got_remote_bundle_uri and - * transport->bundle_uri. We might get nothing. - */ - transport_get_remote_bundle_uri(transport, 1); + if (!bundle_uri) { + /* + * Populate transport->got_remote_bundle_uri and + * transport->bundle_uri. We might get nothing. + */ + transport_get_remote_bundle_uri(transport, 1); + + if (transport->bundles && + hashmap_get_size(&transport->bundles->bundles)) { + /* At this point, we need the_repository to match the cloned repo. */ + if (repo_init(the_repository, git_dir, work_tree)) + warning(_("failed to initialize the repo, skipping bundle URI")); + if (fetch_bundle_list(the_repository, + remote->url[0], + transport->bundles)) + warning(_("failed to fetch advertised bundles")); + } else { + clear_bundle_list(transport->bundles); + FREE_AND_NULL(transport->bundles); + } + } if (mapped_refs) { int hash_algo = hash_algo_by_ptr(transport_get_hash_algo(transport)); diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh index 45f0803ed4d..d1d8139751e 100755 --- a/t/t5601-clone.sh +++ b/t/t5601-clone.sh @@ -795,6 +795,65 @@ test_expect_success 'reject cloning shallow repository using HTTP' ' git clone --no-reject-shallow $HTTPD_URL/smart/repo.git repo ' +test_expect_success 'auto-discover bundle URI from HTTP clone' ' + test_when_finished rm -rf trace.txt repo2 "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" && + git -C src bundle create "$HTTPD_DOCUMENT_ROOT_PATH/everything.bundle" --all && + git clone --bare --no-local src "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + uploadpack.advertiseBundleURIs true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.version 1 && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.mode all && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo2.git" config \ + bundle.everything.uri "$HTTPD_URL/everything.bundle" && + + GIT_TEST_BUNDLE_URI=1 \ + GIT_TRACE2_EVENT="$(pwd)/trace.txt" \ + git -c protocol.version=2 clone \ + $HTTPD_URL/smart/repo2.git repo2 && + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/everything.bundle"\] + EOF + grep -f pattern trace.txt +' + +test_expect_success 'auto-discover multiple bundles from HTTP clone' ' + test_when_finished rm -rf trace.txt repo3 "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" && + + test_commit -C src new && + git -C src bundle create "$HTTPD_DOCUMENT_ROOT_PATH/new.bundle" HEAD~1..HEAD && + git clone --bare --no-local src "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + uploadpack.advertiseBundleURIs true && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.version 1 && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.mode all && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.everything.uri "$HTTPD_URL/everything.bundle" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/repo3.git" config \ + bundle.new.uri "$HTTPD_URL/new.bundle" && + + GIT_TEST_BUNDLE_URI=1 \ + GIT_TRACE2_EVENT="$(pwd)/trace.txt" \ + git -c protocol.version=2 clone \ + $HTTPD_URL/smart/repo3.git repo3 && + + # We should fetch _both_ bundles + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/everything.bundle"\] + EOF + grep -f pattern trace.txt && + cat >pattern <<-EOF && + "event":"child_start".*"argv":\["git-remote-https","$HTTPD_URL/new.bundle"\] + EOF + grep -f pattern trace.txt +' + # DO NOT add non-httpd-specific tests here, because the last part of this # test script is only executed when httpd is available and enabled.