From patchwork Thu May 30 08:21:27 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xing Xin X-Patchwork-Id: 13679893 Received: from mail-wr1-f47.google.com (mail-wr1-f47.google.com [209.85.221.47]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8EA2A1C6A3 for ; Thu, 30 May 2024 08:21:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.221.47 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717057297; cv=none; b=Op+3ncUlSKTKdrok5vC4EKNJwNotstLu4lBoebeefsdK6hrEJxw0N/naWWvKVIKsQx6gROJMi3LL+OzXMrbmp+31cnAXm+gcFGh6AqA39J1mhcmcafeKgT/63KfIroCoOVI7eOsUX4+DQ8B/fzPMDgneKfAGD+gUsn9HuRAgsrQ= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717057297; c=relaxed/simple; bh=/tENE3Kgl1cYhu0N5e4hM8lvQ9F5tFHbMwVAioiliyY=; h=Message-Id:In-Reply-To:References:From:Date:Subject:Content-Type: MIME-Version:To:Cc; b=sjSKY+O72BikTvOBGS8CjrTOh9RizgB9NvwzDiXCcKjLHuaCbdHTLgP2pOndHGFT9ltmGS4HAhVNs3cf2f/XMwLNjZJmrlWrtgWxBj0XA7gibFVon8lhudxuu4LPf3jJ5xteM/XKQbtdWRNIKCCq7Db1IMtD/XQdOtfEjATtgPo= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=atlhm7pK; arc=none smtp.client-ip=209.85.221.47 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="atlhm7pK" Received: by mail-wr1-f47.google.com with SMTP id ffacd0b85a97d-354fb2d8f51so601183f8f.3 for ; Thu, 30 May 2024 01:21:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717057293; x=1717662093; darn=vger.kernel.org; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=O1vTJEyiG2vXL6cdAMlKLrLvnBGFQoqjVV/+UtqDT8E=; b=atlhm7pKGbDMFRqI1Uq/bRXjfm/O5GcZJZ8Nei2D7iJ5ydmrPJxkt/7fm2o0Vdq2l2 Ynf/XqdpAN/26fTpfN8UEJ+4DTwRsSrpdjbI8hH447MZoayfI4CHfKOA4n4MGuADWZw8 Q669PqO5hup6gszg43v46mFU7a0RCsCIJ4AVyYGncbLzqSZXWcoF4ZbMp7pEeHajuH1i VV1oWUhINHcOVy4SqnZdME8P0R1boEbaia5bJMQUHsptGXRkXd1GLPYUl6mHcNfsEhef DlNP46+hj/1bbdQ3TC87jaX6fFy70mNAOqmKwnOow+pzea80fw4T9mX6HnsXNr8WyrLp ZHhw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717057293; x=1717662093; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=O1vTJEyiG2vXL6cdAMlKLrLvnBGFQoqjVV/+UtqDT8E=; b=i3WNaoeO5OmWxcZJqIwVJhP6tlv71KuVUw5FtJOKRaHR5yIZjfRY2Et24aPKOmhQLF gpykV7JiF0VuqeEq+v7e3EJr2VBtpe8p5cKzguxwyZQ3f4hhbtR2hyd17jjOK1b371rd qgvrCkZrN5ui0Ll3ORZCm7sLSTXxXqXATOb+v884NppR/JutkCspc/qcJRkZWhVD7rzn PIslFmogbKgql8j9ESAGxhR6bmQ+/I0bGkAQGZlC6Rlzg2Z5MLvdMl4mry3J7Db+TdWM Jx3Kh3gWY6REyf/QXrV0GYqDHUzMDr0LBApMxrBBBcWQbuauZa9xvpTCI9qGmKaknqqJ vA6w== X-Gm-Message-State: AOJu0YzWjI4rEZ7pztR34DA3le97gGHUbR2p08OyIxfts6iCwDUQZb2/ j+HW6dmSFVMMgmQGcX3LGx8WEr/1QoT2UOlKnAG7plZ+hKwrAH495yf/jQ== X-Google-Smtp-Source: AGHT+IE4aQWpnoYYbfIqbqXTKDjXCJGKD9UUCx3Mj9//TccgWlqbgt5MUBiJmHwbAVGUBwc1fOUUNg== X-Received: by 2002:a5d:5146:0:b0:355:673:dd95 with SMTP id ffacd0b85a97d-35dc009f5bemr914419f8f.38.1717057293269; Thu, 30 May 2024 01:21:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3557dcf03f0sm16852833f8f.98.2024.05.30.01.21.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 May 2024 01:21:32 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 30 May 2024 08:21:27 +0000 Subject: [PATCH v4 1/4] bundle-uri: verify oid before writing refs Fcc: Sent Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To: git@vger.kernel.org Cc: Patrick Steinhardt , Karthik Nayak , blanet , Xing Xin From: Xing Xin From: Xing Xin When using the bundle-uri mechanism with a bundle list containing multiple interrelated bundles, we encountered a bug where tips from downloaded bundles were not discovered, thus resulting in rather slow clones. This was particularly problematic when employing the "creationTokens" heuristic. To reproduce this issue, consider a repository with a single branch "main" pointing to commit "A". Firstly, create a base bundle with: git bundle create base.bundle main Then, add a new commit "B" on top of "A", and create an incremental bundle for "main": git bundle create incr.bundle A..main Now, generate a bundle list with the following content: [bundle] version = 1 mode = all heuristic = creationToken [bundle "base"] uri = base.bundle creationToken = 1 [bundle "incr"] uri = incr.bundle creationToken = 2 A fresh clone with the bundle list above should result in a reference "refs/bundles/main" pointing to "B" in the new repository. However, git would still download everything from the server, as if it had fetched nothing locally. So why the "refs/bundles/main" is not discovered? After some digging I found that: 1. Bundles in bundle list are downloaded to local files via `bundle-uri.c:download_bundle_list` or via `bundle-uri.c:fetch_bundles_by_token` for the "creationToken" heuristic. 2. Each bundle is unbundled via `bundle-uri.c:unbundle_from_file`, which is called by `bundle-uri.c:unbundle_all_bundles` or called within `bundle-uri.c:fetch_bundles_by_token` for the "creationToken" heuristic. 3. To get all prerequisites of the bundle, the bundle header is read inside `bundle-uri.c:unbundle_from_file` to by calling `bundle.c:read_bundle_header`. 4. Then it calls `bundle.c:unbundle`, which calls `bundle.c:verify_bundle` to ensure the repository contains all the prerequisites. 5. `bundle.c:verify_bundle` calls `parse_object`, which eventually invokes `packfile.c:prepare_packed_git` or `packfile.c:reprepare_packed_git`, filling `raw_object_store->packed_git` and setting `packed_git_initialized`. 6. If `bundle.c:unbundle` succeeds, it writes refs via `refs.c:refs_update_ref` with `REF_SKIP_OID_VERIFICATION` set. Here bundle refs which can target arbitrary objects are written to the repository. 7. Finally, in `fetch-pack.c:do_fetch_pack_v2`, the functions `fetch-pack.c:mark_complete_and_common_ref` and `fetch-pack.c:mark_tips` are called with `OBJECT_INFO_QUICK` set to find local tips for negotiation. The `OBJECT_INFO_QUICK` flag prevents `packfile.c:reprepare_packed_git` from being called, resulting in failures to parse OIDs that reside only in the latest bundle. In the example above, when unbunding "incr.bundle", "base.pack" is added to `packed_git` due to prerequisites verification. However, "B" cannot be found for negotiation because it exists in "incr.pack", which is not included in `packed_git`. This commit fixes the bug by removing `REF_SKIP_OID_VERIFICATION` flag when writing bundle refs. When `refs.c:refs_update_ref` is called to to write the corresponding bundle refs, it triggers `refs.c:ref_transaction_commit`. This, in turn, invokes `refs.c:ref_transaction_prepare`, which calls `transaction_prepare` of the refs storage backend. For files backend, this function is `files-backend.c:files_transaction_prepare`, and for reftable backend, it is `reftable-backend.c:reftable_be_transaction_prepare`. Both functions eventually call `object.c:parse_object`, which can invoke `packfile.c:reprepare_packed_git` to refresh `packed_git`. This ensures that bundle refs point to valid objects and that all tips from bundle refs are correctly parsed during subsequent negotiations. A test has been added to demonstrate that bundles with incorrect headers, where refs point to non-existent objects, do not result in any bundle refs being created in the repository. Additionally, a set of negotiation-related tests for fetching with bundle-uri has been included. Reviewed-by: Karthik Nayak Reviewed-by: Patrick Steinhardt Signed-off-by: Xing Xin --- bundle-uri.c | 3 +- t/t5558-clone-bundle-uri.sh | 153 +++++++++++++++++++++++++++++++++++- 2 files changed, 150 insertions(+), 6 deletions(-) diff --git a/bundle-uri.c b/bundle-uri.c index 91b3319a5c1..65666a11d9c 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -400,8 +400,7 @@ static int unbundle_from_file(struct repository *r, const char *file) refs_update_ref(get_main_ref_store(the_repository), "fetched bundle", bundle_ref.buf, oid, has_old ? &old_oid : NULL, - REF_SKIP_OID_VERIFICATION, - UPDATE_REFS_MSG_ON_ERR); + 0, UPDATE_REFS_MSG_ON_ERR); } bundle_header_release(&header); diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh index 1ca5f745e73..8f4f802e4f1 100755 --- a/t/t5558-clone-bundle-uri.sh +++ b/t/t5558-clone-bundle-uri.sh @@ -19,10 +19,19 @@ test_expect_success 'fail to clone from non-bundle file' ' test_expect_success 'create bundle' ' git init clone-from && - git -C clone-from checkout -b topic && - test_commit -C clone-from A && - test_commit -C clone-from B && - git -C clone-from bundle create B.bundle topic + ( + cd clone-from && + git checkout -b topic && + + test_commit A && + git bundle create A.bundle topic && + + test_commit B && + git bundle create B.bundle topic && + + # Create a bundle with reference pointing to non-existent object. + sed "s/$(git rev-parse A)/$(git rev-parse B)/" bad-header.bundle + ) ' test_expect_success 'clone with path bundle' ' @@ -33,6 +42,16 @@ test_expect_success 'clone with path bundle' ' test_cmp expect actual ' +test_expect_success 'clone with bundle that has bad header' ' + git clone --bundle-uri="clone-from/bad-header.bundle" \ + clone-from clone-bad-header 2>err && + # Write bundle ref fails, but clone can still proceed. + commit_b=$(git -C clone-from rev-parse B) && + test_grep "trying to write ref '\''refs/bundles/topic'\'' with nonexistent object $commit_b" err && + git -C clone-bad-header for-each-ref --format="%(refname)" >refs && + ! grep "refs/bundles/" refs +' + test_expect_success 'clone with path bundle and non-default hash' ' test_when_finished "rm -rf clone-path-non-default-hash" && GIT_DEFAULT_HASH=sha256 git clone --bundle-uri="clone-from/B.bundle" \ @@ -259,6 +278,132 @@ test_expect_success 'clone bundle list (file, any mode, all failures)' ' ! grep "refs/bundles/" refs ' +######################################################################### +# Clone negotiation related tests begin here + +test_expect_success 'negotiation: bundle with part of wanted commits' ' + test_when_finished rm -rf trace*.txt && + GIT_TRACE_PACKET="$(pwd)/trace-packet.txt" \ + git clone --no-local --bundle-uri="clone-from/A.bundle" \ + clone-from nego-bundle-part && + git -C nego-bundle-part for-each-ref --format="%(refname)" >refs && + grep "refs/bundles/" refs >actual && + cat >expect <<-\EOF && + refs/bundles/topic + EOF + test_cmp expect actual && + # Ensure that refs/bundles/topic are sent as "have". + grep "clone> have $(git -C clone-from rev-parse A)" trace-packet.txt +' + +test_expect_success 'negotiation: bundle with all wanted commits' ' + test_when_finished rm -rf trace*.txt && + GIT_TRACE_PACKET="$(pwd)/trace-packet.txt" \ + git clone --no-local --single-branch --branch=topic --no-tags \ + --bundle-uri="clone-from/B.bundle" \ + clone-from nego-bundle-all && + git -C nego-bundle-all for-each-ref --format="%(refname)" >refs && + grep "refs/bundles/" refs >actual && + cat >expect <<-\EOF && + refs/bundles/topic + EOF + test_cmp expect actual && + # We already have all needed commits so no "want" needed. + ! grep "clone> want " trace-packet.txt +' + +test_expect_success 'negotiation: bundle list (no heuristic)' ' + test_when_finished rm -f trace*.txt && + cat >bundle-list <<-EOF && + [bundle] + version = 1 + mode = all + + [bundle "bundle-1"] + uri = file://$(pwd)/clone-from/bundle-1.bundle + + [bundle "bundle-2"] + uri = file://$(pwd)/clone-from/bundle-2.bundle + EOF + + GIT_TRACE_PACKET="$(pwd)/trace-packet.txt" \ + git clone --no-local --bundle-uri="file://$(pwd)/bundle-list" \ + clone-from nego-bundle-list-no-heuristic && + + git -C nego-bundle-list-no-heuristic for-each-ref --format="%(refname)" >refs && + grep "refs/bundles/" refs >actual && + cat >expect <<-\EOF && + refs/bundles/base + refs/bundles/left + EOF + test_cmp expect actual && + grep "clone> have $(git -C nego-bundle-list-no-heuristic rev-parse refs/bundles/left)" trace-packet.txt +' + +test_expect_success 'negotiation: bundle list (creationToken)' ' + test_when_finished rm -f trace*.txt && + cat >bundle-list <<-EOF && + [bundle] + version = 1 + mode = all + heuristic = creationToken + + [bundle "bundle-1"] + uri = file://$(pwd)/clone-from/bundle-1.bundle + creationToken = 1 + + [bundle "bundle-2"] + uri = file://$(pwd)/clone-from/bundle-2.bundle + creationToken = 2 + EOF + + GIT_TRACE_PACKET="$(pwd)/trace-packet.txt" \ + git clone --no-local --bundle-uri="file://$(pwd)/bundle-list" \ + clone-from nego-bundle-list-heuristic && + + git -C nego-bundle-list-heuristic for-each-ref --format="%(refname)" >refs && + grep "refs/bundles/" refs >actual && + cat >expect <<-\EOF && + refs/bundles/base + refs/bundles/left + EOF + test_cmp expect actual && + grep "clone> have $(git -C nego-bundle-list-heuristic rev-parse refs/bundles/left)" trace-packet.txt +' + +test_expect_success 'negotiation: bundle list with all wanted commits' ' + test_when_finished rm -f trace*.txt && + cat >bundle-list <<-EOF && + [bundle] + version = 1 + mode = all + heuristic = creationToken + + [bundle "bundle-1"] + uri = file://$(pwd)/clone-from/bundle-1.bundle + creationToken = 1 + + [bundle "bundle-2"] + uri = file://$(pwd)/clone-from/bundle-2.bundle + creationToken = 2 + EOF + + GIT_TRACE_PACKET="$(pwd)/trace-packet.txt" \ + git clone --no-local --single-branch --branch=left --no-tags \ + --bundle-uri="file://$(pwd)/bundle-list" \ + clone-from nego-bundle-list-all && + + git -C nego-bundle-list-all for-each-ref --format="%(refname)" >refs && + grep "refs/bundles/" refs >actual && + cat >expect <<-\EOF && + refs/bundles/base + refs/bundles/left + EOF + test_cmp expect actual && + # We already have all needed commits so no "want" needed. + ! grep "clone> want " trace-packet.txt +' + ######################################################################### # HTTP tests begin here From patchwork Thu May 30 08:21:28 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xing Xin X-Patchwork-Id: 13679894 Received: from mail-wm1-f49.google.com (mail-wm1-f49.google.com [209.85.128.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D408A14387F for ; Thu, 30 May 2024 08:21:35 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.49 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717057298; cv=none; b=OBQSwE3L/fy7B2gf8wthvYwCuWRVTDt/nThrm1XlnFQW18feNpnS0wsitOAhc8W9koNm4S8jDA29jeVb5CwJIyQmHh+GncaLP4bVszbxNtv3C99Gm8RezNNiOQl5LMLhH751R9j0IbWG8BFxEGjWFpx3BsuCuIQvW1+9rdujxlg= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717057298; c=relaxed/simple; bh=fRaGag7KYdvTweUrQaeEgz5yJRt/3RZ9wzL4BXFVz3g=; h=Message-Id:In-Reply-To:References:From:Date:Subject:Content-Type: MIME-Version:To:Cc; b=JupdndGhikmGQmb/MqmkQ+fT/piDVN4oonLTQvl90xr7f3elxTxWOyurKobP5PxgLZUxrWnnfN5doAmitfD1htEzAZbjkB2TZHlX3K+Mb+aUJ48ru9jffPDMyAaZSXHB1LYmXKKzMO81jaITsSUMte4/KgeSmIPOl+x3bkRPhHg= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=gqDyBg77; arc=none smtp.client-ip=209.85.128.49 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="gqDyBg77" Received: by mail-wm1-f49.google.com with SMTP id 5b1f17b1804b1-42120fc8d1dso7197325e9.2 for ; Thu, 30 May 2024 01:21:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717057294; x=1717662094; darn=vger.kernel.org; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=otyCUdQ5630OA1n3Kv2jXIRvqkETG7Cu3OerP6BBMZo=; b=gqDyBg77syk6lIpHQr0wvEdECsxAzEXZmpFxd0U2UtMsPONz0oXBn+Ht5LK6WIDoJK RNIbQrdgu37jK3YFvRopqvzVX6wBWE0CEPgDsVjl/u1GAlAHPfkeGv/PikYPLySlrgPM 0K0uECPsHzePlWF9Y19IGXPTJxy1CRbzCyX4djG8X1tEY4+BQ+nEytl6v3fQTxYmiXHG /h57t63t2lsudMUxBiAx6xYr38ZJcJEKYJjAr4bVKXYEmI3KB+bwW+/G4Ms1gtKGylB8 gi+PsToOOIEy4KY1eyvq3HNb0tFrJgWBqo3Fb7jTfmPuJa2R8Po2VP+drzoOCcrbhLEO HZeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717057294; x=1717662094; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=otyCUdQ5630OA1n3Kv2jXIRvqkETG7Cu3OerP6BBMZo=; b=CIH9CA38WLqhpvm0DuKvcDzWgkW7XmK7CYgKZG5onukZFwfdbzqpDZVhuZSygmoBl+ xHoSWqRY9OYn+WNbugT8gz6x3mItFBQCHYVdaq6UgwRbJ2gAfDRIDFuVvqP+YXVx+0sC mkgMaEzLBIBcsmdZ9S7N4qhLNRvF0SCrC17lkTuFAzG4UiVZZFJSjg+XY2QWkihBOxjp ubdjPD7KhxdYgj8S9Nx3LsJenhY5ePSveF61uVOzjPBUdLNM56LmPyrEX0IjshXHdkOd H1mMS2gG48sIa5RCB5emZW7HrxyjAER3Rcx4D6HZo9JC+VVHxxYyml+CbzVJfxrwDhUx Gi3w== X-Gm-Message-State: AOJu0Yy39ygfDlahvx7YFWFrwaUGdImYr4iNKpcpVby4KfHNinDz9LGy WT0xHarartr9EV9TkIVJT7ID5URjH/pPtfRuP2O3c0pIvPu2Avr2ofyFlw== X-Google-Smtp-Source: AGHT+IFIhWx4Wkx0BOZ+BuSw6xy/lHuSTZaODpm67F2FdNweNgeQQXtLMCHkUgrtoZInt8gI8M3M8w== X-Received: by 2002:a05:600c:470a:b0:41a:e995:b915 with SMTP id 5b1f17b1804b1-421278130cbmr15273325e9.1.1717057293919; Thu, 30 May 2024 01:21:33 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-42127065ba7sm17690805e9.12.2024.05.30.01.21.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 May 2024 01:21:33 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Thu, 30 May 2024 08:21:28 +0000 Subject: [PATCH v4 2/4] unbundle: extend verify_bundle_flags to support fsck-objects Fcc: Sent Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To: git@vger.kernel.org Cc: Patrick Steinhardt , Karthik Nayak , blanet , Xing Xin From: Xing Xin From: Xing Xin This commit extends `verify_bundle_flags` by adding a new option `VERIFY_BUNDLE_FSCK_ALWAYS`, which enables checks for broken objects in `bundle.c:unbundle`. This option is now used as the default for fetches involving bundles, specifically by `transport.c:fetch_refs_from_bundle` for direct bundle fetches and by `bundle-uri.c:unbundle_from_file` for _bundle-uri_ enabled fetches. Upcoming commits will introduce another option as a replacement that fits better with fetch operations. `VERIFY_BUNDLE_FSCK_ALWAYS` will be further used to add "--fsck-objects" support for "git bundle unbundle" and "git bundle verify". Reviewed-by: Patrick Steinhardt Signed-off-by: Xing Xin --- bundle-uri.c | 2 +- bundle.c | 3 +++ bundle.h | 1 + transport.c | 2 +- 4 files changed, 6 insertions(+), 2 deletions(-) diff --git a/bundle-uri.c b/bundle-uri.c index 65666a11d9c..066ff788104 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file) * the prerequisite commits. */ if ((result = unbundle(r, &header, bundle_fd, NULL, - VERIFY_BUNDLE_QUIET))) + VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_ALWAYS))) return 1; /* diff --git a/bundle.c b/bundle.c index 95367c2d0a0..26574e74bdd 100644 --- a/bundle.c +++ b/bundle.c @@ -625,6 +625,9 @@ int unbundle(struct repository *r, struct bundle_header *header, if (header->filter.choice) strvec_push(&ip.args, "--promisor=from-bundle"); + if (flags & VERIFY_BUNDLE_FSCK_ALWAYS) + strvec_push(&ip.args, "--fsck-objects"); + if (extra_index_pack_args) { strvec_pushv(&ip.args, extra_index_pack_args->v); strvec_clear(extra_index_pack_args); diff --git a/bundle.h b/bundle.h index 021adbdcbb3..cf23c8615d3 100644 --- a/bundle.h +++ b/bundle.h @@ -33,6 +33,7 @@ int create_bundle(struct repository *r, const char *path, enum verify_bundle_flags { VERIFY_BUNDLE_VERBOSE = (1 << 0), VERIFY_BUNDLE_QUIET = (1 << 1), + VERIFY_BUNDLE_FSCK_ALWAYS = (1 << 2), }; int verify_bundle(struct repository *r, struct bundle_header *header, diff --git a/transport.c b/transport.c index 0ad04b77fd2..1b3d61ffcec 100644 --- a/transport.c +++ b/transport.c @@ -184,7 +184,7 @@ static int fetch_refs_from_bundle(struct transport *transport, if (!data->get_refs_from_bundle_called) get_refs_from_bundle_inner(transport); ret = unbundle(the_repository, &data->header, data->fd, - &extra_index_pack_args, 0); + &extra_index_pack_args, VERIFY_BUNDLE_FSCK_ALWAYS); transport->hash_algo = data->header.hash_algo; return ret; } From patchwork Thu May 30 08:21:29 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xing Xin X-Patchwork-Id: 13679895 Received: from mail-lf1-f53.google.com (mail-lf1-f53.google.com [209.85.167.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8388B14AD25 for ; Thu, 30 May 2024 08:21:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.53 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717057299; cv=none; b=N9r6Aol5rv9vDdVWbcuzHQIYrEmzwvXuIPvGW4iX599Op0/Xqrs1x81jlOF25bZ0kagTxkIT7XoJ5R9odfTzmrFV0kslrvANXPqCthDbOeLs75Gxw+p7c6RNJ5c5sNUy6nAS6zYd/eGyaGE2D8f5MPRp3wjZbN1+FWmkEs5Ud5o= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717057299; c=relaxed/simple; bh=XLb5xiWA4dAF2Fx/JHqqeBya6F3OqFt/2QpQOAIOTOo=; h=Message-Id:In-Reply-To:References:From:Date:Subject:Content-Type: MIME-Version:To:Cc; b=YZt8s7ENehDmccewF6yiAebWfMJx4xbQxmppGzN/+ijGTV//53SNijumLpGzAa9UcxgEiss++abgUKt4ay/42M0km6wauP04cc9/+yNqx3KmkW8xQEmki5wUJeiJ+6EOfZpQNQkedKyooPjeKbA5wjdTGsOTFiXr2dVbYpaYwd8= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=nExw0WkZ; arc=none smtp.client-ip=209.85.167.53 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="nExw0WkZ" Received: by mail-lf1-f53.google.com with SMTP id 2adb3069b0e04-52ac0c4b62cso627920e87.2 for ; Thu, 30 May 2024 01:21:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717057295; x=1717662095; darn=vger.kernel.org; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=G968azMSMZxhGCb7WK9dg8PxA15ncjcjODXB2hpkuVM=; b=nExw0WkZUV0T3HRy6xmorSpMhfN4YjIEjsNhDm8MHQASVGSH7h/n6emqw34yJBcbl9 eWSEzanwEvuAASbXF4hSg7bI7IsDyqiXl/1UahzYIQJ46AmJf2pqhryt6JHAmT8wc6eD /yYL4qrn9u9i3Jx/BjEcizuxtVH4ZxUgtmjCktsJvCYSOmMdpkfz9cnoK9qE8tH9hx81 Z741Z5Uqnwh1qE9f8Op5UowjZ7G8pItZ2wwNQEVv92B/uiqJtXv7lDRWxHCX5MYHxnNF FdRbmS+3jjn1lpGOzrF2Nt2/gdUVzVvl8nttJYjSxv35vcVNZ1y/IKTDBswpakOI1+xH /1GA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717057295; x=1717662095; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=G968azMSMZxhGCb7WK9dg8PxA15ncjcjODXB2hpkuVM=; b=Ei9gmz8mXqUy+SURvL5d+XR5XYIiVN+bGNF/nBYA4KzQkSrAUgfzOTWQ7oZ9ASjQlH ifxJu40+3xhNJqnLNJSHP5wy/+lMGnKZFaxAMlRa1oVH7YoulhXsyKlviIL4mGQK2GRV kyuZAxRlG2v56mTcG5sJxWwij7C4vls5VywAd6YDIBILzkXLZ/S4uTnxvm1iwIAGXL68 nTonK3EFK3FbqqjfvPGs4ysD08YMAdYJbbJzHt082TkpBQcoiO0b8IuoAaxm77KqdBSH BJ4sj0OLto8HFWRJtkrCMYV4Wp1ECCRgAZ9ke/hbTJqatwBd//84j2DNpwd1KN52kVoI SlOQ== X-Gm-Message-State: AOJu0Yyt1W3OyNzzScoVdUw+NGzDlEofNHk5WaqcxoFLFCPo1xrbzL3v pgx1pHDIpic6R8Mq18sMSrEQrICsn4mrDr3nQBYXcSMES7uWcBSoxJEkag== X-Google-Smtp-Source: AGHT+IHGVBdqkR9g59JodGPel5lRg2/N3+xZaDA61B8nXkXABg8AfwpX6eTeNt2D+fSu8Qx8zt3tlQ== X-Received: by 2002:a2e:8ed5:0:b0:2ea:75c4:5621 with SMTP id 38308e7fff4ca-2ea847754f4mr6187621fa.7.1717057294978; Thu, 30 May 2024 01:21:34 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-35579d7d88esm16903163f8f.19.2024.05.30.01.21.34 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 May 2024 01:21:34 -0700 (PDT) Message-Id: <5ddc894c2c16db109ef96a07d8a2b623b855d95d.1717057290.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 30 May 2024 08:21:29 +0000 Subject: [PATCH v4 3/4] fetch-pack: expose fsckObjects configuration logic Fcc: Sent Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To: git@vger.kernel.org Cc: Patrick Steinhardt , Karthik Nayak , blanet , Xing Xin From: Xing Xin From: Xing Xin Currently, we can use "transfer.fsckObjects" and the more specific "fetch.fsckObjects" to control checks for broken objects in received packs during fetches. However, these configurations were only acknowledged by `fetch-pack.c:get_pack` and did not take effect in direct bundle fetches and fetches with _bundle-uri_ enabled. This commit exposes the fetch-then-transfer configuration logic by adding a new function `fetch_pack_fsck_objects` in fetch-pack.h. This new function is used to replace the assignment for `fsck_objects` in `fetch-pack.c:get_pack`. In the next commit, it will also be used by `bundle.c:unbundle` to better fit fetching scenarios. Helped-by: Junio C Hamano Helped-by: Patrick Steinhardt Signed-off-by: Xing Xin --- fetch-pack.c | 17 +++++++++++------ fetch-pack.h | 5 +++++ 2 files changed, 16 insertions(+), 6 deletions(-) diff --git a/fetch-pack.c b/fetch-pack.c index 7d2aef21add..3acff2baf09 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -954,12 +954,7 @@ static int get_pack(struct fetch_pack_args *args, strvec_push(&cmd.args, alternate_shallow_file); } - if (fetch_fsck_objects >= 0 - ? fetch_fsck_objects - : transfer_fsck_objects >= 0 - ? transfer_fsck_objects - : 0) - fsck_objects = 1; + fsck_objects = fetch_pack_fsck_objects(); if (do_keep || args->from_promisor || index_pack_args || fsck_objects) { if (pack_lockfiles || fsck_objects) @@ -2046,6 +2041,16 @@ static const struct object_id *iterate_ref_map(void *cb_data) return &ref->old_oid; } +int fetch_pack_fsck_objects(void) +{ + fetch_pack_setup(); + if (fetch_fsck_objects >= 0) + return fetch_fsck_objects; + if (transfer_fsck_objects >= 0) + return transfer_fsck_objects; + return 0; +} + struct ref *fetch_pack(struct fetch_pack_args *args, int fd[], const struct ref *ref, diff --git a/fetch-pack.h b/fetch-pack.h index 6775d265175..b5c579cdae2 100644 --- a/fetch-pack.h +++ b/fetch-pack.h @@ -101,4 +101,9 @@ void negotiate_using_fetch(const struct oid_array *negotiation_tips, */ int report_unmatched_refs(struct ref **sought, int nr_sought); +/* + * Return true if checks for broken objects in received pack are required. + */ +int fetch_pack_fsck_objects(void); + #endif From patchwork Thu May 30 08:21:30 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Xing Xin X-Patchwork-Id: 13679896 Received: from mail-wm1-f48.google.com (mail-wm1-f48.google.com [209.85.128.48]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D196C14D2AB for ; Thu, 30 May 2024 08:21:37 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.48 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717057300; cv=none; b=jbytuHUJzOdFlmCjf5gclg27+Gwl0ZGD0nqIPp3Bwf1BZN1VXb/bEGgB6MdMujxVNu76U+P3y2zwDD2b/z6dLrUVxyviF4wRvTdKjWQ1M+gT36i0BV469++61i5YTvyiwq8rsrfsKoMjwJSiOHdIJCc5Dk7lCU0I+5ByG8udzSI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1717057300; c=relaxed/simple; bh=Lms6/Kyx4SluTrDEuGuShr2HpqtO/iLV4zkG9ISJhuM=; h=Message-Id:In-Reply-To:References:From:Date:Subject:Content-Type: MIME-Version:To:Cc; b=eofI2ggIr7Dlgymlc93DkhYwqdk8hCvZBqYAbuJ2ijdPwONHJqwbR6QlRADSqEJpU8vo6DCwRouxBk6hZ0XfGbI7PO47DH4AYYCzchHw+pY78cqLrNAYIoNpbVoyFvH2aG6Z3BZ8bO580RRzX2+mHfelCTgw/7cbTDez6Um8FJ0= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=IJAmNcxM; arc=none smtp.client-ip=209.85.128.48 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="IJAmNcxM" Received: by mail-wm1-f48.google.com with SMTP id 5b1f17b1804b1-42120e3911eso6900045e9.0 for ; Thu, 30 May 2024 01:21:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1717057296; x=1717662096; darn=vger.kernel.org; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:from:to:cc:subject:date :message-id:reply-to; bh=3jTKImdaZpL/bvxfJj6IgP7QGU+M1LUchPwtQfNeF2I=; b=IJAmNcxMR/FpIANhqV9r/HSdYzPrqUegjKA/pW92LKRMtIJERWe/7ubkAOUljTctk+ EMZpYnGLpg0uHgl0V5/GZXXRYEjedQBBY105aIhYO7UhmvVH/XkAPygJ2RuAMB0X7y/0 souzdXkFOwKQ0gUhIK4LDvHHm+xESQBwj+n9Dz1ZVc38ZFSHBvE5+ru7UJOq927DymAg p1p1+bzvyQWLyCQNqjIaj+OsB4Zld/LwdZYxajEomqiv7yRj4s5jTyhjQisZ2iLEz0xK ESrg1EBsg8Ikq1TLGPhi912/A1ke4OVpzLubfjrVFzIJC59f2U64R1aAprsT33rS/VXz 8pYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717057296; x=1717662096; h=cc:to:mime-version:content-transfer-encoding:fcc:subject:date:from :references:in-reply-to:message-id:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=3jTKImdaZpL/bvxfJj6IgP7QGU+M1LUchPwtQfNeF2I=; b=ZHBlZXF997A03xa/cump5rqnrb+Be9TaCCjtSsTQbXpEQLJlPuN9w/DgU2L8I1Ext3 boqPjGEr2d6gyhlJgGsHDZq6Ovt9siQNd63QY08JogZJMGKEACS/tac0SWdpYF9ue3mx w/IU9gUb1EXbjLQzMfPj5aYU2DM2jr2HR/P5wSXm1oCADA4i8Emx5gmzZCc0fsh4M/kg vIHWibIj2buHr3/2T2eUNZ32gtqI0aRo3dM8Kv//KTwIxk37j1aO3DxgunvAIcdhoa5F ioemffFVkTVzVKYPUycgrwVvuV4LZ4afM6+w+gN6eBvVgv4ZzdAAaJ5CbelixI4qo3CK dVGQ== X-Gm-Message-State: AOJu0Yyimg4fll02c+2J9SmuFbnpmsKlG4cGp+IhWRGitK/5hyeWwVHt L4MyzBpeRsBPW1hjbGbMgBL0S7eg9q4Kq3WJaAWn6rOGChmC8D2ZdkrsTQ== X-Google-Smtp-Source: AGHT+IEv6eVSPg8rR1JKu/xPK5FKrsSyZBfp3fHUOz/+1FfPtYAr/n76OVygaXHyN+tx7GiEHV8zkQ== X-Received: by 2002:a05:600c:19d4:b0:420:fff:f4c9 with SMTP id 5b1f17b1804b1-421278185b8mr17719785e9.13.1717057295694; Thu, 30 May 2024 01:21:35 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-4212705993csm17744305e9.2.2024.05.30.01.21.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 30 May 2024 01:21:35 -0700 (PDT) Message-Id: <68b9bca9f8b19897997c2adc9a278ac5052e75cd.1717057290.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Thu, 30 May 2024 08:21:30 +0000 Subject: [PATCH v4 4/4] unbundle: introduce option VERIFY_BUNDLE_FSCK_FOLLOW_FETCH Fcc: Sent Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 To: git@vger.kernel.org Cc: Patrick Steinhardt , Karthik Nayak , blanet , Xing Xin From: Xing Xin From: Xing Xin This commit introduces a new option `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH` to `verify_bundle_flags`. In `bundle.c:unbundle`, this new option controls whether broken object checks should be enabled by invoking `fetch-pack.c:fetch_pack_fsck_objects`. Note that the option `VERIFY_BUNDLE_FSCK_ALWAYS` takes precedence over `VERIFY_BUNDLE_FSCK_FOLLOW_FETCH`. This flag is now used in the fetching process by: - `transport.c:fetch_refs_from_bundle` for direct bundle fetches. - `bundle-uri.c:unbundle_from_file` for bundle-uri enabled fetches. This addition ensures a consistent logic for object verification during fetch operations. Tests have been added to confirm functionality in the scenarios mentioned above. Signed-off-by: Xing Xin --- bundle-uri.c | 2 +- bundle.c | 7 +++++++ bundle.h | 1 + t/t5558-clone-bundle-uri.sh | 35 ++++++++++++++++++++++++++++++++++- t/t5607-clone-bundle.sh | 33 +++++++++++++++++++++++++++++++++ transport.c | 2 +- 6 files changed, 77 insertions(+), 3 deletions(-) diff --git a/bundle-uri.c b/bundle-uri.c index 066ff788104..e7ebac6ce57 100644 --- a/bundle-uri.c +++ b/bundle-uri.c @@ -373,7 +373,7 @@ static int unbundle_from_file(struct repository *r, const char *file) * the prerequisite commits. */ if ((result = unbundle(r, &header, bundle_fd, NULL, - VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_ALWAYS))) + VERIFY_BUNDLE_QUIET | VERIFY_BUNDLE_FSCK_FOLLOW_FETCH))) return 1; /* diff --git a/bundle.c b/bundle.c index 26574e74bdd..53ac73834ea 100644 --- a/bundle.c +++ b/bundle.c @@ -17,6 +17,7 @@ #include "list-objects-filter-options.h" #include "connected.h" #include "write-or-die.h" +#include "fetch-pack.h" static const char v2_bundle_signature[] = "# v2 git bundle\n"; static const char v3_bundle_signature[] = "# v3 git bundle\n"; @@ -615,6 +616,7 @@ int unbundle(struct repository *r, struct bundle_header *header, enum verify_bundle_flags flags) { struct child_process ip = CHILD_PROCESS_INIT; + int fsck_objects = 0; if (verify_bundle(r, header, flags)) return -1; @@ -626,6 +628,11 @@ int unbundle(struct repository *r, struct bundle_header *header, strvec_push(&ip.args, "--promisor=from-bundle"); if (flags & VERIFY_BUNDLE_FSCK_ALWAYS) + fsck_objects = 1; + else if (flags & VERIFY_BUNDLE_FSCK_FOLLOW_FETCH) + fsck_objects = fetch_pack_fsck_objects(); + + if (fsck_objects) strvec_push(&ip.args, "--fsck-objects"); if (extra_index_pack_args) { diff --git a/bundle.h b/bundle.h index cf23c8615d3..a39d8ea1a7e 100644 --- a/bundle.h +++ b/bundle.h @@ -34,6 +34,7 @@ enum verify_bundle_flags { VERIFY_BUNDLE_VERBOSE = (1 << 0), VERIFY_BUNDLE_QUIET = (1 << 1), VERIFY_BUNDLE_FSCK_ALWAYS = (1 << 2), + VERIFY_BUNDLE_FSCK_FOLLOW_FETCH = (1 << 3), }; int verify_bundle(struct repository *r, struct bundle_header *header, diff --git a/t/t5558-clone-bundle-uri.sh b/t/t5558-clone-bundle-uri.sh index 8f4f802e4f1..48be1b18802 100755 --- a/t/t5558-clone-bundle-uri.sh +++ b/t/t5558-clone-bundle-uri.sh @@ -30,7 +30,21 @@ test_expect_success 'create bundle' ' git bundle create B.bundle topic && # Create a bundle with reference pointing to non-existent object. - sed "s/$(git rev-parse A)/$(git rev-parse B)/" bad-header.bundle + sed "s/$(git rev-parse A)/$(git rev-parse B)/" bad-header.bundle && + + cat >data <<-EOF && + tree $(git rev-parse HEAD^{tree}) + parent $(git rev-parse HEAD) + author A U Thor + committer A U Thor + + commit: this is a commit with bad emails + + EOF + git hash-object --literally -t commit -w --stdin commit && + git branch bad $(cat commit) && + git bundle create bad-object.bundle bad && + git update-ref -d refs/heads/bad ) ' @@ -52,6 +66,25 @@ test_expect_success 'clone with bundle that has bad header' ' ! grep "refs/bundles/" refs ' +test_expect_success 'clone with bundle that has bad object' ' + # Unbundle succeeds if no fsckObjects confugured. + git clone --bundle-uri="clone-from/bad-object.bundle" \ + clone-from clone-bad-object-no-fsck && + git -C clone-bad-object-no-fsck for-each-ref --format="%(refname)" >refs && + grep "refs/bundles/" refs >actual && + cat >expect <<-\EOF && + refs/bundles/bad + EOF + test_cmp expect actual && + + # Unbundle fails with fsckObjects set true, but clone can still proceed. + git -c fetch.fsckObjects=true clone --bundle-uri="clone-from/bad-object.bundle" \ + clone-from clone-bad-object-fsck 2>err && + test_grep "missingEmail" err && + git -C clone-bad-object-fsck for-each-ref --format="%(refname)" >refs && + ! grep "refs/bundles/" refs +' + test_expect_success 'clone with path bundle and non-default hash' ' test_when_finished "rm -rf clone-path-non-default-hash" && GIT_DEFAULT_HASH=sha256 git clone --bundle-uri="clone-from/B.bundle" \ diff --git a/t/t5607-clone-bundle.sh b/t/t5607-clone-bundle.sh index 0d1e92d9963..5182efc0b45 100755 --- a/t/t5607-clone-bundle.sh +++ b/t/t5607-clone-bundle.sh @@ -138,6 +138,39 @@ test_expect_success 'fetch SHA-1 from bundle' ' git fetch --no-tags foo/tip.bundle "$(cat hash)" ' +test_expect_success 'clone bundle with different fsckObjects configurations' ' + test_create_repo bundle-fsck && + ( + cd bundle-fsck && + test_commit first && + cat >data <<-EOF && + tree $(git rev-parse HEAD^{tree}) + parent $(git rev-parse HEAD) + author A U Thor + committer A U Thor + + commit: this is a commit with bad emails + + EOF + git hash-object --literally -t commit -w --stdin commit && + git branch bad $(cat commit) && + git bundle create bad.bundle bad + ) && + + git clone bundle-fsck/bad.bundle bundle-no-fsck && + + git -c fetch.fsckObjects=false -c transfer.fsckObjects=true \ + clone bundle-fsck/bad.bundle bundle-fetch-no-fsck && + + test_must_fail git -c fetch.fsckObjects=true \ + clone bundle-fsck/bad.bundle bundle-fetch-fsck 2>err && + test_grep "missingEmail" err && + + test_must_fail git -c transfer.fsckObjects=true \ + clone bundle-fsck/bad.bundle bundle-transfer-fsck 2>err && + test_grep "missingEmail" err +' + test_expect_success 'git bundle uses expected default format' ' git bundle create bundle HEAD^.. && cat >expect <<-EOF && diff --git a/transport.c b/transport.c index 1b3d61ffcec..6cd5683bb45 100644 --- a/transport.c +++ b/transport.c @@ -184,7 +184,7 @@ static int fetch_refs_from_bundle(struct transport *transport, if (!data->get_refs_from_bundle_called) get_refs_from_bundle_inner(transport); ret = unbundle(the_repository, &data->header, data->fd, - &extra_index_pack_args, VERIFY_BUNDLE_FSCK_ALWAYS); + &extra_index_pack_args, VERIFY_BUNDLE_FSCK_FOLLOW_FETCH); transport->hash_algo = data->header.hash_algo; return ret; }