From patchwork Mon Feb 1 17:15:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12059487 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1108CC433E0 for ; Mon, 1 Feb 2021 17:15:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B771464EAC for ; Mon, 1 Feb 2021 17:15:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231659AbhBARPy (ORCPT ); Mon, 1 Feb 2021 12:15:54 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36232 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230055AbhBARPw (ORCPT ); Mon, 1 Feb 2021 12:15:52 -0500 Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id B2550C06174A for ; Mon, 1 Feb 2021 09:15:11 -0800 (PST) Received: by mail-wr1-x42c.google.com with SMTP id a1so17456410wrq.6 for ; Mon, 01 Feb 2021 09:15:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=juCAe5atgoTFygs1wNmd/FGDKpeIjGfc51PxxQjB/ho=; b=qkSz8i8116W3fK2B3ZcfQdg3NKJHj2fy2+aOHQnXXlZM+KHhtWfD59bbmm5smjsWzX hrT4VQABxSDYVhUZ+YkAx2TufGqejq9R3tbCNdKgWyEVRvsBKem3t7AIYBp5T8DGWdoX 1IZRCcSnLczYFAYDfvEXbQ9wSq2Fy50Kc0vJhQ89nD/6Y72rGWlH9xgT4Ba4RRx45gCj eDoxfyLniZPF2t+DjSNdoFuga4fLQ2YDNbSoFIoGss3Cpfoa/Lm5FAuYmZFzKpzh0HbS 3P0qp/dH+lfXPWaD1b2q3AWZZgA3H3/xCYC70uB/WSVoc3B4yv00Oc0HHVV6UeNbqU// agbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=juCAe5atgoTFygs1wNmd/FGDKpeIjGfc51PxxQjB/ho=; b=izksorTCH2XXfqTXQ4oAUcRkNmtwOnGwQmn088b+YojlM2+dtHtVYPvrYoK/6Ipj4T CxdtKvQE3Fc9PCezGUmiFDcS1JaWD1CckoUsjhygkDzkqYW5w+JCfy8EKhuY0j54IVkw B1pbhS7sg7PIq54milG28P0GRM16i5bmUwA5zLMKLtOPbzAEXYM2Gc56PPjaaGL9y9tZ PWwCuA77HrjDrW9uDABhi+V5s7FWFedzCAsJ18zgbl6dsTRc6jAHutdqSoGpDzkEcScr UcvuVqLjR/IfMy0bUlUjxU04FQK+Zqo8HNUZH3JTCNGQxAOpA5xbo5IdLMRXjZT3JREB G/Ug== X-Gm-Message-State: AOAM530KKpdr5mXW7M2HKGuXcMA4Rq5Rf3gmnev4FlzZTfdJP0jIlRc3 fSlkR6t/xEjqJUHgoW70gzIrFdOYi20= X-Google-Smtp-Source: ABdhPJzDYIAyuaRZVHLqm81KfnMr4OWpWMqqLeFXwjvrjg+6LRHyAQ30T4iD+Wy25PQNM6jL+W3WJg== X-Received: by 2002:a5d:51cf:: with SMTP id n15mr18738843wrv.303.1612199710304; Mon, 01 Feb 2021 09:15:10 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r16sm6490239wrt.68.2021.02.01.09.15.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 09:15:09 -0800 (PST) Message-Id: <9c605c99f66cae3c66bcdd4cbeefbfa1ec2be192.1612199707.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 01 Feb 2021 17:15:03 +0000 Subject: [PATCH 1/5] commit-graph: use repo_parse_commit Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: me@ttaylorr.com, peff@peff.net, gister@pobox.com, abhishekkumar8222@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee The write_commit_graph_context has a repository pointer, so use it. Signed-off-by: Derrick Stolee --- commit-graph.c | 10 +++++----- commit.h | 5 +++-- 2 files changed, 8 insertions(+), 7 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index f3bde2ad95a..03e5a987968 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1098,7 +1098,7 @@ static int write_graph_chunk_data(struct hashfile *f, uint32_t packedDate[2]; display_progress(ctx->progress, ++ctx->progress_cnt); - if (parse_commit_no_graph(*list)) + if (repo_parse_commit_no_graph(ctx->r, *list)) die(_("unable to parse commit %s"), oid_to_hex(&(*list)->object.oid)); tree = get_commit_tree_oid(*list); @@ -1411,11 +1411,11 @@ static void close_reachable(struct write_commit_graph_context *ctx) if (!commit) continue; if (ctx->split) { - if ((!parse_commit(commit) && + if ((!repo_parse_commit(ctx->r, commit) && commit_graph_position(commit) == COMMIT_NOT_FROM_GRAPH) || flags == COMMIT_GRAPH_SPLIT_REPLACE) add_missing_parents(ctx, commit); - } else if (!parse_commit_no_graph(commit)) + } else if (!repo_parse_commit_no_graph(ctx->r, commit)) add_missing_parents(ctx, commit); } stop_progress(&ctx->progress); @@ -1710,9 +1710,9 @@ static void copy_oids_to_commits(struct write_commit_graph_context *ctx) continue; if (ctx->split && flags == COMMIT_GRAPH_SPLIT_REPLACE) - parse_commit(ctx->commits.list[ctx->commits.nr]); + repo_parse_commit(ctx->r, ctx->commits.list[ctx->commits.nr]); else - parse_commit_no_graph(ctx->commits.list[ctx->commits.nr]); + repo_parse_commit_no_graph(ctx->r, ctx->commits.list[ctx->commits.nr]); num_parents = commit_list_count(ctx->commits.list[ctx->commits.nr]->parents); if (num_parents > 2) diff --git a/commit.h b/commit.h index 251d877fcf6..b05ab558ce2 100644 --- a/commit.h +++ b/commit.h @@ -89,9 +89,10 @@ static inline int repo_parse_commit(struct repository *r, struct commit *item) return repo_parse_commit_gently(r, item, 0); } -static inline int parse_commit_no_graph(struct commit *commit) +static inline int repo_parse_commit_no_graph(struct repository *r, + struct commit *commit) { - return repo_parse_commit_internal(the_repository, commit, 0, 0); + return repo_parse_commit_internal(r, commit, 0, 0); } #ifndef NO_THE_REPOSITORY_COMPATIBILITY_MACROS From patchwork Mon Feb 1 17:15:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12059497 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CF5A0C433DB for ; Mon, 1 Feb 2021 17:16:04 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9D24764EAC for ; Mon, 1 Feb 2021 17:16:04 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231697AbhBARP5 (ORCPT ); Mon, 1 Feb 2021 12:15:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36240 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229612AbhBARPx (ORCPT ); Mon, 1 Feb 2021 12:15:53 -0500 Received: from mail-wm1-x32d.google.com (mail-wm1-x32d.google.com [IPv6:2a00:1450:4864:20::32d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 17768C0613ED for ; Mon, 1 Feb 2021 09:15:13 -0800 (PST) Received: by mail-wm1-x32d.google.com with SMTP id o10so322921wmc.1 for ; Mon, 01 Feb 2021 09:15:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=Znqfxk2Gop1qR9R898yhT0BQKTbVqCtFqS7Cp/XrlV0=; b=GdIipdiWK9nVYpH8wKh490l/lg/zlIOfS3JsqlBMMgASfg9/VcMiX5W7zg5vF8Oe9C lMbPOsehAdrllIBgfvlDrwxiJATHaQgHUa6VRS24QgXNFDLLWkJT9BYkI9piuyYH90bQ OjQAWarzHFzO1BrSJM0wISLL++eNfL8hTaWmMeTcXuMiIc9OlEHdLPodKEhjtfCAx7gP lQ2RSFXC2mvIhzR4rzzlkA3c6JkwEwgeIONjR1SnqoEbACJrgV//ckgIPc2iVc7fRB3C tHEHIYqhRCnu8NXsyGZMtSjnNLZ6ddmaOH6S1KsffAlGuB1mH1SOuDauZpbZ3Twb4RzF ClmQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=Znqfxk2Gop1qR9R898yhT0BQKTbVqCtFqS7Cp/XrlV0=; b=ulY7b8VJLfAgDvFQmzp0eKfOIZF/ggq5cSr5v0vIgUJLM1SrA7TZOs/gH4Y6BGzQwI JfXXHbCcpo4QiOZhMT28he2ulYV8Vsh7/h7FKEiQ5X491dzWLIuBGLDsoDUVLxVkBNDm u7+uBklkElXd3Eb+xY9LyZP92kt6u+7x1HzPQGn7RfMkbmf7CYfhnM3PyzRcCqTZT8Dk z3GMGzQucXM2qNNhvF5DH56Fswy8AW8GV/68c1rNQSUCKmLKp4WPhuiY7WNZsNLMgIio CfZ24Ygg29V9kfZvtcn9Xd2YDyb6Y8Q5F411xoH1X2sBDQutZu/oBNHdOvDOm+UCS33B JpOg== X-Gm-Message-State: AOAM531QJhuD0dMzuObUhuG17+tYyjVS2adConvLqEONt7gRt/YaOwPD 31Q2WqwrtGliUlF60nU8RsSMl4WK6SE= X-Google-Smtp-Source: ABdhPJw04DTdJ63l0Me3ysJd03WMs1vXBqV9DHc65YEfAM0epkSLFNO16c/Z2hqgbCl3trxjlfJcHA== X-Received: by 2002:a1c:4d13:: with SMTP id o19mr15866099wmh.3.1612199711308; Mon, 01 Feb 2021 09:15:11 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id r12sm27820466wrp.13.2021.02.01.09.15.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 09:15:10 -0800 (PST) Message-Id: <82afa811dffa3f7b3c371de5c220b0529d96ed87.1612199707.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Mon, 01 Feb 2021 17:15:04 +0000 Subject: [PATCH 2/5] commit-graph: always parse before commit_graph_data_at() Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: me@ttaylorr.com, peff@peff.net, gister@pobox.com, abhishekkumar8222@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee There is a subtle failure happening when computing corrected commit dates with --split enabled. It requires a base layer needing the generation_data_overflow chunk. Then, the next layer on top erroneously thinks it needs an overflow chunk due to a bug leading to recalculating all reachable generation numbers. The output of the failure is BUG: commit-graph.c:1912: expected to write 8 bytes to chunk 47444f56, but wrote 0 instead These "expected" 8 bytes are due to re-computing the corrected commit date for the lower layer but the new layer does not need any overflow. Add a test to t5318-commit-graph.sh that demonstrates this bug. However, it does not trigger consistently with the existing code. The generation number data is stored in a slab and accessed by commit_graph_data_at(). This data is initialized when parsing a commit, but is otherwise used assuming it has been populated. The loop in compute_generation_numbers() did not enforce that all reachable commits were parsed and had correct values. This could lead to some problems when writing a commit-graph with corrected commit dates based on a commit-graph without them. It has been difficult to identify the issue here becaus it was so hard to reproduce. It relies on this uninitialized data having a non-zero value, but also on specifically in a way that overwrites the existing data. This patch adds the extra parse to ensure the data is filled before we compute the generation number of a commit. This triggers the new test to fail because the generation number overflow count does not match between this computation and the write for that chunk. The actual fix will follow as the next few changes. Signed-off-by: Derrick Stolee --- commit-graph.c | 16 ++++++++++++---- t/t5318-commit-graph.sh | 21 +++++++++++++++++++++ 2 files changed, 33 insertions(+), 4 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index 03e5a987968..edbb3a0f2cc 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1193,7 +1193,9 @@ static int write_graph_chunk_generation_data(struct hashfile *f, for (i = 0; i < ctx->commits.nr; i++) { struct commit *c = ctx->commits.list[i]; - timestamp_t offset = commit_graph_data_at(c)->generation - c->date; + timestamp_t offset; + repo_parse_commit(ctx->r, c); + offset = commit_graph_data_at(c)->generation - c->date; display_progress(ctx->progress, ++ctx->progress_cnt); if (offset > GENERATION_NUMBER_V2_OFFSET_MAX) { @@ -1444,15 +1446,20 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx) _("Computing commit graph generation numbers"), ctx->commits.nr); for (i = 0; i < ctx->commits.nr; i++) { - uint32_t level = *topo_level_slab_at(ctx->topo_levels, ctx->commits.list[i]); - timestamp_t corrected_commit_date = commit_graph_data_at(ctx->commits.list[i])->generation; + struct commit *c = ctx->commits.list[i]; + uint32_t level; + timestamp_t corrected_commit_date; + + repo_parse_commit(ctx->r, c); + level = *topo_level_slab_at(ctx->topo_levels, c); + corrected_commit_date = commit_graph_data_at(c)->generation; display_progress(ctx->progress, i + 1); if (level != GENERATION_NUMBER_ZERO && corrected_commit_date != GENERATION_NUMBER_ZERO) continue; - commit_list_insert(ctx->commits.list[i], &list); + commit_list_insert(c, &list); while (list) { struct commit *current = list->item; struct commit_list *parent; @@ -1461,6 +1468,7 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx) timestamp_t max_corrected_commit_date = 0; for (parent = current->parents; parent; parent = parent->next) { + repo_parse_commit(ctx->r, parent->item); level = *topo_level_slab_at(ctx->topo_levels, parent->item); corrected_commit_date = commit_graph_data_at(parent->item)->generation; diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index fa27df579a5..2cf29f425a0 100755 --- a/t/t5318-commit-graph.sh +++ b/t/t5318-commit-graph.sh @@ -446,6 +446,27 @@ test_expect_success 'warn on improper hash version' ' ) ' +test_expect_failure 'lower layers have overflow chunk' ' + cd "$TRASH_DIRECTORY/full" && + UNIX_EPOCH_ZERO="@0 +0000" && + FUTURE_DATE="@2147483646 +0000" && + rm -f .git/objects/info/commit-graph && + test_commit --date "$FUTURE_DATE" future-1 && + test_commit --date "$UNIX_EPOCH_ZERO" old-1 && + git commit-graph write --reachable && + test_commit --date "$FUTURE_DATE" future-2 && + test_commit --date "$UNIX_EPOCH_ZERO" old-2 && + git commit-graph write --reachable --split=no-merge && + test_commit extra && + git commit-graph write --reachable --split=no-merge && + git commit-graph write --reachable && + graph_read_expect 16 "generation_data generation_data_overflow extra_edges" && + mv .git/objects/info/commit-graph commit-graph-upgraded && + git commit-graph write --reachable && + graph_read_expect 16 "generation_data generation_data_overflow extra_edges" && + test_cmp .git/objects/info/commit-graph commit-graph-upgraded +' + # the verify tests below expect the commit-graph to contain # exactly the commits reachable from the commits/8 branch. # If the file changes the set of commits in the list, then the From patchwork Mon Feb 1 17:15:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12059489 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D6B1AC433DB for ; Mon, 1 Feb 2021 17:15:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9316264EAC for ; Mon, 1 Feb 2021 17:15:59 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S229534AbhBARP5 (ORCPT ); Mon, 1 Feb 2021 12:15:57 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36252 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230055AbhBARP4 (ORCPT ); Mon, 1 Feb 2021 12:15:56 -0500 Received: from mail-wm1-x32e.google.com (mail-wm1-x32e.google.com [IPv6:2a00:1450:4864:20::32e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 15184C061786 for ; Mon, 1 Feb 2021 09:15:16 -0800 (PST) Received: by mail-wm1-x32e.google.com with SMTP id u14so13841768wmq.4 for ; Mon, 01 Feb 2021 09:15:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=/U6UDAGaH1mCjnX82IwAUqNIZFhOQLAUI+ugKX9Q2tk=; b=kDIv91brKm1ilrz19as9TNlZmc+dQDu5v7Tbm3LyjWOmeMaCoDKvK5bzLX9ZbeY60a PMlxwn/ylsA7oA7FMGxpOhOEB5f6+t4MBOY120wNCVSk5P917YhkxaGq11o3D/SA7/ne a90sR1BeuDcBbmzYwdK9cohQ92dH5XlqlU0JYCsV65SoB5OHeN0qHhOlnbz3MVqsxSFd hNOqs9guGDrspanKtDoZ3a7zWnJtMHsKdYY9NP+BHi4twH7DfF6AdlJGnuFe8T5fh3if NCfGOBJJHfKRIDUtrCmvYM79K6Q0WXtdD6g0nVEJO2H9ig91cL0ftZ7L0lXHkYIrZHKf gFIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=/U6UDAGaH1mCjnX82IwAUqNIZFhOQLAUI+ugKX9Q2tk=; b=MYuKvgsfIvZZNS9nAnPrdl6JMXiMiVRfTZaL6szaCZrApvv1Zqr5WMDmsu2NQabhNZ 0GRKB6vQ7ltX2ktXH25t0u22oiw2xLQWdVUOL1ai1/xXnhdgk3VEii0a/XdWAGqdQztq 4KGz13wtHXvDXhB3FP78pSKB1YGWgERp9htW0Ha3EYy26VYYLj+ya37nZa2zfrz3cwD+ HYp8kle5cg1z7l3vubQ4wErwB5oi/wvWyKkcyUeREZ/wR2PZ6Cdj2OaG1nLbIne22xCu HMTQ0kQsToobXtNnRCbREBup99T5AtBhqK6GKh7JYn5Hn0mVnKZnKZxnylH/zPfTyOsx nDwA== X-Gm-Message-State: AOAM532brHxeoiou0tpKnD5gRAKAHlm7LizSkeDmvCIc0DxTBFIGTfcI DNvjKV91dk27zRLs2unvZy+hIfzZ5mo= X-Google-Smtp-Source: ABdhPJxI2bVuG/h0oxJorAreE5RSbIYrx6UeJo/zE13H2C/nPyk4bbzG52+dBI7gF5OCyHcv87e9Qw== X-Received: by 2002:a1c:4e:: with SMTP id 75mr16125092wma.150.1612199714307; Mon, 01 Feb 2021 09:15:14 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id n9sm27931727wrq.41.2021.02.01.09.15.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 09:15:13 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 01 Feb 2021 17:15:05 +0000 Subject: [PATCH 3/5] commit-graph: validate layers for generation data Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: me@ttaylorr.com, peff@peff.net, gister@pobox.com, abhishekkumar8222@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee We need to be extra careful that we don't use corrected commit dates from any layer of a commit-graph chain if there is a single commit-graph file that is missing the generation_data chunk. Update validate_mixed_generation_chain() to correctly update each layer to ignore the generation_data chunk in this case. It now also returns 1 if all layers have a generation_data chunk. This return value will be used in the next change. Signed-off-by: Derrick Stolee --- commit-graph.c | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index edbb3a0f2cc..13992137dd0 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -614,19 +614,26 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r, return graph_chain; } -static void validate_mixed_generation_chain(struct commit_graph *g) +/* + * returns 1 if and only if all graphs in the chain have + * corrected commit dates stored in the generation_data chunk. + */ +static int validate_mixed_generation_chain(struct commit_graph *g) { - int read_generation_data; + int read_generation_data = 1; + struct commit_graph *p = g; - if (!g) - return; - - read_generation_data = !!g->chunk_generation_data; + while (read_generation_data && p) { + read_generation_data = p->read_generation_data; + p = p->base_graph; + } while (g) { g->read_generation_data = read_generation_data; g = g->base_graph; } + + return read_generation_data; } struct commit_graph *read_commit_graph_one(struct repository *r, From patchwork Mon Feb 1 17:15:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12059491 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id EA60CC433E0 for ; Mon, 1 Feb 2021 17:16:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id AF89864EAC for ; Mon, 1 Feb 2021 17:16:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231803AbhBARP7 (ORCPT ); Mon, 1 Feb 2021 12:15:59 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36258 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231742AbhBARP5 (ORCPT ); Mon, 1 Feb 2021 12:15:57 -0500 Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D8C52C061788 for ; Mon, 1 Feb 2021 09:15:17 -0800 (PST) Received: by mail-wr1-x434.google.com with SMTP id l12so17463855wry.2 for ; Mon, 01 Feb 2021 09:15:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=rcseCZP2x1rpmxdurCQy/lUNoOhUFfcgvsvbOvu3pN0=; b=R6PfCswaoORVD3ev4e7K4a9+567SIpYRtq4i7ejo2pyYneqBQwZF8eNPgLh9A3x2A5 +BUshlmG86HqTm/ECqKcfdl7QrXoZqB599abrkXlPbJYGTckZertDWYhLyZstCL9OMri 4/FplQHVwHOEgY/fWL4Okq+24wKINoIKR/QqPFbCdOC0QcURDPEEShv3XnjezJSPhtpY NcYEzRjVjJAKWciAo+XbZ+Zqvls2YzoMPHd/k+mAIHeFTl1ITyEDetm7GkXOz7azKuX8 VJYHrpv0ojRCEKO+X38mCXndj58LFrZQtSDqnhb+9jsK3F7aFr/tkU7uRE85GFGUewwA SEYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=rcseCZP2x1rpmxdurCQy/lUNoOhUFfcgvsvbOvu3pN0=; b=V3D5cU9fKq4D+JTxHhC6SNJoz8qQAl7iY466cyISjCJ4JafrhE7KqHTajegzMrbLSA ZCtbOXnjBfrzEsad14lrpXaExkjtBEl3tJOCsvAPUxtOcV4jcZDRgUd7aQ5B9GmO0G5P NW9C1ftbIACTC5MqVQxD7Tyk0Q4I6uIym9SFeI3bbp4Zzam7ces+FuihLz/qwvp9lmYM r8a7mpUi6q1XK8J+HpABT1J7hdzjmq6gLk0wb+l7lg0im+2s5jMP97SpzbttdiqxjXzD AxvWcnqDTe85ArtAssrTu3QJ4JriqkDvwBS+hdM3ayVbWL9tNBbRToZo3uurtxjwSCzy TwOg== X-Gm-Message-State: AOAM532t9ZS2DsqIg7Ny3eNxTYVscrTqMb6YxYn6jzwQMPQlDf/Tl8pj /1BueXGI3VK7eIkiS+0tfroeb/41j0E= X-Google-Smtp-Source: ABdhPJyp8Lhaid4weGL1tsolxiEzijLW1yFZDKwA0dBYitwVFQRFshlYllB0lW+ZVFZwygPSjzlboA== X-Received: by 2002:adf:fcc7:: with SMTP id f7mr19060752wrs.420.1612199716403; Mon, 01 Feb 2021 09:15:16 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id q6sm21207166wmj.32.2021.02.01.09.15.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 09:15:15 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 01 Feb 2021 17:15:06 +0000 Subject: [PATCH 4/5] commit-graph: be extra careful about mixed generations Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: me@ttaylorr.com, peff@peff.net, gister@pobox.com, abhishekkumar8222@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When upgrading to a commit-graph with corrected commit dates from one without, there are a few things that need to be considered. When computing generation numbers for the new commit-graph file that expects to add the generation_data chunk with corrected commit dates, we need to ensure that the 'generation' member of the commit_graph_data struct is set to zero for these commits. Unfortunately, the fallback to use topological level for generation number when corrected commit dates are not available are causing us harm here: parsing commits notices that read_generation_data is false and populates 'generation' with the topological level. The solution is to iterate through the commits, parse the commits to populate initial values, then reset the generation values to zero to trigger recalculation. This loop only occurs when the existing commit-graph data has no corrected commit dates. While this improves our situation somewhat, we have not completely solved the issue for correctly computing generation numbers for mixes layers. That follows in the next change. Signed-off-by: Derrick Stolee --- commit-graph.c | 32 +++++++++++++++++++++++--------- 1 file changed, 23 insertions(+), 9 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index 13992137dd0..08148dd17f1 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1033,7 +1033,8 @@ struct write_commit_graph_context { split:1, changed_paths:1, order_by_pack:1, - write_generation_data:1; + write_generation_data:1, + trust_generation_numbers:1; struct topo_level_slab *topo_levels; const struct commit_graph_opts *opts; @@ -1452,6 +1453,15 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx) ctx->progress = start_delayed_progress( _("Computing commit graph generation numbers"), ctx->commits.nr); + + if (ctx->write_generation_data && !ctx->trust_generation_numbers) { + for (i = 0; i < ctx->commits.nr; i++) { + struct commit *c = ctx->commits.list[i]; + repo_parse_commit(ctx->r, c); + commit_graph_data_at(c)->generation = GENERATION_NUMBER_ZERO; + } + } + for (i = 0; i < ctx->commits.nr; i++) { struct commit *c = ctx->commits.list[i]; uint32_t level; @@ -1480,7 +1490,8 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx) corrected_commit_date = commit_graph_data_at(parent->item)->generation; if (level == GENERATION_NUMBER_ZERO || - corrected_commit_date == GENERATION_NUMBER_ZERO) { + (ctx->write_generation_data && + corrected_commit_date == GENERATION_NUMBER_ZERO)) { all_parents_computed = 0; commit_list_insert(parent->item, &list); break; @@ -1500,12 +1511,15 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx) max_level = GENERATION_NUMBER_V1_MAX - 1; *topo_level_slab_at(ctx->topo_levels, current) = max_level + 1; - if (current->date && current->date > max_corrected_commit_date) - max_corrected_commit_date = current->date - 1; - commit_graph_data_at(current)->generation = max_corrected_commit_date + 1; - - if (commit_graph_data_at(current)->generation - current->date > GENERATION_NUMBER_V2_OFFSET_MAX) - ctx->num_generation_data_overflows++; + if (ctx->write_generation_data) { + timestamp_t cur_g; + if (current->date && current->date > max_corrected_commit_date) + max_corrected_commit_date = current->date - 1; + cur_g = commit_graph_data_at(current)->generation + = max_corrected_commit_date + 1; + if (cur_g - current->date > GENERATION_NUMBER_V2_OFFSET_MAX) + ctx->num_generation_data_overflows++; + } } } } @@ -2396,7 +2410,7 @@ int write_commit_graph(struct object_directory *odb, } else ctx->num_commit_graphs_after = 1; - validate_mixed_generation_chain(ctx->r->objects->commit_graph); + ctx->trust_generation_numbers = validate_mixed_generation_chain(ctx->r->objects->commit_graph); compute_generation_numbers(ctx); From patchwork Mon Feb 1 17:15:07 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12059495 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7CF66C433E6 for ; Mon, 1 Feb 2021 17:16:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 48C9764EA2 for ; Mon, 1 Feb 2021 17:16:03 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231776AbhBARQA (ORCPT ); Mon, 1 Feb 2021 12:16:00 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36268 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231754AbhBARP7 (ORCPT ); Mon, 1 Feb 2021 12:15:59 -0500 Received: from mail-wr1-x42b.google.com (mail-wr1-x42b.google.com [IPv6:2a00:1450:4864:20::42b]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id D769CC06178A for ; Mon, 1 Feb 2021 09:15:18 -0800 (PST) Received: by mail-wr1-x42b.google.com with SMTP id v15so17476029wrx.4 for ; Mon, 01 Feb 2021 09:15:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=eyNZXSqDn2Bp9zHj7N/Pnedr4Sz3TiBbmMc7XJKdqSo=; b=PI8DgN4Ham44R0UtHGMKNznYtv1mNbESWjYwuqzT7wQt6x9CU0tZwc0YKpAglKAKH5 l5djG1uKv4uqdUZYzVGAjBHxjTiFSfSUBCOwz4JLtm9RCAUxKqDj1+nemrv+qFLeoSn0 w+HXAFEu+JXlDJJlM+xk+BvpuY2VV+HzyUbeUle9juVzxJBuRd32Lewx7EcWzpgFQFSM RX/yVkW2jSVC/p86e5NJG7OBrQZLd/ZHBSd5ThhXYt+lvf7S5G8OPtl+OwUKt+J90B0R nY2Ab+ZRfh49Fimz/6PT0Mgq8ha6kpKufVDDj9SqpaA+at11gh8jOvFcBpX8/xkJmPfn r+Dg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=eyNZXSqDn2Bp9zHj7N/Pnedr4Sz3TiBbmMc7XJKdqSo=; b=UzqQT1ZBJjNWQ1+3QCDFCPK6Y0yWFV3eq28XqG42iwDiYXWWzyMMzvm3KWVOoHQ4sT f6/4ZOQw835cB6s//R74V7CVzImDLvl8lFGR7ZzPDRqEilmYU359Z84bKmdUzS0FKJR6 vTP0KjW49RBFaMqxdY0u2g+Nq0M0rN3BmVbx65OTuEp7n6LPq+7pfg7H8Gb+Mi1y8Xte ClEZhBeFUjQjWNrDv/T/cicJd+G3PHyR+8if6QD/NMzaaxIgiN2wJKJVYLzVQgcGxn/0 XBzdhqb4xWD33R27PQKCqZeKhKSU5DtxNVm2fsBvU/U/f511juXMFrKsgSa3JSD/jV5p WGXg== X-Gm-Message-State: AOAM5307MrLNw8TY6lRs+1T1l0s3/frcHZsKa1Rf2nDJUq9dIgb2uzaR H9eC8EsX1bxR4HOiGbjCA9is1Ledgng= X-Google-Smtp-Source: ABdhPJzyWc60mh+r4ceRmIvTFWJjIB3+Ztizicv0xhGhPI8STF8CmpOR3siZmUddHRvCPbBjnWniZQ== X-Received: by 2002:adf:f452:: with SMTP id f18mr18821493wrp.11.1612199717421; Mon, 01 Feb 2021 09:15:17 -0800 (PST) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k6sm28635109wro.27.2021.02.01.09.15.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Feb 2021 09:15:16 -0800 (PST) Message-Id: In-Reply-To: References: Date: Mon, 01 Feb 2021 17:15:07 +0000 Subject: [PATCH 5/5] commit-graph: prepare commit graph Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: me@ttaylorr.com, peff@peff.net, gister@pobox.com, abhishekkumar8222@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Before checking if the repository has a commit-graph loaded, be sure to run prepare_commit_graph(). This is necessary because without this instance we would not initialize the topo_levels slab for each of the struct commit_graphs in the chain before we start to parse the commits. This leads to possibly recomputing the topological levels for commits in lower layers even when we are adding a small number of commits on top. By properly initializing the topo_slab, we fix the previously broken case of a split commit graph where a base layer has the generation_data_overflow chunk. Signed-off-by: Derrick Stolee --- commit-graph.c | 10 ++-------- t/t5318-commit-graph.sh | 2 +- 2 files changed, 3 insertions(+), 9 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index 08148dd17f1..858fa5594e3 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -2309,6 +2309,7 @@ int write_commit_graph(struct object_directory *odb, init_topo_level_slab(&topo_levels); ctx->topo_levels = &topo_levels; + prepare_commit_graph(ctx->r); if (ctx->r->objects->commit_graph) { struct commit_graph *g = ctx->r->objects->commit_graph; @@ -2322,7 +2323,6 @@ int write_commit_graph(struct object_directory *odb, ctx->changed_paths = 1; if (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) { struct commit_graph *g; - prepare_commit_graph_one(ctx->r, ctx->odb); g = ctx->r->objects->commit_graph; @@ -2334,10 +2334,7 @@ int write_commit_graph(struct object_directory *odb, } if (ctx->split) { - struct commit_graph *g; - prepare_commit_graph(ctx->r); - - g = ctx->r->objects->commit_graph; + struct commit_graph *g = ctx->r->objects->commit_graph; while (g) { ctx->num_commit_graphs_before++; @@ -2361,9 +2358,6 @@ int write_commit_graph(struct object_directory *odb, ctx->approx_nr_objects = approximate_object_count(); - if (ctx->append) - prepare_commit_graph_one(ctx->r, ctx->odb); - if (ctx->append && ctx->r->objects->commit_graph) { struct commit_graph *g = ctx->r->objects->commit_graph; for (i = 0; i < g->num_commits; i++) { diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh index 2cf29f425a0..5b15f1aa0f6 100755 --- a/t/t5318-commit-graph.sh +++ b/t/t5318-commit-graph.sh @@ -446,7 +446,7 @@ test_expect_success 'warn on improper hash version' ' ) ' -test_expect_failure 'lower layers have overflow chunk' ' +test_expect_success 'lower layers have overflow chunk' ' cd "$TRASH_DIRECTORY/full" && UNIX_EPOCH_ZERO="@0 +0000" && FUTURE_DATE="@2147483646 +0000" &&