From patchwork Fri Oct 5 21:31:20 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10628767 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1257515A6 for ; Fri, 5 Oct 2018 21:31:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0178729C56 for ; Fri, 5 Oct 2018 21:31:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EA10529C5A; Fri, 5 Oct 2018 21:31:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CE66A29C56 for ; Fri, 5 Oct 2018 21:31:40 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727475AbeJFEcN (ORCPT ); Sat, 6 Oct 2018 00:32:13 -0400 Received: from mail-pl1-f202.google.com ([209.85.214.202]:33297 "EHLO mail-pl1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725896AbeJFEcM (ORCPT ); Sat, 6 Oct 2018 00:32:12 -0400 Received: by mail-pl1-f202.google.com with SMTP id g6-v6so11574102plo.0 for ; Fri, 05 Oct 2018 14:31:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=8Uppbwd0luknABY1misofZPsHdAAq25GdpUtk/w+bec=; b=F1KW3ha/DAN4/ND68jUoP1TuHwLhEwzR+hmBcRXKyAUy6xFpZ7DAyyYwyh67RYTh7G 5q1PLq0+6iY2kKOb4pOXP/PTC8UQ2WNlH+KCYx9BFNca/wqh0IzokXPsPD2ZN3xTU3qu s0uyv39j3dUSGfjqMWg4R7t85Yrn1+NBbqRR479XNiQLyXB2o3t289pkBqoUc3levoWX 83fxSaw56jkA29BFFceV+G973o17WueHH9c9tCvtbAAI6r+gr99flojZlHfOmW1UF9dp QixrIsNbjuwS8nz755Vd1JZmsUgQ457MpfIgpOvYKbD7rNpTyLOjKIFTL+8cPQfMkYoA Qq5w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=8Uppbwd0luknABY1misofZPsHdAAq25GdpUtk/w+bec=; b=Lj78RmCIFXQPBazHmadX5BKIdDcpmGEPix/wFnf0aD33TTCTQiV5O1f7b4EvURU7dr GZpGGDNoymm88RLP8hjNUpChIARfuSkyzgraAJ1z+GPuNNhJmwbyFR9rLC4vBw+hS/Uf JN6Ib5VMHHdkXXj+YdphCgdWyWSTDx4SJqgEUY1c+jRUMqHPS9lSFtltH2OkD25brzy0 ubvlyU2RLaw01YlKBCxhAfB6vrSECatMI1iEiVPAniUN9dPeaUOfkIhICF4sRI2D5dsq IFiIc1tjgPp6TQ5s9m6TZiQlLRYxunxL3Iw+OKMBecX//sZU00+zWi5oQVpg4ZV0FZvm cVFA== X-Gm-Message-State: ABuFfoiruKidmRSRxf7Z+YlLoZeWT4HT/xPQi0lp0Vg6+Kqp4lQRfVXZ DqGZ+Hhr7EeOLoBulksIBHpnAw150SDiukvmmuYEYVi6tIWL908rKx78frDsxehoDe+zNw7a+Qi U22nOQLyqffQ5FxwNNWdKqWB5DVs3TL0d5/izFUcdYfogl/vTu81WJt1+9JQ= X-Google-Smtp-Source: ACcGV60RfcaprymgxpkkrJkvuVL73NE4kGm487XU6520pNptwBakkWsGnUZsWcCqz3EsNMYn2ceEI2XQZH20 X-Received: by 2002:a63:7247:: with SMTP id c7-v6mr5385713pgn.3.1538775098449; Fri, 05 Oct 2018 14:31:38 -0700 (PDT) Date: Fri, 5 Oct 2018 14:31:20 -0700 In-Reply-To: Message-Id: <3bff094363c7887bfe6547e520cbe83097cf6f73.1538774738.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v11 1/8] list-objects: store common func args in struct From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This will make utility functions easier to create, as done by the next patch. Signed-off-by: Matthew DeVore --- list-objects.c | 158 +++++++++++++++++++++++-------------------------- 1 file changed, 74 insertions(+), 84 deletions(-) diff --git a/list-objects.c b/list-objects.c index c99c47ac1..584518a3f 100644 --- a/list-objects.c +++ b/list-objects.c @@ -12,20 +12,25 @@ #include "packfile.h" #include "object-store.h" -static void process_blob(struct rev_info *revs, +struct traversal_context { + struct rev_info *revs; + show_object_fn show_object; + show_commit_fn show_commit; + void *show_data; + filter_object_fn filter_fn; + void *filter_data; +}; + +static void process_blob(struct traversal_context *ctx, struct blob *blob, - show_object_fn show, struct strbuf *path, - const char *name, - void *cb_data, - filter_object_fn filter_fn, - void *filter_data) + const char *name) { struct object *obj = &blob->object; size_t pathlen; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; - if (!revs->blob_objects) + if (!ctx->revs->blob_objects) return; if (!obj) die("bad blob object"); @@ -41,21 +46,21 @@ static void process_blob(struct rev_info *revs, * may cause the actual filter to report an incomplete list * of missing objects. */ - if (revs->exclude_promisor_objects && + if (ctx->revs->exclude_promisor_objects && !has_object_file(&obj->oid) && is_promisor_object(&obj->oid)) return; pathlen = path->len; strbuf_addstr(path, name); - if (!(obj->flags & USER_GIVEN) && filter_fn) - r = filter_fn(LOFS_BLOB, obj, - path->buf, &path->buf[pathlen], - filter_data); + if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + r = ctx->filter_fn(LOFS_BLOB, obj, + path->buf, &path->buf[pathlen], + ctx->filter_data); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - show(obj, path->buf, cb_data); + ctx->show_object(obj, path->buf, ctx->show_data); strbuf_setlen(path, pathlen); } @@ -81,26 +86,21 @@ static void process_blob(struct rev_info *revs, * the link, and how to do it. Whether it necessarily makes * any sense what-so-ever to ever do that is another issue. */ -static void process_gitlink(struct rev_info *revs, +static void process_gitlink(struct traversal_context *ctx, const unsigned char *sha1, - show_object_fn show, struct strbuf *path, - const char *name, - void *cb_data) + const char *name) { /* Nothing to do */ } -static void process_tree(struct rev_info *revs, +static void process_tree(struct traversal_context *ctx, struct tree *tree, - show_object_fn show, struct strbuf *base, - const char *name, - void *cb_data, - filter_object_fn filter_fn, - void *filter_data) + const char *name) { struct object *obj = &tree->object; + struct rev_info *revs = ctx->revs; struct tree_desc desc; struct name_entry entry; enum interesting match = revs->diffopt.pathspec.nr == 0 ? @@ -133,14 +133,14 @@ static void process_tree(struct rev_info *revs, } strbuf_addstr(base, name); - if (!(obj->flags & USER_GIVEN) && filter_fn) - r = filter_fn(LOFS_BEGIN_TREE, obj, - base->buf, &base->buf[baselen], - filter_data); + if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + r = ctx->filter_fn(LOFS_BEGIN_TREE, obj, + base->buf, &base->buf[baselen], + ctx->filter_data); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - show(obj, base->buf, cb_data); + ctx->show_object(obj, base->buf, ctx->show_data); if (base->len) strbuf_addch(base, '/'); @@ -157,29 +157,25 @@ static void process_tree(struct rev_info *revs, } if (S_ISDIR(entry.mode)) - process_tree(revs, + process_tree(ctx, lookup_tree(the_repository, entry.oid), - show, base, entry.path, - cb_data, filter_fn, filter_data); + base, entry.path); else if (S_ISGITLINK(entry.mode)) - process_gitlink(revs, entry.oid->hash, - show, base, entry.path, - cb_data); + process_gitlink(ctx, entry.oid->hash, base, entry.path); else - process_blob(revs, + process_blob(ctx, lookup_blob(the_repository, entry.oid), - show, base, entry.path, - cb_data, filter_fn, filter_data); + base, entry.path); } - if (!(obj->flags & USER_GIVEN) && filter_fn) { - r = filter_fn(LOFS_END_TREE, obj, - base->buf, &base->buf[baselen], - filter_data); + if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { + r = ctx->filter_fn(LOFS_END_TREE, obj, + base->buf, &base->buf[baselen], + ctx->filter_data); if (r & LOFR_MARK_SEEN) obj->flags |= SEEN; if (r & LOFR_DO_SHOW) - show(obj, base->buf, cb_data); + ctx->show_object(obj, base->buf, ctx->show_data); } strbuf_setlen(base, baselen); @@ -242,19 +238,15 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree) add_pending_object(revs, &tree->object, ""); } -static void traverse_trees_and_blobs(struct rev_info *revs, - struct strbuf *base, - show_object_fn show_object, - void *show_data, - filter_object_fn filter_fn, - void *filter_data) +static void traverse_trees_and_blobs(struct traversal_context *ctx, + struct strbuf *base) { int i; assert(base->len == 0); - for (i = 0; i < revs->pending.nr; i++) { - struct object_array_entry *pending = revs->pending.objects + i; + for (i = 0; i < ctx->revs->pending.nr; i++) { + struct object_array_entry *pending = ctx->revs->pending.objects + i; struct object *obj = pending->item; const char *name = pending->name; const char *path = pending->path; @@ -262,62 +254,49 @@ static void traverse_trees_and_blobs(struct rev_info *revs, continue; if (obj->type == OBJ_TAG) { obj->flags |= SEEN; - show_object(obj, name, show_data); + ctx->show_object(obj, name, ctx->show_data); continue; } if (!path) path = ""; if (obj->type == OBJ_TREE) { - process_tree(revs, (struct tree *)obj, show_object, - base, path, show_data, - filter_fn, filter_data); + process_tree(ctx, (struct tree *)obj, base, path); continue; } if (obj->type == OBJ_BLOB) { - process_blob(revs, (struct blob *)obj, show_object, - base, path, show_data, - filter_fn, filter_data); + process_blob(ctx, (struct blob *)obj, base, path); continue; } die("unknown pending object %s (%s)", oid_to_hex(&obj->oid), name); } - object_array_clear(&revs->pending); + object_array_clear(&ctx->revs->pending); } -static void do_traverse(struct rev_info *revs, - show_commit_fn show_commit, - show_object_fn show_object, - void *show_data, - filter_object_fn filter_fn, - void *filter_data) +static void do_traverse(struct traversal_context *ctx) { struct commit *commit; struct strbuf csp; /* callee's scratch pad */ strbuf_init(&csp, PATH_MAX); - while ((commit = get_revision(revs)) != NULL) { + while ((commit = get_revision(ctx->revs)) != NULL) { /* * an uninteresting boundary commit may not have its tree * parsed yet, but we are not going to show them anyway */ if (get_commit_tree(commit)) - add_pending_tree(revs, get_commit_tree(commit)); - show_commit(commit, show_data); + add_pending_tree(ctx->revs, get_commit_tree(commit)); + ctx->show_commit(commit, ctx->show_data); - if (revs->tree_blobs_in_commit_order) + if (ctx->revs->tree_blobs_in_commit_order) /* * NEEDSWORK: Adding the tree and then flushing it here * needs a reallocation for each commit. Can we pass the * tree directory without allocation churn? */ - traverse_trees_and_blobs(revs, &csp, - show_object, show_data, - filter_fn, filter_data); + traverse_trees_and_blobs(ctx, &csp); } - traverse_trees_and_blobs(revs, &csp, - show_object, show_data, - filter_fn, filter_data); + traverse_trees_and_blobs(ctx, &csp); strbuf_release(&csp); } @@ -326,7 +305,14 @@ void traverse_commit_list(struct rev_info *revs, show_object_fn show_object, void *show_data) { - do_traverse(revs, show_commit, show_object, show_data, NULL, NULL); + struct traversal_context ctx; + ctx.revs = revs; + ctx.show_commit = show_commit; + ctx.show_object = show_object; + ctx.show_data = show_data; + ctx.filter_fn = NULL; + ctx.filter_data = NULL; + do_traverse(&ctx); } void traverse_commit_list_filtered( @@ -337,14 +323,18 @@ void traverse_commit_list_filtered( void *show_data, struct oidset *omitted) { - filter_object_fn filter_fn = NULL; + struct traversal_context ctx; filter_free_fn filter_free_fn = NULL; - void *filter_data = NULL; - - filter_data = list_objects_filter__init(omitted, filter_options, - &filter_fn, &filter_free_fn); - do_traverse(revs, show_commit, show_object, show_data, - filter_fn, filter_data); - if (filter_data && filter_free_fn) - filter_free_fn(filter_data); + + ctx.revs = revs; + ctx.show_object = show_object; + ctx.show_commit = show_commit; + ctx.show_data = show_data; + ctx.filter_fn = NULL; + + ctx.filter_data = list_objects_filter__init(omitted, filter_options, + &ctx.filter_fn, &filter_free_fn); + do_traverse(&ctx); + if (ctx.filter_data && filter_free_fn) + filter_free_fn(ctx.filter_data); } From patchwork Fri Oct 5 21:31:21 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10628769 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DD8E415A6 for ; Fri, 5 Oct 2018 21:31:43 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CCB5629C56 for ; Fri, 5 Oct 2018 21:31:43 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id BE57529C5A; Fri, 5 Oct 2018 21:31:43 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5C7CB29C56 for ; Fri, 5 Oct 2018 21:31:43 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728139AbeJFEcP (ORCPT ); Sat, 6 Oct 2018 00:32:15 -0400 Received: from mail-io1-f73.google.com ([209.85.166.73]:38142 "EHLO mail-io1-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725896AbeJFEcP (ORCPT ); Sat, 6 Oct 2018 00:32:15 -0400 Received: by mail-io1-f73.google.com with SMTP id a13-v6so1291145ioh.5 for ; Fri, 05 Oct 2018 14:31:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=w9PL1mndR6OdE97ALIWJ7/eMgY6Fzt1CJXNhuiZ9jBs=; b=C6K7fRdhgfXl/GRHzc4B2xP8py5k75p23FWRFtfmpn4glmKUFhf/YTMRCsOx4NWU3n CwfWpoOCHSq3zijEtFfG6kjQgOQ5BUnewk0QBYjrDPkcpmYduDbYal+alY14XsjU338C SXqcu6rZJOAjJlobevYznPE8Py5YnzhCTSYuHem6r6Vshre7meOdY+2sozFw/o3JBWXJ v6D5iiyGYS/cesfVZP8d33TWvYCqaYMF8imP9Zc7z0CKHqie4LT6mZ6mB5AHBnpSRUqx 05dq/k+9ZpKuqYdJzVC5xUZhd+94BwUz9WqEzNfRqVeK5jotvqfruiHE3ph/zDoR3WwH 23DQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=w9PL1mndR6OdE97ALIWJ7/eMgY6Fzt1CJXNhuiZ9jBs=; b=AAGfZh+teuwQGKh74LyVSsx5xgK5RVfthLt5JIcp8DQMz0JKK6pYuagBYhbfbE6Buc kQ9NS664snpdXfBejc7iK/cV/gLxFvfn2acJGM2khnQpW9wmXKE2nW0sjXEAvNP6ufh5 KJcqYl04ujr6gVVNFcPCBNHngZSvi/oA9486vnUPpm2Uuie231t3tRoLnFChktWW08nb hgQyP5uAX9nzeqNm+kcZZeH7cP5P0ym4snAZMFQ2Tsz7vRYLmeLEu5ioAolQDz+ZnpVW 8NnuE4pahG7IT0EAHTQlZZgVQZNkB/3OGWEOVW1hY3aSTl5VLnIDYkrb711baqhVjQhH bzpw== X-Gm-Message-State: ABuFfoj9Spc4AOk1un2WhRBFJsgXM8+RJH9z2DjoeGdjs47zwrjgKJHa EcPLA5AXeDt8gkKRbP+vWkkczPJ3mRzonZQkuRC+8LZ47obFBMkYB0vjgNRhcCotfxXVWOLbbxE HpBoAp9pBT3pO5D8eShWZ3CeSuZJglg/o2LZYBNo948or5B/BlcwykJ1dDDU= X-Google-Smtp-Source: ACcGV61h2Uza+KKq7hnsPGOF8UoDWFZuBDRd0/X4pU3Pwmw5FDLjvt7UihH7R3JLBuIZWNMT5j3b49JgUybB X-Received: by 2002:a24:7996:: with SMTP id z144-v6mr9372090itc.10.1538775100902; Fri, 05 Oct 2018 14:31:40 -0700 (PDT) Date: Fri, 5 Oct 2018 14:31:21 -0700 In-Reply-To: Message-Id: <42ad5623bab52b8207831eab4a4f27e47a318e36.1538774738.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v11 2/8] list-objects: refactor to process_tree_contents From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This will be used in a follow-up patch to reduce indentation needed when invoking the logic conditionally. i.e. rather than: if (foo) { while (...) { /* this is very indented */ } } we will have: if (foo) process_tree_contents(...); Signed-off-by: Matthew DeVore --- list-objects.c | 68 ++++++++++++++++++++++++++++++-------------------- 1 file changed, 41 insertions(+), 27 deletions(-) diff --git a/list-objects.c b/list-objects.c index 584518a3f..ccc529e5e 100644 --- a/list-objects.c +++ b/list-objects.c @@ -94,6 +94,46 @@ static void process_gitlink(struct traversal_context *ctx, /* Nothing to do */ } +static void process_tree(struct traversal_context *ctx, + struct tree *tree, + struct strbuf *base, + const char *name); + +static void process_tree_contents(struct traversal_context *ctx, + struct tree *tree, + struct strbuf *base) +{ + struct tree_desc desc; + struct name_entry entry; + enum interesting match = ctx->revs->diffopt.pathspec.nr == 0 ? + all_entries_interesting : entry_not_interesting; + + init_tree_desc(&desc, tree->buffer, tree->size); + + while (tree_entry(&desc, &entry)) { + if (match != all_entries_interesting) { + match = tree_entry_interesting(&entry, base, 0, + &ctx->revs->diffopt.pathspec); + if (match == all_entries_not_interesting) + break; + if (match == entry_not_interesting) + continue; + } + + if (S_ISDIR(entry.mode)) + process_tree(ctx, + lookup_tree(the_repository, entry.oid), + base, entry.path); + else if (S_ISGITLINK(entry.mode)) + process_gitlink(ctx, entry.oid->hash, + base, entry.path); + else + process_blob(ctx, + lookup_blob(the_repository, entry.oid), + base, entry.path); + } +} + static void process_tree(struct traversal_context *ctx, struct tree *tree, struct strbuf *base, @@ -101,10 +141,6 @@ static void process_tree(struct traversal_context *ctx, { struct object *obj = &tree->object; struct rev_info *revs = ctx->revs; - struct tree_desc desc; - struct name_entry entry; - enum interesting match = revs->diffopt.pathspec.nr == 0 ? - all_entries_interesting: entry_not_interesting; int baselen = base->len; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; int gently = revs->ignore_missing_links || @@ -144,29 +180,7 @@ static void process_tree(struct traversal_context *ctx, if (base->len) strbuf_addch(base, '/'); - init_tree_desc(&desc, tree->buffer, tree->size); - - while (tree_entry(&desc, &entry)) { - if (match != all_entries_interesting) { - match = tree_entry_interesting(&entry, base, 0, - &revs->diffopt.pathspec); - if (match == all_entries_not_interesting) - break; - if (match == entry_not_interesting) - continue; - } - - if (S_ISDIR(entry.mode)) - process_tree(ctx, - lookup_tree(the_repository, entry.oid), - base, entry.path); - else if (S_ISGITLINK(entry.mode)) - process_gitlink(ctx, entry.oid->hash, base, entry.path); - else - process_blob(ctx, - lookup_blob(the_repository, entry.oid), - base, entry.path); - } + process_tree_contents(ctx, tree, base); if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { r = ctx->filter_fn(LOFS_END_TREE, obj, From patchwork Fri Oct 5 21:31:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10628771 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7BE3514BD for ; Fri, 5 Oct 2018 21:31:45 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6BEBB29C56 for ; Fri, 5 Oct 2018 21:31:45 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 5FD3829C5A; Fri, 5 Oct 2018 21:31:45 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 00F4129C56 for ; Fri, 5 Oct 2018 21:31:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728367AbeJFEcR (ORCPT ); Sat, 6 Oct 2018 00:32:17 -0400 Received: from mail-pf1-f202.google.com ([209.85.210.202]:41937 "EHLO mail-pf1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725896AbeJFEcR (ORCPT ); Sat, 6 Oct 2018 00:32:17 -0400 Received: by mail-pf1-f202.google.com with SMTP id 87-v6so10644024pfq.8 for ; Fri, 05 Oct 2018 14:31:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=lRBW4T+Uj1qpBIE5Q/etFfBP2/bdQ1ZPeAB+hIuj5tY=; b=A4BeheiDpJERQoFqW+k6MqoeDF2pDQlRy0exoBIfJKckbuxI5hE5vYOBLp/BxOJP2o RyPzZlQFSdp82h8Rx/khfB76M7MHUOxH+GcLzp73pMPt/bTXWYu1ejaMiHpN7oyq+lJw wYOQ6vwQZrTmOT2FSMDpWKoEc9TFZ+qaR5gSUR3m6xZEsXTTdgHfZ945ZWK8CQgrdmHv poNn19iR2dHth4pslO3vW0y8dGCoS3Oev+8L5H8Bh9W2BZUy2dKIx3GSnJ9POk9HTfMY Kj2PK1AgS8sk8/1t9RAtXPy5lwCJZPx/YV8blZ5JBwoOkCBc1BzG3G/EkMHq3EdWPtLL afWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=lRBW4T+Uj1qpBIE5Q/etFfBP2/bdQ1ZPeAB+hIuj5tY=; b=MivlBGXLzS4mYFZzZf//Gjy/OvctyVPmWRD8R+Y//l3BMm0OCbEkdqHs+bumHR5o3g Px7/ZZCoEb3VoV6QWVnwxXHpQ0j7Ox44Di+Lm6ZsSC41L6K9jqzJPYQ62z8auNsqGtJt o4sByDVT03+eNuWhK+ftIaOo/zhltxFL/VJReATXsFt6Jm49RbVBN/w64vtn9ElCJJ9V J7d55OTaUanK0Y/ITIvvZfmqafDriHhasI5wA/q+QU+yXLN2jLsF9MP+unoF2fzVPWQt D0sdwnNWI1ekxYB4mAPg8yT/HwcyrzeiZEOexKaq4S68JrpmVdiHGKHqmC3T7kO06zfF vWOw== X-Gm-Message-State: ABuFfohY8A0AmyXKQRjEWcAQYFNnjto9n4pEc8GPZ/xf1YHsMkSKZlnv +Yq+8Qumdw1YMojr60FSWtV0FMiwPkwMug2rQD2tW6bO2bB2v80QU7xcTz7McaO1X+Svsmdct1+ AI2n5nnEV9Xc74P5ZCp28U7CuZmWAS6NksrOjMwj9cpQ2WpdkGL1VSckbiGs= X-Google-Smtp-Source: ACcGV628h1kSm9t/Hvbu4RMxsAy0cRhmnGcfyfPdC6qaxY9NT1Nh6LqXhxtgRNvCOGVw2e/OYZmEXSN/CyQX X-Received: by 2002:a63:f553:: with SMTP id e19-v6mr5820679pgk.68.1538775103037; Fri, 05 Oct 2018 14:31:43 -0700 (PDT) Date: Fri, 5 Oct 2018 14:31:22 -0700 In-Reply-To: Message-Id: <0d1285a0ec412846e5417ec524a866f6b2c39d82.1538774738.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v11 3/8] list-objects: always parse trees gently From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If parsing fails when revs->ignore_missing_links and revs->exclude_promisor_objects are both false, we print the OID anyway in the die("bad tree object...") call, so any message printed by parse_tree_gently() is superfluous. Signed-off-by: Matthew DeVore --- list-objects.c | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/list-objects.c b/list-objects.c index ccc529e5e..f9b51db7a 100644 --- a/list-objects.c +++ b/list-objects.c @@ -143,8 +143,6 @@ static void process_tree(struct traversal_context *ctx, struct rev_info *revs = ctx->revs; int baselen = base->len; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; - int gently = revs->ignore_missing_links || - revs->exclude_promisor_objects; if (!revs->tree_objects) return; @@ -152,7 +150,7 @@ static void process_tree(struct traversal_context *ctx, die("bad tree object"); if (obj->flags & (UNINTERESTING | SEEN)) return; - if (parse_tree_gently(tree, gently) < 0) { + if (parse_tree_gently(tree, 1) < 0) { if (revs->ignore_missing_links) return; From patchwork Fri Oct 5 21:31:23 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10628773 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 191CC15A6 for ; Fri, 5 Oct 2018 21:31:49 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 05D1229C56 for ; Fri, 5 Oct 2018 21:31:49 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E8C9429C5A; Fri, 5 Oct 2018 21:31:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2D9B129C56 for ; Fri, 5 Oct 2018 21:31:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728510AbeJFEcU (ORCPT ); Sat, 6 Oct 2018 00:32:20 -0400 Received: from mail-it1-f202.google.com ([209.85.166.202]:44353 "EHLO mail-it1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725896AbeJFEcT (ORCPT ); Sat, 6 Oct 2018 00:32:19 -0400 Received: by mail-it1-f202.google.com with SMTP id b124-v6so3658230itb.9 for ; Fri, 05 Oct 2018 14:31:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=6tgI1NdJGIodijE3NvslojDkAWDPYbod2qW4gr54CuI=; b=pTHNjBVWAEQTp3BGRibBTEo2Rm+wo2TQtdur8npnsZ+AhvRHQOjsm4En5N6oxTOfUs MEZrOohE0R4J81I8ONTcAEF9JIddw2/KJafML749P6jSchA/p+xqcpGRXyFYmfD8ROBP KeSRNZtLQEALoZiJGqw7qIGjDpSuL5ITE1GJSQiGiLc5NsYDrJ8u81tS/LeqFXDaPKSc fvpOqOBbPa9rTextDWi1gB1mP5+p6jJL/fdOu2J1YUBBNCCB3W/uue6hGkpT5Wl1+mVH wMq2n7aqZ/mJiOQAl9RtL1IjpZSrRbnnb4A4Y2gWowKpTxWM6T2vREe+A3rPgs+MbtOr 23iw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=6tgI1NdJGIodijE3NvslojDkAWDPYbod2qW4gr54CuI=; b=XYgUjeMZuyJstcwsuPtuIemIyQGZq7kR6fskpdCP5TLrQCAbFGOO2KZcJf5juI28sa vnLy+10vfuzCt0lEyCjhdX0BB7OTJwBB9wpODya12z5FWn+vEOqLtzmvaYpEI2zEzP/D HOa0azYULjAhQBmHWfyKlWyem69QxKLiUPSgCxAwn6zegwIPnmV70UdnEZqe9FYrWkRV xRcEF2ZY3yinbrK02KKlD/1PR+HANln8+HfLuQx+Cfh1VrRiUm9VXH2qPefWqQgns/EM RM6VZjzq3Ki9H2FR0Rfxa1OHXnqel3FTrTqDOGFedvHMi1iYtcvGIfT6nBmjB/zXv3y8 qu4g== X-Gm-Message-State: ABuFfohCyOjR+t0TUhG+Op+WqIMFV8r6L6FO0s21SmeJkYiwkfgPuw+m yZbUOGx5C8T9KA7IhTviCwpSHuecNoEb9ywwQLhtiyS64ukkLiZON/JDi9n4s51Xrw/S1LKGAML O+GmmXkUZPPFSPui4Xc961uToOvF4xUulVyTtV60loPWXl/58OO6aiRoSN80= X-Google-Smtp-Source: ACcGV63ajDSSArUx4H8N2FMUyrXrDi806qTKv3oOwRQDVxcCV9XOd5UBFO+loih/MxSvGH0fWrn3KHO1QeN9 X-Received: by 2002:a24:2752:: with SMTP id g79-v6mr9587079ita.4.1538775105377; Fri, 05 Oct 2018 14:31:45 -0700 (PDT) Date: Fri, 5 Oct 2018 14:31:23 -0700 In-Reply-To: Message-Id: <793739b7df9c7f46e7b2e6a49640b44266415924.1538774738.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v11 4/8] rev-list: handle missing tree objects properly From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Previously, we assumed only blob objects could be missing. This patch makes rev-list handle missing trees like missing blobs. The --missing=* and --exclude-promisor-objects flags now work for trees as they already do for blobs. This is demonstrated in t6112. Signed-off-by: Matthew DeVore --- builtin/rev-list.c | 11 ++++--- list-objects.c | 11 +++++-- revision.h | 15 +++++++++ t/t0410-partial-clone.sh | 45 ++++++++++++++++++++++++++ t/t5317-pack-objects-filter-objects.sh | 13 ++++++++ t/t6112-rev-list-filters-objects.sh | 22 +++++++++++++ 6 files changed, 110 insertions(+), 7 deletions(-) diff --git a/builtin/rev-list.c b/builtin/rev-list.c index 5b07f3f4a..49d6deed7 100644 --- a/builtin/rev-list.c +++ b/builtin/rev-list.c @@ -6,6 +6,7 @@ #include "list-objects.h" #include "list-objects-filter.h" #include "list-objects-filter-options.h" +#include "object.h" #include "object-store.h" #include "pack.h" #include "pack-bitmap.h" @@ -209,7 +210,8 @@ static inline void finish_object__ma(struct object *obj) */ switch (arg_missing_action) { case MA_ERROR: - die("missing blob object '%s'", oid_to_hex(&obj->oid)); + die("missing %s object '%s'", + type_name(obj->type), oid_to_hex(&obj->oid)); return; case MA_ALLOW_ANY: @@ -222,8 +224,8 @@ static inline void finish_object__ma(struct object *obj) case MA_ALLOW_PROMISOR: if (is_promisor_object(&obj->oid)) return; - die("unexpected missing blob object '%s'", - oid_to_hex(&obj->oid)); + die("unexpected missing %s object '%s'", + type_name(obj->type), oid_to_hex(&obj->oid)); return; default: @@ -235,7 +237,7 @@ static inline void finish_object__ma(struct object *obj) static int finish_object(struct object *obj, const char *name, void *cb_data) { struct rev_list_info *info = cb_data; - if (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) { + if (!has_object_file(&obj->oid)) { finish_object__ma(obj); return 1; } @@ -373,6 +375,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix) init_revisions(&revs, prefix); revs.abbrev = DEFAULT_ABBREV; revs.commit_format = CMIT_FMT_UNSPECIFIED; + revs.do_not_die_on_missing_tree = 1; /* * Scan the argument list before invoking setup_revisions(), so that we diff --git a/list-objects.c b/list-objects.c index f9b51db7a..243192af5 100644 --- a/list-objects.c +++ b/list-objects.c @@ -143,6 +143,7 @@ static void process_tree(struct traversal_context *ctx, struct rev_info *revs = ctx->revs; int baselen = base->len; enum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW; + int failed_parse; if (!revs->tree_objects) return; @@ -150,7 +151,9 @@ static void process_tree(struct traversal_context *ctx, die("bad tree object"); if (obj->flags & (UNINTERESTING | SEEN)) return; - if (parse_tree_gently(tree, 1) < 0) { + + failed_parse = parse_tree_gently(tree, 1); + if (failed_parse) { if (revs->ignore_missing_links) return; @@ -163,7 +166,8 @@ static void process_tree(struct traversal_context *ctx, is_promisor_object(&obj->oid)) return; - die("bad tree object %s", oid_to_hex(&obj->oid)); + if (!revs->do_not_die_on_missing_tree) + die("bad tree object %s", oid_to_hex(&obj->oid)); } strbuf_addstr(base, name); @@ -178,7 +182,8 @@ static void process_tree(struct traversal_context *ctx, if (base->len) strbuf_addch(base, '/'); - process_tree_contents(ctx, tree, base); + if (!failed_parse) + process_tree_contents(ctx, tree, base); if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { r = ctx->filter_fn(LOFS_END_TREE, obj, diff --git a/revision.h b/revision.h index 007278cc1..5910613cb 100644 --- a/revision.h +++ b/revision.h @@ -126,6 +126,21 @@ struct rev_info { line_level_traverse:1, tree_blobs_in_commit_order:1, + /* + * Blobs are shown without regard for their existence. + * But not so for trees: unless exclude_promisor_objects + * is set and the tree in question is a promisor object; + * OR ignore_missing_links is set, the revision walker + * dies with a "bad tree object HASH" message when + * encountering a missing tree. For callers that can + * handle missing trees and want them to be filterable + * and showable, set this to true. The revision walker + * will filter and show such a missing tree as usual, + * but will not attempt to recurse into this tree + * object. + */ + do_not_die_on_missing_tree:1, + /* for internal use only */ exclude_promisor_objects:1; diff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh index 128130066..5bc5b4445 100755 --- a/t/t0410-partial-clone.sh +++ b/t/t0410-partial-clone.sh @@ -186,6 +186,51 @@ test_expect_success 'rev-list stops traversal at missing and promised commit' ' ! grep $FOO out ' +test_expect_success 'missing tree objects with --missing=allow-promisor and --exclude-promisor-objects' ' + rm -rf repo && + test_create_repo repo && + test_commit -C repo foo && + test_commit -C repo bar && + test_commit -C repo baz && + + promise_and_delete $(git -C repo rev-parse bar^{tree}) && + promise_and_delete $(git -C repo rev-parse foo^{tree}) && + + git -C repo config core.repositoryformatversion 1 && + git -C repo config extensions.partialclone "arbitrary string" && + + git -C repo rev-list --missing=allow-promisor --objects HEAD >objs 2>rev_list_err && + test_must_be_empty rev_list_err && + # 3 commits, 3 blobs, and 1 tree + test_line_count = 7 objs && + + # Do the same for --exclude-promisor-objects, but with all trees gone. + promise_and_delete $(git -C repo rev-parse baz^{tree}) && + git -C repo rev-list --exclude-promisor-objects --objects HEAD >objs 2>rev_list_err && + test_must_be_empty rev_list_err && + # 3 commits, no blobs or trees + test_line_count = 3 objs +' + +test_expect_success 'missing non-root tree object and rev-list' ' + rm -rf repo && + test_create_repo repo && + mkdir repo/dir && + echo foo >repo/dir/foo && + git -C repo add dir/foo && + git -C repo commit -m "commit dir/foo" && + + promise_and_delete $(git -C repo rev-parse HEAD:dir) && + + git -C repo config core.repositoryformatversion 1 && + git -C repo config extensions.partialclone "arbitrary string" && + + git -C repo rev-list --missing=allow-any --objects HEAD >objs 2>rev_list_err && + test_must_be_empty rev_list_err && + # 1 commit and 1 tree + test_line_count = 2 objs +' + test_expect_success 'rev-list stops traversal at missing and promised tree' ' rm -rf repo && test_create_repo repo && diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh index 6710c8bc8..9839b48c1 100755 --- a/t/t5317-pack-objects-filter-objects.sh +++ b/t/t5317-pack-objects-filter-objects.sh @@ -59,6 +59,19 @@ test_expect_success 'verify normal and blob:none packfiles have same commits/tre test_cmp observed expected ' +test_expect_success 'get an error for missing tree object' ' + git init r5 && + echo foo >r5/foo && + git -C r5 add foo && + git -C r5 commit -m "foo" && + del=$(git -C r5 rev-parse HEAD^{tree} | sed "s|..|&/|") && + rm r5/.git/objects/$del && + test_must_fail git -C r5 pack-objects --rev --stdout 2>bad_tree <<-EOF && + HEAD + EOF + grep -q "bad tree object" bad_tree +' + # Test blob:limit=[kmg] filter. # We boundary test around the size parameter. The filter is strictly less than # the value, so size 500 and 1000 should have the same results, but 1001 should diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index d4ff0b3be..efe5a2467 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -195,6 +195,28 @@ test_expect_success 'verify sparse:oid=oid-ish omits top-level files' ' test_cmp observed expected ' +test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for trees' ' + TREE=$(git -C r3 rev-parse HEAD:dir1) && + + # Create a spare repo because we will be deleting objects from this one. + git clone r3 r3.b && + + rm r3.b/.git/objects/$(echo $TREE | sed "s|^..|&/|") && + + git -C r3.b rev-list --quiet --missing=print --objects HEAD \ + >missing_objs 2>rev_list_err && + echo "?$TREE" >expected && + test_cmp expected missing_objs && + + # do not complain when a missing tree cannot be parsed + test_must_be_empty rev_list_err && + + git -C r3.b rev-list --missing=allow-any --objects HEAD \ + >objs 2>rev_list_err && + ! grep $TREE objs && + test_must_be_empty rev_list_err +' + # Delete some loose objects and use rev-list, but WITHOUT any filtering. # This models previously omitted objects that we did not receive. From patchwork Fri Oct 5 21:31:24 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10628775 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D33F915A6 for ; Fri, 5 Oct 2018 21:31:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id C1A6329C56 for ; Fri, 5 Oct 2018 21:31:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B53CB29C5A; Fri, 5 Oct 2018 21:31:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2BE9329C56 for ; Fri, 5 Oct 2018 21:31:50 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728572AbeJFEcW (ORCPT ); Sat, 6 Oct 2018 00:32:22 -0400 Received: from mail-pf1-f201.google.com ([209.85.210.201]:52110 "EHLO mail-pf1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725896AbeJFEcW (ORCPT ); Sat, 6 Oct 2018 00:32:22 -0400 Received: by mail-pf1-f201.google.com with SMTP id b22-v6so10592330pfc.18 for ; Fri, 05 Oct 2018 14:31:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ZKuKNSP057nSvDjXzFGLi35kG3kKSmLWoYm+UknR7i4=; b=Ta+LYlGTa9gUOrT/zjY0wKLWckYqNP4zSbuGGItFDKJfo656GnbBzOBvb3SsUPSWkM pewm4dAmHLYaYfYMc6w2uPhkHQH4jMk6rR9zGY1wRZsURqBpPYpOyoozfmeM6+p+kMxW 8akh+GgLbL9GCinNtaJYUzihgeyoYicOKpcAU8Ij4RH3RNFniLGGOUfTkGa3BJDOYEc2 XHl3KANf0nLLctuWlSwzd2ajA5ym+QB0j5RR45lO77InmQEmsb7whWctbtPocEaoUj1B X09zJWA5AIcDKc+gokAlvL4oZUbI/HTeFdGdVvFT/kAsxcdHStI7yPi65Xksi4pyPsVz M7yw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ZKuKNSP057nSvDjXzFGLi35kG3kKSmLWoYm+UknR7i4=; b=UUEmEYIcckiJaMlwU7MlsoJR2LgTOgc8YDEkwtCo0EILVLOyQM1tUMILG5aQ8vC1Ur D3+kfkFaVZM0vtSzjIuA76CmQTDyKCQmZ+iaiv6/iM+FG5NAubNb/XdT9aa/nMQ1/ayM bmM9kkcayCPLjNv7aFjygcDsz5GQMzh0dJB2/sR2sR++zzT6GCS7f+5KQBxTG/rluccE pozcZxrfgivRcM6DsmDz3fZ2UxUdC6bCmjRaTHRRJQTgUlIhVlnO2ITcXomkUlF9/ej5 ALzVT2WR82Lf3TqOKDqKmNSByTssSFrSlHGW9GUjHPe4kicKgjl2st1yYIEA2YQsGHwt /J9g== X-Gm-Message-State: ABuFfogpW/aWoln2M3Cx9ASDQcoHc71R8Du3CXM1feHf8qu5U8EbVbr8 Q/ug+P49EWm2KrQrEKX6Mh3HoDbuBfcAQgFhKx7VPFL2q6MBHbVBYQXRH/g8D2MqBRv5T0fiUrr O3QKOxrKoLxKX/5n49lZXc3W1tPvPXFTjISJaNWy6ZHjH5nuLoM3Zj0qRxw0= X-Google-Smtp-Source: ACcGV61J1dSRqOIu4XtjCMs+rXVgIjPusly0LV9uQgho3260hfd80jTE7vQI6EpH4+X8naqvFSUePZQQ5Jj3 X-Received: by 2002:a62:4853:: with SMTP id v80-v6mr6173842pfa.10.1538775107597; Fri, 05 Oct 2018 14:31:47 -0700 (PDT) Date: Fri, 5 Oct 2018 14:31:24 -0700 In-Reply-To: Message-Id: <9ef4d3ada1f9ce89abd529c83137625469eaf920.1538774738.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v11 5/8] revision: mark non-user-given objects instead From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Currently, list-objects.c incorrectly treats all root trees of commits as USER_GIVEN. Also, it would be easier to mark objects that are non-user-given instead of user-given, since the places in the code where we access an object through a reference are more obvious than the places where we access an object that was given by the user. Resolve these two problems by introducing a flag NOT_USER_GIVEN that marks blobs and trees that are non-user-given, replacing USER_GIVEN. (Only blobs and trees are marked because this mark is only used when filtering objects, and filtering of other types of objects is not supported yet.) This fixes a bug in that git rev-list behaved differently from git pack-objects. pack-objects would *not* filter objects given explicitly on the command line and rev-list would filter. This was because the two commands used a different function to add objects to the rev_info struct. This seems to have been an oversight, and pack-objects has the correct behavior, so I added a test to make sure that rev-list now behaves properly. Signed-off-by: Matthew DeVore --- list-objects.c | 31 +++++++++++++++++------------ revision.c | 1 - revision.h | 11 ++++++++-- t/t6112-rev-list-filters-objects.sh | 12 +++++++++++ 4 files changed, 39 insertions(+), 16 deletions(-) diff --git a/list-objects.c b/list-objects.c index 243192af5..7a1a0929d 100644 --- a/list-objects.c +++ b/list-objects.c @@ -53,7 +53,7 @@ static void process_blob(struct traversal_context *ctx, pathlen = path->len; strbuf_addstr(path, name); - if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) r = ctx->filter_fn(LOFS_BLOB, obj, path->buf, &path->buf[pathlen], ctx->filter_data); @@ -120,17 +120,19 @@ static void process_tree_contents(struct traversal_context *ctx, continue; } - if (S_ISDIR(entry.mode)) - process_tree(ctx, - lookup_tree(the_repository, entry.oid), - base, entry.path); + if (S_ISDIR(entry.mode)) { + struct tree *t = lookup_tree(the_repository, entry.oid); + t->object.flags |= NOT_USER_GIVEN; + process_tree(ctx, t, base, entry.path); + } else if (S_ISGITLINK(entry.mode)) process_gitlink(ctx, entry.oid->hash, base, entry.path); - else - process_blob(ctx, - lookup_blob(the_repository, entry.oid), - base, entry.path); + else { + struct blob *b = lookup_blob(the_repository, entry.oid); + b->object.flags |= NOT_USER_GIVEN; + process_blob(ctx, b, base, entry.path); + } } } @@ -171,7 +173,7 @@ static void process_tree(struct traversal_context *ctx, } strbuf_addstr(base, name); - if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) + if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) r = ctx->filter_fn(LOFS_BEGIN_TREE, obj, base->buf, &base->buf[baselen], ctx->filter_data); @@ -185,7 +187,7 @@ static void process_tree(struct traversal_context *ctx, if (!failed_parse) process_tree_contents(ctx, tree, base); - if (!(obj->flags & USER_GIVEN) && ctx->filter_fn) { + if ((obj->flags & NOT_USER_GIVEN) && ctx->filter_fn) { r = ctx->filter_fn(LOFS_END_TREE, obj, base->buf, &base->buf[baselen], ctx->filter_data); @@ -301,8 +303,11 @@ static void do_traverse(struct traversal_context *ctx) * an uninteresting boundary commit may not have its tree * parsed yet, but we are not going to show them anyway */ - if (get_commit_tree(commit)) - add_pending_tree(ctx->revs, get_commit_tree(commit)); + if (get_commit_tree(commit)) { + struct tree *tree = get_commit_tree(commit); + tree->object.flags |= NOT_USER_GIVEN; + add_pending_tree(ctx->revs, tree); + } ctx->show_commit(commit, ctx->show_data); if (ctx->revs->tree_blobs_in_commit_order) diff --git a/revision.c b/revision.c index de4dce600..72d48a17f 100644 --- a/revision.c +++ b/revision.c @@ -175,7 +175,6 @@ static void add_pending_object_with_path(struct rev_info *revs, strbuf_release(&buf); return; /* do not add the commit itself */ } - obj->flags |= USER_GIVEN; add_object_array_with_path(obj, name, &revs->pending, mode, path); } diff --git a/revision.h b/revision.h index 5910613cb..83e164039 100644 --- a/revision.h +++ b/revision.h @@ -21,9 +21,16 @@ #define SYMMETRIC_LEFT (1u<<8) #define PATCHSAME (1u<<9) #define BOTTOM (1u<<10) -#define USER_GIVEN (1u<<25) /* given directly by the user */ +/* + * Indicates object was reached by traversal. i.e. not given by user on + * command-line or stdin. + * NEEDSWORK: NOT_USER_GIVEN doesn't apply to commits because we only support + * filtering trees and blobs, but it may be useful to support filtering commits + * in the future. + */ +#define NOT_USER_GIVEN (1u<<25) #define TRACK_LINEAR (1u<<26) -#define ALL_REV_FLAGS (((1u<<11)-1) | USER_GIVEN | TRACK_LINEAR) +#define ALL_REV_FLAGS (((1u<<11)-1) | NOT_USER_GIVEN | TRACK_LINEAR) #define DECORATE_SHORT_REFS 1 #define DECORATE_FULL_REFS 2 diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index efe5a2467..ccbc64413 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -30,6 +30,18 @@ test_expect_success 'verify blob:none omits all 5 blobs' ' test_cmp observed expected ' +test_expect_success 'specify blob explicitly prevents filtering' ' + file_3=$(git -C r1 ls-files -s file.3 | + awk -f print_2.awk) && + + file_4=$(git -C r1 ls-files -s file.4 | + awk -f print_2.awk) && + + git -C r1 rev-list --objects --filter=blob:none HEAD $file_3 >observed && + grep -q "$file_3" observed && + test_must_fail grep -q "$file_4" observed +' + test_expect_success 'verify emitted+omitted == all' ' git -C r1 rev-list HEAD --objects \ | awk -f print_1.awk \ From patchwork Fri Oct 5 21:31:25 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10628777 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 129FC14BD for ; Fri, 5 Oct 2018 21:31:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0239E29C56 for ; Fri, 5 Oct 2018 21:31:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id EAA4A29C5A; Fri, 5 Oct 2018 21:31:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9491C29C56 for ; Fri, 5 Oct 2018 21:31:52 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728595AbeJFEcY (ORCPT ); Sat, 6 Oct 2018 00:32:24 -0400 Received: from mail-yw1-f74.google.com ([209.85.161.74]:41749 "EHLO mail-yw1-f74.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725896AbeJFEcY (ORCPT ); Sat, 6 Oct 2018 00:32:24 -0400 Received: by mail-yw1-f74.google.com with SMTP id d23-v6so8280774ywd.8 for ; Fri, 05 Oct 2018 14:31:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=eo+qxEhYTUv3+lsEELI+YrG0h/e6PzpHh4BhzyK/WVc=; b=ETYmtsedM9mqSAX+wVOWfEJmpcRlDSPUgT2ISTsO96hzXPzH/uXklVb3JCU01TsvhM 8oedMY16g89zj3LaFUKDbEkkuhoQbjtpl8JTIrj61GCCBruyjMP+akeI9Wj4t4NE5zC7 7Doy4I5p1+gQDlp+8V20CfxSASGOeVJOYQlRzvGfFBXLy+j8yts2m1PdpeNaVK8YNclZ W/nXsrqsEKio9Zk1lK5ZjjBNe3BbxePv2y1C+yoAn5S2lWbwisUHhu4YjWQWvIyhoNiw RpO8KiCvyUHbIdKgilzsTogw5Lqm3RDqjZZVyC5cPz33WcvPkn2Dx3gl+zXvXWb2K8Fs K0yg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=eo+qxEhYTUv3+lsEELI+YrG0h/e6PzpHh4BhzyK/WVc=; b=XhFPhAGj9bTYYNKRBw/SMmiD16u0BLF+da78iCHYwYPBk9/outfnfQv2JoOYGsfMD8 rXVndkINTcSoopHBisNQUauv8ImGkYfEe7TAFKTjTP91FOLBobe2AMOjoB29XRX5iT9q UVl7MDposbVoHOrYV2Y0kFsq8Q9QzzYCi37kFqlYeo48XRDW51H2dX9lzPZRMqHHdMvf 5ST1XQV9Yv4Q0aIu8RoFaYc4g8QibvuMYumyUtL4VgtyxVcPICCkf8SSw0JdhHg6MAsw k/vNJ3ARsRiVwnsVB9o+7AoG3+acxi8VqzT1+hQj5Ar3hjctbcp2Qt97/4R+Y52rWBk8 sLwA== X-Gm-Message-State: ABuFfoi3gny9l6vFQj8EvguLrpRmhtuvmQVi0p4mB7WN5V6FydfRQV9P GxwuMXIJkn4ubgZNKv+DWRE79JNBrgnBerfGG/ywBKxHjf0RHFx+XJNwKCZTJTAIcZ25gbZbbbw UAOCCF1acpvyxvDT9HQSGfWhzASI3XogVrZKECO0C56byM4wVMpEfgdFBdks= X-Google-Smtp-Source: ACcGV61idjskI2uLtfqVa30PecLJxgbzbU6qbHUJyZYItVWc4IQL6pXalHBgqZoM2H1H3VMUqIuFKgw00Y8A X-Received: by 2002:a25:bdc8:: with SMTP id g8-v6mr7943362ybk.101.1538775110120; Fri, 05 Oct 2018 14:31:50 -0700 (PDT) Date: Fri, 5 Oct 2018 14:31:25 -0700 In-Reply-To: Message-Id: <5b337881c3b6cf0f7a9483e578f613f59c609c44.1538774738.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v11 6/8] list-objects-filter: use BUG rather than die From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In some cases in this file, BUG makes more sense than die. In such cases, a we get there from a coding error rather than a user error. 'return' has been removed following some instances of BUG since BUG does not return. Signed-off-by: Matthew DeVore --- list-objects-filter.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-) diff --git a/list-objects-filter.c b/list-objects-filter.c index a0ba78b20..5f8b1a002 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -44,8 +44,7 @@ static enum list_objects_filter_result filter_blobs_none( switch (filter_situation) { default: - die("unknown filter_situation"); - return LOFR_ZERO; + BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); @@ -102,8 +101,7 @@ static enum list_objects_filter_result filter_blobs_limit( switch (filter_situation) { default: - die("unknown filter_situation"); - return LOFR_ZERO; + BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); @@ -208,8 +206,7 @@ static enum list_objects_filter_result filter_sparse( switch (filter_situation) { default: - die("unknown filter_situation"); - return LOFR_ZERO; + BUG("unknown filter_situation: %d", filter_situation); case LOFS_BEGIN_TREE: assert(obj->type == OBJ_TREE); @@ -389,7 +386,7 @@ void *list_objects_filter__init( assert((sizeof(s_filters) / sizeof(s_filters[0])) == LOFC__COUNT); if (filter_options->choice >= LOFC__COUNT) - die("invalid list-objects filter choice: %d", + BUG("invalid list-objects filter choice: %d", filter_options->choice); init_fn = s_filters[filter_options->choice]; From patchwork Fri Oct 5 21:31:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10628779 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4A88514BD for ; Fri, 5 Oct 2018 21:31:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 3A21C29C56 for ; Fri, 5 Oct 2018 21:31:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 2E79629C5A; Fri, 5 Oct 2018 21:31:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id D36B729C56 for ; Fri, 5 Oct 2018 21:31:54 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728662AbeJFEc1 (ORCPT ); Sat, 6 Oct 2018 00:32:27 -0400 Received: from mail-it1-f201.google.com ([209.85.166.201]:43661 "EHLO mail-it1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725896AbeJFEc1 (ORCPT ); Sat, 6 Oct 2018 00:32:27 -0400 Received: by mail-it1-f201.google.com with SMTP id w137-v6so3679513itc.8 for ; Fri, 05 Oct 2018 14:31:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=PQw5VNRqY67VEQ0o3B/Q/ISsDEZKysHNiMYTvo+2YUI=; b=fMvmXWTZeAfRL0So3JRopHarqNmc6wZVtfbxUY/mj/O2N+NyhXXSsI210HJUh8rdta mFDz493GCRKG8XxOD+2VMDX9JiTPjtOPuMuHpoLaGR2bJkaboxlG3vXa5oza2scy7w4b +IeVAS2KBNPTwNBNVz8r3QMzQ5Dlyq+14+O3ufD2uO8+FMa0jCB5adwiq7bKS4kKXU9m 6zDMmmxGz6p5DMQTSsgGo1JGpn+77gmSCRR8hSoF+EMTza6gc2ko1nIF4DpMezKIS39D 2kRjvzzckYOXCza4KhF/Xmu66zwB6E+NrOBgcN5e+bT33BwHnyzQs84SJy82x082yDxB YtNA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=PQw5VNRqY67VEQ0o3B/Q/ISsDEZKysHNiMYTvo+2YUI=; b=aQB5YR/WGXVGkLXVMYqh78sqBdgRBciZn5lg5m4Ase1VvBuPzSdSrdR+2f/ZQyNShG cyMiXFdFh4fhrt2253qJfPewpzDq8BkAjI1/aiA2EN6Wu+Nur+/l0pEro7wzMBWjNMTL R+u5OCYlhyi/v+Pz7dki+zwROoEdK6rb3tPVar3rh428hgYZZctu9bK+NNXhivmeUEp2 8XbAndOFyGz95RtyGuAfar12l0X6PCbr7tQDQE0mTU6nUfR8je3A1PJiu1N5rI1X5Oyi v339kE1TtDFxYIlwAL+PG//jc/mGFfGS63nPF/J05OsqvPJusCyTgLaptXtKKhsYI5OE kyiA== X-Gm-Message-State: ABuFfoiiEXDN+W3i3olIBGFX/UB1WOoSrMPfFCsIOVepGXhg22iC53rW znLxAm6R79UWVLR4Mv4F2Cc/kybMqeG22Oc+gyIrOaK9cb6RkQye66EjlJbtoGQNB4mbiYHNYdX VcTd/BD5+haWVZWH6WI83g2ksdBllQPB9F1OerO822vmLWzbqRaDxvgR+sAI= X-Google-Smtp-Source: ACcGV61c8oDEmnR/eBsTRKkmRAHH/b/aCuflb+uvFVoY/OzMrDbp2fhczvxrj61KTU8LAW7Cv/tkgdqQuo0I X-Received: by 2002:a24:d49:: with SMTP id 70-v6mr9712668itx.5.1538775112526; Fri, 05 Oct 2018 14:31:52 -0700 (PDT) Date: Fri, 5 Oct 2018 14:31:26 -0700 In-Reply-To: Message-Id: <1a055a109adadaca6eff89514089fbde739d600c.1538774738.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v11 7/8] list-objects-filter-options: do not over-strbuf_init From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The function gently_parse_list_objects_filter is either called with errbuf=STRBUF_INIT or errbuf=NULL, but that function calls strbuf_init when errbuf is not NULL. strbuf_init is only necessary if errbuf contains garbage, and risks a memory leak if errbuf already has a non-STRBUF_INIT state. It should be the caller's responsibility to make sure errbuf is not garbage, since garbage content is easily avoidable with STRBUF_INIT. Signed-off-by: Matthew DeVore --- list-objects-filter-options.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index c0e2bd6a0..d259bdb2c 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -30,7 +30,6 @@ static int gently_parse_list_objects_filter( if (filter_options->choice) { if (errbuf) { - strbuf_init(errbuf, 0); strbuf_addstr( errbuf, _("multiple filter-specs cannot be combined")); @@ -71,10 +70,9 @@ static int gently_parse_list_objects_filter( return 0; } - if (errbuf) { - strbuf_init(errbuf, 0); + if (errbuf) strbuf_addf(errbuf, "invalid filter-spec '%s'", arg); - } + memset(filter_options, 0, sizeof(*filter_options)); return 1; } From patchwork Fri Oct 5 21:31:27 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Matthew DeVore X-Patchwork-Id: 10628781 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 62A0B14BD for ; Fri, 5 Oct 2018 21:31:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 528A629C56 for ; Fri, 5 Oct 2018 21:31:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 4712D29C5A; Fri, 5 Oct 2018 21:31:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 6D43029C56 for ; Fri, 5 Oct 2018 21:31:57 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728826AbeJFEc3 (ORCPT ); Sat, 6 Oct 2018 00:32:29 -0400 Received: from mail-it1-f201.google.com ([209.85.166.201]:51765 "EHLO mail-it1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725896AbeJFEc3 (ORCPT ); Sat, 6 Oct 2018 00:32:29 -0400 Received: by mail-it1-f201.google.com with SMTP id u2-v6so3742840ith.1 for ; Fri, 05 Oct 2018 14:31:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=Lp0eSAdRXCT9/FhKMrpAha2q1c+QYgylCXEowJj2B4Y=; b=hVTnPduh/43zBCoq4n7TtObgQSJnzwtto+o1zzr2B9RH15NaQLEKiMVY1+pb61Hnr4 xB0TzygYQjTjPLjWgk7BM0e/xI1FqGUzXsaWfKuA+elE7NJlAfyPnGLuTRKxXV/fsIHn 1wprKDe1rdIvmhKikEvpa+fbodFhlN3ks68BZ5vcP4/i4ds9inlr6M+gl1WvPjT6y2ut t0pB2iuVAVSoQADq83LCTbIVQ/kNtzcav6mR1mCCC/6zqHMQul7Jb6Oo1c/faC6okABs mD099XFEWC61lAn7YAQT9ofF9ecmwfPb9IOLzVOaN/WyL/lRVAiZ0I5cH3SKrkpqERfK 4ZfQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=Lp0eSAdRXCT9/FhKMrpAha2q1c+QYgylCXEowJj2B4Y=; b=DOjPlfjXGA/N1TACI8+/49z69wEdTizAdUGWfznUMI963/oa9PNntUnmqWrXUMBa1i CVF5xLZG1jllq6E6EzM/5DaX4jV5XXihfilzegZxW1dhVTyVkfBKhe/5hlMeWNiVlOgg 6bC3iy8HGWBOVK1otLUeAgI2S73+7BTCXUtfFEwM0wlmzNzEg7YXMAVgVKDlYfSueQGg LbI+6OUZlMubyE7FCeu01uOyxoUUgR7Gnw4TECZV9KliWBdFn2HgnKhm9f65xGIDdCzz cuwyqe8wW3trKl00UFfHoKmxJB46tvmQKs5ipVlrWr7lwXkGDDizePty2jj+0/ZCgmol Prdw== X-Gm-Message-State: ABuFfog48NkxytsSdWGatGfWhcbbJ3dq2Yr2aA7Z/nJtNWaeWvyEmT+G l0jZ6qf2qRqBnAMOyNJBU/W2LJ+m1BG2VobeO9xG01reNK890a3J0IoLf7HvZEKIQKqiJWSiqET wgGTH0v+CpQO8izbZKpt/1I+/1furs0NhXQlM4hZB/LpNqgcp02AUNkHUGDo= X-Google-Smtp-Source: ACcGV61rBFT19pEkZly/KS1vfxBsydk7Dpg2GXJm/2OgOce+NHdaNjvB1YEUOf5eRjv29DXHHNIpwAALZ1Uw X-Received: by 2002:a24:ee43:: with SMTP id b64-v6mr13588310iti.3.1538775114980; Fri, 05 Oct 2018 14:31:54 -0700 (PDT) Date: Fri, 5 Oct 2018 14:31:27 -0700 In-Reply-To: Message-Id: <6ca50a28e292f32127ea706a2aef39f834ac7702.1538774738.git.matvore@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.605.g01d371f741-goog Subject: [PATCH v11 8/8] list-objects-filter: implement filter tree:0 From: Matthew DeVore To: git@vger.kernel.org Cc: Matthew DeVore , sbeller@google.com, git@jeffhostetler.com, jeffhost@microsoft.com, peff@peff.net, stefanbeller@gmail.com, jonathantanmy@google.com, gitster@pobox.com, pclouds@gmail.com Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Teach list-objects the "tree:0" filter which allows for filtering out all tree and blob objects (unless other objects are explicitly specified by the user). The purpose of this patch is to allow smaller partial clones. The name of this filter - tree:0 - does not explicitly specify that it also filters out all blobs, but this should not cause much confusion because blobs are not at all useful without the trees that refer to them. I also considered only:commits as a name, but this is inaccurate because it suggests that annotated tags are omitted, but actually they are included. The name "tree:0" allows later filtering based on depth, i.e. "tree:1" would filter out all but the root tree and blobs. In order to avoid confusion between 0 and capital O, the documentation was worded in a somewhat round-about way that also hints at this future improvement to the feature. Signed-off-by: Matthew DeVore --- Documentation/rev-list-options.txt | 5 +++ list-objects-filter-options.c | 13 +++++++ list-objects-filter-options.h | 1 + list-objects-filter.c | 49 ++++++++++++++++++++++++++ t/t5317-pack-objects-filter-objects.sh | 28 +++++++++++++++ t/t5616-partial-clone.sh | 42 ++++++++++++++++++++++ t/t6112-rev-list-filters-objects.sh | 15 ++++++++ 7 files changed, 153 insertions(+) diff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt index 7b273635d..5f1672913 100644 --- a/Documentation/rev-list-options.txt +++ b/Documentation/rev-list-options.txt @@ -731,6 +731,11 @@ the requested refs. + The form '--filter=sparse:path=' similarly uses a sparse-checkout specification contained in . ++ +The form '--filter=tree:' omits all blobs and trees whose depth +from the root tree is >= (minimum depth if an object is located +at multiple depths in the commits traversed). Currently, only =0 +is supported, which omits all blobs and trees. --no-filter:: Turn off any previous `--filter=` argument. diff --git a/list-objects-filter-options.c b/list-objects-filter-options.c index d259bdb2c..e8da2e858 100644 --- a/list-objects-filter-options.c +++ b/list-objects-filter-options.c @@ -49,6 +49,19 @@ static int gently_parse_list_objects_filter( return 0; } + } else if (skip_prefix(arg, "tree:", &v0)) { + unsigned long depth; + if (!git_parse_ulong(v0, &depth) || depth != 0) { + if (errbuf) { + strbuf_addstr( + errbuf, + _("only 'tree:0' is supported")); + } + return 1; + } + filter_options->choice = LOFC_TREE_NONE; + return 0; + } else if (skip_prefix(arg, "sparse:oid=", &v0)) { struct object_context oc; struct object_id sparse_oid; diff --git a/list-objects-filter-options.h b/list-objects-filter-options.h index 0000a61f8..af64e5c66 100644 --- a/list-objects-filter-options.h +++ b/list-objects-filter-options.h @@ -10,6 +10,7 @@ enum list_objects_filter_choice { LOFC_DISABLED = 0, LOFC_BLOB_NONE, LOFC_BLOB_LIMIT, + LOFC_TREE_NONE, LOFC_SPARSE_OID, LOFC_SPARSE_PATH, LOFC__COUNT /* must be last */ diff --git a/list-objects-filter.c b/list-objects-filter.c index 5f8b1a002..09b2b05d5 100644 --- a/list-objects-filter.c +++ b/list-objects-filter.c @@ -79,6 +79,54 @@ static void *filter_blobs_none__init( return d; } +/* + * A filter for list-objects to omit ALL trees and blobs from the traversal. + * Can OPTIONALLY collect a list of the omitted OIDs. + */ +struct filter_trees_none_data { + struct oidset *omits; +}; + +static enum list_objects_filter_result filter_trees_none( + enum list_objects_filter_situation filter_situation, + struct object *obj, + const char *pathname, + const char *filename, + void *filter_data_) +{ + struct filter_trees_none_data *filter_data = filter_data_; + + switch (filter_situation) { + default: + BUG("unknown filter_situation: %d", filter_situation); + + case LOFS_BEGIN_TREE: + case LOFS_BLOB: + if (filter_data->omits) + oidset_insert(filter_data->omits, &obj->oid); + return LOFR_MARK_SEEN; /* but not LOFR_DO_SHOW (hard omit) */ + + case LOFS_END_TREE: + assert(obj->type == OBJ_TREE); + return LOFR_ZERO; + + } +} + +static void* filter_trees_none__init( + struct oidset *omitted, + struct list_objects_filter_options *filter_options, + filter_object_fn *filter_fn, + filter_free_fn *filter_free_fn) +{ + struct filter_trees_none_data *d = xcalloc(1, sizeof(*d)); + d->omits = omitted; + + *filter_fn = filter_trees_none; + *filter_free_fn = free; + return d; +} + /* * A filter for list-objects to omit large blobs. * And to OPTIONALLY collect a list of the omitted OIDs. @@ -371,6 +419,7 @@ static filter_init_fn s_filters[] = { NULL, filter_blobs_none__init, filter_blobs_limit__init, + filter_trees_none__init, filter_sparse_oid__init, filter_sparse_path__init, }; diff --git a/t/t5317-pack-objects-filter-objects.sh b/t/t5317-pack-objects-filter-objects.sh index 9839b48c1..510d3537f 100755 --- a/t/t5317-pack-objects-filter-objects.sh +++ b/t/t5317-pack-objects-filter-objects.sh @@ -72,6 +72,34 @@ test_expect_success 'get an error for missing tree object' ' grep -q "bad tree object" bad_tree ' +test_expect_success 'setup for tests of tree:0' ' + mkdir r1/subtree && + echo "This is a file in a subtree" >r1/subtree/file && + git -C r1 add subtree/file && + git -C r1 commit -m subtree +' + +test_expect_success 'verify tree:0 packfile has no blobs or trees' ' + git -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF && + HEAD + EOF + git -C r1 index-pack ../commitsonly.pack && + git -C r1 verify-pack -v ../commitsonly.pack >objs && + ! grep -E "tree|blob" objs +' + +test_expect_success 'grab tree directly when using tree:0' ' + # We should get the tree specified directly but not its blobs or subtrees. + git -C r1 pack-objects --rev --stdout --filter=tree:0 >commitsonly.pack <<-EOF && + HEAD: + EOF + git -C r1 index-pack ../commitsonly.pack && + git -C r1 verify-pack -v ../commitsonly.pack >objs && + awk "/tree|blob/{print \$1}" objs >trees_and_blobs && + git -C r1 rev-parse HEAD: >expected && + test_cmp expected trees_and_blobs +' + # Test blob:limit=[kmg] filter. # We boundary test around the size parameter. The filter is strictly less than # the value, so size 500 and 1000 should have the same results, but 1001 should diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh index bbbe7537d..53fbf7db8 100755 --- a/t/t5616-partial-clone.sh +++ b/t/t5616-partial-clone.sh @@ -154,6 +154,48 @@ test_expect_success 'partial clone with transfer.fsckobjects=1 uses index-pack - grep "git index-pack.*--fsck-objects" trace ' +test_expect_success 'use fsck before and after manually fetching a missing subtree' ' + # push new commit so server has a subtree + mkdir src/dir && + echo "in dir" >src/dir/file.txt && + git -C src add dir/file.txt && + git -C src commit -m "file in dir" && + git -C src push -u srv master && + SUBTREE=$(git -C src rev-parse HEAD:dir) && + + rm -rf dst && + git clone --no-checkout --filter=tree:0 "file://$(pwd)/srv.bare" dst && + git -C dst fsck && + + # Make sure we only have commits, and all trees and blobs are missing. + git -C dst rev-list --missing=allow-any --objects master \ + >fetched_objects && + awk -f print_1.awk fetched_objects | + xargs -n1 git -C dst cat-file -t >fetched_types && + + sort -u fetched_types >unique_types.observed && + echo commit >unique_types.expected && + test_cmp unique_types.expected unique_types.observed && + + # Auto-fetch a tree with cat-file. + git -C dst cat-file -p $SUBTREE >tree_contents && + grep file.txt tree_contents && + + # fsck still works after an auto-fetch of a tree. + git -C dst fsck && + + # Auto-fetch all remaining trees and blobs with --missing=error + git -C dst rev-list --missing=error --objects master >fetched_objects && + test_line_count = 70 fetched_objects && + + awk -f print_1.awk fetched_objects | + xargs -n1 git -C dst cat-file -t >fetched_types && + + sort -u fetched_types >unique_types.observed && + printf "blob\ncommit\ntree\n" >unique_types.expected && + test_cmp unique_types.expected unique_types.observed +' + test_expect_success 'partial clone fetches blobs pointed to by refs even if normally filtered out' ' rm -rf src dst && git init src && diff --git a/t/t6112-rev-list-filters-objects.sh b/t/t6112-rev-list-filters-objects.sh index ccbc64413..c8e3d87c4 100755 --- a/t/t6112-rev-list-filters-objects.sh +++ b/t/t6112-rev-list-filters-objects.sh @@ -229,6 +229,21 @@ test_expect_success 'rev-list W/ --missing=print and --missing=allow-any for tre test_must_be_empty rev_list_err ' +# Test tree:0 filter. + +test_expect_success 'verify tree:0 includes trees in "filtered" output' ' + git -C r3 rev-list --quiet --objects --filter-print-omitted \ + --filter=tree:0 HEAD >revs && + + awk -f print_1.awk revs | + sed s/~// | + xargs -n1 git -C r3 cat-file -t >unsorted_filtered_types && + + sort -u unsorted_filtered_types >filtered_types && + printf "blob\ntree\n" >expected && + test_cmp expected filtered_types +' + # Delete some loose objects and use rev-list, but WITHOUT any filtering. # This models previously omitted objects that we did not receive.