From patchwork Wed Dec 18 11:25:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300301 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 46BCC109A for ; Wed, 18 Dec 2019 11:26:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2377F21582 for ; Wed, 18 Dec 2019 11:26:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="eQsMALcK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726842AbfLRL0K (ORCPT ); Wed, 18 Dec 2019 06:26:10 -0500 Received: from mail-wm1-f68.google.com ([209.85.128.68]:40113 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725930AbfLRL0J (ORCPT ); Wed, 18 Dec 2019 06:26:09 -0500 Received: by mail-wm1-f68.google.com with SMTP id t14so1472264wmi.5 for ; Wed, 18 Dec 2019 03:26:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=sNKzsfCZv+59wRMREPLBIM/8tQ5sQr8zX7izXDKn/SA=; b=eQsMALcK/dRQLorbjSjDRyx6N+6NqletazGega37VISC9YV3WVb/hGyovVrykbPqJ0 m/Bek/YpfaU8/C0f+H0T8pZbX5BdF5wbPNwxZcEwhwrWEjtGUP2IhaIrLdCQHDkp0DFN nOB6wW1ATWSrhLpTUdlHp3hK4SAkDYZzaS8v3hboC2UBLEp/w2DmBenjItOi/66qR135 7mqM/yzIcR7trnuqLqlM9zwpCxDAlfqtGHG9X20HGg5Gqv5e5QFPdfvSo0vZjDDTPsxH kX2us09MjEzQBf2R8IUa0VKWqpff4Mg476pXfTJy7Kj5Nn8EPubKj2wbDAKrjhEiFcvj sGog== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=sNKzsfCZv+59wRMREPLBIM/8tQ5sQr8zX7izXDKn/SA=; b=nVuuJe0VpDzzaCl9zjhHsEq93o3REGgpn2Nm9S0S7HvDxv3v8nvh5wvYqXOtww9PHK x+S9FBVkpONFmdivc2Ws2fcN12wYPkrux8QLkN7ZKtHdY5EzJ1ce9dUFf0E1fo5Qf51a lhBm3rceS+L4QHc0j2Vcq9Af+doqwKqtu9OTE3vOuPs2QzN96NkC58hLBXUb/2sYutQ/ 8Y4kH0DTzdugYq66ycPWsKzRNBzn9nj+EInX0OmepZKYRM8WMOWe+5/MIqgDciQWyPUK 2v7IdKTstef2M7HHQFoqdDj0ivdjlnQIQh8QUaFw99qolsZboUuIgstAIDiLZb30haSE 6oaw== X-Gm-Message-State: APjAAAWMwbuUcTrEh8Pg+ahV9FHMHdHpkTFHQDYhTpsuatx39no6/q6x MeqAa2iCRXiUUjN8sK/d55IpZVFy X-Google-Smtp-Source: APXvYqxupVZbGsIuOCo8Je4FZpr0JU3/ZNikD4tY3o5HO2wpJgK2Ey5TDjwZf+EQ92dN+9OQmh1HzQ== X-Received: by 2002:a05:600c:2218:: with SMTP id z24mr2698072wml.50.1576668367431; Wed, 18 Dec 2019 03:26:07 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:06 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan , James Ramsay Subject: [PATCH v4 01/12] builtin/pack-objects: report reused packfile objects Date: Wed, 18 Dec 2019 12:25:36 +0100 Message-Id: <20191218112547.4974-2-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King To see when packfile reuse kicks in or not, it is useful to show reused packfile objects statistics in the output of upload-pack. Helped-by: James Ramsay Signed-off-by: Jeff King Signed-off-by: Christian Couder --- builtin/pack-objects.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 393c20a2d7..369f46fbc5 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3509,7 +3509,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) if (progress) fprintf_ln(stderr, _("Total %"PRIu32" (delta %"PRIu32")," - " reused %"PRIu32" (delta %"PRIu32")"), - written, written_delta, reused, reused_delta); + " reused %"PRIu32" (delta %"PRIu32")," + " pack-reused %"PRIu32), + written, written_delta, reused, reused_delta, + reuse_packfile_objects); return 0; } From patchwork Wed Dec 18 11:25:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300305 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 67A9014F6 for ; Wed, 18 Dec 2019 11:26:14 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4658F227BF for ; Wed, 18 Dec 2019 11:26:14 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="tNofeJ3V" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726851AbfLRL0M (ORCPT ); Wed, 18 Dec 2019 06:26:12 -0500 Received: from mail-wr1-f66.google.com ([209.85.221.66]:46341 "EHLO mail-wr1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726825AbfLRL0L (ORCPT ); Wed, 18 Dec 2019 06:26:11 -0500 Received: by mail-wr1-f66.google.com with SMTP id z7so1835224wrl.13 for ; Wed, 18 Dec 2019 03:26:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AAeYwn5IvUi1KRAkbfxvOQoJ/vX9+46kRwTaoV5Nr/U=; b=tNofeJ3V7YLQm3LrGbxVCE7XuNQ9KNi9kC4JiP7iC9F7PZUw9xil4ozKYa8s+F2Yxl Ew4iwVwXs4K14g0jGrFZjw+KGs4VmOTDdCA+E6zXLRQIkk4I45PKRPj2mx2kwdWEoHR+ +HvCfFfehefLGdw0o0RfxVy67mhvFQXrIFlCUW6AwgmwDgcyx3qZ+t1tw2TlMUX482uj rgw5tJT5cA7lPTlV9uHqrApPYKkqIYWx8BsNcqu31li8ZvhGHzvU6nysWFqXdEWORDQZ XIL4ucda4s67xJyGsqotZ9OEML00QAhoTHeLivwKHOOOwka1JRGHiYWQ+w/tnSOswLqt njuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AAeYwn5IvUi1KRAkbfxvOQoJ/vX9+46kRwTaoV5Nr/U=; b=nPpHRTEpJ7B15C2SMbog19Z/a5yVnNsxnhjjg01+bd22GjSnPB4YYLIwJK2ufL971q pUSiRPwhF0y+AONK3xmd5ejT1z0YYpZF9J47T2Y/lN0lF93rSQqxQG+qLL9+MECSVbyj f+j7vCwtByuWn9Uu+6O6AfDlWAyJsaijNChqi5MrjTzlN0ijVK5ZvtO+LnRfv+TMivfl DTJRF1pwcf118sKo49H5P06aZ+awNQ149qZEx1ciz6dVcI/MPy5QrTjjay4+5ypZgz78 lbFy1cVkuNs0gIr3XHB4CxicDf3LNZbunpWw0XQMqTsf8PsOpmfhEfzLAmFIP+DQbcp4 m3+A== X-Gm-Message-State: APjAAAU6pzbTJ2NE9z/Y+qbcrWpi6ehyvakWFclwvI6nmZlhMtXLKP+A EVwCOK6Gj1WPK2gvHc/qXk8qbOpI X-Google-Smtp-Source: APXvYqxsGEnm6M2V2yj20rrbGf9jnveRvxWKI2XD7Q9x3yD2WiOoEzLR77rnDfyxw8vWgciRD1XXAw== X-Received: by 2002:adf:f850:: with SMTP id d16mr2283996wrq.161.1576668368573; Wed, 18 Dec 2019 03:26:08 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:08 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 02/12] packfile: expose get_delta_base() Date: Wed, 18 Dec 2019 12:25:37 +0100 Message-Id: <20191218112547.4974-3-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King In a following commit get_delta_base() will be used outside packfile.c, so let's make it non static and declare it in packfile.h. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- packfile.c | 10 +++++----- packfile.h | 3 +++ 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/packfile.c b/packfile.c index 355066de17..81e66847bf 100644 --- a/packfile.c +++ b/packfile.c @@ -1173,11 +1173,11 @@ const struct packed_git *has_packed_and_bad(struct repository *r, return NULL; } -static off_t get_delta_base(struct packed_git *p, - struct pack_window **w_curs, - off_t *curpos, - enum object_type type, - off_t delta_obj_offset) +off_t get_delta_base(struct packed_git *p, + struct pack_window **w_curs, + off_t *curpos, + enum object_type type, + off_t delta_obj_offset) { unsigned char *base_info = use_pack(p, w_curs, *curpos, NULL); off_t base_offset; diff --git a/packfile.h b/packfile.h index fc7904ec81..ec536a4ae5 100644 --- a/packfile.h +++ b/packfile.h @@ -151,6 +151,9 @@ void *unpack_entry(struct repository *r, struct packed_git *, off_t, enum object unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep); unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t); int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *); +off_t get_delta_base(struct packed_git *p, struct pack_window **w_curs, + off_t *curpos, enum object_type type, + off_t delta_obj_offset); void release_pack_memory(size_t); From patchwork Wed Dec 18 11:25:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300303 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D784E14B7 for ; Wed, 18 Dec 2019 11:26:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B40CE227BF for ; Wed, 18 Dec 2019 11:26:13 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="kMn4MUXG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726860AbfLRL0N (ORCPT ); Wed, 18 Dec 2019 06:26:13 -0500 Received: from mail-wm1-f67.google.com ([209.85.128.67]:39211 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725930AbfLRL0L (ORCPT ); Wed, 18 Dec 2019 06:26:11 -0500 Received: by mail-wm1-f67.google.com with SMTP id 20so1353604wmj.4 for ; Wed, 18 Dec 2019 03:26:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=Eg0SAYG3hzVkghScUnwSMvJzjm05LrtCowdPWgs7dro=; b=kMn4MUXGyiyQ88j+JulZRFGFr+Qa0w63TFGSY6qCh5AiLve7AqvqsIELXplCiOd4Ey c4qjhtbWrgnbzXCjr5/KU43tWSrx25eR5FcKMwTqyYUMWTxoJmw5DP68O0EtX+hocBOk NTg+DXCeI6FoJpnn5SlkVk/2MisCBu0m+rvZqX68LR37sMBxRESVichnhyC+kQXbKScY tMY6WYZhqiM25iczoeus29JbiuRXsaqs/egsagg73W3Pk2Uva26GufIh5lIkQuymCLHX QbisAXKOcpxNbzrTbLii8MoeZJbefNL84Eirfnl0SAgL1ioxSWaxC7y7usE58yQGWYp+ NFlQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=Eg0SAYG3hzVkghScUnwSMvJzjm05LrtCowdPWgs7dro=; b=q1BqJ5b6oCZdmcbp79zmfTjNdHFc7CcL/2isiuJn15Z7gDszh+k3TMXhNHPQpbt4Q7 IXaCIJx1pupZxQWkVSFtUo79kC6408LlX4LtVaKatfFi1KJq7fw0OlCjGvMcEYXWHniS odjPvTmoQ+gPq6WPkg5+dvg2Hv6v96mSPdMyZ67TJs2iKF27OzygazMrD3sEolQ95FFV ZgDY82qUu5GDixiyhGJIb/gi+qbbGfLHg/LfAHKoRVNIgkOq/09Bhv/jWslrNhvHR+sz 41yuYBmfp70L14+6eIDGpOBCGOweDmE/RhjCF/FUhkk0k05sjHlxPgROiNOLghkdU+oY KPiA== X-Gm-Message-State: APjAAAXusDh4VJSOSEN/CWygnqbbuYOdLSNsEHwDyaomj+eN7ExQ4mWG dI1XjMLSOSaNjVdgaxIf6+3kx5zY X-Google-Smtp-Source: APXvYqypzMYtjqhX4GKvMVC5Ck7wzjzhwzrwSFByiL8ILiY0QxTgQlhfbpk0aH9SugyiWgMKzQFtVQ== X-Received: by 2002:a1c:f009:: with SMTP id a9mr2579658wmb.73.1576668369706; Wed, 18 Dec 2019 03:26:09 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:09 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 03/12] ewah/bitmap: introduce bitmap_word_alloc() Date: Wed, 18 Dec 2019 12:25:38 +0100 Message-Id: <20191218112547.4974-4-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King In a following commit we will need to allocate a variable number of bitmap words, instead of always 32, so let's add bitmap_word_alloc() for this purpose. Note that we have to adjust the block growth in bitmap_set(), since a caller could now use an initial size of "0" (we don't plan to do that, but it doesn't hurt to be defensive). Helped-by: Jonathan Tan Signed-off-by: Jeff King Signed-off-by: Christian Couder --- ewah/bitmap.c | 13 +++++++++---- ewah/ewok.h | 1 + 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/ewah/bitmap.c b/ewah/bitmap.c index 52f1178db4..b5fed9621f 100644 --- a/ewah/bitmap.c +++ b/ewah/bitmap.c @@ -22,21 +22,26 @@ #define EWAH_MASK(x) ((eword_t)1 << (x % BITS_IN_EWORD)) #define EWAH_BLOCK(x) (x / BITS_IN_EWORD) -struct bitmap *bitmap_new(void) +struct bitmap *bitmap_word_alloc(size_t word_alloc) { struct bitmap *bitmap = xmalloc(sizeof(struct bitmap)); - bitmap->words = xcalloc(32, sizeof(eword_t)); - bitmap->word_alloc = 32; + bitmap->words = xcalloc(word_alloc, sizeof(eword_t)); + bitmap->word_alloc = word_alloc; return bitmap; } +struct bitmap *bitmap_new(void) +{ + return bitmap_word_alloc(32); +} + void bitmap_set(struct bitmap *self, size_t pos) { size_t block = EWAH_BLOCK(pos); if (block >= self->word_alloc) { size_t old_size = self->word_alloc; - self->word_alloc = block * 2; + self->word_alloc = block ? block * 2 : 1; REALLOC_ARRAY(self->words, self->word_alloc); memset(self->words + old_size, 0x0, (self->word_alloc - old_size) * sizeof(eword_t)); diff --git a/ewah/ewok.h b/ewah/ewok.h index 84b2a29faa..1b98b57c8b 100644 --- a/ewah/ewok.h +++ b/ewah/ewok.h @@ -172,6 +172,7 @@ struct bitmap { }; struct bitmap *bitmap_new(void); +struct bitmap *bitmap_word_alloc(size_t word_alloc); void bitmap_set(struct bitmap *self, size_t pos); int bitmap_get(struct bitmap *self, size_t pos); void bitmap_reset(struct bitmap *self); From patchwork Wed Dec 18 11:25:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300307 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 879F0109A for ; Wed, 18 Dec 2019 11:26:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 65632227BF for ; Wed, 18 Dec 2019 11:26:15 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="DhX8kR6j" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726735AbfLRL0O (ORCPT ); Wed, 18 Dec 2019 06:26:14 -0500 Received: from mail-wm1-f41.google.com ([209.85.128.41]:35293 "EHLO mail-wm1-f41.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726141AbfLRL0N (ORCPT ); Wed, 18 Dec 2019 06:26:13 -0500 Received: by mail-wm1-f41.google.com with SMTP id p17so1491187wmb.0 for ; Wed, 18 Dec 2019 03:26:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=cBvBDYGTTds24xnjho/x6hcXxB6lw9adCvPuXB2U0XU=; b=DhX8kR6jyPN7bl10W7Hir9GQzpCFGit3ppFCgnbDR0gOdWIVCSr4BVnCWEgyl82KSb JUlRzuTq78guvd9yFVMhysQtISBBcyWyTcL4uCw4dvnhWLBzqEeRp00QluN22rz0ct3+ VYM6VC/QTdlmOXNeAxCMF3ueMjjp5M9fNL2+kVsqDRmFVyg+5R68dRUOulkH2xl9J030 F+t2HyKq4hl34spjvPkPHTd5dLiCWH8/xte1qY3s3RlsY/5pyi6awsWoq7zdYhTMnQAI X+ARdBSiqm3U201A98hnM2joZ6/8xJBI741w6ZtIeyWSvOcY43aGgLBHrxgi8Stxlamw 59hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=cBvBDYGTTds24xnjho/x6hcXxB6lw9adCvPuXB2U0XU=; b=NWwNgDQcE5SsMzg4Wwh/qvf49F4J8zb6kwBkGHyqUydQVlF67/3AuHDMK6ElgyLRlE pZXOVGlq5Vg6po06xjfBEERAXvE7r/j56A5yiCYY9CHq/m0JpPWkWzXbkbZRy6+zHffp 304F1uNJdj3chzks6SGWpNXpTH+YkLykpDgp1ZUhclAtk6/t9oQmKuHl1EWfAm2AFoLx p/f5XQaJfLZGh1l50C7C6HNutFZ35S7is+0BU2psusOj+wb4b0P1bepIYHcePbf/VTAH sXV2MtP2h+0Vn+YA+iSeyXzu4B0ocGgrxM7sFJ2YJ6W2mUmAI+2crQTk7W/ikdGNAyQ8 ijyw== X-Gm-Message-State: APjAAAUzqPaiuntiWoYcqwJv58gf6anxD//Zx4Bj8E/MRjwGkrCoIc2r 5+w8p9/4QTKj6F1mXiZqRo3JuTTs X-Google-Smtp-Source: APXvYqyoREQmM/KcZPdKTwNhabqSzZOAx+58FpFZVpWY+MUdO+bX2oi92JRROoIKosKtVmnek0Q9XA== X-Received: by 2002:a1c:9d8b:: with SMTP id g133mr2575470wme.27.1576668370765; Wed, 18 Dec 2019 03:26:10 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:10 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 04/12] pack-bitmap: introduce bitmap_walk_contains() Date: Wed, 18 Dec 2019 12:25:39 +0100 Message-Id: <20191218112547.4974-5-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King We will use this helper function in a following commit to tell us if an object is packed. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- pack-bitmap.c | 12 ++++++++++++ pack-bitmap.h | 3 +++ 2 files changed, 15 insertions(+) diff --git a/pack-bitmap.c b/pack-bitmap.c index e07c798879..fb4f6297f2 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -830,6 +830,18 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, return 0; } +int bitmap_walk_contains(struct bitmap_index *bitmap_git, + struct bitmap *bitmap, const struct object_id *oid) +{ + int idx; + + if (!bitmap) + return 0; + + idx = bitmap_position(bitmap_git, oid); + return idx >= 0 && bitmap_get(bitmap, idx); +} + void traverse_bitmap_commit_list(struct bitmap_index *bitmap_git, show_reachable_fn show_reachable) { diff --git a/pack-bitmap.h b/pack-bitmap.h index 466c5afa09..6ab6033dbe 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -3,6 +3,7 @@ #include "ewah/ewok.h" #include "khash.h" +#include "pack.h" #include "pack-objects.h" struct commit; @@ -53,6 +54,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *, int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping, kh_oid_map_t *reused_bitmaps, int show_progress); void free_bitmap_index(struct bitmap_index *); +int bitmap_walk_contains(struct bitmap_index *, + struct bitmap *bitmap, const struct object_id *oid); /* * After a traversal has been performed by prepare_bitmap_walk(), this can be From patchwork Wed Dec 18 11:25:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300309 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9CB6414B7 for ; Wed, 18 Dec 2019 11:26:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7B84A227BF for ; Wed, 18 Dec 2019 11:26:17 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="bf5zVVf9" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726861AbfLRL0Q (ORCPT ); Wed, 18 Dec 2019 06:26:16 -0500 Received: from mail-wm1-f67.google.com ([209.85.128.67]:39217 "EHLO mail-wm1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726856AbfLRL0O (ORCPT ); Wed, 18 Dec 2019 06:26:14 -0500 Received: by mail-wm1-f67.google.com with SMTP id 20so1353733wmj.4 for ; Wed, 18 Dec 2019 03:26:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=f8M8cZ+gRv/rNN8fFjyF98stzkGZ+D510F0bBH0psTM=; b=bf5zVVf9Boxmlyw+Q9rxrvFiNi/SY9wInvo57fxt1TAzze9NuD/MoJfFtqgyOCupIb mm0+H3WfiI84ylf87s1IgAD82fSWJSTbYPy5yo/yKGbYw7moQFRaUQeGH1Xb1i6koOeP 6cym/Pas0mp9kGV3oyf7WhplmC896gZe0+7V16bkgNI9O8iWvW5jMKaECdJdaGJF2cZ4 r8N2s/QYw+ADfwh749Ve11xarDPj/L4BdWKzVblm9WwhkrpJVCsu2NgO/ZVKuSfEz+pH ceKHBlMH/f7sW3JCH6xv7f2Y1BQeVuQAQj+so4H+plt/spS5JNFgCaJ0pucIt9zNXR6r r4QA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=f8M8cZ+gRv/rNN8fFjyF98stzkGZ+D510F0bBH0psTM=; b=mLSpHcFRsdy3Kkn6hTia+j+49tACebdgGGiVPLKb3/0BdtXcWAyg9O81/RoiGmBh3o Whh7kqWluvhwrW2/k/rJJDe09hirsfkb4ZjUzrI+M9ll0a+BftF6vGsCjLCksC4IzV0G 0HTRYG0xiuI6fRXR41REeLbF7h0cqxaPd3sdcCyqAJXuJsPu9pUaxQu01psOSD1XMRkH IM68yDWmDNZ7ZCuWG+g49N4sTl20FPhAPV1rNgjeJ+aZJeq46i17gAWJk4u45U60SRm/ UrnzUa5xtYlDFox+cU4VQmt9lj7PzqEl87uPxceT9ptnuYrLkdiBwUIjv082kwePONNi S3Yg== X-Gm-Message-State: APjAAAX5LUHkpo+urCxFkUbYL5EYSqrxEgnm2Tf68gm6xBx5hUfHGlcJ oyt3DjvhfcBgLoLQCPLW1AJW+Zjo X-Google-Smtp-Source: APXvYqx6U4PcvS3CxdH3mSrHJr0T43Q6UAzTm8RjBZF7iAeQIXS+rfXshQ3MhxcN4F2NJHOE/6UMjg== X-Received: by 2002:a05:600c:21c6:: with SMTP id x6mr2543724wmj.177.1576668371878; Wed, 18 Dec 2019 03:26:11 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:11 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 05/12] pack-bitmap: uninteresting oid can be outside bitmapped packfile Date: Wed, 18 Dec 2019 12:25:40 +0100 Message-Id: <20191218112547.4974-6-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King bitmap_has_oid_in_uninteresting() only used bitmap_position_packfile(), not bitmap_position(). So it wouldn't find objects which weren't in the bitmapped packfile (i.e., ones where we extended the bitmap to handle loose objects, or objects in other packs). As we could reuse a delta against such an object it is suboptimal not to use bitmap_position(), so let's use it instead of bitmap_position_packfile(). Signed-off-by: Jeff King Signed-off-by: Christian Couder --- pack-bitmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index fb4f6297f2..de65f2fc36 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -1137,7 +1137,7 @@ int bitmap_has_oid_in_uninteresting(struct bitmap_index *bitmap_git, if (!bitmap_git->haves) return 0; /* walk had no "haves" */ - pos = bitmap_position_packfile(bitmap_git, oid); + pos = bitmap_position(bitmap_git, oid); if (pos < 0) return 0; From patchwork Wed Dec 18 11:25:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300311 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8007614B7 for ; Wed, 18 Dec 2019 11:26:19 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 5DFEE227BF for ; Wed, 18 Dec 2019 11:26:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SQ+evK3N" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726881AbfLRL0S (ORCPT ); Wed, 18 Dec 2019 06:26:18 -0500 Received: from mail-wm1-f68.google.com ([209.85.128.68]:37167 "EHLO mail-wm1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726591AbfLRL0P (ORCPT ); Wed, 18 Dec 2019 06:26:15 -0500 Received: by mail-wm1-f68.google.com with SMTP id f129so1482791wmf.2 for ; Wed, 18 Dec 2019 03:26:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=+z4UpB8YN+FmCBQgub7FHMUlx3/iCBaiPLp4PPWTdcY=; b=SQ+evK3Nx0/otuklwyYZ7Z659n5FYdtixD6/nKf/JQlI14NlHD9Fivs7qiRCnvW+YI x2YVzHtKWCMBnLC4hxI7Rx05+/777LwoPXq6QloXK5MsyX6jRbQeLxXq5W/ciGn2AQVR 4iUs77EK7zMY++JwfSS0SdMod4BP0VoWAtnKMK62XQe1VUhwAIvA6R1n8FuRBhuw5Vty iJjQc5Ov19dBvUmB8rBBU0tIABYrqoWUe/IAAr3wSomXDzhBpf8gwPIlNTPoyz/cDeV0 ufBF/xVPKfSkxvlke61tC13ybFfPxxhdvlhL3BSFrq1OEKguLsuvMBLmBw66J71HHpiz kojg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=+z4UpB8YN+FmCBQgub7FHMUlx3/iCBaiPLp4PPWTdcY=; b=Q1lTxUfJzCbdu/ggcMlSFHoZkwrInC+bfyGA/+hWiAcCmA3H1o5M2zkbpu8cxw1Eap IKazw5lNnsei/XQls+s+HeCd+/gkZae6jDdZMKeah94aL30Mc+sjcGJ4XOHC8wVXk9Sk FcXPcweVIvgSD3Y8b5C5hWair6o63c8gA4kjhtP43MyCf4uBvytt1KlCeRUTYWt0DXmh oGd8LNMEB4ENbBUOGbGgW1/qu7MEH2pLdTOoUH+E4GmpyLY75gy8w6ZPRiTq80sSwaAg eovuN+XSaeVmFflXoIwd70FphOu0bDNz8j0oTMyxuUmagAMFJGJKORYnGBrhUYqzNwyr zjsw== X-Gm-Message-State: APjAAAVb8HdLLUbCc+c9tlbSXEKWgVgMMpt0qSNzJwSMm38aB+U7s8Sd HsLDDAA6lsvB+lNP0aMnrXQrdjL9 X-Google-Smtp-Source: APXvYqyK8bEiIicCHRTHMSNPm/okTgKvUdwoaNoM68IuPPDOikkxoEuFzWgULHFxRo0VF83qaHukaA== X-Received: by 2002:a05:600c:2c13:: with SMTP id q19mr2587420wmg.144.1576668372992; Wed, 18 Dec 2019 03:26:12 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:12 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 06/12] pack-bitmap: simplify bitmap_has_oid_in_uninteresting() Date: Wed, 18 Dec 2019 12:25:41 +0100 Message-Id: <20191218112547.4974-7-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King Let's refactor bitmap_has_oid_in_uninteresting() using bitmap_walk_contains(). Signed-off-by: Jeff King Signed-off-by: Christian Couder --- pack-bitmap.c | 14 ++------------ 1 file changed, 2 insertions(+), 12 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index de65f2fc36..0bbc651dde 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -1130,16 +1130,6 @@ void free_bitmap_index(struct bitmap_index *b) int bitmap_has_oid_in_uninteresting(struct bitmap_index *bitmap_git, const struct object_id *oid) { - int pos; - - if (!bitmap_git) - return 0; /* no bitmap loaded */ - if (!bitmap_git->haves) - return 0; /* walk had no "haves" */ - - pos = bitmap_position(bitmap_git, oid); - if (pos < 0) - return 0; - - return bitmap_get(bitmap_git->haves, pos); + return bitmap_git && + bitmap_walk_contains(bitmap_git, bitmap_git->haves, oid); } From patchwork Wed Dec 18 11:25:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300315 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 4274C14F6 for ; Wed, 18 Dec 2019 11:26:21 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2029521D7D for ; Wed, 18 Dec 2019 11:26:21 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="dcA9KLAx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726876AbfLRL0S (ORCPT ); Wed, 18 Dec 2019 06:26:18 -0500 Received: from mail-wr1-f67.google.com ([209.85.221.67]:43158 "EHLO mail-wr1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726787AbfLRL0Q (ORCPT ); Wed, 18 Dec 2019 06:26:16 -0500 Received: by mail-wr1-f67.google.com with SMTP id d16so1856249wre.10 for ; Wed, 18 Dec 2019 03:26:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=JQQgqz1yIUMmNdx3lVEw8Zcw26jdsVnZB5A9Dhbx604=; b=dcA9KLAx7KZq4q2B/ClJRO8sr3Wcclzgdocnrh+FdjmH6XbsJJgOXQCJyJNNpkNyy7 VTe2k52ImroWdlJ7Lj2AWSqWBZ96hPVPdGKNpiOddPgNj6hftLWJLAhv6OTVVAm/Ms4I ZxWgQiaWw9XohcIYORn4D75Ogp8+sUreSCveLKXHYAxKfXT/RHvrmFW6KmEj1bZdNERH ZPTSBUQsw4pfdtaliMprauGCu7deUpaIWQDnqcqSJ8lXd0fUKZ2KF/tq2s12yZfert/N 5P3CWn5rvxYlmoc2u+xXT5KkV8oMH6lqIE8UxHoH00qgUnGcc1Z7ykzAlm3ZGEh3W9Eh ZfDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=JQQgqz1yIUMmNdx3lVEw8Zcw26jdsVnZB5A9Dhbx604=; b=rcqLM66wiiE28YaQbnHsfvRXTQjD8+uPd/n4gpgTsvhIZRFkrNe3ZBUDgQr8Duvnzq 9rRQaf3wWexPvmUGdbDOoniuLAJmBzf656DbIKEiFgh8inGVmuu8E+NHty31KZoZQM2z 7+zAwLntA737DgdFl4eKUssgGLAFIJ8dbNxA9q3D0/UsOTp95iz844ri9iPORg7L4VFU I6MMTNIqBpEp/QlrH+ZgOPA4B3TrpLODYyL3vHN97SQZskHjsipmO75+inf8YGmyGwtH nE14ukrtyLETl1hkPIr86Cf9rsCe3bymfQeCPx1nFDv6ZrX3xLZPAoN+Nbeu7tfCxh4e HwbA== X-Gm-Message-State: APjAAAXxO0fiJ5jUznojdIfg0ZXUaxZWNDH7qpAvs3dXlil0u1UySBwS xSkBhdkAKAEdeVF8pk7Dqk71eGi5 X-Google-Smtp-Source: APXvYqxD8vnvV10neS85Amw6reQQkHnljXYWYOKKZEkfRpDYYhITtBZKpdTxxEDqJjR61gpCFtQUuw== X-Received: by 2002:adf:fd87:: with SMTP id d7mr2296529wrr.226.1576668373989; Wed, 18 Dec 2019 03:26:13 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:13 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 07/12] csum-file: introduce hashfile_total() Date: Wed, 18 Dec 2019 12:25:42 +0100 Message-Id: <20191218112547.4974-8-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King We will need this helper function in a following commit to give us total number of bytes fed to the hashfile so far. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- csum-file.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/csum-file.h b/csum-file.h index a98b1eee53..f9cbd317fb 100644 --- a/csum-file.h +++ b/csum-file.h @@ -42,6 +42,15 @@ void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); uint32_t crc32_end(struct hashfile *); +/* + * Returns the total number of bytes fed to the hashfile so far (including ones + * that have not been written out to the descriptor yet). + */ +static inline off_t hashfile_total(struct hashfile *f) +{ + return f->total + f->offset; +} + static inline void hashwrite_u8(struct hashfile *f, uint8_t data) { hashwrite(f, &data, sizeof(data)); From patchwork Wed Dec 18 11:25:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300317 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0A881109A for ; Wed, 18 Dec 2019 11:26:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id DD926227BF for ; Wed, 18 Dec 2019 11:26:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="LqRvmovJ" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726910AbfLRL0V (ORCPT ); Wed, 18 Dec 2019 06:26:21 -0500 Received: from mail-wm1-f42.google.com ([209.85.128.42]:37480 "EHLO mail-wm1-f42.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726674AbfLRL0R (ORCPT ); Wed, 18 Dec 2019 06:26:17 -0500 Received: by mail-wm1-f42.google.com with SMTP id f129so1482898wmf.2 for ; Wed, 18 Dec 2019 03:26:16 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WiHLNDiBzU2tElAJNWa4G3aQ14D40sJWpcuGm33EBgE=; b=LqRvmovJ0ATUayGlBQTj9cTRt/GwiXivn1Zihy5glO3536Zi0pOLuSmj3jc4ix9M1q ZWqtbl4z4kTieLo+cfdvAXIty1AGcIhmlY1EAcfv3AlpbeXb4DKFeHTnAfn4hgU4DoiG gGP0HWUS5f6yUf8UzXhZE2WQzCi9GIQWdx6x8WWz5/TTU/Z3IfV9nVXNjKyVF+O72fLr R5J1eHTvgqcY7KZLBfNnHZMfFANhIEXA9tAncQNght/drrG1DE6NYoulV3p0pjT43HAr wAnZYhxuFVie5q1YtKe2+fk8mipybP1ql7VUEc//Odov+Rvpan99+yTXlgPKe0O09eTK Momw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WiHLNDiBzU2tElAJNWa4G3aQ14D40sJWpcuGm33EBgE=; b=SOklMLE2RBxOCkRDdOtnEcfQZsIE5rndv0TvrY95cf6yGDzancs9pDs8MpuRmu7aYT rytefTUgGz45kc+ZQFNZf4EyelgXwsNoJQAOrTP7a8r+L96Pp6XPokOUMFcMmbQIgyp4 TbZeombnfZFX0HvOUSpNN/y3Lb6pIiM8USBmci9dv7XC51fgq+ChdlRqie6J5NLRcJ8w AfXtVspf4USUwuvDEbLaJwl1KufFsWLQhfO/NWRAkCV3UN6+A1KCrPVRsbB3EOxLdIxa 27MB4w6S9g7nD/L+hJn7oxQmKdUPYMJQfUUh0wZ7sXY5IbDsJTxFz98mPz52FYc4UAB0 5DDQ== X-Gm-Message-State: APjAAAVT7XHclLp1Vi4BcEEADhpnFhBRWRatO06TlSaITN1lsB0S/QlB tDMQXFd1ZtYP6YZc6SXxnwvGkZx7 X-Google-Smtp-Source: APXvYqzFhSYvZ0GV/gDNdFkumai3U5Jq2mSJT1XJdIf/ZxndbfOwQONHL8DY/6k8BeL2zlMvSbST0Q== X-Received: by 2002:a1c:1fd0:: with SMTP id f199mr2667021wmf.113.1576668375195; Wed, 18 Dec 2019 03:26:15 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.14 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:14 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 08/12] pack-objects: introduce pack.allowPackReuse Date: Wed, 18 Dec 2019 12:25:43 +0100 Message-Id: <20191218112547.4974-9-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King Let's make it possible to configure if we want pack reuse or not. The main reason it might not be wanted is probably debugging and performance testing, though pack reuse _might_ cause larger packs, because we wouldn't consider the reused objects as bases for finding new deltas. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- Documentation/config/pack.txt | 7 +++++++ builtin/pack-objects.c | 8 +++++++- 2 files changed, 14 insertions(+), 1 deletion(-) diff --git a/Documentation/config/pack.txt b/Documentation/config/pack.txt index 1d66f0c992..0dac580581 100644 --- a/Documentation/config/pack.txt +++ b/Documentation/config/pack.txt @@ -27,6 +27,13 @@ Note that changing the compression level will not automatically recompress all existing objects. You can force recompression by passing the -F option to linkgit:git-repack[1]. +pack.allowPackReuse:: + When true, and when reachability bitmaps are enabled, + pack-objects will try to send parts of the bitmapped packfile + verbatim. This can reduce memory and CPU usage to serve fetches, + but might result in sending a slightly larger pack. Defaults to + true. + pack.island:: An extended regular expression configuring a set of delta islands. See "DELTA ISLANDS" in linkgit:git-pack-objects[1] diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 369f46fbc5..801e23dfe7 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -96,6 +96,7 @@ static off_t reuse_packfile_offset; static int use_bitmap_index_default = 1; static int use_bitmap_index = -1; +static int allow_pack_reuse = 1; static enum { WRITE_BITMAP_FALSE = 0, WRITE_BITMAP_QUIET, @@ -2699,6 +2700,10 @@ static int git_pack_config(const char *k, const char *v, void *cb) use_bitmap_index_default = git_config_bool(k, v); return 0; } + if (!strcmp(k, "pack.allowpackreuse")) { + allow_pack_reuse = git_config_bool(k, v); + return 0; + } if (!strcmp(k, "pack.threads")) { delta_search_threads = git_config_int(k, v); if (delta_search_threads < 0) @@ -3030,7 +3035,8 @@ static void loosen_unused_packed_objects(void) */ static int pack_options_allow_reuse(void) { - return pack_to_stdout && + return allow_pack_reuse && + pack_to_stdout && allow_ofs_delta && !ignore_packed_keep_on_disk && !ignore_packed_keep_in_core && From patchwork Wed Dec 18 11:25:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300313 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B9154109A for ; Wed, 18 Dec 2019 11:26:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8EF0E21D7D for ; Wed, 18 Dec 2019 11:26:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="t+rlHnQ2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726891AbfLRL0T (ORCPT ); Wed, 18 Dec 2019 06:26:19 -0500 Received: from mail-wm1-f65.google.com ([209.85.128.65]:38337 "EHLO mail-wm1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726856AbfLRL0S (ORCPT ); Wed, 18 Dec 2019 06:26:18 -0500 Received: by mail-wm1-f65.google.com with SMTP id u2so1479798wmc.3 for ; Wed, 18 Dec 2019 03:26:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7lJGHpJsVA6sbzue4xMRRNCnCN/QUiH09Y1A4kW0pU0=; b=t+rlHnQ2aLqnXL+zCJsSGbJa2HzvMbYkSNfjBKFppDMNDArzNPaPK1HNOq3pDP9fML b+UX1ZKj9cPJPez4ju0/qtquUOE9tqVYGHAUoK5YMaz3EEsNb3Bxmwnm+UfuvhmforJk k8tL0MYncBidhOiqMDTbVOlcxBgvBj1JZgxhXMik2PzhwEEKJLBqKTzCXKAYgXvs9Omf 1nZ4wWYA1mGDTev/7WqY+j6+wBW19yJIU9chSvhI8EmntERVFDfm35nrVhrnCirciKNP eGjjl/TKBO5E+ZSqbBgsVUHnw5QZsTaSp1Dg5A0gdhriRMYqpczigzMMxYGIiawHvDMn 6ceQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7lJGHpJsVA6sbzue4xMRRNCnCN/QUiH09Y1A4kW0pU0=; b=s2TgGpE40IShmLKHGcYn4M9sQyzLt9gtG317t6lNuWOlrqoZ15nInCcKbmZSUppjK9 B/usNe05/PI60mKKvSh9/yENabXNcHox3uSwuawcvHpjVBQe3zMyRES7aYCamJQV77E4 1bCbFgoLyzg7MyTi/6TpgzmGGwgDcU+42tE4DPXPRafVB/Vh9Ggr4wDPj/wu3B5LJuyp /sDMzxYvGax5YJYQK/Tfm4HLHlLh8Nq3MxGq++qaBUdbrAF/PyUFiBUOTlTeWZhpQaUQ 2aY4O/f7V246sGRJe6znfz5sACFmbG7ImXmD+89r68dOjrz/1jyXgEIscJU1+f9p6ond flWg== X-Gm-Message-State: APjAAAV95TN/G24jjcp46Lj/I1lkgA7KLIORiCZopUmkD8O5vSUSLvMn iLSevbQPU1yHxGF2Ix4pN3LcwlW0 X-Google-Smtp-Source: APXvYqwUUWiwlHgPV1igXzd6stUC4H+iSVfPd+v99t4TcH9G2G8VkWZwXnvLnesEe//rbwYfWvvnaw== X-Received: by 2002:a1c:7d93:: with SMTP id y141mr2751364wmc.111.1576668376154; Wed, 18 Dec 2019 03:26:16 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.15 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:15 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 09/12] builtin/pack-objects: introduce obj_is_packed() Date: Wed, 18 Dec 2019 12:25:44 +0100 Message-Id: <20191218112547.4974-10-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King Let's refactor the way we check if an object is packed by introducing obj_is_packed(). This function is now a simple wrapper around packlist_find(), but it will evolve in a following commit. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- builtin/pack-objects.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 801e23dfe7..43730154ba 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -2553,6 +2553,11 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size, free(p); } +static int obj_is_packed(const struct object_id *oid) +{ + return !!packlist_find(&to_pack, oid); +} + static void add_tag_chain(const struct object_id *oid) { struct tag *tag; @@ -2564,7 +2569,7 @@ static void add_tag_chain(const struct object_id *oid) * it was included via bitmaps, we would not have parsed it * previously). */ - if (packlist_find(&to_pack, oid)) + if (obj_is_packed(oid)) return; tag = lookup_tag(the_repository, oid); @@ -2588,7 +2593,7 @@ static int add_ref_tag(const char *path, const struct object_id *oid, int flag, if (starts_with(path, "refs/tags/") && /* is a tag? */ !peel_ref(path, &peeled) && /* peelable? */ - packlist_find(&to_pack, &peeled)) /* object packed? */ + obj_is_packed(&peeled)) /* object packed? */ add_tag_chain(oid); return 0; } From patchwork Wed Dec 18 11:25:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300319 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A1FB7109A for ; Wed, 18 Dec 2019 11:26:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 68D51227BF for ; Wed, 18 Dec 2019 11:26:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PHom7KFv" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726912AbfLRL0X (ORCPT ); Wed, 18 Dec 2019 06:26:23 -0500 Received: from mail-wr1-f43.google.com ([209.85.221.43]:45198 "EHLO mail-wr1-f43.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726591AbfLRL0V (ORCPT ); Wed, 18 Dec 2019 06:26:21 -0500 Received: by mail-wr1-f43.google.com with SMTP id j42so1844389wrj.12 for ; Wed, 18 Dec 2019 03:26:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=BkL+XvZ6+5TABk5pAfNVy8BrEJp2zV7gM4DNIbLYyMI=; b=PHom7KFvTRmQyX6p7DW/OCphF4mn8xMeiGC+hHvAJzA8C/p++VDpy5hw6GuJfpnpTY hYFUNfKa0C3Y2XLf3ZDqOFuWXoqoe9vbOyVPwpThCaIl/JUogKko4SYIUs2OOgPETqTg ICnZbpkJT2e2ubPnCQAkptABypTNITDRRZnFZr85e5r2vLDdQx8sy9R258JJj0nTQ9+l 58k81Ixa/K0OMqEPEUao4j7uoP89e7jUI896rGQjHq7P83maeyUfmpopDFCrNb+93fYB 2BOMo9RdmGuOktzRpT2c5eBDKsuiOcBzuUKrzmlgv3tctlqst8nj+J+G6TRU2Ks6eyl4 iHPQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=BkL+XvZ6+5TABk5pAfNVy8BrEJp2zV7gM4DNIbLYyMI=; b=Scor8zra5jliggcPSqUvej+IC9/PyHgAr2DvC1WtkRlC+OUSH5Xg9xgwMB0K4u3v10 9cEfK7y7Xfk4NeLN4JEbn0u5rtJ/ZpGiQXVfLcbCQtZZdubLKZZg3DOjMYExZi+Gkly7 1d7i/UxVLHQP7jHLhobviCKXNXczVfygoaBuN2EMe+nyolVcC9O+hd5S/pgq+MiJDlJx /J+DQIIHqAhgFtxQ7u8QI/MGsaLJSteGGoB4PBnEzvFOzj4Hq6MZAf43GWsZ9D5v5Z+/ snnNdiOLjBtFXhVGWJmzKZnMQzSL8c/Fet3R6FM6GBvBV6I1h48Q6cKvqfIOcaPGesku E4tg== X-Gm-Message-State: APjAAAVJn6BabdWjZ07eaEJEo7wgo5/qDn5c8uQ7MvjtX6zISEVh56Df 98I4UKaU/hzYwdecNYdgUCnjKk4P X-Google-Smtp-Source: APXvYqzioaLgHBPi+y6qtmmGD39i5nxLzmEBSNeFE1rFhk91sTPD/ofvxKBOTdQd2nXNKJue7eHEeA== X-Received: by 2002:adf:f78d:: with SMTP id q13mr2244636wrp.365.1576668377501; Wed, 18 Dec 2019 03:26:17 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:16 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 10/12] pack-objects: improve partial packfile reuse Date: Wed, 18 Dec 2019 12:25:45 +0100 Message-Id: <20191218112547.4974-11-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King The old code to reuse deltas from an existing packfile just tried to dump a whole segment of the pack verbatim. That's faster than the traditional way of actually adding objects to the packing list, but it didn't kick in very often. This new code is really going for a middle ground: do _some_ per-object work, but way less than we'd traditionally do. The general strategy of the new code is to make a bitmap of objects from the packfile we'll include, and then iterate over it, writing out each object exactly as it is in our on-disk pack, but _not_ adding it to our packlist (which costs memory, and increases the search space for deltas). One complication is that if we're omitting some objects, we can't set a delta against a base that we're not sending. So we have to check each object in try_partial_reuse() to make sure we have its delta. About performance, in the worst case we might have interleaved objects that we are sending or not sending, and we'd have as many chunks as objects. But in practice we send big chunks. For instance, packing torvalds/linux on GitHub servers now reused 6.5M objects, but only needed ~50k chunks. Helped-by: Jonathan Tan Signed-off-by: Jeff King Signed-off-by: Christian Couder --- builtin/pack-objects.c | 214 ++++++++++++++++++++++++++++++++--------- pack-bitmap.c | 150 +++++++++++++++++++++-------- pack-bitmap.h | 3 +- 3 files changed, 281 insertions(+), 86 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 43730154ba..c80c1fac94 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -92,7 +92,7 @@ static struct progress *progress_state; static struct packed_git *reuse_packfile; static uint32_t reuse_packfile_objects; -static off_t reuse_packfile_offset; +static struct bitmap *reuse_packfile_bitmap; static int use_bitmap_index_default = 1; static int use_bitmap_index = -1; @@ -785,57 +785,185 @@ static struct object_entry **compute_write_order(void) return wo; } -static off_t write_reused_pack(struct hashfile *f) + +/* + * A reused set of objects. All objects in a chunk have the same + * relative position in the original packfile and the generated + * packfile. + */ + +static struct reused_chunk { + /* The offset of the first object of this chunk in the original + * packfile. */ + off_t original; + /* The offset of the first object of this chunk in the generated + * packfile minus "original". */ + off_t difference; +} *reused_chunks; +static int reused_chunks_nr; +static int reused_chunks_alloc; + +static void record_reused_object(off_t where, off_t offset) +{ + if (reused_chunks_nr && reused_chunks[reused_chunks_nr-1].difference == offset) + return; + + ALLOC_GROW(reused_chunks, reused_chunks_nr + 1, + reused_chunks_alloc); + reused_chunks[reused_chunks_nr].original = where; + reused_chunks[reused_chunks_nr].difference = offset; + reused_chunks_nr++; +} + +/* + * Binary search to find the chunk that "where" is in. Note + * that we're not looking for an exact match, just the first + * chunk that contains it (which implicitly ends at the start + * of the next chunk. + */ +static off_t find_reused_offset(off_t where) { - unsigned char buffer[8192]; - off_t to_write, total; - int fd; + int lo = 0, hi = reused_chunks_nr; + while (lo < hi) { + int mi = lo + ((hi - lo) / 2); + if (where == reused_chunks[mi].original) + return reused_chunks[mi].difference; + if (where < reused_chunks[mi].original) + hi = mi; + else + lo = mi + 1; + } - if (!is_pack_valid(reuse_packfile)) - die(_("packfile is invalid: %s"), reuse_packfile->pack_name); + /* + * The first chunk starts at zero, so we can't have gone below + * there. + */ + assert(lo); + return reused_chunks[lo-1].difference; +} - fd = git_open(reuse_packfile->pack_name); - if (fd < 0) - die_errno(_("unable to open packfile for reuse: %s"), - reuse_packfile->pack_name); +static void write_reused_pack_one(size_t pos, struct hashfile *out, + struct pack_window **w_curs) +{ + off_t offset, next, cur; + enum object_type type; + unsigned long size; - if (lseek(fd, sizeof(struct pack_header), SEEK_SET) == -1) - die_errno(_("unable to seek in reused packfile")); + offset = reuse_packfile->revindex[pos].offset; + next = reuse_packfile->revindex[pos + 1].offset; - if (reuse_packfile_offset < 0) - reuse_packfile_offset = reuse_packfile->pack_size - the_hash_algo->rawsz; + record_reused_object(offset, offset - hashfile_total(out)); - total = to_write = reuse_packfile_offset - sizeof(struct pack_header); + cur = offset; + type = unpack_object_header(reuse_packfile, w_curs, &cur, &size); + assert(type >= 0); - while (to_write) { - int read_pack = xread(fd, buffer, sizeof(buffer)); + if (type == OBJ_OFS_DELTA) { + off_t base_offset; + off_t fixup; + + unsigned char header[MAX_PACK_OBJECT_HEADER]; + unsigned len; + + base_offset = get_delta_base(reuse_packfile, w_curs, &cur, type, offset); + assert(base_offset != 0); + + /* Convert to REF_DELTA if we must... */ + if (!allow_ofs_delta) { + int base_pos = find_revindex_position(reuse_packfile, base_offset); + const unsigned char *base_sha1 = + nth_packed_object_sha1(reuse_packfile, + reuse_packfile->revindex[base_pos].nr); + + len = encode_in_pack_object_header(header, sizeof(header), + OBJ_REF_DELTA, size); + hashwrite(out, header, len); + hashwrite(out, base_sha1, 20); + copy_pack_data(out, reuse_packfile, w_curs, cur, next - cur); + return; + } - if (read_pack <= 0) - die_errno(_("unable to read from reused packfile")); + /* Otherwise see if we need to rewrite the offset... */ + fixup = find_reused_offset(offset) - + find_reused_offset(base_offset); + if (fixup) { + unsigned char ofs_header[10]; + unsigned i, ofs_len; + off_t ofs = offset - base_offset - fixup; - if (read_pack > to_write) - read_pack = to_write; + len = encode_in_pack_object_header(header, sizeof(header), + OBJ_OFS_DELTA, size); - hashwrite(f, buffer, read_pack); - to_write -= read_pack; + i = sizeof(ofs_header) - 1; + ofs_header[i] = ofs & 127; + while (ofs >>= 7) + ofs_header[--i] = 128 | (--ofs & 127); + + ofs_len = sizeof(ofs_header) - i; + + hashwrite(out, header, len); + hashwrite(out, ofs_header + sizeof(ofs_header) - ofs_len, ofs_len); + copy_pack_data(out, reuse_packfile, w_curs, cur, next - cur); + return; + } + + /* ...otherwise we have no fixup, and can write it verbatim */ + } + + copy_pack_data(out, reuse_packfile, w_curs, offset, next - offset); +} + +static size_t write_reused_pack_verbatim(struct hashfile *out, + struct pack_window **w_curs) +{ + size_t pos = 0; + + while (pos < reuse_packfile_bitmap->word_alloc && + reuse_packfile_bitmap->words[pos] == (eword_t)~0) + pos++; + + if (pos) { + off_t to_write; + + written = (pos * BITS_IN_EWORD); + to_write = reuse_packfile->revindex[written].offset + - sizeof(struct pack_header); + + /* We're recording one chunk, not one object. */ + record_reused_object(sizeof(struct pack_header), 0); + hashflush(out); + copy_pack_data(out, reuse_packfile, w_curs, + sizeof(struct pack_header), to_write); - /* - * We don't know the actual number of objects written, - * only how many bytes written, how many bytes total, and - * how many objects total. So we can fake it by pretending all - * objects we are writing are the same size. This gives us a - * smooth progress meter, and at the end it matches the true - * answer. - */ - written = reuse_packfile_objects * - (((double)(total - to_write)) / total); display_progress(progress_state, written); } + return pos; +} + +static void write_reused_pack(struct hashfile *f) +{ + size_t i = 0; + uint32_t offset; + struct pack_window *w_curs = NULL; + + if (allow_ofs_delta) + i = write_reused_pack_verbatim(f, &w_curs); + + for (; i < reuse_packfile_bitmap->word_alloc; ++i) { + eword_t word = reuse_packfile_bitmap->words[i]; + size_t pos = (i * BITS_IN_EWORD); + + for (offset = 0; offset < BITS_IN_EWORD; ++offset) { + if ((word >> offset) == 0) + break; + + offset += ewah_bit_ctz64(word >> offset); + write_reused_pack_one(pos + offset, f, &w_curs); + display_progress(progress_state, ++written); + } + } - close(fd); - written = reuse_packfile_objects; - display_progress(progress_state, written); - return reuse_packfile_offset - sizeof(struct pack_header); + unuse_pack(&w_curs); } static const char no_split_warning[] = N_( @@ -868,11 +996,9 @@ static void write_pack_file(void) offset = write_pack_header(f, nr_remaining); if (reuse_packfile) { - off_t packfile_size; assert(pack_to_stdout); - - packfile_size = write_reused_pack(f); - offset += packfile_size; + write_reused_pack(f); + offset = hashfile_total(f); } nr_written = 0; @@ -2661,6 +2787,7 @@ static void prepare_pack(int window, int depth) if (nr_deltas && n > 1) { unsigned nr_done = 0; + if (progress) progress_state = start_progress(_("Compressing objects"), nr_deltas); @@ -3042,7 +3169,6 @@ static int pack_options_allow_reuse(void) { return allow_pack_reuse && pack_to_stdout && - allow_ofs_delta && !ignore_packed_keep_on_disk && !ignore_packed_keep_in_core && (!local || !have_non_local_packs) && @@ -3059,7 +3185,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs) bitmap_git, &reuse_packfile, &reuse_packfile_objects, - &reuse_packfile_offset)) { + &reuse_packfile_bitmap)) { assert(reuse_packfile_objects); nr_result += reuse_packfile_objects; display_progress(progress_state, nr_result); diff --git a/pack-bitmap.c b/pack-bitmap.c index 0bbc651dde..a1d43d367a 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -326,6 +326,13 @@ static int load_pack_bitmap(struct bitmap_index *bitmap_git) munmap(bitmap_git->map, bitmap_git->map_size); bitmap_git->map = NULL; bitmap_git->map_size = 0; + + kh_destroy_oid_map(bitmap_git->bitmaps); + bitmap_git->bitmaps = NULL; + + kh_destroy_oid_pos(bitmap_git->ext_index.positions); + bitmap_git->ext_index.positions = NULL; + return -1; } @@ -768,65 +775,126 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs) return NULL; } -int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, - struct packed_git **packfile, - uint32_t *entries, - off_t *up_to) +static void try_partial_reuse(struct bitmap_index *bitmap_git, + size_t pos, + struct bitmap *reuse, + struct pack_window **w_curs) { + struct revindex_entry *revidx; + off_t offset; + enum object_type type; + unsigned long size; + + if (pos >= bitmap_git->pack->num_objects) + return; /* not actually in the pack */ + + revidx = &bitmap_git->pack->revindex[pos]; + offset = revidx->offset; + type = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size); + if (type < 0) + return; /* broken packfile, punt */ + + if (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) { + off_t base_offset; + int base_pos; + + /* + * Find the position of the base object so we can look it up + * in our bitmaps. If we can't come up with an offset, or if + * that offset is not in the revidx, the pack is corrupt. + * There's nothing we can do, so just punt on this object, + * and the normal slow path will complain about it in + * more detail. + */ + base_offset = get_delta_base(bitmap_git->pack, w_curs, + &offset, type, revidx->offset); + if (!base_offset) + return; + base_pos = find_revindex_position(bitmap_git->pack, base_offset); + if (base_pos < 0) + return; + + /* + * We assume delta dependencies always point backwards. This + * lets us do a single pass, and is basically always true + * due to the way OFS_DELTAs work. You would not typically + * find REF_DELTA in a bitmapped pack, since we only bitmap + * packs we write fresh, and OFS_DELTA is the default). But + * let's double check to make sure the pack wasn't written with + * odd parameters. + */ + if (base_pos >= pos) + return; + + /* + * And finally, if we're not sending the base as part of our + * reuse chunk, then don't send this object either. The base + * would come after us, along with other objects not + * necessarily in the pack, which means we'd need to convert + * to REF_DELTA on the fly. Better to just let the normal + * object_entry code path handle it. + */ + if (!bitmap_get(reuse, base_pos)) + return; + } + /* - * Reuse the packfile content if we need more than - * 90% of its objects + * If we got here, then the object is OK to reuse. Mark it. */ - static const double REUSE_PERCENT = 0.9; + bitmap_set(reuse, pos); +} +int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, + struct packed_git **packfile_out, + uint32_t *entries, + struct bitmap **reuse_out) +{ struct bitmap *result = bitmap_git->result; - uint32_t reuse_threshold; - uint32_t i, reuse_objects = 0; + struct bitmap *reuse; + struct pack_window *w_curs = NULL; + size_t i = 0; + uint32_t offset; assert(result); - for (i = 0; i < result->word_alloc; ++i) { - if (result->words[i] != (eword_t)~0) { - reuse_objects += ewah_bit_ctz64(~result->words[i]); - break; - } + while (i < result->word_alloc && result->words[i] == (eword_t)~0) + i++; - reuse_objects += BITS_IN_EWORD; - } + /* Don't mark objects not in the packfile */ + if (i > bitmap_git->pack->num_objects / BITS_IN_EWORD) + i = bitmap_git->pack->num_objects / BITS_IN_EWORD; -#ifdef GIT_BITMAP_DEBUG - { - const unsigned char *sha1; - struct revindex_entry *entry; + reuse = bitmap_word_alloc(i); + memset(reuse->words, 0xFF, i * sizeof(eword_t)); - entry = &bitmap_git->reverse_index->revindex[reuse_objects]; - sha1 = nth_packed_object_sha1(bitmap_git->pack, entry->nr); - - fprintf(stderr, "Failed to reuse at %d (%016llx)\n", - reuse_objects, result->words[i]); - fprintf(stderr, " %s\n", hash_to_hex(sha1)); - } -#endif + for (; i < result->word_alloc; ++i) { + eword_t word = result->words[i]; + size_t pos = (i * BITS_IN_EWORD); - if (!reuse_objects) - return -1; + for (offset = 0; offset < BITS_IN_EWORD; ++offset) { + if ((word >> offset) == 0) + break; - if (reuse_objects >= bitmap_git->pack->num_objects) { - bitmap_git->reuse_objects = *entries = bitmap_git->pack->num_objects; - *up_to = -1; /* reuse the full pack */ - *packfile = bitmap_git->pack; - return 0; + offset += ewah_bit_ctz64(word >> offset); + try_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs); + } } - reuse_threshold = bitmap_popcount(bitmap_git->result) * REUSE_PERCENT; + unuse_pack(&w_curs); - if (reuse_objects < reuse_threshold) + *entries = bitmap_popcount(reuse); + if (!*entries) { + bitmap_free(reuse); return -1; + } - bitmap_git->reuse_objects = *entries = reuse_objects; - *up_to = bitmap_git->pack->revindex[reuse_objects].offset; - *packfile = bitmap_git->pack; - + /* + * Drop any reused objects from the result, since they will not + * need to be handled separately. + */ + bitmap_and_not(result, reuse); + *packfile_out = bitmap_git->pack; + *reuse_out = reuse; return 0; } diff --git a/pack-bitmap.h b/pack-bitmap.h index 6ab6033dbe..bcd03b8993 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -50,7 +50,8 @@ void test_bitmap_walk(struct rev_info *revs); struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs); int reuse_partial_packfile_from_bitmap(struct bitmap_index *, struct packed_git **packfile, - uint32_t *entries, off_t *up_to); + uint32_t *entries, + struct bitmap **reuse_out); int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping, kh_oid_map_t *reused_bitmaps, int show_progress); void free_bitmap_index(struct bitmap_index *); From patchwork Wed Dec 18 11:25:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300323 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 44FC514B7 for ; Wed, 18 Dec 2019 11:26:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2209D227BF for ; Wed, 18 Dec 2019 11:26:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="m10ixN2v" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726921AbfLRL0Z (ORCPT ); Wed, 18 Dec 2019 06:26:25 -0500 Received: from mail-wr1-f65.google.com ([209.85.221.65]:36539 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726895AbfLRL0U (ORCPT ); Wed, 18 Dec 2019 06:26:20 -0500 Received: by mail-wr1-f65.google.com with SMTP id z3so1893667wru.3 for ; Wed, 18 Dec 2019 03:26:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ZU0Fq2ekjvI6QJ11VjQCePiqYvjvOa6DerBBZH6+aEc=; b=m10ixN2vBc60PDGF0bgaSjxbFuPz/YJ2P/J/vFX0F6yDN34gBLrUXi3KqJua9QIVSi LKg9y1WuyjdvDrvRJ+J4I4cyVCAY+7f74HlwdDRNTQoo6RKk9ain2VPVfmAsPVO50v0H C1bJCo5TWmLG9LsObOXrMD5b2+94aJIaapfvcTUkcVQRKEadUteCx8u/6OPkPdoSTNUB tinc1XUZQNMATtHWnqLFvISjyF+aZe4ELKZPM4yT0CXXBv9hLsCcHDLarKRd5aamMgBf +4XnPmS9H8wNF/V+9YJPg1s0XeaaOy0lOSrwUVqL7RNphjgoq/3llbCV7E2k5UtTh29O 4ngA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ZU0Fq2ekjvI6QJ11VjQCePiqYvjvOa6DerBBZH6+aEc=; b=cNlKt69XdmdINBmwT70mZAjw19+yewxak0z5TG01RAqIP6ULXCGwtAOuc5Vjlj9PQB I+olrRB4KtVokyAf+eU0v6WMoL931aiAN5dzodid3jkOfxXl9cK9RY4JldF5IQm6Aos8 pBGkdva2bBf87FdrF9rTHdGnx4R8rEhv8aOKJ03t110CUeLK3LvIuokwB8YY5uPYnrlA xkfDj/AC++yJHpN9uOnD2+qwRqeMUJ/LVsGUE+n36D5vZcPV8Jsn7oqxdozIswU+cklv nzFNwfFJRrcjzLGR0NLar5zBR8PbG21WCIg6m24E8SuCrBYD4Cp0lsT5VtaB6oAcqwu2 RQ8g== X-Gm-Message-State: APjAAAXI7SmnFZRmA+JaUcLh2uDqhsoPuPWrCPo1dHzWT3v7I2ZvM1e+ l/31vTh9rWudq3XrRS+72KdOhKUa X-Google-Smtp-Source: APXvYqwqTtno13MiyDnuhYed+nwRcYOn/hcEqwuEHduPbhLsZvbvP92h2eFvYKXVt/Xif//1djqQLw== X-Received: by 2002:a5d:4a8c:: with SMTP id o12mr2244413wrq.43.1576668378540; Wed, 18 Dec 2019 03:26:18 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.17 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:17 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 11/12] pack-objects: add checks for duplicate objects Date: Wed, 18 Dec 2019 12:25:46 +0100 Message-Id: <20191218112547.4974-12-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King Additional checks are added in have_duplicate_entry() and obj_is_packed() to avoid duplicate objects in the reuse bitmap. It was probably buggy to not have such a check before. Git as a client would never both asks for a tag by sha1 and specify "include-tag", but libgit2 will, so a libgit2 client cloning from a Git server would trigger the bug. If a client both asks for a tag by sha1 and specifies "include-tag", we may end up including the tag in the reuse bitmap (due to the first thing), and then later adding it to the packlist (due to the second). This results in duplicate objects in the pack, which git chokes on. We should notice that we are already including it when doing the include-tag portion, and avoid adding it to the packlist. The simplest place to fix this is right in add_ref_tag(), where we could avoid peeling the tag at all if we know that we are already including it. However, this pushes the check instead into have_duplicate_entry(). This fixes not only this case, but also means that we cannot have any similar problems lurking in other code. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- builtin/pack-objects.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index c80c1fac94..b1998202fb 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1127,6 +1127,10 @@ static int have_duplicate_entry(const struct object_id *oid, { struct object_entry *entry; + if (reuse_packfile_bitmap && + bitmap_walk_contains(bitmap_git, reuse_packfile_bitmap, oid)) + return 1; + entry = packlist_find(&to_pack, oid); if (!entry) return 0; @@ -2681,7 +2685,9 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size, static int obj_is_packed(const struct object_id *oid) { - return !!packlist_find(&to_pack, oid); + return packlist_find(&to_pack, oid) || + (reuse_packfile_bitmap && + bitmap_walk_contains(bitmap_git, reuse_packfile_bitmap, oid)); } static void add_tag_chain(const struct object_id *oid) From patchwork Wed Dec 18 11:25:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11300321 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 65FE2109A for ; Wed, 18 Dec 2019 11:26:25 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4402B21D7D for ; Wed, 18 Dec 2019 11:26:25 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="MNCCvPTS" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726916AbfLRL0Y (ORCPT ); Wed, 18 Dec 2019 06:26:24 -0500 Received: from mail-wr1-f65.google.com ([209.85.221.65]:38376 "EHLO mail-wr1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726856AbfLRL0W (ORCPT ); Wed, 18 Dec 2019 06:26:22 -0500 Received: by mail-wr1-f65.google.com with SMTP id y17so1881987wrh.5 for ; Wed, 18 Dec 2019 03:26:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=7ZF7E9SDkzTDxY+2XMD91lyyqEH3cakGYfsZejGNZcg=; b=MNCCvPTSrRqqcymmanakYTzldGUeVryFqNhI3mEJTO0/Amk9k9bvFfYrybNYcGnDIx /3RwqIXCh3BeasE4+ADEf74UxykB2/cU4BYjcdF8hqsvO1AHgajZYj/CBLS4ykdc3zvL eKaBmJrbg+QnG0ujYSRNb60fnBJT/7Y/oMda4JReNd5fttnEyaiiwJiGLCtXe4GAwJKn RXrjv6p3YTKNp+ktQ24GWnuZmZzo4LBG9eU+O98IKnFp0JNI0tB9QJtebtc7T0syt5Ye 0emsCdmISIUpsAQvjOOpB5dRXamCIbiKdN7ovy4FNx29DuAnJFSQ3i9/SAx7jPG2yJg4 gVyQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=7ZF7E9SDkzTDxY+2XMD91lyyqEH3cakGYfsZejGNZcg=; b=W6fbpFFfYVhoI826na4dOSDNfK+bQMZ/9Vk9ZcDfa8wQ7IcUlQH3eMKzHBOX2xwo4U 6NJyTjsfVvLJuo8UgbuPot04W0JLGgcxjIFzDzYmasWC3BABr9uY0AmZiOjztoHgE1B5 NwIOu/pA7aw2kTm8zgOXDtLG0XQqnXJE2eRWmTRuu4D4HF2JmKsw1aeUbJgjZwSodimP jDHgZPq9hVNx6fM+hTKtINeSQVqheMUfTVzYM6+8LdmQfioqHxPAr/VSEuwAB4TxtQXC DhKBs4HuRIAynLBgvjhMORaYVAdGH9Mj+/zB/wijomtEBDFfAWKrF9aF9CRJ/vHEaxrm eohg== X-Gm-Message-State: APjAAAWSs/VGL6cYWafbvNVwzOQuKCJznx/iip4E5Ups5xW36k1SbBe3 96QhHVru980g4YiSNnSTE4Jv6lyA X-Google-Smtp-Source: APXvYqydYYPMrqyGWjvcdvkEpUB9a6vF1J9ogpoMOfCTEMLgLjmqQYjbUszyaigoDeXwEFscSr+8GQ== X-Received: by 2002:adf:f604:: with SMTP id t4mr2384095wrp.33.1576668379731; Wed, 18 Dec 2019 03:26:19 -0800 (PST) Received: from localhost.localdomain ([2a04:cec0:116b:f417:15eb:66f2:c98d:a463]) by smtp.gmail.com with ESMTPSA id f127sm1204647wma.4.2019.12.18.03.26.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 18 Dec 2019 03:26:19 -0800 (PST) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , Jonathan Tan Subject: [PATCH v4 12/12] pack-bitmap: don't rely on bitmap_git->reuse_objects Date: Wed, 18 Dec 2019 12:25:47 +0100 Message-Id: <20191218112547.4974-13-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.24.1.498.g561400140f In-Reply-To: <20191218112547.4974-1-chriscool@tuxfamily.org> References: <20191218112547.4974-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King We no longer compute bitmap_git->reuse_objects, so we cannot rely on it anymore to terminate the loop early; we have to iterate to the end. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- pack-bitmap.c | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index a1d43d367a..5a8689cdf8 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -629,7 +629,7 @@ static void show_objects_for_type( enum object_type object_type, show_reachable_fn show_reach) { - size_t pos = 0, i = 0; + size_t i = 0; uint32_t offset; struct ewah_iterator it; @@ -637,13 +637,15 @@ static void show_objects_for_type( struct bitmap *objects = bitmap_git->result; - if (bitmap_git->reuse_objects == bitmap_git->pack->num_objects) - return; - ewah_iterator_init(&it, type_filter); - while (i < objects->word_alloc && ewah_iterator_next(&filter, &it)) { + for (i = 0; i < objects->word_alloc && + ewah_iterator_next(&filter, &it); i++) { eword_t word = objects->words[i] & filter; + size_t pos = (i * BITS_IN_EWORD); + + if (!word) + continue; for (offset = 0; offset < BITS_IN_EWORD; ++offset) { struct object_id oid; @@ -655,9 +657,6 @@ static void show_objects_for_type( offset += ewah_bit_ctz64(word >> offset); - if (pos + offset < bitmap_git->reuse_objects) - continue; - entry = &bitmap_git->pack->revindex[pos + offset]; nth_packed_object_oid(&oid, bitmap_git->pack, entry->nr); @@ -666,9 +665,6 @@ static void show_objects_for_type( show_reach(&oid, object_type, 0, hash, bitmap_git->pack, entry->offset); } - - pos += BITS_IN_EWORD; - i++; } }