From patchwork Fri Sep 13 13:02:17 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11144745 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B9CDB912 for ; Fri, 13 Sep 2019 13:02:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 97E36206BB for ; Fri, 13 Sep 2019 13:02:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="QuxB0vpi" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388162AbfIMNCx (ORCPT ); Fri, 13 Sep 2019 09:02:53 -0400 Received: from mail-ed1-f66.google.com ([209.85.208.66]:37740 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387878AbfIMNCw (ORCPT ); Fri, 13 Sep 2019 09:02:52 -0400 Received: by mail-ed1-f66.google.com with SMTP id i1so27021496edv.4 for ; Fri, 13 Sep 2019 06:02:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=vbJCW7SAsR7LZySePzU4Jn4AHu3TVJFjRIctZSV/Dj0=; b=QuxB0vpi98Q3qH0Yx/aoJxKAt4iX2ax71MV2SAxFGfuMFNTqrhzQbBLbZgwYaSvcI6 nY6Nk2Od9cQ6BEO79DEIdVan4Md1aoc+p7fwA0WaRaYLkukn3vTmnQJ7A8NUnksARpsR F402NCdjiLl6pPfv7iMQ71czJM1MEDqNCRSgju+KOVxYyxP2VHvAHWtE0H3W45X6lAK8 QhV9twdYohUse0KM571L+rNCesM7wtm01R+09ld4KvJKdhbV6vvap/0LoxXv3D2jGP3v 71kdRZSz7H4tb5n9MCp2xxdKhRZmO88JKUsxKedZMOSH74gozAfa6T2TK01cHri3Zzxd INeA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=vbJCW7SAsR7LZySePzU4Jn4AHu3TVJFjRIctZSV/Dj0=; b=gDDLT4pDz8IvT+410EZLyP8CtJfOwnQpWF7LnmhaEUCkYmCiiBvJMurNY28pLThdFT 9Wh+61OJ0hBnn1SQq1xChGrkthkwwvPXgcmDFk1PenE9eJkzFd4G39RW0lHPvQNFchFL GaiD6bPuRPW0xRL21aAIb7BwIin6ORP/2l/d7+N6K8uDIvw+azd2rp2gwQ6+N/CUV3/r pSFSPVp4Bpk9+P31OB8YkBqPgmEfFhv/pbEGH+vc17h7/ld328iTo2mWe+ZlUqBDmU/B 1WZDLvjYhDDvn1luurlxnsyy/R5AtsqIkxtsU2oagsmZ8Rck8vr54MvImSlD8ity7SfX oqgw== X-Gm-Message-State: APjAAAVpfC0uUdLm05RBAfJs6PbSAky6ZwctN6UfE6YvxEthrnMapqc3 gWm71V2m8aw/qJJgsBVEQnpXVpuB X-Google-Smtp-Source: APXvYqyijGQz2YAblPLzHQ6vUVUSfq977LKlycBOUjUe1uDI8aNaqrF6nGVyZiK+Y1WVZ2MsKX6PqQ== X-Received: by 2002:a50:8f81:: with SMTP id y1mr47378968edy.181.1568379770154; Fri, 13 Sep 2019 06:02:50 -0700 (PDT) Received: from ubuntu-N150ZU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id c14sm2846570ejx.31.2019.09.13.06.02.49 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2019 06:02:49 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones , James Ramsay Subject: [RFC PATCH 01/10] builtin/pack-objects: report reused packfile objects Date: Fri, 13 Sep 2019 15:02:17 +0200 Message-Id: <20190913130226.7449-2-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.23.0.46.gd213b4aca1.dirty In-Reply-To: <20190913130226.7449-1-chriscool@tuxfamily.org> References: <20190913130226.7449-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King To see when packfile reuse kicks in or not, it is useful to show reused packfile objects statistics in the output of upload-pack. Helped-by: James Ramsay Signed-off-by: Jeff King Signed-off-by: Christian Couder --- builtin/pack-objects.c | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 76ce906946..c11b2ea8d4 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -3525,7 +3525,9 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) if (progress) fprintf_ln(stderr, _("Total %"PRIu32" (delta %"PRIu32")," - " reused %"PRIu32" (delta %"PRIu32")"), - written, written_delta, reused, reused_delta); + " reused %"PRIu32" (delta %"PRIu32")," + " pack-reused %"PRIu32), + written, written_delta, reused, reused_delta, + reuse_packfile_objects); return 0; } From patchwork Fri Sep 13 13:02:18 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11144747 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E6A4F17E6 for ; Fri, 13 Sep 2019 13:02:53 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id C4ED6206BB for ; Fri, 13 Sep 2019 13:02:53 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="AbBjAGnc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388175AbfIMNCx (ORCPT ); Fri, 13 Sep 2019 09:02:53 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:38787 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387714AbfIMNCx (ORCPT ); Fri, 13 Sep 2019 09:02:53 -0400 Received: by mail-ed1-f67.google.com with SMTP id a23so24753698edv.5 for ; Fri, 13 Sep 2019 06:02:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=FE65YJQO7zrKfVbH++0ZWusJuB9EyAxSzwJdZ0cmVTI=; b=AbBjAGncAOgboC7D3yPlYZ2UHWUKvADVG5m+LlAGGTTJ+hiAVLdDLmqIPYbLixY3/r ub6gWc/GJhjkyT5bWacsJrPBcfySZXpJq49O1HbAGceEgxjGFUJeRpM1ijlftJmG1YUt 1KXBlmuPX72nFczsWh3mSXe+4w7z3xafUmcNIBD8fqEAyeQhSy71XO/U4EJccM78Iadu sY854heiFWgCJjXopRtmvjozxcPf++l+FuXU6SpTUI7LsIILVgi6k4ZOUTxNthhkQybp eWq6X0Vcm8/930NMpEiOL2ckzWmkgJIJ1DQb33ICK/ALrYLpmesxZl4DLTCxlQ/qnA42 v82Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=FE65YJQO7zrKfVbH++0ZWusJuB9EyAxSzwJdZ0cmVTI=; b=pFSjZczAGXFvSzcAdLXGytiAaNgHdaeeBbxcL8bByddvZfmYMItP6Wd+rFVqGdn8jb Z3PGrDwf7JvJQR8bVjNXiVpfDV5b4nmCxuR+ND2kEmIQdJ0EOn713ia7Gelk+Xax90y2 i4xJEICwx+Xh+qXlzAd71Wd8FQ7dCXohGhw3RiRj6j+dI2GddYm7kNxqQCBo+PA4OdyW ao+Y+KDZz1o89B6MO531uRZ82DloCboWlw/0t82a+Eb87VTE387uOPHLGEI8nRu0bNHQ cj70LLSJnDkBvF7S17wNIjgCjFvEdQhmJZjzr9diPBOKcBxjS6yhEdKP+1z5Z08dPv1V bcYQ== X-Gm-Message-State: APjAAAVKCPggnjYohlNTDN3e3hJjfFW8Oy2Dd17m/7dS9tUdam/jS0AU G4la7axAHTiqYhFzkjv9pWhnHJBd X-Google-Smtp-Source: APXvYqzWhXxFL/ELy/3lw9Te3kLPS3FRY0bBMQ4tJiwuQiY8X3w/pCp/Umll/EzU1HfCG2c/RAwfsw== X-Received: by 2002:a17:906:76c2:: with SMTP id q2mr26568554ejn.202.1568379770792; Fri, 13 Sep 2019 06:02:50 -0700 (PDT) Received: from ubuntu-N150ZU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id c14sm2846570ejx.31.2019.09.13.06.02.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2019 06:02:50 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones Subject: [RFC PATCH 02/10] packfile: expose get_delta_base() Date: Fri, 13 Sep 2019 15:02:18 +0200 Message-Id: <20190913130226.7449-3-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.23.0.46.gd213b4aca1.dirty In-Reply-To: <20190913130226.7449-1-chriscool@tuxfamily.org> References: <20190913130226.7449-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King In a following commit get_delta_base() will be used outside packfile.c, so let's make it non static and declare it in packfile.h. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- packfile.c | 10 +++++----- packfile.h | 3 +++ 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/packfile.c b/packfile.c index fc43a6c52c..5a6e8d54f1 100644 --- a/packfile.c +++ b/packfile.c @@ -1191,11 +1191,11 @@ const struct packed_git *has_packed_and_bad(struct repository *r, return NULL; } -static off_t get_delta_base(struct packed_git *p, - struct pack_window **w_curs, - off_t *curpos, - enum object_type type, - off_t delta_obj_offset) +off_t get_delta_base(struct packed_git *p, + struct pack_window **w_curs, + off_t *curpos, + enum object_type type, + off_t delta_obj_offset) { unsigned char *base_info = use_pack(p, w_curs, *curpos, NULL); off_t base_offset; diff --git a/packfile.h b/packfile.h index 3e98910bdd..8049202f4c 100644 --- a/packfile.h +++ b/packfile.h @@ -151,6 +151,9 @@ void *unpack_entry(struct repository *r, struct packed_git *, off_t, enum object unsigned long unpack_object_header_buffer(const unsigned char *buf, unsigned long len, enum object_type *type, unsigned long *sizep); unsigned long get_size_from_delta(struct packed_git *, struct pack_window **, off_t); int unpack_object_header(struct packed_git *, struct pack_window **, off_t *, unsigned long *); +off_t get_delta_base(struct packed_git *p, struct pack_window **w_curs, + off_t *curpos, enum object_type type, + off_t delta_obj_offset); void release_pack_memory(size_t); From patchwork Fri Sep 13 13:02:19 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11144757 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 705341395 for ; Fri, 13 Sep 2019 13:02:59 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 4FB9E2084F for ; Fri, 13 Sep 2019 13:02:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="pVCw3wRo" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388191AbfIMNCz (ORCPT ); Fri, 13 Sep 2019 09:02:55 -0400 Received: from mail-ed1-f47.google.com ([209.85.208.47]:44050 "EHLO mail-ed1-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2387714AbfIMNCz (ORCPT ); Fri, 13 Sep 2019 09:02:55 -0400 Received: by mail-ed1-f47.google.com with SMTP id p2so25761883edx.11 for ; Fri, 13 Sep 2019 06:02:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QYPNFBHT1ek2V+d9R1QZYL+nVum/w0iFSuFz73KLjMc=; b=pVCw3wRo53TVcAjP+9CCJ1wiabVYIbBXqmDlgXW6j2USN0tn/xZAaNwzhti59afTuX DGJ66sClPjGTgkwlz2OsyJDHzlKGSpxrDN32POrCn+BGurX1BlHX0fSBtuhMdcxsh3Ht lPLGsWdfIsgu7oCDL2NvMCF4g6sEZhANPtGDBYALJhFOQDE/UiOWI3iIO+2rUDGa9snA Kx+9DRERBQf/YDBB4jpXxDWc3bIkaegTqTeVCFHTtM84XVeGxu+RUPry/Kb7//NyT07H aFXQMRpp+XSkTl2TgR5U5lomrFvHyAGxuym8Ks0ws7ygzYJS3FaAsKzLK29GCha3bci9 i8kw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QYPNFBHT1ek2V+d9R1QZYL+nVum/w0iFSuFz73KLjMc=; b=Ea6Yppk49QOV7tSfBvI2aqy9uSki/Ll6UoCT9zXZx3XjaKA6CUQ2dWMxqstZTdBxZG MgGN3SsF6bDu4P2ALkaQ4Q9bm6uja3243VMnhDqP5SslaLdQVoU3D1KgnXTJcNj54x9G XQAlypG1VYun2sKnhzXM0xh1rpWaqKP0QhBEd+Ng3KdR9X+4sab9MhoXUP7d6Hkm80CG X10WmMP6KoP/CH2SfnfULaCTkPjGqp5rPKn8kD3ojINMKy/v/fcB8k+XoA1OpmT76ZRx uyjGSl4TguYAZ9R4Hk3kr3K9nvQGgMdf8LSmsjPxb3HOvTs7w+Utrb4dEllMhizoXaex nfLg== X-Gm-Message-State: APjAAAVjf2vPY8eDmsZofoM/BbfOH73evNZnpYdJCbW1D96Yg7bVksGA hipEp+AV01m+6lFWwqTOoa9YHbLG X-Google-Smtp-Source: APXvYqwDLXWjGbTdZyU0irINPjhWWGTNAG3zTad5ema2lcOqdx6TRP19wVaYPsc4KgY+xySjzHOJdA== X-Received: by 2002:a17:906:d9d9:: with SMTP id qk25mr40536479ejb.275.1568379771429; Fri, 13 Sep 2019 06:02:51 -0700 (PDT) Received: from ubuntu-N150ZU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id c14sm2846570ejx.31.2019.09.13.06.02.50 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2019 06:02:50 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones Subject: [RFC PATCH 03/10] ewah/bitmap: introduce bitmap_word_alloc() Date: Fri, 13 Sep 2019 15:02:19 +0200 Message-Id: <20190913130226.7449-4-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.23.0.46.gd213b4aca1.dirty In-Reply-To: <20190913130226.7449-1-chriscool@tuxfamily.org> References: <20190913130226.7449-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King In a following patch we will need to allocate a variable number of bitmap words, instead of always 32, so let's add bitmap_word_alloc() for this purpose. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- ewah/bitmap.c | 11 ++++++++--- ewah/ewok.h | 1 + 2 files changed, 9 insertions(+), 3 deletions(-) diff --git a/ewah/bitmap.c b/ewah/bitmap.c index 52f1178db4..143dc71419 100644 --- a/ewah/bitmap.c +++ b/ewah/bitmap.c @@ -22,14 +22,19 @@ #define EWAH_MASK(x) ((eword_t)1 << (x % BITS_IN_EWORD)) #define EWAH_BLOCK(x) (x / BITS_IN_EWORD) -struct bitmap *bitmap_new(void) +struct bitmap *bitmap_word_alloc(size_t word_alloc) { struct bitmap *bitmap = xmalloc(sizeof(struct bitmap)); - bitmap->words = xcalloc(32, sizeof(eword_t)); - bitmap->word_alloc = 32; + bitmap->words = xcalloc(word_alloc, sizeof(eword_t)); + bitmap->word_alloc = word_alloc; return bitmap; } +struct bitmap *bitmap_new(void) +{ + return bitmap_word_alloc(32); +} + void bitmap_set(struct bitmap *self, size_t pos) { size_t block = EWAH_BLOCK(pos); diff --git a/ewah/ewok.h b/ewah/ewok.h index 84b2a29faa..1b98b57c8b 100644 --- a/ewah/ewok.h +++ b/ewah/ewok.h @@ -172,6 +172,7 @@ struct bitmap { }; struct bitmap *bitmap_new(void); +struct bitmap *bitmap_word_alloc(size_t word_alloc); void bitmap_set(struct bitmap *self, size_t pos); int bitmap_get(struct bitmap *self, size_t pos); void bitmap_reset(struct bitmap *self); From patchwork Fri Sep 13 13:02:20 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11144751 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8A9FC1395 for ; Fri, 13 Sep 2019 13:02:57 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6883E206BB for ; Fri, 13 Sep 2019 13:02:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="J1GEaYNc" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388202AbfIMNC4 (ORCPT ); Fri, 13 Sep 2019 09:02:56 -0400 Received: from mail-ed1-f66.google.com ([209.85.208.66]:35037 "EHLO mail-ed1-f66.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388177AbfIMNCz (ORCPT ); Fri, 13 Sep 2019 09:02:55 -0400 Received: by mail-ed1-f66.google.com with SMTP id f24so9363802edv.2 for ; Fri, 13 Sep 2019 06:02:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=c435ouO4rFUck2s9Wz+ve8vT5VdFElRswz0pkOdLSpY=; b=J1GEaYNcl6o2CkCnW0OvJ0FPS67AYLFo/8XffmOHVcXK7HZuxBLEZdeSxHXSqXbDlV GBseJjERoyv6N6UnsY5XGrSAyIWbYZaILaZT9AkFKVG3Tdu9LjtLPhkZtb8jufvXMIl1 HkukT3El5hVXY9BDGTZLhG3PiNI36iA/5jWoXLI6WvNscig6AiN6LJEHpR7mwWwoHu3p PeakTTC3QCYksZwZkigSFyH1Ao+m8GXaBczvaFTjMbtJvVX+Knz8UtslegPWlyEB+SeQ TtdWODYQr1x3ddDot5vevyJ/bh2fqvvW2d6RYZNgxXaBq+1P0S1kGF6PjUWoyFwftteH 8wxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=c435ouO4rFUck2s9Wz+ve8vT5VdFElRswz0pkOdLSpY=; b=ElyEox5SAJ3vMoxgS/VeZLTBcU6xt5A2sHCb9IbqE8a01S1Gimu4RGb8JMMeAKb1L2 AofwtYr7KF/en392tuHSiXC+EQW66qtPGSLpVPfyfjLgUPCcoMHwEa+qv5ofJr2vlh2a XQcWUetZ4olTQ64rEJBi5q96Ir+5cb8mI7qng/RGm2CIMDZmFqw3/E5VCzrP4qvoIMWc mDBfzXi+OAoy9dNoRPy2N3vJcfTPZr6BKbUroaX7DcT6tdii/De4d+MJLvEB+oxjzseO 5V/azHrLlnbrDaWCjp/EMuFRmizxfN5doVJOJKHbPAiXRoO6osL1O9I4vW0Zac8AMLcK ZPbg== X-Gm-Message-State: APjAAAVlBEYj0nyBh+Tc9PI9jFFgk/ZhZ4h+OGHvXxgb9i2iodfCtXoP tkOjO9rLuZASCjckTfmHFKYHJXOZ X-Google-Smtp-Source: APXvYqxklacZmwB/wR8cASkj0KDH7/wwymVQnIW/EQ9Azpzyvpm1D2N4AJHHlqNYb2r4dvwxHdV++w== X-Received: by 2002:a50:e806:: with SMTP id e6mr46388012edn.162.1568379772071; Fri, 13 Sep 2019 06:02:52 -0700 (PDT) Received: from ubuntu-N150ZU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id c14sm2846570ejx.31.2019.09.13.06.02.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2019 06:02:51 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones Subject: [RFC PATCH 04/10] ewah/bitmap: always allocate 2 more words Date: Fri, 13 Sep 2019 15:02:20 +0200 Message-Id: <20190913130226.7449-5-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.23.0.46.gd213b4aca1.dirty In-Reply-To: <20190913130226.7449-1-chriscool@tuxfamily.org> References: <20190913130226.7449-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King In a following patch we will allocate a variable number of words in some bitmaps. When iterating over the words we will need a mark to tell us when to stop iterating. Let's always allocate 2 more words, that will always contain 0, as that mark. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- ewah/bitmap.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/ewah/bitmap.c b/ewah/bitmap.c index 143dc71419..eac05485f1 100644 --- a/ewah/bitmap.c +++ b/ewah/bitmap.c @@ -41,7 +41,7 @@ void bitmap_set(struct bitmap *self, size_t pos) if (block >= self->word_alloc) { size_t old_size = self->word_alloc; - self->word_alloc = block * 2; + self->word_alloc = (block + 1) * 2; REALLOC_ARRAY(self->words, self->word_alloc); memset(self->words + old_size, 0x0, (self->word_alloc - old_size) * sizeof(eword_t)); From patchwork Fri Sep 13 13:02:21 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11144759 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8352C1395 for ; Fri, 13 Sep 2019 13:03:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 618E3206BB for ; Fri, 13 Sep 2019 13:03:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ecdv582R" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388220AbfIMNC7 (ORCPT ); Fri, 13 Sep 2019 09:02:59 -0400 Received: from mail-ed1-f68.google.com ([209.85.208.68]:33166 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388187AbfIMNC4 (ORCPT ); Fri, 13 Sep 2019 09:02:56 -0400 Received: by mail-ed1-f68.google.com with SMTP id o9so27029269edq.0 for ; Fri, 13 Sep 2019 06:02:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=TAPD2AriovQ0IcJpCJVxop9CRVM9q4TT+plx+M33Ipk=; b=ecdv582RfjSCGFow9VJ5K46PwKUlwxiVw+/xUk88i6FzPHO2iy5NE0D2ld7kyS9TtH kLcpklZ7eCR6Lo+OA5CluUBhT1aISVXNbxMGrYryqLaQgVKl5qnyQQWJiolCB6d29zLV AR8Bn+W2FRFOh88cgCyR8N3KqsPN+EpPOUv/xDud4MJcXURx/NUX7m5+0qgsM3TnaaC6 9U/Lf82A9dfg79unERr4bf3krNDIURxctU1/+34KaaulnyQBKifG0iYywyE/eFjrGx6J JQN1eOY7BD8JuyHmrl2eaevzqvQArKzSyHcdyVlP8vVladR3xYVpywZE7qmvWxU9/Lcm hKsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=TAPD2AriovQ0IcJpCJVxop9CRVM9q4TT+plx+M33Ipk=; b=HFFRKFutJI27ThpoLLQyoCpYaJ70ng7o0E4MilJPVBzE5gvJGcnEwvvvf5K7t67l3j iIZyxDq/9C3QNvWNtbj/pI+cx/R/BcA4zKndsUYlvBNsFZInWZwZzLxz0x/gJQHbSi+P lw6KBkDt9QFpmxEztvjFnVuDrZgV3navT2vXduIjFTYTooDhwpaKA5CCzzp17V8dqFVd HyWuObncNlYEEAfR013DDDKMLV0x5rPiB9rx+7e1igFFnhyuDr/HCFAY+QiZUg50KUdX iyJxnwj4laOuxGKpfdUt6brtke8vQV/4ShCFrmnicpv6Zh+rQdU3IO/I9GELHcjr7iR3 uyow== X-Gm-Message-State: APjAAAURZ6/OaOcx68Sj6TPiddrH3LIOxBhPOxyvqDNq28lmfjf96+4G IArQV4I+1mUrjWRlvIO/oM1WUk6V X-Google-Smtp-Source: APXvYqwlHMYAe9yCDUzP1vlctXuf2NRIYD/3UNfRWuVg9YNDf4LNcikymac9L/gHl+N+tieUvwbUbA== X-Received: by 2002:a50:baab:: with SMTP id x40mr22135084ede.60.1568379772659; Fri, 13 Sep 2019 06:02:52 -0700 (PDT) Received: from ubuntu-N150ZU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id c14sm2846570ejx.31.2019.09.13.06.02.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2019 06:02:52 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones Subject: [RFC PATCH 05/10] pack-bitmap: don't rely on bitmap_git->reuse_objects Date: Fri, 13 Sep 2019 15:02:21 +0200 Message-Id: <20190913130226.7449-6-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.23.0.46.gd213b4aca1.dirty In-Reply-To: <20190913130226.7449-1-chriscool@tuxfamily.org> References: <20190913130226.7449-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King As we now allocate 2 more words than necessary for each bitmap to serve as marks telling us that we can stop iterating over the words, we don't need to rely on bitmap_git->reuse_objects to stop iterating over the words. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- pack-bitmap.c | 18 +++++++----------- 1 file changed, 7 insertions(+), 11 deletions(-) diff --git a/pack-bitmap.c b/pack-bitmap.c index ed2befaac6..b2422fed8f 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -622,7 +622,7 @@ static void show_objects_for_type( enum object_type object_type, show_reachable_fn show_reach) { - size_t pos = 0, i = 0; + size_t i = 0; uint32_t offset; struct ewah_iterator it; @@ -630,13 +630,15 @@ static void show_objects_for_type( struct bitmap *objects = bitmap_git->result; - if (bitmap_git->reuse_objects == bitmap_git->pack->num_objects) - return; - ewah_iterator_init(&it, type_filter); - while (i < objects->word_alloc && ewah_iterator_next(&filter, &it)) { + for (i = 0; i < objects->word_alloc && + ewah_iterator_next(&filter, &it); i++) { eword_t word = objects->words[i] & filter; + size_t pos = (i * BITS_IN_EWORD); + + if (!word) + continue; for (offset = 0; offset < BITS_IN_EWORD; ++offset) { struct object_id oid; @@ -648,9 +650,6 @@ static void show_objects_for_type( offset += ewah_bit_ctz64(word >> offset); - if (pos + offset < bitmap_git->reuse_objects) - continue; - entry = &bitmap_git->pack->revindex[pos + offset]; nth_packed_object_oid(&oid, bitmap_git->pack, entry->nr); @@ -659,9 +658,6 @@ static void show_objects_for_type( show_reach(&oid, object_type, 0, hash, bitmap_git->pack, entry->offset); } - - pos += BITS_IN_EWORD; - i++; } } From patchwork Fri Sep 13 13:02:22 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11144749 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 89EF01395 for ; Fri, 13 Sep 2019 13:02:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 6726C206BB for ; Fri, 13 Sep 2019 13:02:56 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="SfgsLQNO" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388195AbfIMNCz (ORCPT ); Fri, 13 Sep 2019 09:02:55 -0400 Received: from mail-ed1-f65.google.com ([209.85.208.65]:43926 "EHLO mail-ed1-f65.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388176AbfIMNCz (ORCPT ); Fri, 13 Sep 2019 09:02:55 -0400 Received: by mail-ed1-f65.google.com with SMTP id c19so26921581edy.10 for ; Fri, 13 Sep 2019 06:02:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=nbIYPYfvlZmjFMk31c6i2QEUyWC7Lyz7Rjql2VovRmI=; b=SfgsLQNOYhaf9jgQ8sXtJE4w18WxWiwJuvf9NlbVahyBXcFAK8dAab0plOXEn2RT+4 HvWHyt1wm8ZmdmD9jCi4CUAZoPWPNk5pu3mfnkAwQiZMaOEL6f0bFkNgEdLbeKTtLsWI zRwA/dmDfPWhqt3dHflnpDXWMAeuQBv63qPv95I0pCHQ7H9K04XidS/8ykxJ0LSdzZfs kV+BxvJ9O3PxQXXnn0teeoa97VLC4hlWRylPlQHBiTu+xCLZEsor4yV0JASNFw66TjQq BLeaY6/BAcgxZDbzGGbV75yJKAJD6K7HPK2IotoXh4KacqkP4/m4ti2hFzwRz8pfdMX9 ZoAw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=nbIYPYfvlZmjFMk31c6i2QEUyWC7Lyz7Rjql2VovRmI=; b=q5myxXErhCqgOiz91iR7/O30Sn+PRRemL4FQ7Vca4JlnY7nUvl1ka1l1LeIOpTP/l3 JbZpznytiGgbEuFn3keMclI10FhNKrDtpUeLom7dYn12fXOJUrgXcvJ2UhyvUGHQ/BqY pK13zUJiP9qpQoikohh25JI7tMWc4Qs2l6nk2OW2NS2nqUn7JPWcROa2aaLCXDqi1w5n yw0F+MOnwhjGSrAq9vtV+JSFg1fX8/wUy6obQWRBN98zYhrx4wrYqMpADUT61o+umhUu j1hPw6XzSOoFhvfOonY4OPxjFVpGijv8grV2RZMfCD9Xpj2tBXxlXLmiyruckLBz1krs feYQ== X-Gm-Message-State: APjAAAVF011VpzCo+CAWfozeO9uoJj2J/JRaSkA5PJB6RIHY27fgGZi2 gJIaJZr7PWXSrG6PtpD+nslRjLMZ X-Google-Smtp-Source: APXvYqxZiErAo4ui1dXbzWNTXD4Z0xeD5GDR6R+aDLo6zLJLDBPeckf+yuI92NOVi/ldTakIaTQIlQ== X-Received: by 2002:a17:906:79ca:: with SMTP id m10mr3712986ejo.292.1568379773221; Fri, 13 Sep 2019 06:02:53 -0700 (PDT) Received: from ubuntu-N150ZU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id c14sm2846570ejx.31.2019.09.13.06.02.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2019 06:02:52 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones Subject: [RFC PATCH 06/10] pack-bitmap: introduce bitmap_walk_contains() Date: Fri, 13 Sep 2019 15:02:22 +0200 Message-Id: <20190913130226.7449-7-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.23.0.46.gd213b4aca1.dirty In-Reply-To: <20190913130226.7449-1-chriscool@tuxfamily.org> References: <20190913130226.7449-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King We will use this helper function in a following patch to tell us if an object is packed. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- pack-bitmap.c | 12 ++++++++++++ pack-bitmap.h | 3 +++ 2 files changed, 15 insertions(+) diff --git a/pack-bitmap.c b/pack-bitmap.c index b2422fed8f..1833971dc7 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -828,6 +828,18 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, return 0; } +int bitmap_walk_contains(struct bitmap_index *bitmap_git, + struct bitmap *bitmap, const struct object_id *oid) +{ + int idx; + + if (!bitmap) + return 0; + + idx = bitmap_position(bitmap_git, oid); + return idx >= 0 && bitmap_get(bitmap, idx); +} + void traverse_bitmap_commit_list(struct bitmap_index *bitmap_git, show_reachable_fn show_reachable) { diff --git a/pack-bitmap.h b/pack-bitmap.h index 00de3ec8e4..5425767f0f 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -3,6 +3,7 @@ #include "ewah/ewok.h" #include "khash.h" +#include "pack.h" #include "pack-objects.h" struct commit; @@ -53,6 +54,8 @@ int reuse_partial_packfile_from_bitmap(struct bitmap_index *, int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping, kh_oid_map_t *reused_bitmaps, int show_progress); void free_bitmap_index(struct bitmap_index *); +int bitmap_walk_contains(struct bitmap_index *, + struct bitmap *bitmap, const struct object_id *oid); /* * After a traversal has been performed by prepare_bitmap_walk(), this can be From patchwork Fri Sep 13 13:02:23 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11144755 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C3DA917E6 for ; Fri, 13 Sep 2019 13:02:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A2C8D2084F for ; Fri, 13 Sep 2019 13:02:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="HE8xia3O" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388205AbfIMNC5 (ORCPT ); Fri, 13 Sep 2019 09:02:57 -0400 Received: from mail-ed1-f68.google.com ([209.85.208.68]:43933 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388196AbfIMNC4 (ORCPT ); Fri, 13 Sep 2019 09:02:56 -0400 Received: by mail-ed1-f68.google.com with SMTP id c19so26921650edy.10 for ; Fri, 13 Sep 2019 06:02:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=feuJkzJI/SWfV7q4mtYOBaQBB5fBDWpeiqXQ3Lw+ILY=; b=HE8xia3O13wOYxUZ9JcjjJX4GdGkg0SFvzSielw3m0z4XbqpKFxlYX7wDB2/sTuh2P 20hVR5N5A2rZwRkouSDKjZmYMK9d6SbeFvJhNYvOrAnAgGPS7Shi/MXymNl3PqzSLewv MMucDrAbTngey2oQBeC2sm4KlpP32XGqRpmOgZ+4T13y+W4mO24FTjGipZiOJRzUFsBB 6WFpOQY7K22jC8a4okmz3brGml/UDmL/3eK51WOlDg4v81XTSuR7/YiD6NDbNElzBhHY MjO/ah0vRHc1mzFzP7mndTifpiU/tAUZ10yGfEZUNtiAExznjEJ1Sz6zEeIwyJHzXWFj Zyiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=feuJkzJI/SWfV7q4mtYOBaQBB5fBDWpeiqXQ3Lw+ILY=; b=WlpR0Cmzahk95nole7IBESQV5M2Bq1Nm84Gh+X34+PIJRY9Yaw/QkViU4UOfU0FIhy 6MwfNnnTQsBP/WCCXrsZ7QJADjvp9380LwP0/m+DOIEv8fDgDyPshIylyBHl+mcX2/ST B6WRYiTkjeXXACo6oxd3hIE+BWd2siy2HSCRYXoXNWuJ5qr8Wtg+rB8IwlfQ2J0D5o7S AIfjcK68aWcpcRzYIbPRK+hUshVLuJO76jgySrBl4bh1fHpKsWydCj9LMiTvkVWiy/6s vRFcK0tzUvsFdyv6UNvyJfrJij9xbI/z3uetu+zNIuSdKdo1XsSgF1BTcmlHHcYiDpBf 8qoQ== X-Gm-Message-State: APjAAAWYtU5KoYSDR7A7SsOeLEW47oo6sK7+4GXLFxcgWUJb4Rrr+Pcq Y2L3N+M8XkPs/oF/uYOQcr3HpLgj X-Google-Smtp-Source: APXvYqzqT5yrIV/zzlA42J9tsaW9je/oRwi9maOjoMX6sWsdQIBoB+z7KeHzkImuO8BP3u8QZfK25Q== X-Received: by 2002:a50:9734:: with SMTP id c49mr48057458edb.93.1568379774506; Fri, 13 Sep 2019 06:02:54 -0700 (PDT) Received: from ubuntu-N150ZU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id c14sm2846570ejx.31.2019.09.13.06.02.53 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2019 06:02:53 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones Subject: [RFC PATCH 07/10] csum-file: introduce hashfile_total() Date: Fri, 13 Sep 2019 15:02:23 +0200 Message-Id: <20190913130226.7449-8-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.23.0.46.gd213b4aca1.dirty In-Reply-To: <20190913130226.7449-1-chriscool@tuxfamily.org> References: <20190913130226.7449-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King We will need this helper function in a following patch to give us total number of bytes fed to the hashfile so far. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- csum-file.h | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/csum-file.h b/csum-file.h index a98b1eee53..f9cbd317fb 100644 --- a/csum-file.h +++ b/csum-file.h @@ -42,6 +42,15 @@ void hashflush(struct hashfile *f); void crc32_begin(struct hashfile *); uint32_t crc32_end(struct hashfile *); +/* + * Returns the total number of bytes fed to the hashfile so far (including ones + * that have not been written out to the descriptor yet). + */ +static inline off_t hashfile_total(struct hashfile *f) +{ + return f->total + f->offset; +} + static inline void hashwrite_u8(struct hashfile *f, uint8_t data) { hashwrite(f, &data, sizeof(data)); From patchwork Fri Sep 13 13:02:24 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11144753 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5A6DE912 for ; Fri, 13 Sep 2019 13:02:58 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 399512084F for ; Fri, 13 Sep 2019 13:02:58 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ER8D/Unw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388211AbfIMNC5 (ORCPT ); Fri, 13 Sep 2019 09:02:57 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:37753 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388176AbfIMNC5 (ORCPT ); Fri, 13 Sep 2019 09:02:57 -0400 Received: by mail-ed1-f67.google.com with SMTP id i1so27021746edv.4 for ; Fri, 13 Sep 2019 06:02:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=qdwQgh7pzl+xQZbSdut6vaCCldhq0LrCbRM8FplS/8E=; b=ER8D/UnwQ0W9HKzYFlNewEwgj0KxMwdBmmQbkghH5yHHaDu8PNgDe/yGcRHA118qlp 1yZUf7Y5S3A773BMwiZbRLV7lcmNhTEi7uPCCpxFmi26QTnM9FaCLZJoahaZixki6uxh 94ppG3EQT9otlaPudL5JXqaOJcCGZ1rt2UzD0Y1BtzuAT7bldrEimgNUT9XQf8ZwzZbG 1tI0JlnxC5TlZNMKCEiXdf0rc0j93CvlcfOdkUUQClLBhIvg5v7a97UJtNAzqeo7Xhlv ir2F11/32V6JyfPoktx/x1Rrh2syXECWY6MzaWPX0A6Pq00u+vKGekT6qM393ORrbXL7 PLgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=qdwQgh7pzl+xQZbSdut6vaCCldhq0LrCbRM8FplS/8E=; b=Uy6a1KAr2ZFXQnZPNMv6j1Kgz8DU4OIg/XNJxEnCftifyi/mxPzWwd+FQz0YfyJuf6 td1A4KLozqqjWydyxXy3eC6DY9pOKVRmmCHLpYdRUTgG9XMZLegweLR2X8hEPHgH7N6m gdxYUfH2GqaiNwpvOmVpHw0gs551vM5iBEqwOLMkY6Yqh6ZLc62XGaO2taUA/sP6P6uV U8nxlEzVcc9jFZrnG8Mu05uAbgjN3JGC7sqf+JnhNscYkmZnQWnC5NNJI5ixI0vrLdyI eUWFAMpXSwArQzNtD8CxHuMAvsL7Ac5mdYoqqH10nkWKZMM02Elu80qMb+BsYL+QBtWL 3mwA== X-Gm-Message-State: APjAAAVsMhrZFBcM6IlBYEKPpvBSSdDtjCUUUPQ97K7w8CMObiBuzL71 NNjgYVrtQ/euUrzFnxzkVadFnge2 X-Google-Smtp-Source: APXvYqzfah16AKpOknv164EAUG2jhB0VD/jJbiPgyHM989c9JTF16vWSxqqie9+j7j8qg0Qt6TNaow== X-Received: by 2002:aa7:c24f:: with SMTP id y15mr49157710edo.152.1568379775126; Fri, 13 Sep 2019 06:02:55 -0700 (PDT) Received: from ubuntu-N150ZU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id c14sm2846570ejx.31.2019.09.13.06.02.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2019 06:02:54 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones Subject: [RFC PATCH 08/10] pack-objects: introduce pack.allowPackReuse Date: Fri, 13 Sep 2019 15:02:24 +0200 Message-Id: <20190913130226.7449-9-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.23.0.46.gd213b4aca1.dirty In-Reply-To: <20190913130226.7449-1-chriscool@tuxfamily.org> References: <20190913130226.7449-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King Let's make it possible to configure if we want pack reuse or not. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- builtin/pack-objects.c | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index c11b2ea8d4..1664969c97 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -96,6 +96,7 @@ static off_t reuse_packfile_offset; static int use_bitmap_index_default = 1; static int use_bitmap_index = -1; +static int allow_pack_reuse = 1; static enum { WRITE_BITMAP_FALSE = 0, WRITE_BITMAP_QUIET, @@ -2719,6 +2720,10 @@ static int git_pack_config(const char *k, const char *v, void *cb) sparse = git_config_bool(k, v); return 0; } + if (!strcmp(k, "pack.allowpackreuse")) { + allow_pack_reuse = git_config_bool(k, v); + return 0; + } if (!strcmp(k, "pack.threads")) { delta_search_threads = git_config_int(k, v); if (delta_search_threads < 0) @@ -3063,7 +3068,8 @@ static int get_object_list_from_bitmap(struct rev_info *revs) if (!(bitmap_git = prepare_bitmap_walk(revs))) return -1; - if (pack_options_allow_reuse() && + if (allow_pack_reuse && + pack_options_allow_reuse() && !reuse_partial_packfile_from_bitmap( bitmap_git, &reuse_packfile, From patchwork Fri Sep 13 13:02:25 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11144761 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C4A0F912 for ; Fri, 13 Sep 2019 13:03:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A236E206BB for ; Fri, 13 Sep 2019 13:03:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Jeht5qBq" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388218AbfIMNC7 (ORCPT ); Fri, 13 Sep 2019 09:02:59 -0400 Received: from mail-ed1-f68.google.com ([209.85.208.68]:45975 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388206AbfIMNC6 (ORCPT ); Fri, 13 Sep 2019 09:02:58 -0400 Received: by mail-ed1-f68.google.com with SMTP id f19so26938871eds.12 for ; Fri, 13 Sep 2019 06:02:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ES7lEP9eyIvOOHzZKKec59nJYdx78UblLRKgS/tqyUg=; b=Jeht5qBqsRFOi6TlzG4feicPJ2dfQ+wyP/pKSIoKJMm056+gbmwDmBvOG63rOOUgTG PVwKlcjWeonCk9D3XNlZ7MXxtcixoBFFW1j8j4HUcgy0XnpJ1aUFoFWgJA+re2zsoxiV J6o0iQGmQTu63RCPpeb7IYoRcpNE+IpK2w1kP46sJe+okqvA5jMckl2RIyy/2H8vB/Tn iQn22RdPMyqz228cjlRVG86bZFy2CBljtkmSwcfZc8hIlhYeuhITfz3T10lSk8tBoMhR icHJuvaxQXjDVFIEh2E9nB045MnOwG/h65Sozoqax+0xo0OqM3MJQ/9F2UM15um1cYBo DRZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ES7lEP9eyIvOOHzZKKec59nJYdx78UblLRKgS/tqyUg=; b=Zh8odMFXk9UJ9KjCItLGwr8Fg9sMgOz1SUQgu6AiXZEg1MQk136xh/xqf+HZtqsrww Dc4pH68a3+3sByruoI2U9scw6/ZH7wHehMFYPmw13BJv4adYXuYXSiYlSO9mAx7v4B0m pnDB/f4OtGaULhDV6vBnM+T0ZKXYI0pJ9BD2n5xpNcEaxYIxt+lzN0sBhdUOUHKIwEZP o5/5nbmSUpohNjJbSlheLfoxc7UlYyGdBqP1R9l4Q88eSd1lh8dJHZ9VqqqAIGGPYnpt 9geGdJTlEJWW4RnfgjdlPDF0ymGUc18jZ+MNQXZS7HLh7YBB3uCNLFUAjHbvSeH1NuSw O8Yw== X-Gm-Message-State: APjAAAVp3HkCjtPzhXRRSVSW62lKd/rCGvdVxQCOrFYRpPh/lIaFluOA kgBfvFtvu2hlumIctQHKjcquscTH X-Google-Smtp-Source: APXvYqyOa2BlO2NLpAMTPninmGmuaRXJESu6Rh89sO+tYoS8sdQx/xAphyN0My7nC1A6lbhMtAyAfQ== X-Received: by 2002:aa7:dc59:: with SMTP id g25mr48119944edu.183.1568379776468; Fri, 13 Sep 2019 06:02:56 -0700 (PDT) Received: from ubuntu-N150ZU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id c14sm2846570ejx.31.2019.09.13.06.02.55 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2019 06:02:55 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones Subject: [RFC PATCH 09/10] builtin/pack-objects: introduce obj_is_packed() Date: Fri, 13 Sep 2019 15:02:25 +0200 Message-Id: <20190913130226.7449-10-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.23.0.46.gd213b4aca1.dirty In-Reply-To: <20190913130226.7449-1-chriscool@tuxfamily.org> References: <20190913130226.7449-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King Let's refactor the way we check if an object is packed by introducing obj_is_packed(). This function is now a simple wrapper around packlist_find(), but it will evolve in a following commit. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- builtin/pack-objects.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 1664969c97..5ee0b3092d 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -2569,6 +2569,11 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size, free(p); } +static int obj_is_packed(const struct object_id *oid) +{ + return !!packlist_find(&to_pack, oid, NULL); +} + static void add_tag_chain(const struct object_id *oid) { struct tag *tag; @@ -2580,7 +2585,7 @@ static void add_tag_chain(const struct object_id *oid) * it was included via bitmaps, we would not have parsed it * previously). */ - if (packlist_find(&to_pack, oid, NULL)) + if (obj_is_packed(oid)) return; tag = lookup_tag(the_repository, oid); @@ -2604,7 +2609,7 @@ static int add_ref_tag(const char *path, const struct object_id *oid, int flag, if (starts_with(path, "refs/tags/") && /* is a tag? */ !peel_ref(path, &peeled) && /* peelable? */ - packlist_find(&to_pack, &peeled, NULL)) /* object packed? */ + obj_is_packed(&peeled)) /* object packed? */ add_tag_chain(oid); return 0; } From patchwork Fri Sep 13 13:02:26 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Christian Couder X-Patchwork-Id: 11144763 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1F52118A6 for ; Fri, 13 Sep 2019 13:03:01 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id E78F4206BB for ; Fri, 13 Sep 2019 13:03:00 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="TlQtLDOh" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S2388222AbfIMNDA (ORCPT ); Fri, 13 Sep 2019 09:03:00 -0400 Received: from mail-ed1-f68.google.com ([209.85.208.68]:33175 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S2388176AbfIMNDA (ORCPT ); Fri, 13 Sep 2019 09:03:00 -0400 Received: by mail-ed1-f68.google.com with SMTP id o9so27029511edq.0 for ; Fri, 13 Sep 2019 06:02:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=uwrKVMIsJycWJBPIImLVMCIiKPd40aIozZjKsFZrgxY=; b=TlQtLDOh73UZ3RDmNZhZmCV8tnyt6bM7KYYPE9QAiDw1z2KCOzr6fkDjkN6XmOzVvm NQPCJUwz+OB0AHZRO3tSVoeNey+9LvyYz6WkkO5jSlBTe5YcXlii+yZceruOu3rYjDXA fwYDqcIP/g9T2HrXAEZxUzEQuySt+dDnQdRZPdpt6RH0E+fr5W5EpqygbJuutZdi+hT1 lmHxQlh+uDzCTCM5gk1+nfwJ3801jj3fxYCL0dhWOwj/01Jw8/oKsHO2i8+T8NpnPNci t5wlvj3hB7ba/jK+HWucEWQwUvkvtsb+8PEdAf2oleopo09zT1/rt2dnrQtp+flt6bDx iiuA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=uwrKVMIsJycWJBPIImLVMCIiKPd40aIozZjKsFZrgxY=; b=E1HiNV9xFpXQ6HG22AXP3+fVZUH3bdpy+kOAotMEXRwmiQKcs2b4RtHF5DyQW3OHjA racXPvvqQ4k4ftUaudVsWI128GZyIURRSMNTRoJespxx6dVaSzI3WY9zF3IQow4lTMhd Y2+3tHRCcx/5HURgqI8HLOUx3WYoKSQC3Ytff3I0kHhJlRXF3wPOp+ovYx03NcHotJI3 IkmB6mSaTPWYvOjUgO03G9lSsaV/ilnlAvrBV+J1nN+Hz2FRWnSkLhrPaABjYIYqLm05 iB3dDln4A9VlrDdYlqSL9QvCfE+4sQF+P9TQYTk5NFQ1c/jkOGNKEYYKRvq8ClNWg0bH Eobw== X-Gm-Message-State: APjAAAWm9aZ6Fs/ZOfZ0wdVzyh0yljzoYZrjzeFPnxZOnD5wEJHMs21X MKhc4An9zlKJunWXyHfMlsb6HvTI X-Google-Smtp-Source: APXvYqzdEcLdNBbbxWqUCRJPOnNhb2TO0lctRoKX2KI33HVxgQv9DiTz9UaL/x1WMjJU62Xtyjesdg== X-Received: by 2002:a05:6402:125a:: with SMTP id l26mr10775572edw.95.1568379777584; Fri, 13 Sep 2019 06:02:57 -0700 (PDT) Received: from ubuntu-N150ZU.booking.pcln.com (access-114.38.rev.fr.colt.net. [213.41.38.114]) by smtp.gmail.com with ESMTPSA id c14sm2846570ejx.31.2019.09.13.06.02.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 13 Sep 2019 06:02:56 -0700 (PDT) From: Christian Couder X-Google-Original-From: Christian Couder To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , Christian Couder , Ramsay Jones Subject: [RFC PATCH 10/10] pack-objects: improve partial packfile reuse Date: Fri, 13 Sep 2019 15:02:26 +0200 Message-Id: <20190913130226.7449-11-chriscool@tuxfamily.org> X-Mailer: git-send-email 2.23.0.46.gd213b4aca1.dirty In-Reply-To: <20190913130226.7449-1-chriscool@tuxfamily.org> References: <20190913130226.7449-1-chriscool@tuxfamily.org> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King Let's store the chunks of the packfile that we reuse in a dynamic array of `struct reused_chunk`, and let's use a reuse_packfile_bitmap to speed up reusing parts of packfiles. Signed-off-by: Jeff King Signed-off-by: Christian Couder --- builtin/pack-objects.c | 227 +++++++++++++++++++++++++++++++++-------- pack-bitmap.c | 150 +++++++++++++++++++-------- pack-bitmap.h | 3 +- 3 files changed, 293 insertions(+), 87 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 5ee0b3092d..6822ac1026 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -92,7 +92,7 @@ static struct progress *progress_state; static struct packed_git *reuse_packfile; static uint32_t reuse_packfile_objects; -static off_t reuse_packfile_offset; +static struct bitmap *reuse_packfile_bitmap; static int use_bitmap_index_default = 1; static int use_bitmap_index = -1; @@ -785,57 +785,189 @@ static struct object_entry **compute_write_order(void) return wo; } -static off_t write_reused_pack(struct hashfile *f) +/* + * Record the offsets needed in our reused packfile chunks due to + * "gaps" where we omitted some objects. + */ +static struct reused_chunk { + off_t start; + off_t offset; +} *reused_chunks; +static int reused_chunks_nr; +static int reused_chunks_alloc; + +static void record_reused_object(off_t where, off_t offset) { - unsigned char buffer[8192]; - off_t to_write, total; - int fd; + if (reused_chunks_nr && reused_chunks[reused_chunks_nr-1].offset == offset) + return; - if (!is_pack_valid(reuse_packfile)) - die(_("packfile is invalid: %s"), reuse_packfile->pack_name); + ALLOC_GROW(reused_chunks, reused_chunks_nr + 1, + reused_chunks_alloc); + reused_chunks[reused_chunks_nr].start = where; + reused_chunks[reused_chunks_nr].offset = offset; + reused_chunks_nr++; +} - fd = git_open(reuse_packfile->pack_name); - if (fd < 0) - die_errno(_("unable to open packfile for reuse: %s"), - reuse_packfile->pack_name); +/* + * Binary search to find the chunk that "where" is in. Note + * that we're not looking for an exact match, just the first + * chunk that contains it (which implicitly ends at the start + * of the next chunk. + */ +static off_t find_reused_offset(off_t where) +{ + int lo = 0, hi = reused_chunks_nr; + while (lo < hi) { + int mi = lo + ((hi - lo) / 2); + if (where == reused_chunks[mi].start) + return reused_chunks[mi].offset; + if (where < reused_chunks[mi].start) + hi = mi; + else + lo = mi + 1; + } - if (lseek(fd, sizeof(struct pack_header), SEEK_SET) == -1) - die_errno(_("unable to seek in reused packfile")); + /* + * The first chunk starts at zero, so we can't have gone below + * there. + */ + assert(lo); + return reused_chunks[lo-1].offset; +} - if (reuse_packfile_offset < 0) - reuse_packfile_offset = reuse_packfile->pack_size - the_hash_algo->rawsz; +static void write_reused_pack_one(size_t pos, struct hashfile *out, + struct pack_window **w_curs) +{ + off_t offset, next, cur; + enum object_type type; + unsigned long size; - total = to_write = reuse_packfile_offset - sizeof(struct pack_header); + offset = reuse_packfile->revindex[pos].offset; + next = reuse_packfile->revindex[pos + 1].offset; - while (to_write) { - int read_pack = xread(fd, buffer, sizeof(buffer)); + record_reused_object(offset, offset - hashfile_total(out)); - if (read_pack <= 0) - die_errno(_("unable to read from reused packfile")); + cur = offset; + type = unpack_object_header(reuse_packfile, w_curs, &cur, &size); + assert(type >= 0); - if (read_pack > to_write) - read_pack = to_write; + if (type == OBJ_OFS_DELTA) { + off_t base_offset; + off_t fixup; + + unsigned char header[MAX_PACK_OBJECT_HEADER]; + unsigned len; + + base_offset = get_delta_base(reuse_packfile, w_curs, &cur, type, offset); + assert(base_offset != 0); + + /* Convert to REF_DELTA if we must... */ + if (!allow_ofs_delta) { + int base_pos = find_revindex_position(reuse_packfile, base_offset); + const unsigned char *base_sha1 = + nth_packed_object_sha1(reuse_packfile, + reuse_packfile->revindex[base_pos].nr); + + len = encode_in_pack_object_header(header, sizeof(header), + OBJ_REF_DELTA, size); + hashwrite(out, header, len); + hashwrite(out, base_sha1, 20); + copy_pack_data(out, reuse_packfile, w_curs, cur, next - cur); + return; + } - hashwrite(f, buffer, read_pack); - to_write -= read_pack; + /* Otherwise see if we need to rewrite the offset... */ + fixup = find_reused_offset(offset) - + find_reused_offset(base_offset); + if (fixup) { + unsigned char ofs_header[10]; + unsigned i, ofs_len; + off_t ofs = offset - base_offset - fixup; + + len = encode_in_pack_object_header(header, sizeof(header), + OBJ_OFS_DELTA, size); + + i = sizeof(ofs_header) - 1; + ofs_header[i] = ofs & 127; + while (ofs >>= 7) + ofs_header[--i] = 128 | (--ofs & 127); + + ofs_len = sizeof(ofs_header) - i; + + if (0) { + off_t expected_size = cur - offset; + + if (len + ofs_len < expected_size) { + unsigned max_pad = (len >= 4) ? 9 : 5; + header[len - 1] |= 0x80; + while (len < max_pad && len + ofs_len < expected_size) + header[len++] = 0x80; + header[len - 1] &= 0x7F; + } + } + + hashwrite(out, header, len); + hashwrite(out, ofs_header + sizeof(ofs_header) - ofs_len, ofs_len); + copy_pack_data(out, reuse_packfile, w_curs, cur, next - cur); + return; + } + + /* ...otherwise we have no fixup, and can write it verbatim */ + } + + copy_pack_data(out, reuse_packfile, w_curs, offset, next - offset); +} + +static size_t write_reused_pack_verbatim(struct hashfile *out, + struct pack_window **w_curs) +{ + size_t pos = 0; + + while (pos < reuse_packfile_bitmap->word_alloc && + reuse_packfile_bitmap->words[pos] == (eword_t)~0) + pos++; + + if (pos) { + off_t to_write; + + written = (pos * BITS_IN_EWORD); + to_write = reuse_packfile->revindex[written].offset + - sizeof(struct pack_header); + + record_reused_object(sizeof(struct pack_header), 0); + hashflush(out); + copy_pack_data(out, reuse_packfile, w_curs, + sizeof(struct pack_header), to_write); - /* - * We don't know the actual number of objects written, - * only how many bytes written, how many bytes total, and - * how many objects total. So we can fake it by pretending all - * objects we are writing are the same size. This gives us a - * smooth progress meter, and at the end it matches the true - * answer. - */ - written = reuse_packfile_objects * - (((double)(total - to_write)) / total); display_progress(progress_state, written); } + return pos; +} + +static void write_reused_pack(struct hashfile *f) +{ + size_t i = 0; + uint32_t offset; + struct pack_window *w_curs = NULL; + + if (allow_ofs_delta) + i = write_reused_pack_verbatim(f, &w_curs); + + for (; i < reuse_packfile_bitmap->word_alloc; ++i) { + eword_t word = reuse_packfile_bitmap->words[i]; + size_t pos = (i * BITS_IN_EWORD); + + for (offset = 0; offset < BITS_IN_EWORD; ++offset) { + if ((word >> offset) == 0) + break; + + offset += ewah_bit_ctz64(word >> offset); + write_reused_pack_one(pos + offset, f, &w_curs); + display_progress(progress_state, ++written); + } + } - close(fd); - written = reuse_packfile_objects; - display_progress(progress_state, written); - return reuse_packfile_offset - sizeof(struct pack_header); + unuse_pack(&w_curs); } static const char no_split_warning[] = N_( @@ -868,11 +1000,9 @@ static void write_pack_file(void) offset = write_pack_header(f, nr_remaining); if (reuse_packfile) { - off_t packfile_size; assert(pack_to_stdout); - - packfile_size = write_reused_pack(f); - offset += packfile_size; + write_reused_pack(f); + offset = hashfile_total(f); } nr_written = 0; @@ -1002,6 +1132,10 @@ static int have_duplicate_entry(const struct object_id *oid, { struct object_entry *entry; + if (reuse_packfile_bitmap && + bitmap_walk_contains(bitmap_git, reuse_packfile_bitmap, oid)) + return 1; + entry = packlist_find(&to_pack, oid, index_pos); if (!entry) return 0; @@ -1192,6 +1326,7 @@ static int add_object_entry(const struct object_id *oid, enum object_type type, create_object_entry(oid, type, pack_name_hash(name), exclude, name && no_try_delta(name), index_pos, found_pack, found_offset); + return 1; } @@ -2571,7 +2706,9 @@ static void ll_find_deltas(struct object_entry **list, unsigned list_size, static int obj_is_packed(const struct object_id *oid) { - return !!packlist_find(&to_pack, oid, NULL); + return packlist_find(&to_pack, oid, NULL) || + (reuse_packfile_bitmap && + bitmap_walk_contains(bitmap_git, reuse_packfile_bitmap, oid)); } static void add_tag_chain(const struct object_id *oid) @@ -2677,6 +2814,7 @@ static void prepare_pack(int window, int depth) if (nr_deltas && n > 1) { unsigned nr_done = 0; + if (progress) progress_state = start_progress(_("Compressing objects"), nr_deltas); @@ -3061,7 +3199,6 @@ static void loosen_unused_packed_objects(void) static int pack_options_allow_reuse(void) { return pack_to_stdout && - allow_ofs_delta && !ignore_packed_keep_on_disk && !ignore_packed_keep_in_core && (!local || !have_non_local_packs) && @@ -3079,7 +3216,7 @@ static int get_object_list_from_bitmap(struct rev_info *revs) bitmap_git, &reuse_packfile, &reuse_packfile_objects, - &reuse_packfile_offset)) { + &reuse_packfile_bitmap)) { assert(reuse_packfile_objects); nr_result += reuse_packfile_objects; display_progress(progress_state, nr_result); diff --git a/pack-bitmap.c b/pack-bitmap.c index 1833971dc7..1d4c95ebc1 100644 --- a/pack-bitmap.c +++ b/pack-bitmap.c @@ -326,6 +326,13 @@ static int load_pack_bitmap(struct bitmap_index *bitmap_git) munmap(bitmap_git->map, bitmap_git->map_size); bitmap_git->map = NULL; bitmap_git->map_size = 0; + + kh_destroy_oid_map(bitmap_git->bitmaps); + bitmap_git->bitmaps = NULL; + + kh_destroy_oid_pos(bitmap_git->ext_index.positions); + bitmap_git->ext_index.positions = NULL; + return -1; } @@ -766,65 +773,126 @@ struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs) return NULL; } -int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, - struct packed_git **packfile, - uint32_t *entries, - off_t *up_to) +static void try_partial_reuse(struct bitmap_index *bitmap_git, + size_t pos, + struct bitmap *reuse, + struct pack_window **w_curs) { + struct revindex_entry *revidx; + off_t offset; + enum object_type type; + unsigned long size; + + if (pos >= bitmap_git->pack->num_objects) + return; /* not actually in the pack */ + + revidx = &bitmap_git->pack->revindex[pos]; + offset = revidx->offset; + type = unpack_object_header(bitmap_git->pack, w_curs, &offset, &size); + if (type < 0) + return; /* broken packfile, punt */ + + if (type == OBJ_REF_DELTA || type == OBJ_OFS_DELTA) { + off_t base_offset; + int base_pos; + + /* + * Find the position of the base object so we can look it up + * in our bitmaps. If we can't come up with an offset, or if + * that offset is not in the revidx, the pack is corrupt. + * There's nothing we can do, so just punt on this object, + * and the normal slow path will complain about it in + * more detail. + */ + base_offset = get_delta_base(bitmap_git->pack, w_curs, + &offset, type, revidx->offset); + if (!base_offset) + return; + base_pos = find_revindex_position(bitmap_git->pack, base_offset); + if (base_pos < 0) + return; + + /* + * We assume delta dependencies always point backwards. This + * lets us do a single pass, and is basically always true + * due to the way OFS_DELTAs work. You would not typically + * find REF_DELTA in a bitmapped pack, since we only bitmap + * packs we write fresh, and OFS_DELTA is the default). But + * let's double check to make sure the pack wasn't written with + * odd parameters. + */ + if (base_pos >= pos) + return; + + /* + * And finally, if we're not sending the base as part of our + * reuse chunk, then don't send this object either. The base + * would come after us, along with other objects not + * necessarily in the pack, which means we'd need to convert + * to REF_DELTA on the fly. Better to just let the normal + * object_entry code path handle it. + */ + if (!bitmap_get(reuse, base_pos)) + return; + } + /* - * Reuse the packfile content if we need more than - * 90% of its objects + * If we got here, then the object is OK to reuse. Mark it. */ - static const double REUSE_PERCENT = 0.9; + bitmap_set(reuse, pos); +} +int reuse_partial_packfile_from_bitmap(struct bitmap_index *bitmap_git, + struct packed_git **packfile_out, + uint32_t *entries, + struct bitmap **reuse_out) +{ struct bitmap *result = bitmap_git->result; - uint32_t reuse_threshold; - uint32_t i, reuse_objects = 0; + struct bitmap *reuse; + struct pack_window *w_curs = NULL; + size_t i = 0; + uint32_t offset; assert(result); - for (i = 0; i < result->word_alloc; ++i) { - if (result->words[i] != (eword_t)~0) { - reuse_objects += ewah_bit_ctz64(~result->words[i]); - break; - } - - reuse_objects += BITS_IN_EWORD; - } + while (i < result->word_alloc && result->words[i] == (eword_t)~0) + i++; -#ifdef GIT_BITMAP_DEBUG - { - const unsigned char *sha1; - struct revindex_entry *entry; + /* Don't mark objects not in the packfile */ + if (i > bitmap_git->pack->num_objects / BITS_IN_EWORD) + i = bitmap_git->pack->num_objects / BITS_IN_EWORD; - entry = &bitmap_git->reverse_index->revindex[reuse_objects]; - sha1 = nth_packed_object_sha1(bitmap_git->pack, entry->nr); + reuse = bitmap_word_alloc(i); + memset(reuse->words, 0xFF, i * sizeof(eword_t)); - fprintf(stderr, "Failed to reuse at %d (%016llx)\n", - reuse_objects, result->words[i]); - fprintf(stderr, " %s\n", hash_to_hex(sha1)); - } -#endif + for (; i < result->word_alloc; ++i) { + eword_t word = result->words[i]; + size_t pos = (i * BITS_IN_EWORD); - if (!reuse_objects) - return -1; + for (offset = 0; offset < BITS_IN_EWORD; ++offset) { + if ((word >> offset) == 0) + break; - if (reuse_objects >= bitmap_git->pack->num_objects) { - bitmap_git->reuse_objects = *entries = bitmap_git->pack->num_objects; - *up_to = -1; /* reuse the full pack */ - *packfile = bitmap_git->pack; - return 0; + offset += ewah_bit_ctz64(word >> offset); + try_partial_reuse(bitmap_git, pos + offset, reuse, &w_curs); + } } - reuse_threshold = bitmap_popcount(bitmap_git->result) * REUSE_PERCENT; + unuse_pack(&w_curs); - if (reuse_objects < reuse_threshold) + *entries = bitmap_popcount(reuse); + if (!*entries) { + bitmap_free(reuse); return -1; + } - bitmap_git->reuse_objects = *entries = reuse_objects; - *up_to = bitmap_git->pack->revindex[reuse_objects].offset; - *packfile = bitmap_git->pack; - + /* + * Drop any reused objects from the result, since they will not + * need to be handled separately. + */ + bitmap_and_not(result, reuse); + *packfile_out = bitmap_git->pack; + *reuse_out = reuse; return 0; } diff --git a/pack-bitmap.h b/pack-bitmap.h index 5425767f0f..7af7335f2e 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -50,7 +50,8 @@ void test_bitmap_walk(struct rev_info *revs); struct bitmap_index *prepare_bitmap_walk(struct rev_info *revs); int reuse_partial_packfile_from_bitmap(struct bitmap_index *, struct packed_git **packfile, - uint32_t *entries, off_t *up_to); + uint32_t *entries, + struct bitmap **bitmap); int rebuild_existing_bitmaps(struct bitmap_index *, struct packing_data *mapping, kh_oid_map_t *reused_bitmaps, int show_progress); void free_bitmap_index(struct bitmap_index *);