From patchwork Tue May 14 19:56:50 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13664392 Received: from mail-qk1-f171.google.com (mail-qk1-f171.google.com [209.85.222.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5EF6F181318 for ; Tue, 14 May 2024 19:56:53 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.171 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716614; cv=none; b=AsBLVRri2MxAOquY7Q58AAxA1g+AhsWNoVTQakYxoqftLAI0MXTKgEZefd/cn0frPVQafdGIJ4U4+5QPHCY2CzV5ODTl2zIlgFEXqxX/Eqgor42wNJBQyi4+UwLmk2cja/R/FoY5bGUgOTHXrfxRtHlJpKEImKVzWCKuuj9CVzE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716614; c=relaxed/simple; bh=DYG1kSZpoP8dujbKTgaMzbrHG4xxn51AurGQIm2FDbo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Ws71tsNtbLgOENhpEeKjpO4OeoWLYMsLkPDUYp9LN03/XcFM/5yw56gnLAvclg3kWWLzpz2VU4GG/jnwFiScyJ4IaW8F2YgW9J0YMqc698EyUBIcldOD33jJb/zIA+B2S8zH7OaUrZUrtd5JjOOflHbHU95LZ4212OHHhaBJHCQ= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=KBCa3D5Q; arc=none smtp.client-ip=209.85.222.171 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="KBCa3D5Q" Received: by mail-qk1-f171.google.com with SMTP id af79cd13be357-792bae4137bso585695785a.3 for ; Tue, 14 May 2024 12:56:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1715716612; x=1716321412; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=OPWI2ewRqgvvqwLoJKcz2JhceFqF8ks+9PQeNLr/UvM=; b=KBCa3D5Qe/1RMjvearQt4bGFxg7oouJeZss++ckrfOEBe1osxZ6t1GfiHcH7Uks3jb kWvMozmXEeb0rrF0FKp5dSEb2E2U5zG5A8TYUdr1S7wPj1fIPYJGhbF7enfeW6M48SAn FoNQmNjt9l6a4PVj7/iIEdUoy1fw31ptA5rbm7xYfzTL8akluTs3fhkbniG1eM79aRW4 /r4vEUIQ2fHLFc4Olds//gdlEa1LsVVE5j7FPJECT4DhfzURHtb6YJ6pc7OjS6hBewQz 2ywKqMEY/JzvYZ5/wPJPh4FwyxMBGBI/XKRYWb+NwTO9el2gUqRYJlSziSuPRGIOKJvf 1f2Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715716612; x=1716321412; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=OPWI2ewRqgvvqwLoJKcz2JhceFqF8ks+9PQeNLr/UvM=; b=XBKTZhdHaZfFXwoygb0s9++dCVLjwzrEh/NNzP/KbLYKM1Xf2/4ihpChQyXeRA0zgw lEa9ML4aPWf04HhYS6Mqulyx1BCBp93/XLMFktF+NDUqR7o7/CklMa+Xeuc2grq/5keR a+kJzJVP8Bm0RtWffuvUclup8Sf7wQQrXzIFt1EWTCK6psG6FazBPhy5TIuPGxuXm1rg QVZikvyHjeXXAwPMO9VtwyJ9K25UB978D/ELpV2H4EPSgpdBBMXVoysLbNuyg/TNG4Tq FJ2y7eC2Z6D/YG65xwrcwp5bILPS+k1p3mlOd7KuemgILqaCbBwTVmpSCOswOTepWbXh 5OXQ== X-Gm-Message-State: AOJu0YxbvuKBygKzLqcWre1EQLm6C+ACjz6aZxzxCWnjXMjU5mR7Owq1 QSZdzU3DRw8FswSQuFZkCLA8jLKn3PhF9mPrPkVqOzqsOQAOvFih+LV4rKMbjQVBj95gSIm6Ear B X-Google-Smtp-Source: AGHT+IE5lrQ0jkyyb1zAEYMK2aHxPOa9zRHGn58tV/rPTFFjs61LOEXlGtTx1zFgCYijkJHkd0i3pg== X-Received: by 2002:a05:620a:9364:b0:792:bf92:b78a with SMTP id af79cd13be357-792c75773b4mr1445007585a.1.1715716611920; Tue, 14 May 2024 12:56:51 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id af79cd13be357-792bf1cd143sm601181085a.0.2024.05.14.12.56.51 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 May 2024 12:56:51 -0700 (PDT) Date: Tue, 14 May 2024 15:56:50 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Junio C Hamano Subject: [PATCH 1/6] object.h: add flags allocated by pack-bitmap.h Message-ID: <38c96fc1909162a4d9c188f55b7c708cfc1b14b9.1715716605.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In commit 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21) the NEEDS_BITMAP flag was introduced into pack-bitmap.h, but no object flags allocation table existed at the time. In 208acbfb82f (object.h: centralize object flag allocation, 2014-03-25) when that table was first introduced, we never added the flags from 7cc8f971085, which has remained the case since. Rectify this by including the flag bit used by pack-bitmap.h into the centralized table in object.h. Signed-off-by: Taylor Blau --- object.h | 1 + 1 file changed, 1 insertion(+) diff --git a/object.h b/object.h index 9293e703cc..99b9c8f114 100644 --- a/object.h +++ b/object.h @@ -81,6 +81,7 @@ void object_array_init(struct object_array *array); * reflog.c: 10--12 * builtin/show-branch.c: 0-------------------------------------------26 * builtin/unpack-objects.c: 2021 + * pack-bitmap.h: 22 */ #define FLAG_BITS 28 From patchwork Tue May 14 19:56:53 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13664393 Received: from mail-qk1-f177.google.com (mail-qk1-f177.google.com [209.85.222.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 713D4181318 for ; Tue, 14 May 2024 19:56:56 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716618; cv=none; b=qWLcQ24ZYf4dk4Gi7uYcoaBNi7D2OSP7hPc6WefzH2wU8eWtyQFTpoNZv3rBcPIsZBjdyjZ5oFkgKnAcjgVCybWORXqXbBMfn1LR/vARv7Rqx6zCPxXcNr5tfOSCdkIC5sZKtrDRC0Yufzh52AcScYDM7img7n05YDkvK8g5XPI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716618; c=relaxed/simple; bh=4Vz6cCvmiUoddiTWDKwFmI6fIPXoqUL/ACHnWR/H/bI=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=k2qr3OX48iSaTXZieCZcTk+9xszPBfzOaONaSTHGZoXFFIzC/9PvEKZyay4OywbdMcRPP5t9culEm08A5FCbIuUh7KWFgMAtQ1Yl6o28imx1p4RPCqPUwKx+Yh70r5pgg7sSCmE/lYgmYeZgVghzvDCVz6jS31Fn/O72voP+vrs= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=gspRVBX3; arc=none smtp.client-ip=209.85.222.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="gspRVBX3" Received: by mail-qk1-f177.google.com with SMTP id af79cd13be357-792b934de39so520195285a.3 for ; Tue, 14 May 2024 12:56:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1715716615; x=1716321415; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=DtJ+lV7XgBbq32BQT3DpjqKGF9XYzIDWJvQRTihEwQI=; b=gspRVBX3G1QFlgbY3gpgmeawX5uMmt+5NIgYxXL+gtDvQjgP249YKAqJ4gTvRK1/Mu bvT/6HxClgoKsJrMcrtHdTsdHIBlk+gHT4T8lcp0VR/6hfw5IxuxjWdVAkimwVXKgsgA AJbBhAYNo0JGW+RNufF4uPUbDy11/9TzFaoYzO+IBieYpFn1SiqBfXVn3YQFr5DIrmCq 28BH7rr90V/HlUPUo8xq4I69emmuGqtqhE9j5M2hfdxQQVmb2NoARb2gF7QrMzEL7ul3 /Q9XnFuCzUKRUWsa3yb5vY/7NTzosfRqnSQpKjkxOiQ1b4TaMDFGfIw+BaxX0k5iMOgh at4w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715716615; x=1716321415; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=DtJ+lV7XgBbq32BQT3DpjqKGF9XYzIDWJvQRTihEwQI=; b=YykbgD9NaEEyZu/ISphEENZj3mPSuYMARVrkfIaVceqFcglX2+DV1bzZf0kqnw4JO0 QOZwzv36qvyTtazZyCZbGFbMXVpddz4fsv/2GtxXNr+3KMR4XEAQqevM5FHXsi1tCuOA LXQZ6TA46idPUAruq8MoeoLNnpSTE3ZB3R/7Ndd7qj4+xKZ2EYBtOty6hgEIhQp/fqn7 o9qcB9oFgYs5j1MttiL9VZIdnVzTdgse3ByAQIQI1bHjgyUYycUz52IHzP8nkcBCu/i3 sJL3pQnxH2FXBOvZjVpqw5qxOJM8ele0/C5z4/mtwdjVaRI6KMuqtU0QyKF+dUDeFEzv Q4Cg== X-Gm-Message-State: AOJu0YzIQL7jPHGnzqIUl32dfbwOWlp30SKMjSbYw/06sfEVqOQAHJ4V Kk/0JAjIN9WFzvzA+/E+Eev1F1Jl5uuebAs0aIRbm6QEAbOCcQuBU2OpLR799/Qybf3TMkeWEDc m X-Google-Smtp-Source: AGHT+IEQcq97Gb9C6sOoO+QYXeBKPYQV5X8cVjkNSsAe7JG9ukXPhg+LmGZ1A2yMSF2sJ71sl7MTJg== X-Received: by 2002:a05:620a:3cb:b0:792:a1c5:659f with SMTP id af79cd13be357-792c75abda4mr1483890385a.32.1715716615066; Tue, 14 May 2024 12:56:55 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id af79cd13be357-792bf3106c2sm597786285a.106.2024.05.14.12.56.54 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 May 2024 12:56:54 -0700 (PDT) Date: Tue, 14 May 2024 15:56:53 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Junio C Hamano Subject: [PATCH 2/6] pack-bitmap-write.c: move commit_positions into commit_pos fields Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: In 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21), the bitmapped_commit struct was introduced, including the 'commit_pos' field, which has been unused ever since its introduction more than a decade ago. Instead, we have used the nearby `commit_positions` array leaving the bitmapped_commit struct with an unused 4-byte field. We could drop the `commit_pos` field as unused, and continue to store the values in the auxiliary array. But we could also drop the array and store the data for each bitmapped_commit struct inside of the structure itself, which is what this patch does. In any spot that we previously read `commit_positions[i]`, we can now instead read `writer.selected[i].commit_pos`. There are a few spots that need changing as a result: - write_selected_commits_v1() is a simple transformation, since we're just reading the field. As a result, the function no longer needs an explicit argument to pass the commit_positions array. - write_lookup_table() also no longer needs the explicit commit_positions array passed in as an argument. But it still needs to sort an array of indices into the writer.selected array to read them in commit_pos order, so table_cmp() is adjusted accordingly. - bitmap_writer_finish() no longer needs to allocate, populate, and free the commit_positions table. Instead, we can just write the data directly into each struct bitmapped_commit. Signed-off-by: Taylor Blau --- pack-bitmap-write.c | 41 ++++++++++++++++------------------------- 1 file changed, 16 insertions(+), 25 deletions(-) diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index c6c8f94cc5..2ae82b8696 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -670,9 +670,7 @@ static const struct object_id *oid_access(size_t pos, const void *table) return &index[pos]->oid; } -static void write_selected_commits_v1(struct hashfile *f, - uint32_t *commit_positions, - off_t *offsets) +static void write_selected_commits_v1(struct hashfile *f, off_t *offsets) { int i; @@ -682,7 +680,7 @@ static void write_selected_commits_v1(struct hashfile *f, if (offsets) offsets[i] = hashfile_total(f); - hashwrite_be32(f, commit_positions[i]); + hashwrite_be32(f, stored->commit_pos); hashwrite_u8(f, stored->xor_offset); hashwrite_u8(f, stored->flags); @@ -690,23 +688,20 @@ static void write_selected_commits_v1(struct hashfile *f, } } -static int table_cmp(const void *_va, const void *_vb, void *_data) +static int table_cmp(const void *_va, const void *_vb) { - uint32_t *commit_positions = _data; - uint32_t a = commit_positions[*(uint32_t *)_va]; - uint32_t b = commit_positions[*(uint32_t *)_vb]; + struct bitmapped_commit *a = &writer.selected[*(uint32_t *)_va]; + struct bitmapped_commit *b = &writer.selected[*(uint32_t *)_vb]; - if (a > b) + if (a->commit_pos < b->commit_pos) + return -1; + else if (a->commit_pos > b->commit_pos) return 1; - else if (a < b) - return -1; return 0; } -static void write_lookup_table(struct hashfile *f, - uint32_t *commit_positions, - off_t *offsets) +static void write_lookup_table(struct hashfile *f, off_t *offsets) { uint32_t i; uint32_t *table, *table_inv; @@ -722,7 +717,7 @@ static void write_lookup_table(struct hashfile *f, * bitmap corresponds to j'th bitmapped commit (among the selected * commits) in lex order of OIDs. */ - QSORT_S(table, writer.selected_nr, table_cmp, commit_positions); + QSORT(table, writer.selected_nr, table_cmp); /* table_inv helps us discover that relationship (i'th bitmap * to j'th commit by j = table_inv[i]) @@ -753,7 +748,7 @@ static void write_lookup_table(struct hashfile *f, xor_row = 0xffffffff; } - hashwrite_be32(f, commit_positions[table[i]]); + hashwrite_be32(f, writer.selected[table[i]].commit_pos); hashwrite_be64(f, (uint64_t)offsets[table[i]]); hashwrite_be32(f, xor_row); } @@ -789,7 +784,6 @@ void bitmap_writer_finish(struct pack_idx_entry **index, static uint16_t flags = BITMAP_OPT_FULL_DAG; struct strbuf tmp_file = STRBUF_INIT; struct hashfile *f; - uint32_t *commit_positions = NULL; off_t *offsets = NULL; uint32_t i; @@ -814,22 +808,20 @@ void bitmap_writer_finish(struct pack_idx_entry **index, if (options & BITMAP_OPT_LOOKUP_TABLE) CALLOC_ARRAY(offsets, index_nr); - ALLOC_ARRAY(commit_positions, writer.selected_nr); - for (i = 0; i < writer.selected_nr; i++) { struct bitmapped_commit *stored = &writer.selected[i]; - int commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access); + int commit_pos = oid_pos(&stored->commit->object.oid, index, + index_nr, oid_access); if (commit_pos < 0) BUG(_("trying to write commit not in index")); - - commit_positions[i] = commit_pos; + stored->commit_pos = commit_pos; } - write_selected_commits_v1(f, commit_positions, offsets); + write_selected_commits_v1(f, offsets); if (options & BITMAP_OPT_LOOKUP_TABLE) - write_lookup_table(f, commit_positions, offsets); + write_lookup_table(f, offsets); if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); @@ -844,6 +836,5 @@ void bitmap_writer_finish(struct pack_idx_entry **index, die_errno("unable to rename temporary bitmap file to '%s'", filename); strbuf_release(&tmp_file); - free(commit_positions); free(offsets); } From patchwork Tue May 14 19:56:56 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13664394 Received: from mail-qt1-f177.google.com (mail-qt1-f177.google.com [209.85.160.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id DEC43182C82 for ; Tue, 14 May 2024 19:56:59 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.160.177 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716622; cv=none; b=pb601B0qNzOvQVDtj+DaCvG3D62opXrIkjs3aF+Fnl1pQ+GLmLRuu+afO46ITYDB+69P26JdakGL55QtYF5kCPLsI95b04zrj75v66yse63hMA0s040sebDvwWmZ6HIfNxcat2fUXCqxtDld89FxGGI1wEuWyUVMkcHlRFIGuAI= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716622; c=relaxed/simple; bh=CNg4deGubRUAz+pRbeXT+ffC+76ZH/AizxXckhP7Dc4=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Pc232sWPCctujSinX8/jbR5q2g2H+9nBBbPdpI4hmtscYlv3nmjA1sZko44LTJremsSt3e7P2B1cTFPspHxMuY0oQRu6miX3odAylIbzPSdFbYKKrvtkJsTmHLOO4HpWp0rsNcP3/UiWLYOw4OocEmylX5mGcRNmUcY5jCQWuWM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=Q6jkkfks; arc=none smtp.client-ip=209.85.160.177 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="Q6jkkfks" Received: by mail-qt1-f177.google.com with SMTP id d75a77b69052e-43e1e98aaf0so11895851cf.2 for ; Tue, 14 May 2024 12:56:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1715716618; x=1716321418; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=DThBoV6ZkeVEdLxgfRTlNqQ/RRoQh0N1BBK/XCcO1E8=; b=Q6jkkfksm2hPqLPc6mzaVn5co1EflhQUSL4f8wqUpLyfOc37Z+d7l87aNFSjmS+n0q inhqfmHwg/gawf5O2Wllm3u84ZbGJhOQ7jWU74qRmQuvqBb0YleJ3hg+gVbHVXPWtlAK xPVjIC7XPUtAyYz1Byj1TyrnY2rSsM0sj/s47d7iaJLqUka/SwhpkBhzzmkA/wR0tQQr icTdCBw1+GqN+Gqw2feTHBUEwgCHu2dy2oRc/2Dy2LfrNJTHMi129J3Ss25EmEDk9Vdm ewoiVD2vlB0TUraYWjJmY7VxeteExPm/c+dle4/nr09MXSqsxfEDvpU4l/1yrmLgAee8 MgjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715716618; x=1716321418; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=DThBoV6ZkeVEdLxgfRTlNqQ/RRoQh0N1BBK/XCcO1E8=; b=TQe7mhrHMogaKTsT0q8+pTXXgeJsRZYfmkg1P5n/OVhm65DzqJL4VcpwGs5WrDEx9O MpqeSLqqtFjhKpQdQ/3JhXef5Hvyf5J5AQNXYTSbCganJETJB5V+7NJevkGnQ5dUqFhv nJskUqhAiogv61hgLpfrMeIUL2qZIKQy1690eQouwf91Vqr98KhCBl2rYnfw+K5nvQn5 u6uhGnlwwEWZCsAOMFLU0/WldJYNb4BUQF3fYgaWpin6FlApouwO1OpMGrq/GDMUrArV 6W5xAmAsx0PNl+oiR0XAS9R0xOW5SC0g1ECO48XZiamSU4g6+u4SGjtp6QkTtXU0dt1g lWtg== X-Gm-Message-State: AOJu0Yw0D+bdD2ZKpGVZ5P4+LhE4U3IK5J5bJZKQd3XupgsdoP+IPyU+ Yew3RPW3f5ITE5D8RGNWSGpU3Frz+RddYGza12CkeekgTFBy2vxEQgc/3zmcWeHdfMVAaFSnzi7 R X-Google-Smtp-Source: AGHT+IH99Gwe+tsUitvc0B/syml+D3yKlJSYZmXrSsiqlEpmX/XjClEOiAUtaMqlWnUPsNeQjHq58A== X-Received: by 2002:ac8:7f0a:0:b0:43b:4bc:6606 with SMTP id d75a77b69052e-43dfdae7392mr163400851cf.24.1715716618343; Tue, 14 May 2024 12:56:58 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id d75a77b69052e-43e2467c2ecsm17874631cf.72.2024.05.14.12.56.57 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 May 2024 12:56:57 -0700 (PDT) Date: Tue, 14 May 2024 15:56:56 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Junio C Hamano Subject: [PATCH 3/6] pack-bitmap: avoid use of static `bitmap_writer` Message-ID: <65ee69acfeb9c66404aa85f0c4b09891d3b8fc27.1715716605.git.me@ttaylorr.com> References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The pack-bitmap machinery uses a structure called 'bitmap_writer' to collect the data necessary to write out .bitmap files. Since its introduction in 7cc8f971085 (pack-objects: implement bitmap writing, 2013-12-21), there has been a single static bitmap_writer structure, which is responsible for all bitmap writing-related operations. In practice, this is OK, since we are only ever writing a single .bitmap file in a single process (e.g., `git multi-pack-index write --bitmap`, `git pack-objects --write-bitmap-index`, `git repack -b`, etc.). However, having a single static variable makes issues like data ownership unclear, when to free variables, what has/hasn't been initialized unclear. Refactor this code to be written in terms of a given bitmap_writer structure instead of relying on a static global. Note that this exposes the structure definition of the bitmap_writer at the pack-bitmap.h level. We could work around this by, e.g., forcing callers to declare their writers as: struct bitmap_writer *writer; bitmap_writer_init(&bitmap_writer); and then declaring `bitmap_writer_init()` as taking in a double-pointer like so: void bitmap_writer_init(struct bitmap_writer **writer); which would avoid us having to expose the definition of the structure itself. This patch takes a different approach, since future patches (like for the ongoing pseudo-merge bitmaps work) will want to modify the innards of this structure (in the previous example, via pseudo-merge.c). Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 19 ++-- midx-write.c | 16 ++-- pack-bitmap-write.c | 209 +++++++++++++++++++++-------------------- pack-bitmap.h | 38 ++++++-- 4 files changed, 159 insertions(+), 123 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index baf0090fc8..ba4c93d241 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1245,6 +1245,7 @@ static void write_pack_file(void) uint32_t nr_remaining = nr_result; time_t last_mtime = 0; struct object_entry **write_order; + struct bitmap_writer bitmap_writer; if (progress > pack_to_stdout) progress_state = start_progress(_("Writing objects"), nr_result); @@ -1339,8 +1340,9 @@ static void write_pack_file(void) hash_to_hex(hash)); if (write_bitmap_index) { - bitmap_writer_set_checksum(hash); - bitmap_writer_build_type_index( + bitmap_writer_init(&bitmap_writer); + bitmap_writer_set_checksum(&bitmap_writer, hash); + bitmap_writer_build_type_index(&bitmap_writer, &to_pack, written_list, nr_written); } @@ -1358,11 +1360,16 @@ static void write_pack_file(void) strbuf_addstr(&tmpname, "bitmap"); stop_progress(&progress_state); - bitmap_writer_show_progress(progress); - bitmap_writer_select_commits(indexed_commits, indexed_commits_nr, -1); - if (bitmap_writer_build(&to_pack) < 0) + bitmap_writer_show_progress(&bitmap_writer, + progress); + bitmap_writer_select_commits(&bitmap_writer, + indexed_commits, + indexed_commits_nr, + -1); + if (bitmap_writer_build(&bitmap_writer, &to_pack) < 0) die(_("failed to write bitmap index")); - bitmap_writer_finish(written_list, nr_written, + bitmap_writer_finish(&bitmap_writer, + written_list, nr_written, tmpname.buf, write_bitmap_options); write_bitmap_index = 0; strbuf_setlen(&tmpname, tmpname_len); diff --git a/midx-write.c b/midx-write.c index 65e69d2de7..5fdc8f2ff5 100644 --- a/midx-write.c +++ b/midx-write.c @@ -798,6 +798,7 @@ static int write_midx_bitmap(const char *midx_name, { int ret, i; uint16_t options = 0; + struct bitmap_writer writer; struct pack_idx_entry **index; char *bitmap_name = xstrfmt("%s-%s.bitmap", midx_name, hash_to_hex(midx_hash)); @@ -819,8 +820,10 @@ static int write_midx_bitmap(const char *midx_name, for (i = 0; i < pdata->nr_objects; i++) index[i] = &pdata->objects[i].idx; - bitmap_writer_show_progress(flags & MIDX_PROGRESS); - bitmap_writer_build_type_index(pdata, index, pdata->nr_objects); + bitmap_writer_init(&writer); + bitmap_writer_show_progress(&writer, flags & MIDX_PROGRESS); + bitmap_writer_build_type_index(&writer, pdata, index, + pdata->nr_objects); /* * bitmap_writer_finish expects objects in lex order, but pack_order @@ -838,13 +841,14 @@ static int write_midx_bitmap(const char *midx_name, for (i = 0; i < pdata->nr_objects; i++) index[pack_order[i]] = &pdata->objects[i].idx; - bitmap_writer_select_commits(commits, commits_nr, -1); - ret = bitmap_writer_build(pdata); + bitmap_writer_select_commits(&writer, commits, commits_nr, -1); + ret = bitmap_writer_build(&writer, pdata); if (ret < 0) goto cleanup; - bitmap_writer_set_checksum(midx_hash); - bitmap_writer_finish(index, pdata->nr_objects, bitmap_name, options); + bitmap_writer_set_checksum(&writer, midx_hash); + bitmap_writer_finish(&writer, index, pdata->nr_objects, bitmap_name, + options); cleanup: free(index); diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index 2ae82b8696..e22fa70745 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -27,43 +27,30 @@ struct bitmapped_commit { uint32_t commit_pos; }; -struct bitmap_writer { - struct ewah_bitmap *commits; - struct ewah_bitmap *trees; - struct ewah_bitmap *blobs; - struct ewah_bitmap *tags; - - kh_oid_map_t *bitmaps; - struct packing_data *to_pack; - - struct bitmapped_commit *selected; - unsigned int selected_nr, selected_alloc; - - struct progress *progress; - int show_progress; - unsigned char pack_checksum[GIT_MAX_RAWSZ]; -}; - -static struct bitmap_writer writer; +void bitmap_writer_init(struct bitmap_writer *writer) +{ + memset(writer, 0, sizeof(struct bitmap_writer)); +} -void bitmap_writer_show_progress(int show) +void bitmap_writer_show_progress(struct bitmap_writer *writer, int show) { - writer.show_progress = show; + writer->show_progress = show; } /** * Build the initial type index for the packfile or multi-pack-index */ -void bitmap_writer_build_type_index(struct packing_data *to_pack, +void bitmap_writer_build_type_index(struct bitmap_writer *writer, + struct packing_data *to_pack, struct pack_idx_entry **index, uint32_t index_nr) { uint32_t i; - writer.commits = ewah_new(); - writer.trees = ewah_new(); - writer.blobs = ewah_new(); - writer.tags = ewah_new(); + writer->commits = ewah_new(); + writer->trees = ewah_new(); + writer->blobs = ewah_new(); + writer->tags = ewah_new(); ALLOC_ARRAY(to_pack->in_pack_pos, to_pack->nr_objects); for (i = 0; i < index_nr; ++i) { @@ -88,19 +75,19 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack, switch (real_type) { case OBJ_COMMIT: - ewah_set(writer.commits, i); + ewah_set(writer->commits, i); break; case OBJ_TREE: - ewah_set(writer.trees, i); + ewah_set(writer->trees, i); break; case OBJ_BLOB: - ewah_set(writer.blobs, i); + ewah_set(writer->blobs, i); break; case OBJ_TAG: - ewah_set(writer.tags, i); + ewah_set(writer->tags, i); break; default: @@ -115,23 +102,25 @@ void bitmap_writer_build_type_index(struct packing_data *to_pack, * Compute the actual bitmaps */ -static inline void push_bitmapped_commit(struct commit *commit) +static inline void push_bitmapped_commit(struct bitmap_writer *writer, + struct commit *commit) { - if (writer.selected_nr >= writer.selected_alloc) { - writer.selected_alloc = (writer.selected_alloc + 32) * 2; - REALLOC_ARRAY(writer.selected, writer.selected_alloc); + if (writer->selected_nr >= writer->selected_alloc) { + writer->selected_alloc = (writer->selected_alloc + 32) * 2; + REALLOC_ARRAY(writer->selected, writer->selected_alloc); } - writer.selected[writer.selected_nr].commit = commit; - writer.selected[writer.selected_nr].bitmap = NULL; - writer.selected[writer.selected_nr].flags = 0; + writer->selected[writer->selected_nr].commit = commit; + writer->selected[writer->selected_nr].bitmap = NULL; + writer->selected[writer->selected_nr].flags = 0; - writer.selected_nr++; + writer->selected_nr++; } -static uint32_t find_object_pos(const struct object_id *oid, int *found) +static uint32_t find_object_pos(struct bitmap_writer *writer, + const struct object_id *oid, int *found) { - struct object_entry *entry = packlist_find(writer.to_pack, oid); + struct object_entry *entry = packlist_find(writer->to_pack, oid); if (!entry) { if (found) @@ -143,17 +132,17 @@ static uint32_t find_object_pos(const struct object_id *oid, int *found) if (found) *found = 1; - return oe_in_pack_pos(writer.to_pack, entry); + return oe_in_pack_pos(writer->to_pack, entry); } -static void compute_xor_offsets(void) +static void compute_xor_offsets(struct bitmap_writer *writer) { static const int MAX_XOR_OFFSET_SEARCH = 10; int i, next = 0; - while (next < writer.selected_nr) { - struct bitmapped_commit *stored = &writer.selected[next]; + while (next < writer->selected_nr) { + struct bitmapped_commit *stored = &writer->selected[next]; int best_offset = 0; struct ewah_bitmap *best_bitmap = stored->bitmap; @@ -166,7 +155,7 @@ static void compute_xor_offsets(void) break; test_xor = ewah_pool_new(); - ewah_xor(writer.selected[curr].bitmap, stored->bitmap, test_xor); + ewah_xor(writer->selected[curr].bitmap, stored->bitmap, test_xor); if (test_xor->buffer_size < best_bitmap->buffer_size) { if (best_bitmap != stored->bitmap) @@ -348,7 +337,8 @@ static void bitmap_builder_clear(struct bitmap_builder *bb) bb->commits_nr = bb->commits_alloc = 0; } -static int fill_bitmap_tree(struct bitmap *bitmap, +static int fill_bitmap_tree(struct bitmap_writer *writer, + struct bitmap *bitmap, struct tree *tree) { int found; @@ -360,7 +350,7 @@ static int fill_bitmap_tree(struct bitmap *bitmap, * If our bit is already set, then there is nothing to do. Both this * tree and all of its children will be set. */ - pos = find_object_pos(&tree->object.oid, &found); + pos = find_object_pos(writer, &tree->object.oid, &found); if (!found) return -1; if (bitmap_get(bitmap, pos)) @@ -375,12 +365,12 @@ static int fill_bitmap_tree(struct bitmap *bitmap, while (tree_entry(&desc, &entry)) { switch (object_type(entry.mode)) { case OBJ_TREE: - if (fill_bitmap_tree(bitmap, + if (fill_bitmap_tree(writer, bitmap, lookup_tree(the_repository, &entry.oid)) < 0) return -1; break; case OBJ_BLOB: - pos = find_object_pos(&entry.oid, &found); + pos = find_object_pos(writer, &entry.oid, &found); if (!found) return -1; bitmap_set(bitmap, pos); @@ -397,7 +387,8 @@ static int fill_bitmap_tree(struct bitmap *bitmap, static int reused_bitmaps_nr; -static int fill_bitmap_commit(struct bb_commit *ent, +static int fill_bitmap_commit(struct bitmap_writer *writer, + struct bb_commit *ent, struct commit *commit, struct prio_queue *queue, struct prio_queue *tree_queue, @@ -436,7 +427,7 @@ static int fill_bitmap_commit(struct bb_commit *ent, * Mark ourselves and queue our tree. The commit * walk ensures we cover all parents. */ - pos = find_object_pos(&c->object.oid, &found); + pos = find_object_pos(writer, &c->object.oid, &found); if (!found) return -1; bitmap_set(ent->bitmap, pos); @@ -444,7 +435,8 @@ static int fill_bitmap_commit(struct bb_commit *ent, repo_get_commit_tree(the_repository, c)); for (p = c->parents; p; p = p->next) { - pos = find_object_pos(&p->item->object.oid, &found); + pos = find_object_pos(writer, &p->item->object.oid, + &found); if (!found) return -1; if (!bitmap_get(ent->bitmap, pos)) { @@ -455,29 +447,31 @@ static int fill_bitmap_commit(struct bb_commit *ent, } while (tree_queue->nr) { - if (fill_bitmap_tree(ent->bitmap, + if (fill_bitmap_tree(writer, ent->bitmap, prio_queue_get(tree_queue)) < 0) return -1; } return 0; } -static void store_selected(struct bb_commit *ent, struct commit *commit) +static void store_selected(struct bitmap_writer *writer, + struct bb_commit *ent, struct commit *commit) { - struct bitmapped_commit *stored = &writer.selected[ent->idx]; + struct bitmapped_commit *stored = &writer->selected[ent->idx]; khiter_t hash_pos; int hash_ret; stored->bitmap = bitmap_to_ewah(ent->bitmap); - hash_pos = kh_put_oid_map(writer.bitmaps, commit->object.oid, &hash_ret); + hash_pos = kh_put_oid_map(writer->bitmaps, commit->object.oid, &hash_ret); if (hash_ret == 0) die("Duplicate entry when writing index: %s", oid_to_hex(&commit->object.oid)); - kh_value(writer.bitmaps, hash_pos) = stored; + kh_value(writer->bitmaps, hash_pos) = stored; } -int bitmap_writer_build(struct packing_data *to_pack) +int bitmap_writer_build(struct bitmap_writer *writer, + struct packing_data *to_pack) { struct bitmap_builder bb; size_t i; @@ -488,11 +482,12 @@ int bitmap_writer_build(struct packing_data *to_pack) uint32_t *mapping; int closed = 1; /* until proven otherwise */ - writer.bitmaps = kh_init_oid_map(); - writer.to_pack = to_pack; + writer->bitmaps = kh_init_oid_map(); + writer->to_pack = to_pack; - if (writer.show_progress) - writer.progress = start_progress("Building bitmaps", writer.selected_nr); + if (writer->show_progress) + writer->progress = start_progress("Building bitmaps", + writer->selected_nr); trace2_region_enter("pack-bitmap-write", "building_bitmaps_total", the_repository); @@ -502,23 +497,23 @@ int bitmap_writer_build(struct packing_data *to_pack) else mapping = NULL; - bitmap_builder_init(&bb, &writer, old_bitmap); + bitmap_builder_init(&bb, writer, old_bitmap); for (i = bb.commits_nr; i > 0; i--) { struct commit *commit = bb.commits[i-1]; struct bb_commit *ent = bb_data_at(&bb.data, commit); struct commit *child; int reused = 0; - if (fill_bitmap_commit(ent, commit, &queue, &tree_queue, + if (fill_bitmap_commit(writer, ent, commit, &queue, &tree_queue, old_bitmap, mapping) < 0) { closed = 0; break; } if (ent->selected) { - store_selected(ent, commit); + store_selected(writer, ent, commit); nr_stored++; - display_progress(writer.progress, nr_stored); + display_progress(writer->progress, nr_stored); } while ((child = pop_commit(&ent->reverse_edges))) { @@ -549,10 +544,10 @@ int bitmap_writer_build(struct packing_data *to_pack) trace2_data_intmax("pack-bitmap-write", the_repository, "building_bitmaps_reused", reused_bitmaps_nr); - stop_progress(&writer.progress); + stop_progress(&writer->progress); if (closed) - compute_xor_offsets(); + compute_xor_offsets(writer); return closed ? 0 : -1; } @@ -590,7 +585,8 @@ static int date_compare(const void *_a, const void *_b) return (long)b->date - (long)a->date; } -void bitmap_writer_select_commits(struct commit **indexed_commits, +void bitmap_writer_select_commits(struct bitmap_writer *writer, + struct commit **indexed_commits, unsigned int indexed_commits_nr, int max_bitmaps) { @@ -600,12 +596,12 @@ void bitmap_writer_select_commits(struct commit **indexed_commits, if (indexed_commits_nr < 100) { for (i = 0; i < indexed_commits_nr; ++i) - push_bitmapped_commit(indexed_commits[i]); + push_bitmapped_commit(writer, indexed_commits[i]); return; } - if (writer.show_progress) - writer.progress = start_progress("Selecting bitmap commits", 0); + if (writer->show_progress) + writer->progress = start_progress("Selecting bitmap commits", 0); for (;;) { struct commit *chosen = NULL; @@ -615,8 +611,8 @@ void bitmap_writer_select_commits(struct commit **indexed_commits, if (i + next >= indexed_commits_nr) break; - if (max_bitmaps > 0 && writer.selected_nr >= max_bitmaps) { - writer.selected_nr = max_bitmaps; + if (max_bitmaps > 0 && writer->selected_nr >= max_bitmaps) { + writer->selected_nr = max_bitmaps; break; } @@ -638,13 +634,13 @@ void bitmap_writer_select_commits(struct commit **indexed_commits, } } - push_bitmapped_commit(chosen); + push_bitmapped_commit(writer, chosen); i += next + 1; - display_progress(writer.progress, i); + display_progress(writer->progress, i); } - stop_progress(&writer.progress); + stop_progress(&writer->progress); } @@ -670,12 +666,13 @@ static const struct object_id *oid_access(size_t pos, const void *table) return &index[pos]->oid; } -static void write_selected_commits_v1(struct hashfile *f, off_t *offsets) +static void write_selected_commits_v1(struct bitmap_writer *writer, + struct hashfile *f, off_t *offsets) { int i; - for (i = 0; i < writer.selected_nr; ++i) { - struct bitmapped_commit *stored = &writer.selected[i]; + for (i = 0; i < writer->selected_nr; ++i) { + struct bitmapped_commit *stored = &writer->selected[i]; if (offsets) offsets[i] = hashfile_total(f); @@ -688,10 +685,11 @@ static void write_selected_commits_v1(struct hashfile *f, off_t *offsets) } } -static int table_cmp(const void *_va, const void *_vb) +static int table_cmp(const void *_va, const void *_vb, void *_data) { - struct bitmapped_commit *a = &writer.selected[*(uint32_t *)_va]; - struct bitmapped_commit *b = &writer.selected[*(uint32_t *)_vb]; + struct bitmap_writer *writer = _data; + struct bitmapped_commit *a = &writer->selected[*(uint32_t *)_va]; + struct bitmapped_commit *b = &writer->selected[*(uint32_t *)_vb]; if (a->commit_pos < b->commit_pos) return -1; @@ -701,15 +699,16 @@ static int table_cmp(const void *_va, const void *_vb) return 0; } -static void write_lookup_table(struct hashfile *f, off_t *offsets) +static void write_lookup_table(struct bitmap_writer *writer, struct hashfile *f, + off_t *offsets) { uint32_t i; uint32_t *table, *table_inv; - ALLOC_ARRAY(table, writer.selected_nr); - ALLOC_ARRAY(table_inv, writer.selected_nr); + ALLOC_ARRAY(table, writer->selected_nr); + ALLOC_ARRAY(table_inv, writer->selected_nr); - for (i = 0; i < writer.selected_nr; i++) + for (i = 0; i < writer->selected_nr; i++) table[i] = i; /* @@ -717,17 +716,17 @@ static void write_lookup_table(struct hashfile *f, off_t *offsets) * bitmap corresponds to j'th bitmapped commit (among the selected * commits) in lex order of OIDs. */ - QSORT(table, writer.selected_nr, table_cmp); + QSORT_S(table, writer->selected_nr, table_cmp, writer); /* table_inv helps us discover that relationship (i'th bitmap * to j'th commit by j = table_inv[i]) */ - for (i = 0; i < writer.selected_nr; i++) + for (i = 0; i < writer->selected_nr; i++) table_inv[table[i]] = i; trace2_region_enter("pack-bitmap-write", "writing_lookup_table", the_repository); - for (i = 0; i < writer.selected_nr; i++) { - struct bitmapped_commit *selected = &writer.selected[table[i]]; + for (i = 0; i < writer->selected_nr; i++) { + struct bitmapped_commit *selected = &writer->selected[table[i]]; uint32_t xor_offset = selected->xor_offset; uint32_t xor_row; @@ -748,7 +747,7 @@ static void write_lookup_table(struct hashfile *f, off_t *offsets) xor_row = 0xffffffff; } - hashwrite_be32(f, writer.selected[table[i]].commit_pos); + hashwrite_be32(f, writer->selected[table[i]].commit_pos); hashwrite_be64(f, (uint64_t)offsets[table[i]]); hashwrite_be32(f, xor_row); } @@ -770,12 +769,14 @@ static void write_hash_cache(struct hashfile *f, } } -void bitmap_writer_set_checksum(const unsigned char *sha1) +void bitmap_writer_set_checksum(struct bitmap_writer *writer, + const unsigned char *sha1) { - hashcpy(writer.pack_checksum, sha1); + hashcpy(writer->pack_checksum, sha1); } -void bitmap_writer_finish(struct pack_idx_entry **index, +void bitmap_writer_finish(struct bitmap_writer *writer, + struct pack_idx_entry **index, uint32_t index_nr, const char *filename, uint16_t options) @@ -796,20 +797,20 @@ void bitmap_writer_finish(struct pack_idx_entry **index, memcpy(header.magic, BITMAP_IDX_SIGNATURE, sizeof(BITMAP_IDX_SIGNATURE)); header.version = htons(default_version); header.options = htons(flags | options); - header.entry_count = htonl(writer.selected_nr); - hashcpy(header.checksum, writer.pack_checksum); + header.entry_count = htonl(writer->selected_nr); + hashcpy(header.checksum, writer->pack_checksum); hashwrite(f, &header, sizeof(header) - GIT_MAX_RAWSZ + the_hash_algo->rawsz); - dump_bitmap(f, writer.commits); - dump_bitmap(f, writer.trees); - dump_bitmap(f, writer.blobs); - dump_bitmap(f, writer.tags); + dump_bitmap(f, writer->commits); + dump_bitmap(f, writer->trees); + dump_bitmap(f, writer->blobs); + dump_bitmap(f, writer->tags); if (options & BITMAP_OPT_LOOKUP_TABLE) CALLOC_ARRAY(offsets, index_nr); - for (i = 0; i < writer.selected_nr; i++) { - struct bitmapped_commit *stored = &writer.selected[i]; + for (i = 0; i < writer->selected_nr; i++) { + struct bitmapped_commit *stored = &writer->selected[i]; int commit_pos = oid_pos(&stored->commit->object.oid, index, index_nr, oid_access); @@ -818,10 +819,10 @@ void bitmap_writer_finish(struct pack_idx_entry **index, stored->commit_pos = commit_pos; } - write_selected_commits_v1(f, offsets); + write_selected_commits_v1(writer, f, offsets); if (options & BITMAP_OPT_LOOKUP_TABLE) - write_lookup_table(f, offsets); + write_lookup_table(writer, f, offsets); if (options & BITMAP_OPT_HASH_CACHE) write_hash_cache(f, index, index_nr); diff --git a/pack-bitmap.h b/pack-bitmap.h index c7dea13217..5a1890a2c5 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -97,9 +97,29 @@ int bitmap_has_oid_in_uninteresting(struct bitmap_index *, const struct object_i off_t get_disk_usage_from_bitmap(struct bitmap_index *, struct rev_info *); -void bitmap_writer_show_progress(int show); -void bitmap_writer_set_checksum(const unsigned char *sha1); -void bitmap_writer_build_type_index(struct packing_data *to_pack, +struct bitmap_writer { + struct ewah_bitmap *commits; + struct ewah_bitmap *trees; + struct ewah_bitmap *blobs; + struct ewah_bitmap *tags; + + kh_oid_map_t *bitmaps; + struct packing_data *to_pack; + + struct bitmapped_commit *selected; + unsigned int selected_nr, selected_alloc; + + struct progress *progress; + int show_progress; + unsigned char pack_checksum[GIT_MAX_RAWSZ]; +}; + +void bitmap_writer_init(struct bitmap_writer *writer); +void bitmap_writer_show_progress(struct bitmap_writer *writer, int show); +void bitmap_writer_set_checksum(struct bitmap_writer *writer, + const unsigned char *sha1); +void bitmap_writer_build_type_index(struct bitmap_writer *writer, + struct packing_data *to_pack, struct pack_idx_entry **index, uint32_t index_nr); uint32_t *create_bitmap_mapping(struct bitmap_index *bitmap_git, @@ -109,10 +129,14 @@ int rebuild_bitmap(const uint32_t *reposition, struct bitmap *dest); struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git, struct commit *commit); -void bitmap_writer_select_commits(struct commit **indexed_commits, - unsigned int indexed_commits_nr, int max_bitmaps); -int bitmap_writer_build(struct packing_data *to_pack); -void bitmap_writer_finish(struct pack_idx_entry **index, +void bitmap_writer_select_commits(struct bitmap_writer *writer, + struct commit **indexed_commits, + unsigned int indexed_commits_nr, + int max_bitmaps); +int bitmap_writer_build(struct bitmap_writer *writer, + struct packing_data *to_pack); +void bitmap_writer_finish(struct bitmap_writer *writer, + struct pack_idx_entry **index, uint32_t index_nr, const char *filename, uint16_t options); From patchwork Tue May 14 19:57:00 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13664395 Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 3EBC8182C8F for ; Tue, 14 May 2024 19:57:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.170 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716624; cv=none; b=FgwVxW1+raLb0j3FCdW8yhyXHMcwLq/PNeMDF2OjzLHEK3yUfAXZLyp94UUbCpS5I8TwQBYIpwNVQgSE91GqmuCksN1V3xdb9sGkg1BMMKdSTom9pZi97hh/17UDvb0AYe8EJjlPX8r9f6y8fzbtxSJ50ftl23YEgBvFj4ezq/M= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716624; c=relaxed/simple; bh=YNa2k3iqRN4nGVtedkaIMsojc0JqtDEh1dZf6YF94Wk=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=qONQ3GW9IknaC6tLUm4ddjmESADbGuQ9p4HHI31YYa2LHVlSQjxqIarklXRxz4B8y4jNP1U5r8mkJbmPhcqsOAzgqKVp/ZY2T5W3UVEFyWb7duap/V7CiO5JbCd5paKxcSgYHCrdpvdfQ1n0+WRdLONZEIo3GkQ5lG9uNvDGu8w= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=LwnlLetn; arc=none smtp.client-ip=209.85.222.170 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="LwnlLetn" Received: by mail-qk1-f170.google.com with SMTP id af79cd13be357-78f043eaee9so402922285a.3 for ; Tue, 14 May 2024 12:57:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1715716622; x=1716321422; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=Zs46K1THFTAvWTI4osLAXGh4TVt6cHIlmU1oU5tOwcA=; b=LwnlLetnUCoc/xsQuNbrBekvFc6PTc+DWnpPLHDjfyIVL8JJS3WJGCkAA28q/cs+ND eZdpE8sxNyXU8rhJiE2B3RgaWvn6APpbG9sgczSwpQACn1FFq3aec5/msRRc4yoZ8mzZ 2iRfJnqkmT92BpPGKCzskLgn246XPKRPQVo5tKlKrkQXrOQG4YI/jLqztVU/U7VyTf6I yhwCrDH7/dOYrkBURea8kYIovqrrEAuMx2d+LJr1BUUAfax/xWtwDzJmMc3U9llIkoDz P5UFFeusE8SsUav0pxGlOIMMCLWcqmzATvJzrt3xrI6d87Asbn6rS78Obt6uWgq4THmh aMXA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715716622; x=1716321422; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Zs46K1THFTAvWTI4osLAXGh4TVt6cHIlmU1oU5tOwcA=; b=tvWvBajdw/sOYoWKUkn+s7OAX9ucTcRXM++jSBQQ1lRRaUWhvxb4fCntOPWd2g6Kcw XFdT0TaeiQ/rT+U0xHWEzB1yfMa8cWNlvZEINzMlmIj6FlrKaHfCXLF0sf4f1ZMQI3PM smAlUp9si/YyiuwCvSgMR9UPsmXG0CFsrGatcLY26nRYPcVQ30euj8pMl01tefGyZ4OH U2FIsAwVUx+hCyOyHczMc8loS+8+L4WBg7dt/2sC1vP8ydoFnnBHzqvo2ADooflJu2PQ eJTil9fjp30p1qQHOJVQ/2nNy6GKW9R5QMq3nOyMLjSWRKUWAMLvuonkqqILZimAn6lg K8lA== X-Gm-Message-State: AOJu0YwJM3BxeNJdnOYLGhvgYKBn18Ds5a/zU0hgJN/01YsXhwDxmyoy xG08K4oWurcb2cifbHzSJ9w03GqAvPnlGy9HmIdtiNJbu4Qo357r0rnbG183SUB0XK9DtI96QDf P X-Google-Smtp-Source: AGHT+IEVvvUT80kIBK4E83i62I5BA7wcuRfpkN3pD2cpKtgk+trUtaWALVwEkgMujdeboWG4+VtDXw== X-Received: by 2002:a05:620a:a45:b0:790:e65f:915f with SMTP id af79cd13be357-792c75ac087mr1521922285a.41.1715716621931; Tue, 14 May 2024 12:57:01 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id af79cd13be357-792d7866176sm335369885a.123.2024.05.14.12.57.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 May 2024 12:57:01 -0700 (PDT) Date: Tue, 14 May 2024 15:57:00 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Junio C Hamano Subject: [PATCH 4/6] pack-bitmap: drop unused `max_bitmaps` parameter Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: The `max_bitmaps` parameter in `bitmap_writer_select_commits()` was introduced back in 7cc8f97108 (pack-objects: implement bitmap writing, 2013-12-21), making it original to the bitmap implementation in Git itself. When that patch was merged via 0f9e62e084 (Merge branch 'jk/pack-bitmap', 2014-02-27), its sole caller in builtin/pack-objects.c passed a value of "-1" for `max_bitmaps`, indicating no limit. Since then, the only other caller (in midx.c, added via c528e17966 (pack-bitmap: write multi-pack bitmaps, 2021-08-31)) also uses a value of "-1" for `max_bitmaps`. Since no callers have needed a finite limit for the `max_bitmaps` parameter in the nearly decade that has passed since 0f9e62e084, let's remove the parameter and any dead pieces of code connected to it. Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 3 +-- midx-write.c | 2 +- pack-bitmap-write.c | 8 +------- pack-bitmap.h | 3 +-- 4 files changed, 4 insertions(+), 12 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index ba4c93d241..10e69fdc8e 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1364,8 +1364,7 @@ static void write_pack_file(void) progress); bitmap_writer_select_commits(&bitmap_writer, indexed_commits, - indexed_commits_nr, - -1); + indexed_commits_nr); if (bitmap_writer_build(&bitmap_writer, &to_pack) < 0) die(_("failed to write bitmap index")); bitmap_writer_finish(&bitmap_writer, diff --git a/midx-write.c b/midx-write.c index 5fdc8f2ff5..78fb0a2c8c 100644 --- a/midx-write.c +++ b/midx-write.c @@ -841,7 +841,7 @@ static int write_midx_bitmap(const char *midx_name, for (i = 0; i < pdata->nr_objects; i++) index[pack_order[i]] = &pdata->objects[i].idx; - bitmap_writer_select_commits(&writer, commits, commits_nr, -1); + bitmap_writer_select_commits(&writer, commits, commits_nr); ret = bitmap_writer_build(&writer, pdata); if (ret < 0) goto cleanup; diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index e22fa70745..c0087dab12 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -587,8 +587,7 @@ static int date_compare(const void *_a, const void *_b) void bitmap_writer_select_commits(struct bitmap_writer *writer, struct commit **indexed_commits, - unsigned int indexed_commits_nr, - int max_bitmaps) + unsigned int indexed_commits_nr) { unsigned int i = 0, j, next; @@ -611,11 +610,6 @@ void bitmap_writer_select_commits(struct bitmap_writer *writer, if (i + next >= indexed_commits_nr) break; - if (max_bitmaps > 0 && writer->selected_nr >= max_bitmaps) { - writer->selected_nr = max_bitmaps; - break; - } - if (next == 0) { chosen = indexed_commits[i]; } else { diff --git a/pack-bitmap.h b/pack-bitmap.h index 5a1890a2c5..b2d13d40eb 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -131,8 +131,7 @@ struct ewah_bitmap *bitmap_for_commit(struct bitmap_index *bitmap_git, struct commit *commit); void bitmap_writer_select_commits(struct bitmap_writer *writer, struct commit **indexed_commits, - unsigned int indexed_commits_nr, - int max_bitmaps); + unsigned int indexed_commits_nr); int bitmap_writer_build(struct bitmap_writer *writer, struct packing_data *to_pack); void bitmap_writer_finish(struct bitmap_writer *writer, From patchwork Tue May 14 19:57:03 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13664396 Received: from mail-qk1-f180.google.com (mail-qk1-f180.google.com [209.85.222.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 8A9B0181318 for ; Tue, 14 May 2024 19:57:06 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.222.180 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716627; cv=none; b=LmCTICpT+gO/N+ZNGPuEdSxobfy+Z84kB8mDWJf5ggghHbC/zV0J6aOC67L4CdD/M2pNZ2xQmE0knKLMGGVifvt10xwTR9OecJnzqcYGZpJZa5N7X0Zm+xHqQpu4CyXVV48DIJrshO44npsJ0iEx8hLKyOyMZLWCsw0YN1lTxcE= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716627; c=relaxed/simple; bh=ehEDVivmep7w6WGkel4lNQPZf1Xt4TZePOhGGrZXy5g=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=P75CldDwjZ3DrHajIl6JX5F2Gp+qOW/SE049zYP1I8lnyH8/gSd6KlCWaKNlsRA/dHZNQMUZaNHQhdZZ2jhGB+BgZ4nsUTMWKY6TE5ERpNMnB/FJw+SszccXDqTt05a6HhVq0+0wvOBXOWzguHstbj38MsXiqXg+tx2yqFvo+RU= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=dddn6YLj; arc=none smtp.client-ip=209.85.222.180 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="dddn6YLj" Received: by mail-qk1-f180.google.com with SMTP id af79cd13be357-792c34f891eso506895085a.1 for ; Tue, 14 May 2024 12:57:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1715716625; x=1716321425; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=zXpcyD6elpXjKhMRArstPpg3pnBL+H9lyD65uiqsPYo=; b=dddn6YLjkoN2ENaKF7XQe++slBq3Z05QZSk9Qe/FHt0XB9LHEkJIwNvtwx1mQW8lab B44onIVq0d/sGww5eIvUK7XUHxudUcLv4w0QPB2FCqBFqyc0CchZ7BaYWYc9fEiwCUtc hru8vtfJFKWjH7Xui2M8EtSuaDGOsWSv1x5jNupdlq7cefwguP/C6gThL3PAz23fmKkb Dl14b7FAKslZHfAlcxdszsQTGkAQc0K8QVOG6Pwm1w4Nj2h4ydEp891wrD5pCfpISxcs mCBs+j6nFnT90enIiSwNGYg6dXEd+p7QNtlnwCPt7cLsdN0pDXiNEY7Z6QNghZ5FRg8M 9WpA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715716625; x=1716321425; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=zXpcyD6elpXjKhMRArstPpg3pnBL+H9lyD65uiqsPYo=; b=vKTiHzvYKUIlYcmvPgTY223uYb7Z40fo/9o3QuW9MICBCM0cXSsz9X0ZFY3FNdm7cz UMLuivYz3hd86Iidn5QqVPZKVN/1CyJIKb9pq9S2dFy6zliTB/d5oIWIrP2nw4v4M9/i ics01Rrjn5+dl5+1FFQp30yc4P9XZKW6Rd0oUbp+KXASqaiY3/QdZdDmW+ImpPvKzvTK gR0z6Ay5i1MtEjIV17R1Q+N0dGMeskPD0d4h/ODEpkPe13jt+s6TKBCvtNxNVqNnFoq7 P67dT3OlPmCQwibsqInolXWO8MaiMCApNV0buET5utz+8o961JhqS44efu9QnzopWm64 MgLQ== X-Gm-Message-State: AOJu0Yyvv2QTl1E+j8C6mazYXzjrGZvJfdgtxiA6P67jZ0TybREpgd03 IVCW0GjNHVVOAoc6+dLaNPyqJ3/42urMhEC3zqu30rceEVjINYMy3HQDa0DtMYNcyNmG4TN9MFP m X-Google-Smtp-Source: AGHT+IFZzahmB72zz7OcBqPCZUa4VvXfsTszbH4/YcS+DJNhRdKglN0BxJq9GAR4seG/mHkFLFpRnQ== X-Received: by 2002:a05:620a:9:b0:792:905f:667e with SMTP id af79cd13be357-792c759adfbmr1693785785a.28.1715716625267; Tue, 14 May 2024 12:57:05 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id af79cd13be357-792c3ef7caesm541282285a.57.2024.05.14.12.57.04 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 May 2024 12:57:04 -0700 (PDT) Date: Tue, 14 May 2024 15:57:03 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Junio C Hamano Subject: [PATCH 5/6] pack-bitmap-write.c: avoid uninitialized 'write_as' field Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Prepare to free() memory associated with bitmapped_commit structs by zero'ing the 'write_as' field. In ideal cases, it is fine to do something like: for (i = 0; i < writer->selected_nr; i++) { struct bitmapped_commit *bc = &writer->selected[i]; if (bc->write_as != bc->bitmap) ewah_free(bc->write_as); ewah_free(bc->bitmap); } but if not all of the 'write_as' fields were populated (e.g., because the packing_data given does not form a reachability closure), then we may attempt to free uninitialized memory. Guard against this by preemptively zero'ing this field just in case. Signed-off-by: Taylor Blau --- pack-bitmap-write.c | 1 + 1 file changed, 1 insertion(+) diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index c0087dab12..420f17c2e0 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -112,6 +112,7 @@ static inline void push_bitmapped_commit(struct bitmap_writer *writer, writer->selected[writer->selected_nr].commit = commit; writer->selected[writer->selected_nr].bitmap = NULL; + writer->selected[writer->selected_nr].write_as = NULL; writer->selected[writer->selected_nr].flags = 0; writer->selected_nr++; From patchwork Tue May 14 19:57:06 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 13664397 Received: from mail-qv1-f42.google.com (mail-qv1-f42.google.com [209.85.219.42]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5FE3C181318 for ; Tue, 14 May 2024 19:57:10 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.219.42 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716631; cv=none; b=KqVsb2oa3UAvUSQzZgzFoSnLhMafcQDXKx4p1+PDBThoS+y0aaPKMIq0hACkIovhLMDaYPqXd3EZ+4VlYjAIDGCfaujjEEh0quOlPnRoTVAovoDBB9dXYSP6wmvBmDL+3vGUmwVmIrwatJO73RmYfNvPn50cJ01dFYy/mhGxJzo= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1715716631; c=relaxed/simple; bh=T1NtMAMuReya1kZzMDsEaWvRJdt5rM5vXjCrr0ZDE98=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Apsp4BuM6EHtYaEZ3LrUD/06WSaJtNkpKIvxY9UMUV2dXJHM4eJnmf4Zwod6AJGurwea38GbPjoS4tmGA6U9709wnfZod6Zab2EVfpuNQvxmUMN4xMux13d4B8nZJ87KsTU/jwDdXraSynrI/o0LQYncmGIUZe2r7K9R2OqvvmM= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com; spf=none smtp.mailfrom=ttaylorr.com; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b=2z/aLNmG; arc=none smtp.client-ip=209.85.219.42 Authentication-Results: smtp.subspace.kernel.org; dmarc=none (p=none dis=none) header.from=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; spf=none smtp.mailfrom=ttaylorr.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20230601.gappssmtp.com header.i=@ttaylorr-com.20230601.gappssmtp.com header.b="2z/aLNmG" Received: by mail-qv1-f42.google.com with SMTP id 6a1803df08f44-69b4454e2f1so19043156d6.0 for ; Tue, 14 May 2024 12:57:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20230601.gappssmtp.com; s=20230601; t=1715716629; x=1716321429; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=NIxL665DLt7fNGh/Hz5PY1VgAibPgHVSqFosj4OscVs=; b=2z/aLNmG5pB33RsEVxnq8ctes98mA1omvU+wX6cLF7++Zjq2+S/prGStrbggdbT0mu jUU5zBF/SBqxSaKU0/TBiAp/eVzcWXYyKpMWb8zd+jphLMy06mFkDkzy6f+WVR3leKXu lp1Z/UnzeVXg7Cl4fvVo9ImA/b22nX9/mt0d0YV2Tzl3z2OSTX4F5iAflJjyZ39kAAB/ P0nqHhkoBnzN/yfVg7vmb6xsn4T8Ki0z6qQh209Z5MGK+nx9LizaglY9g0Ko7kLeB2Du EaMpevvFbE0ibjAdfjRK6/VkIGMMcFoLvSr5r98JLA4yLE1MTiH1yGIogMPMWEboYcrW 3bGg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1715716629; x=1716321429; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=NIxL665DLt7fNGh/Hz5PY1VgAibPgHVSqFosj4OscVs=; b=vVW5GwLpP4thC6HsNpsVZNJu42sI3s+PqSP18H8Y6zihZ993fvIiw2jD6tBGV30qE8 fDACtr+WlvOrJo15ff9oELXOs0sma+IQPnTCN6JBMIAMLcDeDjS8Fyf5VJLFfdnnbANd Wuh5Hvpz3U+e07HMn38X9cvi2s8bM/BsJThjL0WsROOQeke7H/GaqaPhM4ZUcT0p1VqK 6oJqb+yEeSEs/WbNtfi0K9vdO+AbaO6T0npTRvOaNA35BbSxlL6+IQiA13+4KCQWu0as rNC5i8qfcWgLQrNlGfEVUmsc52FRUjtO1qk8OaJQDgPw+/c2bmXB2G03A3B6+OEA6OqP dc1w== X-Gm-Message-State: AOJu0YwOJlMzn0IqZZ16M0ii4NHT4nqgho4PAJNLlm3ODjdm5cNJoA37 ibNt2Pj1sP0l+PWMzTosUFSZK6jmDUJ5T5VmRMBpBTCc5Y61yEIQJYZ0FBQAwDiw62aT8FV97d7 q X-Google-Smtp-Source: AGHT+IHnTEF8rMPuPzTok8LQKbS2Dcq4oDMeJTQLIRPY98WBu7dJ2dHNmfrjsGbQdPnr+tTh7aVm7A== X-Received: by 2002:a05:6214:5988:b0:6a0:4fda:af51 with SMTP id 6a1803df08f44-6a1681b4514mr173170306d6.37.1715716628733; Tue, 14 May 2024 12:57:08 -0700 (PDT) Received: from localhost (104-178-186-189.lightspeed.milwwi.sbcglobal.net. [104.178.186.189]) by smtp.gmail.com with ESMTPSA id 6a1803df08f44-6a15f1cd22dsm56630156d6.100.2024.05.14.12.57.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 14 May 2024 12:57:08 -0700 (PDT) Date: Tue, 14 May 2024 15:57:06 -0400 From: Taylor Blau To: git@vger.kernel.org Cc: Jeff King , Patrick Steinhardt , Junio C Hamano Subject: [PATCH 6/6] pack-bitmap: introduce `bitmap_writer_free()` Message-ID: References: Precedence: bulk X-Mailing-List: git@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Now that there is clearer memory ownership around the bitmap_writer structure, introduce a bitmap_writer_free() function that callers may use to free any memory associated with their instance of the bitmap_writer structure. Signed-off-by: Taylor Blau --- builtin/pack-objects.c | 3 ++- midx-write.c | 1 + pack-bitmap-write.c | 23 +++++++++++++++++++++++ pack-bitmap.h | 1 + 4 files changed, 27 insertions(+), 1 deletion(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index 10e69fdc8e..26a6d0d791 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -1245,7 +1245,6 @@ static void write_pack_file(void) uint32_t nr_remaining = nr_result; time_t last_mtime = 0; struct object_entry **write_order; - struct bitmap_writer bitmap_writer; if (progress > pack_to_stdout) progress_state = start_progress(_("Writing objects"), nr_result); @@ -1315,6 +1314,7 @@ static void write_pack_file(void) if (!pack_to_stdout) { struct stat st; struct strbuf tmpname = STRBUF_INIT; + struct bitmap_writer bitmap_writer; char *idx_tmp_name = NULL; /* @@ -1370,6 +1370,7 @@ static void write_pack_file(void) bitmap_writer_finish(&bitmap_writer, written_list, nr_written, tmpname.buf, write_bitmap_options); + bitmap_writer_free(&bitmap_writer); write_bitmap_index = 0; strbuf_setlen(&tmpname, tmpname_len); } diff --git a/midx-write.c b/midx-write.c index 78fb0a2c8c..7c0c08c64b 100644 --- a/midx-write.c +++ b/midx-write.c @@ -853,6 +853,7 @@ static int write_midx_bitmap(const char *midx_name, cleanup: free(index); free(bitmap_name); + bitmap_writer_free(&writer); trace2_region_leave("midx", "write_midx_bitmap", the_repository); diff --git a/pack-bitmap-write.c b/pack-bitmap-write.c index 420f17c2e0..6cae670412 100644 --- a/pack-bitmap-write.c +++ b/pack-bitmap-write.c @@ -32,6 +32,29 @@ void bitmap_writer_init(struct bitmap_writer *writer) memset(writer, 0, sizeof(struct bitmap_writer)); } +void bitmap_writer_free(struct bitmap_writer *writer) +{ + uint32_t i; + + if (!writer) + return; + + ewah_free(writer->commits); + ewah_free(writer->trees); + ewah_free(writer->blobs); + ewah_free(writer->tags); + + kh_destroy_oid_map(writer->bitmaps); + + for (i = 0; i < writer->selected_nr; i++) { + struct bitmapped_commit *bc = &writer->selected[i]; + if (bc->write_as != bc->bitmap) + ewah_free(bc->write_as); + ewah_free(bc->bitmap); + } + free(writer->selected); +} + void bitmap_writer_show_progress(struct bitmap_writer *writer, int show) { writer->show_progress = show; diff --git a/pack-bitmap.h b/pack-bitmap.h index b2d13d40eb..3091095f33 100644 --- a/pack-bitmap.h +++ b/pack-bitmap.h @@ -139,6 +139,7 @@ void bitmap_writer_finish(struct bitmap_writer *writer, uint32_t index_nr, const char *filename, uint16_t options); +void bitmap_writer_free(struct bitmap_writer *writer); char *midx_bitmap_filename(struct multi_pack_index *midx); char *pack_bitmap_filename(struct packed_git *p);