From patchwork Sun Jan 6 16:45:30 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ren=C3=A9_Scharfe?= X-Patchwork-Id: 10749575 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BEB5C13BF for ; Sun, 6 Jan 2019 16:45:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9FB0C288F8 for ; Sun, 6 Jan 2019 16:45:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 938E228900; Sun, 6 Jan 2019 16:45:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 32909288F8 for ; Sun, 6 Jan 2019 16:45:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726041AbfAFQpn (ORCPT ); Sun, 6 Jan 2019 11:45:43 -0500 Received: from mout.web.de ([217.72.192.78]:54829 "EHLO mout.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725986AbfAFQpn (ORCPT ); Sun, 6 Jan 2019 11:45:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1546793133; bh=O517+Q5YbblqExnLmtC8PkM4DI+zsFptNKNX0La8tqw=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=iO8oOOLoE8G2Jtd5kACUzPEl1eVSQ2A6QK2Km/fiIavoUJw/waHlIFtDjMrMW5x3e s7evNaXuTHAdHZfuRgetZCbtbq9rjc0pP4AWwGw2nhLzCCqiBG64y3FZzwcQj6nRsz qV18FbngusjnpofOtjA3yBFCoQmpbaWWtB7mwT+8= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from [192.168.178.36] ([91.20.59.41]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0MCImL-1gXcxp44PZ-0095qc; Sun, 06 Jan 2019 17:45:33 +0100 Subject: [PATCH 1/3] object-store: factor out odb_loose_cache() To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= References: <3512c798-aa42-6fba-ee82-d33a8985be91@web.de> From: =?utf-8?q?Ren=C3=A9_Scharfe?= Message-ID: Date: Sun, 6 Jan 2019 17:45:30 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <3512c798-aa42-6fba-ee82-d33a8985be91@web.de> Content-Language: en-US X-Provags-ID: V03:K1:Oy+eQGTDOT7qvHkHWRAbqZ0f/fwyiGDSL9e2YK7vFrOaluzNDak xOA0mQ4h8XoWo6G0EMVSyi5NzB4AqpBkGLM7U3oHWHaiK+NTJGEzP0cb8EardNkJBphuV6Y sG1X+3pI4Ega1B3vt2xP0uL1KcFT32wuzCcm7U/ECvvLmY3BjNEIrY0EZLT6yLkoU7tfxNI c7opHN+N/DUy+xdRFOIkg== X-UI-Out-Filterresults: notjunk:1;V03:K0:C2ckw3j6Lic=:HG2S8i34G8th4UHFPnvftX 1vvaTvaD9J18OYPQ3kkAIlF8JqTtAMAv5E4Dn2295Jq4cOWutPGW/Z0/lez2+uUF9lfHAWhNj EFN1/ATbVpsUeM0xcDS3I+0kLQ5r6kWZRt+Tz4nxbY72DA/XRAQ3xGzWopMcJkwLf1nFFECE7 +GAlrxIAC8omzB1+j+RoI4s1jguOosyT1FYOTO/x3TlEVU1tBoyoTKKn6kQR1KqwbeVAkcYE6 QcqyK1MNUpgyJOt5/XEtA4QLKtxtd+2wKWo3yObYM0n7jP2+9+kdorO7tB6AkTAXMrIbi+UDP C3pj1qvBojhWMMeXY4rvmpNC+NYGCQ13f/pDe9wLBzA6rDalMo4Wso4XgTl8DqyQRTfjWc6kY MYt67YCaWV6nkEPkUZ9KvkTL8c/WvM88HlXaulxc+qoindOCDelEvHwe25KHrOKzsKEg+cmR+ rycHvwb5BC6V9CDCUbpF6HVZObVWEAMHwBB0nDWM0UNPjP76I3nSXT/YOCF+kC9BVtUepvx8b YI4dIImOVFxwDF/f6bL9XA55jKW9HuGTMQxuZT5i1FaQSE18iVLwo+FRGmxunEIKmwtF75bw2 1BAdhc3LXP7E98HvqBU1PDaQHiFLbpQgH+5FBO72VApEYOTFFXEoCfC7EIyOe4+kIg4wImYXs vdtywP1WbvVvBEjB2ZdXyuLxcuqSRU87xZuCaz5WB2ky4/T6kndSpgvXHrJi2tfQsVB6GPLut N8aKxhuf+d0eHGRDWO49L7qfaXDNon79qj3HFVb+1CYU9jRQJn7vw9+DF2jzQWz93Vkwi1sSt wDPG7as8aIRF8OZmwUrYDNQlxa/I+dG3ZM/MaRWgP5t5rChFcNiBjZapoq9XOVeAniJJUzyx8 QDU5VysyvitXT5UELAfV4z0BlG7TO9bkImIYUYFFVZZ7jNILGTYdLaujm0sh5G Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add and use a function for loading the entries if a loose object subdirectory for a given object ID. It frees callers from deriving the fanout key; they can use the returned oid_array reference for lookups or forward range scans. Suggested-by: Jeff King Signed-off-by: Rene Scharfe --- object-store.h | 7 +++++++ sha1-file.c | 12 +++++++++--- sha1-name.c | 10 +++++----- 3 files changed, 21 insertions(+), 8 deletions(-) diff --git a/object-store.h b/object-store.h index 60758efad8..7236c571c0 100644 --- a/object-store.h +++ b/object-store.h @@ -54,6 +54,13 @@ void add_to_alternates_memory(const char *dir); */ void odb_load_loose_cache(struct object_directory *odb, int subdir_nr); +/* + * Populate and return the loose object cache array corresponding to the + * given object ID. + */ +struct oid_array *odb_loose_cache(struct object_directory *odb, + const struct object_id *oid); + struct packed_git { struct packed_git *next; struct list_head mru; diff --git a/sha1-file.c b/sha1-file.c index 5a272f70de..cb8583b634 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -924,7 +924,6 @@ static int open_sha1_file(struct repository *r, static int quick_has_loose(struct repository *r, const unsigned char *sha1) { - int subdir_nr = sha1[0]; struct object_id oid; struct object_directory *odb; @@ -932,8 +931,7 @@ static int quick_has_loose(struct repository *r, prepare_alt_odb(r); for (odb = r->objects->odb; odb; odb = odb->next) { - odb_load_loose_cache(odb, subdir_nr); - if (oid_array_lookup(&odb->loose_objects_cache, &oid) >= 0) + if (oid_array_lookup(odb_loose_cache(odb, &oid), &oid) >= 0) return 1; } return 0; @@ -2152,6 +2150,14 @@ static int append_loose_object(const struct object_id *oid, const char *path, return 0; } +struct oid_array *odb_loose_cache(struct object_directory *odb, + const struct object_id *oid) +{ + int subdir_nr = oid->hash[0]; + odb_load_loose_cache(odb, subdir_nr); + return &odb->loose_objects_cache; +} + void odb_load_loose_cache(struct object_directory *odb, int subdir_nr) { struct strbuf buf = STRBUF_INIT; diff --git a/sha1-name.c b/sha1-name.c index b24502811b..a656481c6a 100644 --- a/sha1-name.c +++ b/sha1-name.c @@ -87,21 +87,21 @@ static int match_sha(unsigned, const unsigned char *, const unsigned char *); static void find_short_object_filename(struct disambiguate_state *ds) { - int subdir_nr = ds->bin_pfx.hash[0]; struct object_directory *odb; for (odb = the_repository->objects->odb; odb && !ds->ambiguous; odb = odb->next) { int pos; + struct oid_array *loose_objects; - odb_load_loose_cache(odb, subdir_nr); - pos = oid_array_lookup(&odb->loose_objects_cache, &ds->bin_pfx); + loose_objects = odb_loose_cache(odb, &ds->bin_pfx); + pos = oid_array_lookup(loose_objects, &ds->bin_pfx); if (pos < 0) pos = -1 - pos; - while (!ds->ambiguous && pos < odb->loose_objects_cache.nr) { + while (!ds->ambiguous && pos < loose_objects->nr) { const struct object_id *oid; - oid = odb->loose_objects_cache.oid + pos; + oid = loose_objects->oid + pos; if (!match_sha(ds->len, ds->bin_pfx.hash, oid->hash)) break; update_candidates(ds, oid); From patchwork Sun Jan 6 16:45:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ren=C3=A9_Scharfe?= X-Patchwork-Id: 10749577 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8BF2E13AD for ; Sun, 6 Jan 2019 16:45:55 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 7B457288F8 for ; Sun, 6 Jan 2019 16:45:55 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6FD7A28900; Sun, 6 Jan 2019 16:45:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1C660288F8 for ; Sun, 6 Jan 2019 16:45:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726056AbfAFQpv (ORCPT ); Sun, 6 Jan 2019 11:45:51 -0500 Received: from mout.web.de ([217.72.192.78]:40017 "EHLO mout.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725986AbfAFQpu (ORCPT ); Sun, 6 Jan 2019 11:45:50 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1546793143; bh=sehSlELMZnzQ77WDbHA5lcmtATsZ55HkGypDIAiUe/U=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=h497JqI6ClAuHGMkr1qmqwhEVqjR95AHesgQp5+5jduXmUDeUoK7NBN/6dNOOaIWJ 5tgYzlcDi5hTjCQNCflA3StXnyt88DH41DtVLwJsTgC4p19nKMReYCaZHWbfc5S86d yWfvwG0zwR2ucWGZCRGVTUzHq9unYzIGt0tgdntk= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from [192.168.178.36] ([91.20.59.41]) by smtp.web.de (mrweb101 [213.165.67.124]) with ESMTPSA (Nemesis) id 0M5Oct-1hVoOv3s4d-00zVdI; Sun, 06 Jan 2019 17:45:43 +0100 Subject: [PATCH 2/3] object-store: factor out odb_clear_loose_cache() To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= References: <3512c798-aa42-6fba-ee82-d33a8985be91@web.de> From: =?utf-8?q?Ren=C3=A9_Scharfe?= Message-ID: <6e8c3e47-4f11-e26f-64a6-ea4cfd8e2942@web.de> Date: Sun, 6 Jan 2019 17:45:39 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <3512c798-aa42-6fba-ee82-d33a8985be91@web.de> Content-Language: en-US X-Provags-ID: V03:K1:/i84EemPjfioFtJ3PdjpaWd097C/FjvMUq3H5M+rHBwXY2z12WD vYYnrqxJyQzIx6KLzR55Fw9JQGXxPLWxOJIW+9QZ6pfdaYQGZyi4BPuWSazf7q67/hXhwNm yvf00HZ+h9yAD8L2vJuNyGeU+tPgHIozJvrsf2yI4DjUhw38tJ/Dblv00MeOWxK5zbH9ny7 gkOcV6N93UsVh7IIwWj0w== X-UI-Out-Filterresults: notjunk:1;V03:K0:FT94YgP5Tl0=:hWM3wvZFe4AwcU7t1vAQlt kGHMtfX5AXdNDiQINNOl4JJdv48oqP2yWq3WXonDdjCA4La2rM3t9kZMTjcXi5JyIQGLmd8yL EZo9p+zLO9v0nSYd9gcmqIw4IqgItBtgktiyJIpxxzaH/NM7FjF6OSxuPYRVbzvzNfyB7jgig ldTiiCtrutFU5tB9fIiDhuuzJyZVXnr08wbSGQbAiZcrienGQGTQ3Ysc0NUwJWW4FJCCxPAwt OKun2MnEPfD2j/kksaJwkFG18OXlL6XEdMZ+2pM2b9RvEfole6XBGCOHCv3nRHYqAmleGezWS MdK8iXWd7Zey3YBDXUG7xKQ9TeI7OUgHX3l75JF2YXlEcfm45S3FKBTCWBnKgMDfFP7smjmUZ pVgnBIKbDgqQrDsov3Nt0GHM6sP4jGVq4QqE/6h86IkuKt9XpmrCmWJKhMVNDLJed5Vch03w8 5HUqVe8eDGGW8OXlIV4xyouo/SlXgno76xIcXnKJB2tfILo6KPdXBQjcOB4WuU6N1YMEbKbx4 CbrRlaYpUhFsi0aYsRi/w5Tzxh5XC9+NM5GvrOCki2Iw79uAft6L8Nls/s/+uAsWaBGstKoqE HFR27lNRnU5DEFTDvxuV8JUZ79deTiC7RHE+vJvtAsjDjq7JCQdJ7WK+/H7lTuAb2c9lHa3nt pIQpgin164r3RWyTwiqUXlzGLrR3gG5Q3bSaEnS+WY7EGPh43veHgbDEcdP+2KOHmwHxBbJuM iX4DQFHU2OTgehKcXraWrfeiv5eMzIj91AyEFiDHCR+4iUsdSs1+49eZqPLY31yig+Pouaj7H unbP2THnJL5FQHVmJpSQcz96UcW4GxyqC6dF79bTcMeKYCAjQJ6e2GhDkorZB4hGE8yYj6eb0 rzaHOB+i0hObg7/tGMRPSbylkAnm1T+EHAN7KJLsSec7BESnQVNoQrh6htIiHv Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Add and use a function for emptying the loose object cache, so callers don't have to know any of its implementation details. Signed-off-by: Rene Scharfe --- object-store.h | 3 +++ object.c | 2 +- packfile.c | 7 ++----- sha1-file.c | 7 +++++++ 4 files changed, 13 insertions(+), 6 deletions(-) diff --git a/object-store.h b/object-store.h index 7236c571c0..709bf856b6 100644 --- a/object-store.h +++ b/object-store.h @@ -61,6 +61,9 @@ void odb_load_loose_cache(struct object_directory *odb, int subdir_nr); struct oid_array *odb_loose_cache(struct object_directory *odb, const struct object_id *oid); +/* Empty the loose object cache for the specified object directory. */ +void odb_clear_loose_cache(struct object_directory *odb); + struct packed_git { struct packed_git *next; struct list_head mru; diff --git a/object.c b/object.c index 79d636091c..a5c5cf830f 100644 --- a/object.c +++ b/object.c @@ -485,7 +485,7 @@ struct raw_object_store *raw_object_store_new(void) static void free_object_directory(struct object_directory *odb) { free(odb->path); - oid_array_clear(&odb->loose_objects_cache); + odb_clear_loose_cache(odb); free(odb); } diff --git a/packfile.c b/packfile.c index 8c6b47cc77..0fe9c21bf1 100644 --- a/packfile.c +++ b/packfile.c @@ -994,11 +994,8 @@ void reprepare_packed_git(struct repository *r) { struct object_directory *odb; - for (odb = r->objects->odb; odb; odb = odb->next) { - oid_array_clear(&odb->loose_objects_cache); - memset(&odb->loose_objects_subdir_seen, 0, - sizeof(odb->loose_objects_subdir_seen)); - } + for (odb = r->objects->odb; odb; odb = odb->next) + odb_clear_loose_cache(odb); r->objects->approximate_object_count_valid = 0; r->objects->packed_git_initialized = 0; diff --git a/sha1-file.c b/sha1-file.c index cb8583b634..2f965b2688 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -2178,6 +2178,13 @@ void odb_load_loose_cache(struct object_directory *odb, int subdir_nr) strbuf_release(&buf); } +void odb_clear_loose_cache(struct object_directory *odb) +{ + oid_array_clear(&odb->loose_objects_cache); + memset(&odb->loose_objects_subdir_seen, 0, + sizeof(odb->loose_objects_subdir_seen)); +} + static int check_stream_sha1(git_zstream *stream, const char *hdr, unsigned long size, From patchwork Sun Jan 6 16:45:52 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: =?utf-8?q?Ren=C3=A9_Scharfe?= X-Patchwork-Id: 10749579 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 465B491E for ; Sun, 6 Jan 2019 16:51:13 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 2EDDE287A4 for ; Sun, 6 Jan 2019 16:51:13 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 209C628904; Sun, 6 Jan 2019 16:51:13 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id AF02D287A4 for ; Sun, 6 Jan 2019 16:51:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726036AbfAFQvJ (ORCPT ); Sun, 6 Jan 2019 11:51:09 -0500 Received: from mout.web.de ([217.72.192.78]:45877 "EHLO mout.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726012AbfAFQvJ (ORCPT ); Sun, 6 Jan 2019 11:51:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1546793468; bh=TjC+/jrMfVIuwuXEwMRFTnLRIzlLm1rnVMal6zc7bbk=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=NUoplDNlJNhMN0urT15+jsqBYdHPkzxc1/FpuqsxkYlES/LLdn4bbb9FyfnFuSifK HhRkAtxlnF8luZYMEuFElo+uOmK5h2EkUWGiFoJF8HdiFpGyeyki+RMlnMDrv8qIhd W9JNdjc4/mQ7fW+ignmB7FZVGIuhh4552CINhWB4= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from [192.168.178.36] ([91.20.59.41]) by smtp.web.de (mrweb103 [213.165.67.124]) with ESMTPSA (Nemesis) id 0LaCbS-1h9S2u06u6-00m0uy; Sun, 06 Jan 2019 17:45:54 +0100 Subject: [PATCH 3/3] object-store: use one oid_array per subdirectory for loose cache To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= References: <3512c798-aa42-6fba-ee82-d33a8985be91@web.de> From: =?utf-8?q?Ren=C3=A9_Scharfe?= Message-ID: Date: Sun, 6 Jan 2019 17:45:52 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <3512c798-aa42-6fba-ee82-d33a8985be91@web.de> Content-Language: en-US X-Provags-ID: V03:K1:a5yqLVtLJmWdbgp43UD6uhuyNEgK4HOptWU/KIeBYEfVxo3287q bwjFTuqCamJhzSttM2TF49pmpSMoqu7ev5VN87YiLq97Oh+8uZRuvB6SEwuNZZt1JNqLtC7 7iag3p6cZjv+BlSwMraKj366BhAvlGBIOypKF0+YyJpD3ZvBINgCXAzbvbro3/OaZ2Z8llf 7svlGZ90VFmHhh7rWYZeg== X-UI-Out-Filterresults: notjunk:1;V03:K0:y9P8xTMudWk=:OnzqHl6YwF0o7U0x535mnA N5SjXxmbs0Noyag2yjisaMeELZa+8kJT/8W5Ww0fodPoZz18uPEi52EipcwluFx6jZUb4T4RT RAJ/N+eul/T5wM2SLGro/jGCPdtjXmQnGBaNzqSwXrinWgdBysilwsDa0fYKl/2O2ej8BlfbG YBwEs54+JEr7RUvsntm8N1OccboNtcmILKE2glRO+4ptNMvcDMI3TRJICp/GDGRcDQtOarsqy KsxYq10x6zwU6SUs7yqlJDUgWJpGr7AF1BSJXEklRULKUaU7+Clja1trATK5Sc6PMswTwp4lC hGg59AEbkerwGEUK5E+4kORUpEjea7d8r6w5RlcIGXGO6ueb8vv+aN1Jm/aOc7a23U/qNyNsd Ymm1LWJh8UFqo8CxbDP0RhYNsKoNlsyaOhjc59ANcCW+gCXWpFg4kSXT1tRe+Ys+gj7IGxHZm SdjOmrxTH9tvcxFJLozvPA6WzEQNEG9VbD8BnImGXfb3R8VH0HNQs9Bq4H4NWmPq3TNqM5fmK ZdBZuN4et3MjWPs2Kjv8FeKhseNKBQMvlIshZlpo+2CjuWCHW5qRJXnnYXimxI4sjLwMFCeYv Ebm6frgSS26qFYhXc7WQkdohWpodhDfwd93rsdtdIs4tN2W7SQh0Uu9AyLrOVMKBPZi1dHhp2 8UT+bXDEFerL02pOXn0S8lFErV+4Ee0NpH5NCgySvaAOW8r71tJ+X8cbq0C5IWLNt829EhEs3 1gYX5xzzyYXFyxB5zfUjxDdyskmhg0dujFzywTt9CzGe0XStyzcfZxKgpDdIjOxiEDx8gunzw NUkmhVrXtfFQnCJ2PYBmgkAYmJx98mRQPAuZc6y7MpbC+t8gjI6rB5w7+Qtc4DepbPNaGnV5Q wHqwAx+MSbOzgdU3gfs5GQpWDhsSWWZLygSh50epEXDWshECupfsrvEhYz1rRB Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The loose objects cache is filled one subdirectory at a time as needed. It is stored in an oid_array, which has to be resorted after each add operation. So when querying a wide range of objects, the partially filled array needs to be resorted up to 255 times, which takes over 100 times longer than sorting once. Use one oid_array for each subdirectory. This ensures that entries have to only be sorted a single time. It also avoids eight binary search steps for each cache lookup as a small bonus. The cache is used for collision checks for the log placeholders %h, %t and %p, and we can see the change speeding them up in a repository with ca. 100 objects per subdirectory: $ git count-objects 26733 objects, 68808 kilobytes Test HEAD^ HEAD -------------------------------------------------------------------- 4205.1: log with %H 0.51(0.47+0.04) 0.51(0.49+0.02) +0.0% 4205.2: log with %h 0.84(0.82+0.02) 0.60(0.57+0.03) -28.6% 4205.3: log with %T 0.53(0.49+0.04) 0.52(0.48+0.03) -1.9% 4205.4: log with %t 0.84(0.80+0.04) 0.60(0.59+0.01) -28.6% 4205.5: log with %P 0.52(0.48+0.03) 0.51(0.50+0.01) -1.9% 4205.6: log with %p 0.85(0.78+0.06) 0.61(0.56+0.05) -28.2% 4205.7: log with %h-%h-%h 0.96(0.92+0.03) 0.69(0.64+0.04) -28.1% Reported-by: Ævar Arnfjörð Bjarmason Signed-off-by: Rene Scharfe --- object-store.h | 2 +- sha1-file.c | 9 ++++++--- 2 files changed, 7 insertions(+), 4 deletions(-) diff --git a/object-store.h b/object-store.h index 709bf856b6..2fb6c0e4db 100644 --- a/object-store.h +++ b/object-store.h @@ -20,7 +20,7 @@ struct object_directory { * Be sure to call odb_load_loose_cache() before using. */ char loose_objects_subdir_seen[256]; - struct oid_array loose_objects_cache; + struct oid_array loose_objects_cache[256]; /* * Path to the alternative object store. If this is a relative path, diff --git a/sha1-file.c b/sha1-file.c index 2f965b2688..c3c6e50704 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -2155,7 +2155,7 @@ struct oid_array *odb_loose_cache(struct object_directory *odb, { int subdir_nr = oid->hash[0]; odb_load_loose_cache(odb, subdir_nr); - return &odb->loose_objects_cache; + return &odb->loose_objects_cache[subdir_nr]; } void odb_load_loose_cache(struct object_directory *odb, int subdir_nr) @@ -2173,14 +2173,17 @@ void odb_load_loose_cache(struct object_directory *odb, int subdir_nr) for_each_file_in_obj_subdir(subdir_nr, &buf, append_loose_object, NULL, NULL, - &odb->loose_objects_cache); + &odb->loose_objects_cache[subdir_nr]); odb->loose_objects_subdir_seen[subdir_nr] = 1; strbuf_release(&buf); } void odb_clear_loose_cache(struct object_directory *odb) { - oid_array_clear(&odb->loose_objects_cache); + int i; + + for (i = 0; i < ARRAY_SIZE(odb->loose_objects_cache); i++) + oid_array_clear(&odb->loose_objects_cache[i]); memset(&odb->loose_objects_subdir_seen, 0, sizeof(odb->loose_objects_subdir_seen)); } From patchwork Mon Jan 7 17:29:16 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: =?utf-8?q?Ren=C3=A9_Scharfe?= X-Patchwork-Id: 10750927 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 069FE746 for ; Mon, 7 Jan 2019 17:29:35 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E647828922 for ; Mon, 7 Jan 2019 17:29:34 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id DA81D28924; Mon, 7 Jan 2019 17:29:34 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8399428922 for ; Mon, 7 Jan 2019 17:29:34 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727888AbfAGR3c (ORCPT ); Mon, 7 Jan 2019 12:29:32 -0500 Received: from mout.web.de ([212.227.17.11]:52479 "EHLO mout.web.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727370AbfAGR3c (ORCPT ); Mon, 7 Jan 2019 12:29:32 -0500 X-Greylist: delayed 89008 seconds by postgrey-1.27 at vger.kernel.org; Mon, 07 Jan 2019 12:29:31 EST DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=web.de; s=dbaedf251592; t=1546882158; bh=M9IMDCCVg2aFgCOLyTFoZd8TjdQLkoH6wae7VaJ6/Dc=; h=X-UI-Sender-Class:Subject:To:Cc:References:From:Date:In-Reply-To; b=G+riCAGC/tjoVOXy+XjMZV9s6VMFn7xBvTd28ZuiZAZKKw/YRCUmQtDL4AD0kLfYK 4BO+vimzO3w3ifhGwp7NleQlOi3cuxi/R+IAzEx9PHElNJW95M3yXYUfuK9yg5rXK+ 7vgVpHlvOK1MKqwOSxnh/GLRV2rQA0Bf/1liwj5Y= X-UI-Sender-Class: c548c8c5-30a9-4db5-a2e7-cb6cb037b8f9 Received: from [192.168.178.36] ([91.20.59.41]) by smtp.web.de (mrweb101 [213.165.67.124]) with ESMTPSA (Nemesis) id 0M7bYJ-1hTB7p0ckR-00xIZp; Mon, 07 Jan 2019 18:29:18 +0100 Subject: [PATCH 4/3] object-store: retire odb_load_loose_cache() To: git@vger.kernel.org Cc: Junio C Hamano , Jeff King , =?utf-8?b?w4Z2YXIgQXJuZmrDtnLDsCBCamFybWFzb24=?= References: <3512c798-aa42-6fba-ee82-d33a8985be91@web.de> From: =?utf-8?q?Ren=C3=A9_Scharfe?= Message-ID: Date: Mon, 7 Jan 2019 18:29:16 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <3512c798-aa42-6fba-ee82-d33a8985be91@web.de> Content-Language: en-US X-Provags-ID: V03:K1:V0Z97OkjCT2jc1RXGoQ4Ed8p4eE6zptHydh43hBcRCcXJaCaSgb Ir+zz06zo6mQpJqsGawLGtdEfqKxIRJVcf3YO0MS0CDX5Q3b0acN7GQrIGGMT8I4ruUmNOl pSTy7CguGiXEtqUa9/2nbVPZBYd99ayPSK2juMbviWIkBNiNIgkJ8ztm+9IH+q0QHMlUBk4 P7kz2uSLxMtcWFLoyXzLw== X-UI-Out-Filterresults: notjunk:1;V03:K0:5OXrhE5XZNA=:C0UVtxruo89tdKgPxRXOJp lzC4M9z+foiEr5nLNlNbJaFnXjfZw8Cl8DDya0S428ZHaPPQ0K0zm4E+2hCPdaaMV7dS/3Qiv UDPNvqhmlvAr9m3bM4QmL7jbPkYZJr6DOpeBcBm0Co4j+FwT+jh8MOIe9uVuXHDgg4vpaLaOI qftOjHFrsk60DPdDRF+jqfzzS/vkfKt28O1N0OnftqIFgvcU/796THnjFZEGSqlfpZRtKLCe/ QZCySRko38DVIROM8TAz21OY5y27KsPOhad0ijW6YkjQMuHNHCmtBF9HgfyKF2qFXYDxBtu7x zYTkNbzumWDqBmuZ++I6XRxZCHwydTabrLdCNiElMCDWk21GfPsuwFMKmMjnZ7Hq7sXdGdFXK ym5iRv3KN/+7yRAkaYNojvFeyN0UK5JhYDOaXV4760PhP83xDQv7vbFkYVDFrI4NDeIXSVHDZ BhbKrYzG/LppPQ3aDwp64gHRfYyxbCZ7Sdin37LPFYjAgZ2uRzKgKzfoBKjDsp05GHiSUy0B6 QI3XDFRCiiFba7P4+R5kFFKPl+opwZP0qh7JVYqyQAx2taA+5TTDqUTows4mtx+qJ6NUqJQSf 6ru2oIPsY2BfwepwwbaEVdJ0x680z6dAmg7q6mn8eP3ih6lE00OdoeaUqPGOYxj8ebppeBEIg qLYBO9pV4azvdy91kh/vcbWfR33/bM4Kbjw2cY1fsMqKzxY13ymd1sfxRyaDP9EFe8MAThjnd M2kyXSe7bXn2XuQ/qwvUCgZC1udOql5iNYf5ESnLZzF0uX8Zhfby1JQ1W5OIJ26VpRsrbXOI3 Vh5IWSlIvBmd2uqLNt5r/hjvu0YBnbps43+7JmKZwBzdZozJpX9OIOS/o7DDSizs3c9IYoD8C ZHrz3XfI0DbeKtKjB+IG2iCdauynkm9d2oaG3Sxj+AGjBc5Vvyzsgers2aksld Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Inline odb_load_loose_cache() into its only remaining caller, odb_loose_cache(). The latter offers a nicer interface for loading the cache, as it doesn't require callers to deal with fanout directory numbers directly. Signed-off-by: Rene Scharfe --- object-store.h | 7 ------- sha1-file.c | 9 ++------- 2 files changed, 2 insertions(+), 14 deletions(-) diff --git a/object-store.h b/object-store.h index 2fb6c0e4db..e16aa38cae 100644 --- a/object-store.h +++ b/object-store.h @@ -47,13 +47,6 @@ void add_to_alternates_file(const char *dir); */ void add_to_alternates_memory(const char *dir); -/* - * Populate an odb's loose object cache for one particular subdirectory (i.e., - * the one that corresponds to the first byte of objects you're interested in, - * from 0 to 255 inclusive). - */ -void odb_load_loose_cache(struct object_directory *odb, int subdir_nr); - /* * Populate and return the loose object cache array corresponding to the * given object ID. diff --git a/sha1-file.c b/sha1-file.c index c3c6e50704..efcb2cbe74 100644 --- a/sha1-file.c +++ b/sha1-file.c @@ -2154,12 +2154,6 @@ struct oid_array *odb_loose_cache(struct object_directory *odb, const struct object_id *oid) { int subdir_nr = oid->hash[0]; - odb_load_loose_cache(odb, subdir_nr); - return &odb->loose_objects_cache[subdir_nr]; -} - -void odb_load_loose_cache(struct object_directory *odb, int subdir_nr) -{ struct strbuf buf = STRBUF_INIT; if (subdir_nr < 0 || @@ -2167,7 +2161,7 @@ void odb_load_loose_cache(struct object_directory *odb, int subdir_nr) BUG("subdir_nr out of range"); if (odb->loose_objects_subdir_seen[subdir_nr]) - return; + return &odb->loose_objects_cache[subdir_nr]; strbuf_addstr(&buf, odb->path); for_each_file_in_obj_subdir(subdir_nr, &buf, @@ -2176,6 +2170,7 @@ void odb_load_loose_cache(struct object_directory *odb, int subdir_nr) &odb->loose_objects_cache[subdir_nr]); odb->loose_objects_subdir_seen[subdir_nr] = 1; strbuf_release(&buf); + return &odb->loose_objects_cache[subdir_nr]; } void odb_clear_loose_cache(struct object_directory *odb)