From patchwork Tue Nov 20 06:11:47 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Nieder X-Patchwork-Id: 10689905 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 8D63715A7 for ; Tue, 20 Nov 2018 06:11:52 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 73DE42993B for ; Tue, 20 Nov 2018 06:11:52 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 60D102995D; Tue, 20 Nov 2018 06:11:52 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,FSL_HELO_FAKE,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CB0D82993B for ; Tue, 20 Nov 2018 06:11:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730857AbeKTQjN (ORCPT ); Tue, 20 Nov 2018 11:39:13 -0500 Received: from mail-pg1-f196.google.com ([209.85.215.196]:33341 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726883AbeKTQjN (ORCPT ); Tue, 20 Nov 2018 11:39:13 -0500 Received: by mail-pg1-f196.google.com with SMTP id z11so438944pgu.0 for ; Mon, 19 Nov 2018 22:11:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=4QV3tjF1dLhoYiGEnKVkdAEUBkl/LTRlEmdjv5PrZ6E=; b=jpnwUhkCFTsXPpS6Eo19ksckyUggO/DhUhGf4FRYNiC7yZFAfoheIiYIf3uojBlPMC 7LB+q00unBiXa66cuBK6WKhzS9ZIQHbISIsqifCywhp9kQcofTDreDsHHTCGg+6w43zW ZkISTe7bTNbUuLVIb/wCDNOYBu2gGGWHV+fFwewRrZpseVNJephlGQUOFuxYlE4wpDLl Vj8FHdo/iWq4myhFObPae4DRaPrj7ws7dN2HjGdnLEDbQTkz7RwNWFqFG/1xVEFL/GRA n5oL4hstftqE118YmkZhhJMBs8JhXM5RrDQjwWf0uCpabwLi2dhwSMLGomZmJqPcsMkC 1xCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=4QV3tjF1dLhoYiGEnKVkdAEUBkl/LTRlEmdjv5PrZ6E=; b=XFvUBlt0K5FtjvV4zvDg5rx0fAue3ZCsOtgcNdJeCVrMagknX8QoUQ4SQ71+7J+j99 Zxkp9YL7ZLlPwcThqqK1r+BDl33pL0Dxw5NXReB0Qa7ZIghP42ca8NppWEflDLWhL/8o UaXpLKhaHh3VgVS66vgLjfIuZo3v6xIEGIJwlNVibh8oqZ2RTDamOWi4wcU7QzXPoOwS 1RAkL8IS0nL29rF/c8ztVI0FZHWd599mFsEb+wUpcpzsyQjPHtMu+v/L9TZtKvpg783n +y0HCnsOQMPB8EbT8iFBRaRuUGKSxSqxpa5ZavmJgheuV4QSMEcFv7PSwn3s0Q8a9ZxH FhxA== X-Gm-Message-State: AA+aEWbYgto0kbTOzgPo9M0CVBxvtNUYFEPcmVDN+npEnnZ1DfeRqMKv lXmMJIHkt/bX1ZPnX8wy8kpZQ9XQ X-Google-Smtp-Source: AFSGD/XtWzjspzZGmlsiWk9xlWkxIuLHgTXcHRQIXqMGDE9XPVbbnQ4I8c/72HITBTYNcjCKIkZpcA== X-Received: by 2002:a63:86c2:: with SMTP id x185mr707538pgd.230.1542694309363; Mon, 19 Nov 2018 22:11:49 -0800 (PST) Received: from google.com ([2620:0:100e:913:3fb0:1473:cdbf:42]) by smtp.gmail.com with ESMTPSA id v89sm10651306pfk.12.2018.11.19.22.11.48 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 19 Nov 2018 22:11:48 -0800 (PST) Date: Mon, 19 Nov 2018 22:11:47 -0800 From: Jonathan Nieder To: Junio C Hamano Cc: Ben Peart , git@vger.kernel.org, pclouds@gmail.com, Ben Peart , jonathantanmy@google.com Subject: [PATCH 1/5] eoie: default to not writing EOIE section Message-ID: <20181120061147.GB144753@google.com> References: <20180823154053.20212-1-benpeart@microsoft.com> <20181010155938.20996-1-peartben@gmail.com> <20181113003817.GA170017@google.com> <20181113003938.GC170017@google.com> <20181120060920.GA144753@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20181120060920.GA144753@google.com> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Since 3b1d9e04 (eoie: add End of Index Entry (EOIE) extension, 2018-10-10) Git defaults to writing the new EOIE section when writing out an index file. Usually that is a good thing because it improves threaded performance, but when a Git repository is shared with older versions of Git, it produces a confusing warning: $ git status ignoring EOIE extension HEAD detached at 371ed0defa nothing to commit, working tree clean Let's introduce the new index extension more gently. First we'll roll out the new version of Git that understands it, and then once sufficiently many users are using such a version, we can flip the default to writing it by default. Introduce a '[index] recordEndOfIndexEntries' configuration variable to allow interested users to benefit from this index extension early. Signed-off-by: Jonathan Nieder --- Rebased. No other change from v1. As Jonathan pointed out, it would be nice to have tests here. Ben, any advice for how I could write some in a followup change? E.g. does Derrick Stolee's tracing based testing trick apply here? Documentation/config/index.txt | 7 +++++++ read-cache.c | 11 ++++++++++- t/t1700-split-index.sh | 11 +++++++---- 3 files changed, 24 insertions(+), 5 deletions(-) diff --git a/Documentation/config/index.txt b/Documentation/config/index.txt index 4b94b6bedc..8e138aba7a 100644 --- a/Documentation/config/index.txt +++ b/Documentation/config/index.txt @@ -1,3 +1,10 @@ +index.recordEndOfIndexEntries:: + Specifies whether the index file should include an "End Of Index + Entry" section. This reduces index load time on multiprocessor + machines but produces a message "ignoring EOIE extension" when + reading the index using Git versions before 2.20. Defaults to + 'false'. + index.threads:: Specifies the number of threads to spawn when loading the index. This is meant to reduce index load time on multiprocessor machines. diff --git a/read-cache.c b/read-cache.c index 4ca81286c0..1e9c772603 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2689,6 +2689,15 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile rollback_lock_file(lockfile); } +static int record_eoie(void) +{ + int val; + + if (!git_config_get_bool("index.recordendofindexentries", &val)) + return val; + return 0; +} + /* * On success, `tempfile` is closed. If it is the temporary file * of a `struct lock_file`, we will therefore effectively perform @@ -2936,7 +2945,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, * read. Write it out regardless of the strip_extensions parameter as we need it * when loading the shared index. */ - if (offset) { + if (offset && record_eoie()) { struct strbuf sb = STRBUF_INIT; write_eoie_extension(&sb, &eoie_c, offset); diff --git a/t/t1700-split-index.sh b/t/t1700-split-index.sh index 2ac47aa0e4..0cbac64e28 100755 --- a/t/t1700-split-index.sh +++ b/t/t1700-split-index.sh @@ -25,14 +25,17 @@ test_expect_success 'enable split index' ' git update-index --split-index && test-tool dump-split-index .git/index >actual && indexversion=$(test-tool index-version <.git/index) && + + # NEEDSWORK: Stop hard-coding checksums. if test "$indexversion" = "4" then - own=3527df833c6c100d3d1d921a9a782d62a8be4b58 - base=746f7ab2ed44fb839efdfbffcf399d0b113fb4cb + own=432ef4b63f32193984f339431fd50ca796493569 + base=508851a7f0dfa8691e9f69c7f055865389012491 else - own=5e9b60117ece18da410ddecc8b8d43766a0e4204 - base=4370042739b31cd17a5c5cd6043a77c9a00df113 + own=8299b0bcd1ac364e5f1d7768efb62fa2da79a339 + base=39d890139ee5356c7ef572216cebcd27aa41f9df fi && + cat >expect <<-EOF && own $own base $base From patchwork Tue Nov 20 06:12:22 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Nieder X-Patchwork-Id: 10689907 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B4DD213BF for ; Tue, 20 Nov 2018 06:12:26 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 9EE842994C for ; Tue, 20 Nov 2018 06:12:26 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9388829965; Tue, 20 Nov 2018 06:12:26 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,FSL_HELO_FAKE,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 361812994C for ; Tue, 20 Nov 2018 06:12:26 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731188AbeKTQjs (ORCPT ); Tue, 20 Nov 2018 11:39:48 -0500 Received: from mail-pl1-f196.google.com ([209.85.214.196]:44435 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730902AbeKTQjs (ORCPT ); Tue, 20 Nov 2018 11:39:48 -0500 Received: by mail-pl1-f196.google.com with SMTP id s5-v6so457654plq.11 for ; Mon, 19 Nov 2018 22:12:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=AiXYsn7ACzT9CmqR9XDYyAlUWi1R2eSflKxOlx0UgeQ=; b=Q5solX1lufHghId7dYm92pj3vzwnYQ1d/vgIUnshtud9eTuKFc65TvFP6rJDzMDkSI 0nIQCrQxNxnAyYQHywsxr95jlnm9dKHYrFLlbqF1ActVL+NJcwIuvozvkvVpZti20Hg4 vnoYICsg38p8kD/Tjl/4TyuS1l9PWAl0ZAPuk4k/9ruNGT66WWDzS083bUz+C1e1WMX/ RSL4BMdplN7EpD11/CpAW9w4G/vYH7xYoO3lQJrmiWh7kAFsPm6tr65tcKMv+c0OTfkh oLwmWtUzLoXohmatEVcwu88CGWR3f9Ddhf6IsyC74K7J3LXRylH392H1XalUPI8LrfN5 5/3Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=AiXYsn7ACzT9CmqR9XDYyAlUWi1R2eSflKxOlx0UgeQ=; b=tDmsH+5FLakn95ArvPVytvSEoxEBhCemhp8GAd3jWSlO69FkDMHRxchhs73Ps3VK2M LNUlfeW8oYxB3hyiEeVH2a5oGIJ5lUktFsGHin37q+PPmd/Kfvtl/J3eDlZJd6Ngmqyc B1lUK5tkGKkzeSRqAYy69Jqnlo8MFlLexn/aIpiDdhlviI3OZjurSLBCXQAssG7IEkn+ 1dWyWurGt5Fuc7PgT8Hj6ppzaHA70Bf/VQtwCx1RLdDALWnEdCobOVN9L7OklRkLG21D PauISkXmk50GxsjYcCgE5hsHFwPk/iVNTwWKwDYWIWVm40bCh0CCzYYhrxU2nkGM9RO/ NtkQ== X-Gm-Message-State: AA+aEWYjw2IgfRhk2QjD8cRAAUyf+AZ+fO09GpU06HbXSR/Vk1TQxVYO j6Z9MpMjPG747l4o32WrriE= X-Google-Smtp-Source: AFSGD/WDccefAIajEUHmWo5Y5OJreRwFP9RecAGOJE2OmYXPZcP11OElMd6Wa8bzIkVPEpAwQM5d3g== X-Received: by 2002:a17:902:aa0a:: with SMTP id be10mr875888plb.266.1542694344122; Mon, 19 Nov 2018 22:12:24 -0800 (PST) Received: from google.com ([2620:0:100e:913:3fb0:1473:cdbf:42]) by smtp.gmail.com with ESMTPSA id x26sm21703410pfn.50.2018.11.19.22.12.23 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 19 Nov 2018 22:12:23 -0800 (PST) Date: Mon, 19 Nov 2018 22:12:22 -0800 From: Jonathan Nieder To: Junio C Hamano Cc: Ben Peart , git@vger.kernel.org, pclouds@gmail.com, Ben Peart , jonathantanmy@google.com Subject: [PATCH 2/5] ieot: default to not writing IEOT section Message-ID: <20181120061221.GC144753@google.com> References: <20180823154053.20212-1-benpeart@microsoft.com> <20181010155938.20996-1-peartben@gmail.com> <20181113003817.GA170017@google.com> <20181113003938.GC170017@google.com> <20181120060920.GA144753@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20181120060920.GA144753@google.com> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP As with EOIE, popular versions of Git do not support the new IEOT extension yet. When accessing a Git repository written by a more modern version of Git, they correctly ignore the unrecognized section, but in the process they loudly warn ignoring IEOT extension resulting in confusion for users. Introduce the index extension more gently by not writing it yet in this first version with support for it. Soon, once sufficiently many users are running a modern version of Git, we can flip the default so users benefit from this index extension by default. Introduce a '[index] recordOffsetTable' configuration variable to control whether the new index extension is written. Signed-off-by: Jonathan Nieder --- As with patch 1/5, no change from v1 other than rebasing. Documentation/config/index.txt | 7 +++++++ read-cache.c | 11 ++++++++++- 2 files changed, 17 insertions(+), 1 deletion(-) diff --git a/Documentation/config/index.txt b/Documentation/config/index.txt index 8e138aba7a..de44183235 100644 --- a/Documentation/config/index.txt +++ b/Documentation/config/index.txt @@ -5,6 +5,13 @@ index.recordEndOfIndexEntries:: reading the index using Git versions before 2.20. Defaults to 'false'. +index.recordOffsetTable:: + Specifies whether the index file should include an "Index Entry + Offset Table" section. This reduces index load time on + multiprocessor machines but produces a message "ignoring IEOT + extension" when reading the index using Git versions before 2.20. + Defaults to 'false'. + index.threads:: Specifies the number of threads to spawn when loading the index. This is meant to reduce index load time on multiprocessor machines. diff --git a/read-cache.c b/read-cache.c index 1e9c772603..f3d5638d9e 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2698,6 +2698,15 @@ static int record_eoie(void) return 0; } +static int record_ieot(void) +{ + int val; + + if (!git_config_get_bool("index.recordoffsettable", &val)) + return val; + return 0; +} + /* * On success, `tempfile` is closed. If it is the temporary file * of a `struct lock_file`, we will therefore effectively perform @@ -2761,7 +2770,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, else nr_threads = 1; - if (nr_threads != 1) { + if (nr_threads != 1 && record_ieot()) { int ieot_blocks, cpus; /* From patchwork Tue Nov 20 06:12:51 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Nieder X-Patchwork-Id: 10689909 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 87D5515A7 for ; Tue, 20 Nov 2018 06:12:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 76F0229965 for ; Tue, 20 Nov 2018 06:12:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 6B4A529989; Tue, 20 Nov 2018 06:12:56 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,FSL_HELO_FAKE,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 15CA529965 for ; Tue, 20 Nov 2018 06:12:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731267AbeKTQkS (ORCPT ); Tue, 20 Nov 2018 11:40:18 -0500 Received: from mail-pl1-f194.google.com ([209.85.214.194]:46696 "EHLO mail-pl1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730664AbeKTQkS (ORCPT ); Tue, 20 Nov 2018 11:40:18 -0500 Received: by mail-pl1-f194.google.com with SMTP id t13so451377ply.13 for ; Mon, 19 Nov 2018 22:12:54 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=Gnxos7238RK9JBXTMLYdE1/r2Mmy5C60jUNvaKJJuRs=; b=FeR+51R+RlXxHVZnuMl1kLoKvtJjUnfXoM/xXOxlicyqHdcawNjXpgVVC7ZTucvBD8 QtAxt26YNFF5oHYtWJ+DffOkjmgj/rLTeUQfCEnO4a+Seim35ixGbuSCyt1UKcKmO1qB oxdTnlGGkCeqCvVAzf5UrpOld2Ty+SdAhvVTB3Qyz3UyMMv8+XhBFi0jylUMqSQoj6AT c4fkwNLR3cG4txhVrpqlRE6O/q2tjaRjTf7DM4KiYfVOexnMs6rXo631hhSQapUZ86bR WNuU8IC90PpWTN8QYBDeGGWZKvWncwaBP33e2exAcUuHietpqqOiu5KJuTCmbXSCJ1mC jG9Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=Gnxos7238RK9JBXTMLYdE1/r2Mmy5C60jUNvaKJJuRs=; b=WHbdpjorNHadXk1u+G3a5017pxipJTZrMXawAjwFQsarfLpOAirwFFJntcyVaPEutA kifRfVPMCCvzgmyaO2OVLH56utI4z0m9s+H3BReAHKfPQni49NAros4g2SG1zbnMxHvV 0ja35IBMbdejOsHRKxKaFcq5bYRDJgjDMAWVp34Gr3u+o3kPij1swMw9cmYijXEHTe14 zSauIB5OrfgZ6RTPqZ4+L3+7fwDoUafoSE0/k6qrx3HDsazr0qPkiTpyqgsEKK2bN5Nf PJfcR167h5jV21jPGtCjw0nOxcaSuhswBoaaqmBBA2FwlkEKLuY2OkJKQozJlGp6p+LT II+A== X-Gm-Message-State: AA+aEWaknoq/f8cCvNf2K9lJUgEW+Vw10navEyhMmWN4EoBvgdKCTi0M gQkf7qEgIdXAZwjuwEwIhkE= X-Google-Smtp-Source: AFSGD/Vm5T9CxDM7dTmF3Naj887D9AcdV8PorRLy/KNtBvBv8/YfYq+6/cTs9a/2qxo8RZeVfOGh5A== X-Received: by 2002:a17:902:b60a:: with SMTP id b10mr812132pls.303.1542694374043; Mon, 19 Nov 2018 22:12:54 -0800 (PST) Received: from google.com ([2620:0:100e:913:3fb0:1473:cdbf:42]) by smtp.gmail.com with ESMTPSA id q1sm23928718pfb.96.2018.11.19.22.12.53 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 19 Nov 2018 22:12:53 -0800 (PST) Date: Mon, 19 Nov 2018 22:12:51 -0800 From: Jonathan Nieder To: Junio C Hamano Cc: Ben Peart , git@vger.kernel.org, pclouds@gmail.com, Ben Peart , jonathantanmy@google.com Subject: [PATCH 3/5] index: do not warn about unrecognized extensions Message-ID: <20181120061251.GD144753@google.com> References: <20180823154053.20212-1-benpeart@microsoft.com> <20181010155938.20996-1-peartben@gmail.com> <20181113003817.GA170017@google.com> <20181113003938.GC170017@google.com> <20181120060920.GA144753@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20181120060920.GA144753@google.com> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Documentation/technical/index-format explains: 4-byte extension signature. If the first byte is 'A'..'Z' the extension is optional and can be ignored. This allows gracefully introducing a new index extension without having to rely on all readers having support for it. Mandatory extensions start with a lowercase letter and optional ones start with a capital. Thus the versions of Git acting on a shared local repository do not have to upgrade in lockstep. We almost obey that convention, but there is a problem: when encountering an unrecognized optional extension, we write ignoring FNCY extension to stderr, which alarms users. This means that in practice we have had to introduce index extensions in two steps: first add read support, and then a while later, start writing by default. This delays when users can benefit from improvements to the index format. We cannot change the past, but for index extensions of the future, there is a straightforward improvement: silence that message except when tracing. This way, the message is still available when debugging, but in everyday use it does not show up so (once most Git users have this patch) we can turn on new optional extensions right away without alarming people. Signed-off-by: Jonathan Nieder --- Unchanged. read-cache.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/read-cache.c b/read-cache.c index f3d5638d9e..83d24357a6 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1726,7 +1726,7 @@ static int read_index_extension(struct index_state *istate, if (*ext < 'A' || 'Z' < *ext) return error("index uses %.4s extension, which we do not understand", ext); - fprintf(stderr, "ignoring %.4s extension\n", ext); + trace_printf("ignoring %.4s extension\n", ext); break; } return 0; From patchwork Tue Nov 20 06:14:26 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Nieder X-Patchwork-Id: 10689911 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BCFE615A7 for ; Tue, 20 Nov 2018 06:14:42 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A8BFC29F02 for ; Tue, 20 Nov 2018 06:14:42 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 9785829FA9; Tue, 20 Nov 2018 06:14:42 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.7 required=2.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FROM,FSL_HELO_FAKE,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id E697C29F02 for ; Tue, 20 Nov 2018 06:14:31 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731421AbeKTQly (ORCPT ); Tue, 20 Nov 2018 11:41:54 -0500 Received: from mail-pl1-f195.google.com ([209.85.214.195]:36142 "EHLO mail-pl1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730652AbeKTQly (ORCPT ); Tue, 20 Nov 2018 11:41:54 -0500 Received: by mail-pl1-f195.google.com with SMTP id y6-v6so477961plt.3 for ; Mon, 19 Nov 2018 22:14:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=czKKctsjZPaanumPTLI0mweRRcFUGrjfM19q4DkYANk=; b=UOQPeRXDNuIDJpPolIN1U4Nmj6Xm1qAvmcmno+KJkjRhNYrXCTyX5zG00pd+jzxw2+ j93h9Q65We9KzX8uEfazlIrR5ZEUPiZjY1Rd9ySfTGenTSpTVVo5qO3Cj3HM+o9MbnoH KrpDTsMn3lzEp1uf7Ily/j4fqDm60K+bc0htiSN8xJCwXydGXxWGg4/Hzx52HjOtXIp2 otbRD8fUCHqitWpTSNiu36cuGdHlAEE36omRYSBWJnINW4OafSSEwoAeZwkfamTmjkXz sjbs2eUSegdzt95qyPZBiy3X62O2nVOK7cpmOhJIOxq2n12N3ryfW2LuO9A+/fuCUmJ3 j3TA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=czKKctsjZPaanumPTLI0mweRRcFUGrjfM19q4DkYANk=; b=hQPqfps3wUAF5f2864wcZji4djPJOPzjykfvK6kpYZdY17q1ytkroYl5oP76vHucQ8 lUBMvaXoUmmYj4geFvSz0vaqbjrnUkMS6LpD5lnNMqJbAHUzLFBnhtycalr8g2DdomUc NJrr7B2noSNSqkYiSzymoEEsVH6Bql/11HMknqlReKjaduTzbdhLeV60DHTdVm6adrJc 3aI8haM/A7Op2h7BIypgs1IzdgCyKcoAdVkwZduh3A/x5ZPD0IO0uDQ6nqa17bQDI2Wy Mr7Qd30ac4nKeT4dWCS97KWSB6wbbX87UF5Q/8wABsM882yWM/lj1EJ9j0uuOlpGXXkz p9yg== X-Gm-Message-State: AA+aEWaK3iGwacDJ9o1JC7h1oG+52l3siGIKaHxVD4I4l5+RLRD2GIKe pzoIgSwCez2OZxJ2xpEvaAE= X-Google-Smtp-Source: AFSGD/WDDTT1cNA0oo5qBCFcq9To+o+inUKG9Wum9Y1268RZpXIFfr2Bcqvi4rtN180rYI4Lt3wvGQ== X-Received: by 2002:a17:902:1a8:: with SMTP id b37-v6mr883474plb.314.1542694469658; Mon, 19 Nov 2018 22:14:29 -0800 (PST) Received: from google.com ([2620:0:100e:913:3fb0:1473:cdbf:42]) by smtp.gmail.com with ESMTPSA id t90sm28000472pfj.23.2018.11.19.22.14.28 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 19 Nov 2018 22:14:29 -0800 (PST) Date: Mon, 19 Nov 2018 22:14:26 -0800 From: Jonathan Nieder To: Junio C Hamano Cc: Ben Peart , git@vger.kernel.org, pclouds@gmail.com, Ben Peart , jonathantanmy@google.com Subject: [PATCH 4/5] index: make index.threads=true enable ieot and eoie Message-ID: <20181120061426.GE144753@google.com> References: <20180823154053.20212-1-benpeart@microsoft.com> <20181010155938.20996-1-peartben@gmail.com> <20181113003817.GA170017@google.com> <20181113003938.GC170017@google.com> <20181120060920.GA144753@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20181120060920.GA144753@google.com> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP If a user explicitly sets [index] threads = true to read the index using multiple threads, ensure that index writes include the offset table by default to make that possible. This ensures that the user's intent of turning on threading is respected. In other words, permit the following configurations: - index.threads and index.recordOffsetTable unspecified: do not write the offset table yet (to avoid alarming the user with "ignoring IEOT extension" messages when an older version of Git accesses the repository) but do make use of multiple threads to read the index if the supporting offset table is present. This can also be requested explicitly by setting index.threads=true, 0, or >1 and index.recordOffsetTable=false. - index.threads=false or 1: do not write the offset table, and do not make use of the offset table. One can set index.recordOffsetTable=false as well, to be more explicit. - index.threads=true, 0, or >1 and index.recordOffsetTable unspecified: write the offset table and make use of threads at read time. This can also be requested by setting index.threads=true, 0, >1, or unspecified and index.recordOffsetTable=true. Fortunately the complication is temporary: once most Git installations have upgraded to a version with support for the IEOT and EOIE extensions, we can flip the defaults for index.recordEndOfIndexEntries and index.recordOffsetTable to true and eliminate the settings. Helped-by: Ben Peart Signed-off-by: Jonathan Nieder --- New, based on Ben Peart's feedback. Turned out simpler than I feared --- thanks, Ben, for pushing for this. Documentation/config/index.txt | 6 ++++-- config.c | 17 ++++++++++------- config.h | 2 +- read-cache.c | 23 +++++++++++++++++------ 4 files changed, 32 insertions(+), 16 deletions(-) diff --git a/Documentation/config/index.txt b/Documentation/config/index.txt index de44183235..f181503041 100644 --- a/Documentation/config/index.txt +++ b/Documentation/config/index.txt @@ -3,14 +3,16 @@ index.recordEndOfIndexEntries:: Entry" section. This reduces index load time on multiprocessor machines but produces a message "ignoring EOIE extension" when reading the index using Git versions before 2.20. Defaults to - 'false'. + 'true' if index.threads has been explicitly enabled, 'false' + otherwise. index.recordOffsetTable:: Specifies whether the index file should include an "Index Entry Offset Table" section. This reduces index load time on multiprocessor machines but produces a message "ignoring IEOT extension" when reading the index using Git versions before 2.20. - Defaults to 'false'. + Defaults to 'true' if index.threads has been explicitly enabled, + 'false' otherwise. index.threads:: Specifies the number of threads to spawn when loading the index. diff --git a/config.c b/config.c index 04286f7717..ff521eb27a 100644 --- a/config.c +++ b/config.c @@ -2294,22 +2294,25 @@ int git_config_get_fsmonitor(void) return 0; } -int git_config_get_index_threads(void) +int git_config_get_index_threads(int *dest) { - int is_bool, val = 0; + int is_bool, val; val = git_env_ulong("GIT_TEST_INDEX_THREADS", 0); - if (val) - return val; + if (val) { + *dest = val; + return 0; + } if (!git_config_get_bool_or_int("index.threads", &is_bool, &val)) { if (is_bool) - return val ? 0 : 1; + *dest = val ? 0 : 1; else - return val; + *dest = val; + return 0; } - return 0; /* auto */ + return 1; } NORETURN diff --git a/config.h b/config.h index a06027e69b..ee5d3fa7b4 100644 --- a/config.h +++ b/config.h @@ -246,11 +246,11 @@ extern int git_config_get_bool(const char *key, int *dest); extern int git_config_get_bool_or_int(const char *key, int *is_bool, int *dest); extern int git_config_get_maybe_bool(const char *key, int *dest); extern int git_config_get_pathname(const char *key, const char **dest); +extern int git_config_get_index_threads(int *dest); extern int git_config_get_untracked_cache(void); extern int git_config_get_split_index(void); extern int git_config_get_max_percent_split_change(void); extern int git_config_get_fsmonitor(void); -extern int git_config_get_index_threads(void); /* This dies if the configured or default date is in the future */ extern int git_config_get_expiry(const char *key, const char **output); diff --git a/read-cache.c b/read-cache.c index 83d24357a6..002ed2c1e4 100644 --- a/read-cache.c +++ b/read-cache.c @@ -2176,7 +2176,8 @@ int do_read_index(struct index_state *istate, const char *path, int must_exist) src_offset = sizeof(*hdr); - nr_threads = git_config_get_index_threads(); + if (git_config_get_index_threads(&nr_threads)) + nr_threads = 1; /* TODO: does creating more threads than cores help? */ if (!nr_threads) { @@ -2695,7 +2696,13 @@ static int record_eoie(void) if (!git_config_get_bool("index.recordendofindexentries", &val)) return val; - return 0; + + /* + * As a convenience, the end of index entries extension + * used for threading is written by default if the user + * explicitly requested threaded index reads. + */ + return !git_config_get_index_threads(&val) && val != 1; } static int record_ieot(void) @@ -2704,7 +2711,13 @@ static int record_ieot(void) if (!git_config_get_bool("index.recordoffsettable", &val)) return val; - return 0; + + /* + * As a convenience, the offset table used for threading is + * written by default if the user explicitly requested + * threaded index reads. + */ + return !git_config_get_index_threads(&val) && val != 1; } /* @@ -2765,9 +2778,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile, if (ce_write(&c, newfd, &hdr, sizeof(hdr)) < 0) return -1; - if (HAVE_THREADS) - nr_threads = git_config_get_index_threads(); - else + if (!HAVE_THREADS || git_config_get_index_threads(&nr_threads)) nr_threads = 1; if (nr_threads != 1 && record_ieot()) { From patchwork Tue Nov 20 06:15:44 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Nieder X-Patchwork-Id: 10689913 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3CFC913BB for ; Tue, 20 Nov 2018 06:15:50 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 262E829B8D for ; Tue, 20 Nov 2018 06:15:50 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 13A8829F9C; Tue, 20 Nov 2018 06:15:50 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FROM,FSL_HELO_FAKE,MAILING_LIST_MULTI, RCVD_IN_DNSWL_HI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8B30229B8D for ; Tue, 20 Nov 2018 06:15:49 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1731494AbeKTQnM (ORCPT ); Tue, 20 Nov 2018 11:43:12 -0500 Received: from mail-pg1-f196.google.com ([209.85.215.196]:36485 "EHLO mail-pg1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1730652AbeKTQnM (ORCPT ); Tue, 20 Nov 2018 11:43:12 -0500 Received: by mail-pg1-f196.google.com with SMTP id n2so436525pgm.3 for ; Mon, 19 Nov 2018 22:15:47 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=0cHq+ouaS9kEFB2KDSShfYT6IcBItf7xgS/gEdwjJ6w=; b=gUEmQIZpOeedrpmAJo+96QVzmX/t0pm1fesFfU7m5QP6e5HMuVGqbFUdbiI2vLv6fF R2kLPvXmzgd5epSdd1XZyD/Y4uI329QJmpTdIVeKIWdLhvlKrfF5Fj2+UTsY4hBVnu5u Aoya4pADD83teJk3YnfBuDrEDlRRs17Fi8N8CzgR1qS6WzaXm62Ejz734dGv8PP6CeNV NsYXIW8iEMslRoXdclqr0S3eP5XRVYPxvOmsp/D52qSMnhQiHQXld3tCmYaVebH3x27g DrYABsJN8Q4336QJ0Z3r8b3UfVw5fAJKkhJ4oI9bLbO0FuGN5gzqreddW2GCiTR3kd7P JrAA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=0cHq+ouaS9kEFB2KDSShfYT6IcBItf7xgS/gEdwjJ6w=; b=jBR61Ry//ZroS24ALfKBKRHQFl5Jr7NB2mx/Lr3qFx26f+ck+9I5GikIzkHdsYqC1u oDaecFV07avzpN6Y3u8AFBlVfutTkwUaBr9ERdLHrVWF++dHDcJYqZMpNrO18umy/Oqv EJZlTeKdABjnDjWLvfPKf3cqDK2k3X2f3AFAkc2J9P31X5hNTsgBhiGya+yH8WeLmqGt gC+zZ05b95SS6J6YKC0EnHlu6XECaTeaSF/ZYwSzLIZeXeY7BNKWDZnTRUd8XW1S6q53 Q52ug1PqorkfWhxJWqur9yyFNbfHSXEMy+0v6iLLE+SDcu9p1walFqLX9dCz2NZNfJm0 WeyA== X-Gm-Message-State: AA+aEWbbNNdKxX14ns6vuezrLOoIKCJhgexYrnOLeJqreHHShaVN3/gV 6+Fng0bTnmwMzKpqkm5y/iY= X-Google-Smtp-Source: AFSGD/W/iskvnSDZSUmnteRfYSsKzpQkIdSAy650p9Q41N/qjIRcTtiyTLzkjyQ2OeN26lPpkgu7Ug== X-Received: by 2002:a63:62c3:: with SMTP id w186mr742856pgb.345.1542694547093; Mon, 19 Nov 2018 22:15:47 -0800 (PST) Received: from google.com ([2620:0:100e:913:3fb0:1473:cdbf:42]) by smtp.gmail.com with ESMTPSA id l187-v6sm48180317pfc.79.2018.11.19.22.15.46 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Mon, 19 Nov 2018 22:15:46 -0800 (PST) Date: Mon, 19 Nov 2018 22:15:44 -0800 From: Jonathan Nieder To: Junio C Hamano Cc: Ben Peart , git@vger.kernel.org, pclouds@gmail.com, Ben Peart , jonathantanmy@google.com Subject: [PATCH 5/5] index: offer advice for unknown index extensions Message-ID: <20181120061544.GF144753@google.com> References: <20180823154053.20212-1-benpeart@microsoft.com> <20181010155938.20996-1-peartben@gmail.com> <20181113003817.GA170017@google.com> <20181113003938.GC170017@google.com> <20181120060920.GA144753@google.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20181120060920.GA144753@google.com> User-Agent: Mutt/1.10.1 (2018-07-13) Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP It is not unusual for multiple distinct versions of Git to act on a single repository. For example, some IDEs bundle a particular version of Git, which can be a different version from the system copy of Git, or on a Mac, /usr/bin/git quickly goes out of sync with the Homebrew git in /usr/local/bin/git. When a newer version of Git writes an index file that an older version of Git does not know how to make full use of, this is a teaching opportunity. The user may not be aware of what version of Git they are using. Print an advice message to help the user to use the most full featured version of Git (e.g. by futzing with their PATH). warning: ignoring optional IEOT index extension hint: This is likely due to the file having been written by a newer hint: version of Git than is reading it. You can upgrade Git to hint: take advantage of performance improvements from the updated hint: file format. hint: hint: You can run "git config advice.unknownIndexExtension false" hint: to suppress this message. This replaces the message ignoring IEOT extension that existed previously and did not provide enough detail for a user to act on it or suppress it. Helped-by: Junio C Hamano Signed-off-by: Jonathan Nieder --- New, based on Junio's hints about the message removed in patch 3/5. That's the end of the series. Thanks for reading, and thanks again for your help so far. advice.c | 2 ++ advice.h | 1 + read-cache.c | 11 +++++++++++ 3 files changed, 14 insertions(+) diff --git a/advice.c b/advice.c index 5f35656409..91a55046fd 100644 --- a/advice.c +++ b/advice.c @@ -24,6 +24,7 @@ int advice_add_embedded_repo = 1; int advice_ignored_hook = 1; int advice_waiting_for_editor = 1; int advice_graft_file_deprecated = 1; +int advice_unknown_index_extension = 1; int advice_checkout_ambiguous_remote_branch_name = 1; static int advice_use_color = -1; @@ -78,6 +79,7 @@ static struct { { "ignoredHook", &advice_ignored_hook }, { "waitingForEditor", &advice_waiting_for_editor }, { "graftFileDeprecated", &advice_graft_file_deprecated }, + { "unknownIndexExtension", &advice_unknown_index_extension }, { "checkoutAmbiguousRemoteBranchName", &advice_checkout_ambiguous_remote_branch_name }, /* make this an alias for backward compatibility */ diff --git a/advice.h b/advice.h index 696bf0e7d2..8da0845cfc 100644 --- a/advice.h +++ b/advice.h @@ -24,6 +24,7 @@ extern int advice_add_embedded_repo; extern int advice_ignored_hook; extern int advice_waiting_for_editor; extern int advice_graft_file_deprecated; +extern int advice_unknown_index_extension; extern int advice_checkout_ambiguous_remote_branch_name; int git_default_advice_config(const char *var, const char *value); diff --git a/read-cache.c b/read-cache.c index 002ed2c1e4..d1d903e5a1 100644 --- a/read-cache.c +++ b/read-cache.c @@ -1727,6 +1727,17 @@ static int read_index_extension(struct index_state *istate, return error("index uses %.4s extension, which we do not understand", ext); trace_printf("ignoring %.4s extension\n", ext); + if (advice_unknown_index_extension) { + warning(_("ignoring optional %.4s index extension"), ext); + advise(_("This is likely due to the file having been written by a newer\n" + "version of Git than is reading it. You can upgrade Git to\n" + "take advantage of performance improvements from the updated\n" + "file format.\n" + "\n" + "Run \"%s\"\n" + "to suppress this message."), + "git config advice.unknownIndexExtension false"); + } break; } return 0;