From patchwork Tue Jun 29 02:13:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348853 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C239AC11F64 for ; Tue, 29 Jun 2021 02:13:11 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9222F61CD4 for ; Tue, 29 Jun 2021 02:13:11 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231194AbhF2CPg (ORCPT ); Mon, 28 Jun 2021 22:15:36 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51734 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229910AbhF2CPg (ORCPT ); Mon, 28 Jun 2021 22:15:36 -0400 Received: from mail-wm1-x32f.google.com (mail-wm1-x32f.google.com [IPv6:2a00:1450:4864:20::32f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id CB7ACC061760 for ; Mon, 28 Jun 2021 19:13:09 -0700 (PDT) Received: by mail-wm1-x32f.google.com with SMTP id n33so1136446wms.2 for ; Mon, 28 Jun 2021 19:13:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=f/KiyvI/Q1CXlsOXJtPYGGM2ZxsZJqz3uYZNz5JOQt8=; b=TNO7ORVqcPaRlD48f7df0RlmilDjwYykGVNJ96b+BgzEm6nDk4VFO1npAVx1J91g7V q+u4yqR/iEoj6yvWVMBK7vvPFTrkgrzg8Z/qKn94fTuv4ZgnqSuJsaTvgoGdiGAZMKmP Ld5RLHWfmkgx8SPvtJUG+rx99w3Z3DCw0j7jsuWltWZ5mCIONFE/5XfivuuCJidsuYZP th2VadblkreHAmt+I/manmeEcR73bNTN0r+OcZWY14uS3taUZq5jQ+xauseVy5lBDgNC 3OYgNqZq0/a0+q8Hx8qeIopxWnwFIuS/Vn9oM52UFTcERzdRb/XZOp64duTu4zuVoAsG PGtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=f/KiyvI/Q1CXlsOXJtPYGGM2ZxsZJqz3uYZNz5JOQt8=; b=HadQ+m8ugQCYyu6E/OH4ZjF9cdb1o+quiq2bVUjBH9XtG14PJyXNJI44W10rQ04snX XamM8G3eVKrkqpOPiasCg+dhv2a0v9n5CzPUnNa6S18WlwJ0ZzkOvWx8t9XvME61i7ZI M/ORCnlDEcZAvQAH0sqwaFGB04Ds13iX1uhqixCfr7+zZ4EDEz+odeFglJ4i1IuR8vA+ zn5CfOngo+sUf86TxXbNcUpq3in4CFp7xG4jiosQtvNPfFSrtAMl9KDetE2Sha+JTLSm eZD9WfX1gHWCFfhugRnFnqiHGcfaoTw9pLP8+ZyKzq2e05ZtxKX17Imxk2lKhacSPywK Na8Q== X-Gm-Message-State: AOAM531xBfTjcnkJLzlIEyE8wQIdSUko5mDB+LoqK9nH9ULox2pIIrLh y/d8gdCx952sHmymGRFQQHkatqZIJDA= X-Google-Smtp-Source: ABdhPJxSzPv8tnVSQHTkVzjbOQEiBmGs8UX/dk6tbAkqflNzDq4aUGiahnok+4erenJls7KlGX63iA== X-Received: by 2002:a1c:988a:: with SMTP id a132mr11824102wme.175.1624932788470; Mon, 28 Jun 2021 19:13:08 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id k6sm14724155wms.8.2021.06.28.19.13.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:13:08 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:13:02 +0000 Subject: [PATCH 1/5] p2000: add 'git checkout -' test and decrease depth Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee As we increase our list of commands to test in p2000-sparse-operations.sh, we will want to have a slightly smaller test repository. Reduce the size by a factor of four by reducing the depth of the step that creates a big index around a moderately-sized repository. Also add a step to run 'git checkout -' on repeat. This requires having a previous location in the reflog, so add that to the initialization steps. Signed-off-by: Derrick Stolee --- t/perf/p2000-sparse-operations.sh | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh index 94513c97748..f7f8c012103 100755 --- a/t/perf/p2000-sparse-operations.sh +++ b/t/perf/p2000-sparse-operations.sh @@ -6,7 +6,7 @@ test_description="test performance of Git operations using the index" test_perf_default_repo -SPARSE_CONE=f2/f4/f1 +SPARSE_CONE=f2/f4 test_expect_success 'setup repo and indexes' ' git reset --hard HEAD && @@ -27,7 +27,7 @@ test_expect_success 'setup repo and indexes' ' OLD_COMMIT=$(git rev-parse HEAD) && OLD_TREE=$(git rev-parse HEAD^{tree}) && - for i in $(test_seq 1 4) + for i in $(test_seq 1 3) do cat >in <<-EOF && 100755 blob $BLOB a @@ -43,14 +43,23 @@ test_expect_success 'setup repo and indexes' ' done && git sparse-checkout init --cone && - git branch -f wide $OLD_COMMIT && + git sparse-checkout set $SPARSE_CONE && + git checkout -b wide $OLD_COMMIT && + + for l2 in f1 f2 f3 f4 + do + echo more bogus >>$SPARSE_CONE/$l2/a && + git commit -a -m "edit $SPARSE_CONE/$l2/a" || return 1 + done && + git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . full-index-v3 && ( cd full-index-v3 && git sparse-checkout init --cone && git sparse-checkout set $SPARSE_CONE && git config index.version 3 && - git update-index --index-version=3 + git update-index --index-version=3 && + git checkout HEAD~4 ) && git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . full-index-v4 && ( @@ -58,7 +67,8 @@ test_expect_success 'setup repo and indexes' ' git sparse-checkout init --cone && git sparse-checkout set $SPARSE_CONE && git config index.version 4 && - git update-index --index-version=4 + git update-index --index-version=4 && + git checkout HEAD~4 ) && git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . sparse-index-v3 && ( @@ -66,7 +76,8 @@ test_expect_success 'setup repo and indexes' ' git sparse-checkout init --cone --sparse-index && git sparse-checkout set $SPARSE_CONE && git config index.version 3 && - git update-index --index-version=3 + git update-index --index-version=3 && + git checkout HEAD~4 ) && git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . sparse-index-v4 && ( @@ -74,7 +85,8 @@ test_expect_success 'setup repo and indexes' ' git sparse-checkout init --cone --sparse-index && git sparse-checkout set $SPARSE_CONE && git config index.version 4 && - git update-index --index-version=4 + git update-index --index-version=4 && + git checkout HEAD~4 ) ' @@ -97,5 +109,6 @@ test_perf_on_all git status test_perf_on_all git add -A test_perf_on_all git add . test_perf_on_all git commit -a -m A +test_perf_on_all git checkout -f - test_done From patchwork Tue Jun 29 02:13:03 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348857 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC0D3C11F67 for ; Tue, 29 Jun 2021 02:13:12 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id C04EE61CED for ; Tue, 29 Jun 2021 02:13:12 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231347AbhF2CPi (ORCPT ); Mon, 28 Jun 2021 22:15:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51736 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231264AbhF2CPh (ORCPT ); Mon, 28 Jun 2021 22:15:37 -0400 Received: from mail-wr1-x42e.google.com (mail-wr1-x42e.google.com [IPv6:2a00:1450:4864:20::42e]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 4F68DC061766 for ; Mon, 28 Jun 2021 19:13:10 -0700 (PDT) Received: by mail-wr1-x42e.google.com with SMTP id i8so1482633wrc.0 for ; Mon, 28 Jun 2021 19:13:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=CjCwvYNfaftKNON/5Qnsv7Bb6epxLgdnNa3qG/hBIJI=; b=R6F5u1fd9r3KA4+G5u81vc9FCOzJWG8Mag1ZoDVopvhUNoH+xqHyXALUgqqpZDUDye Eq6i7s9ze5aXxiYzALk3uU1ZnS+ip7/QKRGsQHfl6HcHDcwk6q7+umu5QVm6/Haa6G6K sYb1KuKf7OgLdJnuAv8+W6YDgFAACvjmNPC6LrFUqB+yWQwSODrWfJmMl9cXgUygD4eO CIcJsGjDDa6l/6NuaX/0bqy800QHQEYq67VgV94XUn+ORLatWYxAeeVUMQRyCVeiIlLM aSL5PCM8IGFTeVyOknhywufEQzTm8b6ffdR07eeLlI0M6oDn/fAC8vWxIlvBVO4JN4FV +3Sw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=CjCwvYNfaftKNON/5Qnsv7Bb6epxLgdnNa3qG/hBIJI=; b=MPun1iPlEXVlscnywkKX1yrcpVHa47sx63SA2A5cOBtD83f2GuFIYxX0QsQDiZhtXV 3gIlpmW0Qh+sGS8p5nI5ghaAndxFYVPnKYU8tkVQaaeJjNJhN3n7VSiyX3/FroWTvhXJ 0E64VaroLrlfETDafOvt6xgHXs/t/mLgXD8e6/krggM9qLlDFiwjmY8CxrKbNDG8pnJv 0HPuRYVCtUppkzllxNmSO+q5pSOgBcjhbDJEuDCRw+HAtn0RTR+UA2oTwuQ4aQFh5exb 7xghhd8ZeB1TiYphtggCItKxr3heSZMRVCHU0pcwQWZoj4zqivY9TSwZEy0vcNTI5tAc FL/Q== X-Gm-Message-State: AOAM532rfqJdDiAjKEROxtYDdOFQO/VkGFnj0RknNhgEBE3fI6Qt/7uZ pEipFCtnhpPGA7csXXOmPFg+iazN5bU= X-Google-Smtp-Source: ABdhPJxt1NFjM2T575t/gBUrLsHajl3JxzHcOHySXbIuWP/N21dPvU6ZI3U49MK5cV8IgSKENCULxA== X-Received: by 2002:adf:ea07:: with SMTP id q7mr30498352wrm.153.1624932788967; Mon, 28 Jun 2021 19:13:08 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id h206sm1097268wmh.33.2021.06.28.19.13.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:13:08 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:13:03 +0000 Subject: [PATCH 2/5] p2000: compress repo names Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee By using shorter names for the test repos, we will get a slightly more compressed performance summary without comprimising clarity. Signed-off-by: Derrick Stolee --- t/perf/p2000-sparse-operations.sh | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/t/perf/p2000-sparse-operations.sh b/t/perf/p2000-sparse-operations.sh index f7f8c012103..597626276fb 100755 --- a/t/perf/p2000-sparse-operations.sh +++ b/t/perf/p2000-sparse-operations.sh @@ -52,36 +52,36 @@ test_expect_success 'setup repo and indexes' ' git commit -a -m "edit $SPARSE_CONE/$l2/a" || return 1 done && - git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . full-index-v3 && + git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . full-v3 && ( - cd full-index-v3 && + cd full-v3 && git sparse-checkout init --cone && git sparse-checkout set $SPARSE_CONE && git config index.version 3 && git update-index --index-version=3 && git checkout HEAD~4 ) && - git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . full-index-v4 && + git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . full-v4 && ( - cd full-index-v4 && + cd full-v4 && git sparse-checkout init --cone && git sparse-checkout set $SPARSE_CONE && git config index.version 4 && git update-index --index-version=4 && git checkout HEAD~4 ) && - git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . sparse-index-v3 && + git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . sparse-v3 && ( - cd sparse-index-v3 && + cd sparse-v3 && git sparse-checkout init --cone --sparse-index && git sparse-checkout set $SPARSE_CONE && git config index.version 3 && git update-index --index-version=3 && git checkout HEAD~4 ) && - git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . sparse-index-v4 && + git -c core.sparseCheckoutCone=true clone --branch=wide --sparse . sparse-v4 && ( - cd sparse-index-v4 && + cd sparse-v4 && git sparse-checkout init --cone --sparse-index && git sparse-checkout set $SPARSE_CONE && git config index.version 4 && @@ -92,8 +92,8 @@ test_expect_success 'setup repo and indexes' ' test_perf_on_all () { command="$@" - for repo in full-index-v3 full-index-v4 \ - sparse-index-v3 sparse-index-v4 + for repo in full-v3 full-v4 \ + sparse-v3 sparse-v4 do test_perf "$command ($repo)" " ( From patchwork Tue Jun 29 02:13:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348859 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8F2D4C11F65 for ; Tue, 29 Jun 2021 02:13:13 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 72BDD61CEB for ; Tue, 29 Jun 2021 02:13:13 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231350AbhF2CPi (ORCPT ); Mon, 28 Jun 2021 22:15:38 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51742 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231304AbhF2CPi (ORCPT ); Mon, 28 Jun 2021 22:15:38 -0400 Received: from mail-wr1-x42d.google.com (mail-wr1-x42d.google.com [IPv6:2a00:1450:4864:20::42d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DB02DC061574 for ; Mon, 28 Jun 2021 19:13:10 -0700 (PDT) Received: by mail-wr1-x42d.google.com with SMTP id y3so4308660wrq.3 for ; Mon, 28 Jun 2021 19:13:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=JKN67tv2Als0pC/pVRxM3lKN49x/Ibl77IIvVabOAvA=; b=k+alzN/vu2leZSbjbJKS5lAocR5F6EXmcZFOpQNg6qkLs4/dVw5mGWcmJNFf7oXc6Q iwGAgXosW5nUWyYLOAd/SD2OhiF4oFC+2YnFsyqMLMvdmQ+9pUPr62gIC4QHbs3V7qqx 19bMcOBB60mSKLBmcIqRJXZwojHC7f/Za5cJ7JlY7EuzbCrFurciZ72y1OFBSu/orduV CKoKtRpp/SAjhYBM0nN0H7gDNgzI/Ps/6/NlVCRdfn5I0ocu2w2vaJlRzkj4o0Nv+Tf8 8TYCEcgngppM1wAKJMV5mcULe2/3BQSsn+hOEM6FEDdwV8JYHqeShNrIFwA9ZecfMMAm k63g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=JKN67tv2Als0pC/pVRxM3lKN49x/Ibl77IIvVabOAvA=; b=IpKXxgz9IgLX0addM1lw0MIXS6+3lvBRhvgcfnLbXtrh0AOFBjkD7/NiUYAWtcr/0k rxCMtdbpjf9IMJfVJdUngNXU3VCn5hij6wZWcSzD7iULYgpM2SI2Fb8ieHinHUh8iKNY nnN+Y5csiWw6iA+dDVoTn6D3t6S7eo02y3Rvp/+cwZ7GTybX5EGL8NZC3PIHAO+eX5z2 IzBtbVg1ZkSz4gFD49LjWMOT5yP0Yo6lhdpsIHKlrcraEA3qLF2dhV8lL3XV5HG0vmqh iuEjyiBUcJ1k93A49yY9+Q42+MOjiDKvvJYQ5Fm9mC24RvETzIBH4dORA9K3cGDlvn4i WnYA== X-Gm-Message-State: AOAM532wEGmeZqtGDwQimEdZ7r1mNeJPG1kDKa5A7vY7RT7SaPFNt9xT fpQh2d0QJ7JF8giOisRRvJgsX/+hiw0= X-Google-Smtp-Source: ABdhPJyh1phAIjHOo4jR4Of39oIifCgeIizUUwjP2zt2z+c3/bx2u1InGFLj62ucUEWmAp3Cy7cg8A== X-Received: by 2002:a05:6000:120e:: with SMTP id e14mr10795080wrx.139.1624932789520; Mon, 28 Jun 2021 19:13:09 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id l10sm16652243wrv.82.2021.06.28.19.13.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:13:09 -0700 (PDT) Message-Id: <413babe6e774eb4e09b091532219980e1cca94cb.1624932786.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 02:13:04 +0000 Subject: [PATCH 3/5] commit: integrate with sparse-index Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Update 'git commit' to allow using the sparse-index in memory without expanding to a full one. The only place that had an ensure_full_index() call was in cache_tree_update(). The recursive algorithm for update_one() was already updated in 2de37c536 (cache-tree: integrate with sparse directory entries, 2021-03-03) to handle sparse directory entries in the index. Most of this change involves testing different command-line options that allow specifying which on-disk changes should be included in the commit. This includes no options (only take currently-staged changes), -a (take all tracked changes), and --include (take a list of specific changes). To simplify testing that these options do not expand the index, update the test that previously verified that 'git status' does not expand the index with a helper method, ensure_not_expanded(). This allows 'git commit' to operate much faster when the sparse-checkout cone is much smaller than the full list of files at HEAD. Here are the relevant lines from p2000-sparse-operations.sh: Test HEAD~1 HEAD ---------------------------------------------------------------------------------- 2000.14: git commit -a -m A (full-v3) 0.35(0.26+0.06) 0.36(0.28+0.07) +2.9% 2000.15: git commit -a -m A (full-v4) 0.32(0.26+0.05) 0.34(0.28+0.06) +6.3% 2000.16: git commit -a -m A (sparse-v3) 0.63(0.59+0.06) 0.04(0.05+0.05) -93.7% 2000.17: git commit -a -m A (sparse-v4) 0.64(0.59+0.08) 0.04(0.04+0.04) -93.8% It is important to compare the full-index case to the sparse-index case, so the improvement for index version v4 is actually an 88% improvement in this synthetic example. In a real repository with over two million files at HEAD and 60,000 files in the sparse-checkout definition, the time for 'git commit -a' went from 2.61 seconds to 134ms. I compared this to the result if the index only contained the paths in the sparse-checkout definition and found the theoretical optimum to be 120ms, so the out-of-cone paths only add a 12% overhead. Signed-off-by: Derrick Stolee --- builtin/commit.c | 3 ++ cache-tree.c | 2 - t/t1092-sparse-checkout-compatibility.sh | 47 ++++++++++++++++++++++-- 3 files changed, 46 insertions(+), 6 deletions(-) diff --git a/builtin/commit.c b/builtin/commit.c index 12f51db158a..0bc64892505 100644 --- a/builtin/commit.c +++ b/builtin/commit.c @@ -1682,6 +1682,9 @@ int cmd_commit(int argc, const char **argv, const char *prefix) if (argc == 2 && !strcmp(argv[1], "-h")) usage_with_options(builtin_commit_usage, builtin_commit_options); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + status_init_config(&s, git_commit_config); s.commit_template = 1; status_format = STATUS_FORMAT_NONE; /* Ignore status.short */ diff --git a/cache-tree.c b/cache-tree.c index 45e58666afc..577b18d8811 100644 --- a/cache-tree.c +++ b/cache-tree.c @@ -461,8 +461,6 @@ int cache_tree_update(struct index_state *istate, int flags) if (i) return i; - ensure_full_index(istate); - if (!istate->cache_tree) istate->cache_tree = cache_tree(); diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index df217a2d10b..11c6dc8ec1f 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -275,6 +275,34 @@ test_expect_success 'add, commit, checkout' ' test_all_match git checkout - ' +test_expect_success 'commit including unstaged changes' ' + init_repos && + + write_script edit-file <<-\EOF && + echo $1 >$2 + EOF + + run_on_all ../edit-file 1 a && + run_on_all ../edit-file 1 deep/a && + + test_all_match git commit -m "-a" -a && + test_all_match git status --porcelain=v2 && + + run_on_all ../edit-file 2 a && + run_on_all ../edit-file 2 deep/a && + + test_all_match git commit -m "--include" --include deep/a && + test_all_match git status --porcelain=v2 && + test_all_match git commit -m "--include" --include a && + test_all_match git status --porcelain=v2 && + + run_on_all ../edit-file 3 a && + run_on_all ../edit-file 3 deep/a && + + test_all_match git commit -m "--amend" -a --amend && + test_all_match git status --porcelain=v2 +' + test_expect_success 'status/add: outside sparse cone' ' init_repos && @@ -535,14 +563,25 @@ test_expect_success 'sparse-index is expanded and converted back' ' test_region index ensure_full_index trace2.txt ' -test_expect_success 'sparse-index is not expanded' ' - init_repos && - +ensure_not_expanded () { rm -f trace2.txt && echo >>sparse-index/untracked.txt && GIT_TRACE2_EVENT="$(pwd)/trace2.txt" GIT_TRACE2_EVENT_NESTING=10 \ - git -C sparse-index status && + git -C sparse-index "$@" && test_region ! index ensure_full_index trace2.txt +} + +test_expect_success 'sparse-index is not expanded' ' + init_repos && + + ensure_not_expanded status && + ensure_not_expanded commit --allow-empty -m empty && + echo >>sparse-index/a && + ensure_not_expanded commit -a -m a && + echo >>sparse-index/a && + ensure_not_expanded commit --include a -m a && + echo >>sparse-index/deep/deeper1/a && + ensure_not_expanded commit --include deep/deeper1/a -m deeper ' test_expect_success 'reset mixed and checkout orphan' ' From patchwork Tue Jun 29 02:13:05 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348861 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7B22C11F65 for ; Tue, 29 Jun 2021 02:13:15 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 9B4F961CE4 for ; Tue, 29 Jun 2021 02:13:15 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231559AbhF2CPk (ORCPT ); Mon, 28 Jun 2021 22:15:40 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51744 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231304AbhF2CPj (ORCPT ); Mon, 28 Jun 2021 22:15:39 -0400 Received: from mail-wr1-x435.google.com (mail-wr1-x435.google.com [IPv6:2a00:1450:4864:20::435]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 633E7C061760 for ; Mon, 28 Jun 2021 19:13:11 -0700 (PDT) Received: by mail-wr1-x435.google.com with SMTP id u8so10438885wrq.8 for ; Mon, 28 Jun 2021 19:13:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=lX+phGtUUNbbXIoUjtKs9N0xtQbw6A/3yCrL+FgxfnM=; b=AWwslph0JnXtOD852oLlfRuJSw4QT5cUuGptsrGQNHtW7YIDoJuT95GWbmfAJaquMH 1+qI1x6XFT4TVUHdhl2fbtol2adh97YAJHpjSFj84zC3TyVnWIogvEGVlsq0rLTKy3Zz NpG52j2MKAtUVTsKpHQpMGclQ77vsVO5TN0mnd1kuMWuAZUZDT0CsPMiB5xlf47xS/W1 OWzkOy3VSuMJQovnTAARZMCPjjx9ab0FaYrhHu9P9eGN9Psh3dzr4UaALegVEkEo2Bw4 sIwKpwBUClS2xDmn/YFExoJG5zMt9WKfu6PCP/9Cg/1YbY/3bZFwZZOC9vRpmMrvE0BM MK2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=lX+phGtUUNbbXIoUjtKs9N0xtQbw6A/3yCrL+FgxfnM=; b=ECuMvkufyg+HqjkQJDWNn0A1y8JyLT9eKWLy7l76vROjXW6N0VdBhxodHrbL0blFBE o90mk0GSZDxuHZLxQTi+wxKtgzQCKYzWp1lvvxaIEVMJibZ8Ka+YzRXnaP4B2q04aYXz JAMKNgeEwOBPSvi/Xt+vpnhJoYS8RKtCgAL/VrQP1QEG5x3uJ1BLLMr9GNAuXanPrxGy Xlq715CFdH+mOTFTJl++WZOngdN5mNcBasIWj/kdCdn2XFjaa6OD80kHVNVj1rK59KxR gfz5r87m1WzMciMllcZadVsZAXkXVlNUfui5cMKcm5U3BAZDzpjiFtmwvnymZQ+EEG4Z FvCg== X-Gm-Message-State: AOAM533Idbw1pFET8Ykovu3oWQhY6Y3GkPGxP6WuS0HQrktZwC8qfYZJ j/MuZIwyP15M8qD16DlOSKSBz0vOYYw= X-Google-Smtp-Source: ABdhPJzl57MH4kVyqmEgSUXIbI/fKkaYEt80nBDaP+MNqwa9YTAMhM5lI2Oco7Es3f75p8+ZkiDbQQ== X-Received: by 2002:adf:ff8e:: with SMTP id j14mr30223818wrr.374.1624932790065; Mon, 28 Jun 2021 19:13:10 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id s7sm8448712wrp.97.2021.06.28.19.13.09 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:13:09 -0700 (PDT) Message-Id: In-Reply-To: References: Date: Tue, 29 Jun 2021 02:13:05 +0000 Subject: [PATCH 4/5] sparse-index: recompute cache-tree Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee When some commands run with command_requires_full_index=1, then the index can get in a state where the in-memory cache tree is actually equal to the sparse index's cache tree instead of the full one. This results in incorrect entry_count values. By clearing the cache tree before converting to sparse, we avoid this issue. Signed-off-by: Derrick Stolee --- sparse-index.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparse-index.c b/sparse-index.c index 53c8f711ccc..c6b4feec413 100644 --- a/sparse-index.c +++ b/sparse-index.c @@ -170,6 +170,8 @@ int convert_to_sparse(struct index_state *istate) if (index_has_unmerged_entries(istate)) return 0; + /* Clear and recompute the cache-tree */ + cache_tree_free(&istate->cache_tree); if (cache_tree_update(istate, 0)) { warning(_("unable to update cache-tree, staying full")); return -1; From patchwork Tue Jun 29 02:13:06 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Derrick Stolee X-Patchwork-Id: 12348863 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 23742C11F67 for ; Tue, 29 Jun 2021 02:13:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 05DF561CD4 for ; Tue, 29 Jun 2021 02:13:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231593AbhF2CPl (ORCPT ); Mon, 28 Jun 2021 22:15:41 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:51748 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231478AbhF2CPj (ORCPT ); Mon, 28 Jun 2021 22:15:39 -0400 Received: from mail-wr1-x432.google.com (mail-wr1-x432.google.com [IPv6:2a00:1450:4864:20::432]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 102FEC061766 for ; Mon, 28 Jun 2021 19:13:12 -0700 (PDT) Received: by mail-wr1-x432.google.com with SMTP id j2so23739482wrs.12 for ; Mon, 28 Jun 2021 19:13:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:in-reply-to:references:from:date:subject:fcc :content-transfer-encoding:mime-version:to:cc; bh=SQPTNKDRI4ssneXDiJuN5/cJooiS3xOo4L8xI+dA0QU=; b=qPG3rDjHYmn651EPiC82qrgXry0TC7V+ndD4EHQVlUSEjfgZCs4AbKCRosPulyln45 9vneamYbPXl2QQbIOeg+RAVtef1cQe+OE/HrxqmedfDEq1YWUHT1Mk/VuhRz/6yRAFht pFvkxnr04Kce7EOGv4Xd+OklhmkpobcvDRcUNvIeMv7UiwIs0jIP467CAwtfVNZIG61q 5Ir4jT3jRB0IDIiipM1DMmxcFtFVF+elQIzu6mQ8qPm7OlDuPQBSkU0kkhWTzvpGenPc v5IH50Vg0A8G+CVW9ox1NNppMnREOz3KbzS30OE3+LY+wI4uMG1QmZEqimz/K29ISgjf x38Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:in-reply-to:references:from:date :subject:fcc:content-transfer-encoding:mime-version:to:cc; bh=SQPTNKDRI4ssneXDiJuN5/cJooiS3xOo4L8xI+dA0QU=; b=q0hkUkSqR3VRpSJlocervtimxMOt3oSErFo4GBFSkxlARFnBWEjBnaDSBRhfiO1xX/ tYQykf7/4vSpU/7dJLumI6edearhZEKVJ18Ptug/FzvwKTHjh+tM2Qtm7LeYWt3LJ8JN csOG5hJUEyY+x7qCFg/Hwyo2pnE6VjXNbLOy2StMSLWsT8T1s/DedBy8x3txxVjWVpV5 p3SjTRsgtPr9x2ZZfWQN0bLUMW1MOvnB/6eez3/0jIHg2zFEe7uVrkucQBrZnhILDyH5 xANoLUuIPPAG6pwB33gytpwiO2sWaTySqCs9AbtDZfOFGo3UmmLpvXc5ANMcByjuhsyY Wp/A== X-Gm-Message-State: AOAM531bGLLdJLQ10I34Z+nVTB24Fuk/8ONWhR0mn/t+hy6Mf0kMqx0M PUydGEjoi5cCSzqrpkQgXAL0gjEnSog= X-Google-Smtp-Source: ABdhPJyF28g5K0KlY6EYm+m1jUvOT9oGpOyV0++LfM6n00W8xY639S0ZhuGc1B9i8hNk3e+MDRmV2A== X-Received: by 2002:a5d:65d1:: with SMTP id e17mr1471803wrw.275.1624932790686; Mon, 28 Jun 2021 19:13:10 -0700 (PDT) Received: from [127.0.0.1] ([13.74.141.28]) by smtp.gmail.com with ESMTPSA id x1sm15392666wmc.31.2021.06.28.19.13.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 28 Jun 2021 19:13:10 -0700 (PDT) Message-Id: <8710fee36b7821e0c36196b8e7b9a27d5994ae72.1624932787.git.gitgitgadget@gmail.com> In-Reply-To: References: Date: Tue, 29 Jun 2021 02:13:06 +0000 Subject: [PATCH 5/5] checkout: stop expanding sparse indexes Fcc: Sent MIME-Version: 1.0 To: git@vger.kernel.org Cc: gitster@pobox.com, newren@gmail.com, matheus.bernardino@usp.br, stolee@gmail.com, Derrick Stolee , Derrick Stolee Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Derrick Stolee From: Derrick Stolee Previous changes did the necessary improvements to unpack-trees.c and diff-lib.c in order to modify a sparse index based on its comparision with a tree. The only remaining work is to remove some ensure_full_index() calls and add tests that verify that the index is not expanded in our interesting cases. Include 'switch' and 'restore' in these tests, as they share a base implementation with 'checkout'. Here are the relevant performance results from p2000-sparse-operations.sh: Test HEAD~1 HEAD -------------------------------------------------------------------------------- 2000.18: git checkout -f - (full-v3) 0.49(0.43+0.03) 0.47(0.39+0.05) -4.1% 2000.19: git checkout -f - (full-v4) 0.45(0.37+0.06) 0.42(0.37+0.05) -6.7% 2000.20: git checkout -f - (sparse-v3) 0.76(0.71+0.07) 0.04(0.03+0.04) -94.7% 2000.21: git checkout -f - (sparse-v4) 0.75(0.72+0.04) 0.05(0.06+0.04) -93.3% It is important to compare the full index case to the sparse index case, as the previous results for the sparse index were inflated by the index expansion. For index v4, this is an 88% improvement. On an internal repository with over two million paths at HEAD and a sparse-checkout definition containing ~60,000 of those paths, 'git checkout' went from 3.5s to 297ms with this change. The theoretical optimum where only those ~60,000 paths exist was 275ms, so the extra sparse directory entries contribute a 22ms overhead. Signed-off-by: Derrick Stolee --- builtin/checkout.c | 8 +++----- t/t1092-sparse-checkout-compatibility.sh | 10 +++++++++- 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/builtin/checkout.c b/builtin/checkout.c index f4cd7747d35..b5d477919a7 100644 --- a/builtin/checkout.c +++ b/builtin/checkout.c @@ -378,9 +378,6 @@ static int checkout_worktree(const struct checkout_opts *opts, if (pc_workers > 1) init_parallel_checkout(); - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(&the_index); - for (pos = 0; pos < active_nr; pos++) { struct cache_entry *ce = active_cache[pos]; if (ce->ce_flags & CE_MATCHED) { @@ -530,8 +527,6 @@ static int checkout_paths(const struct checkout_opts *opts, * Make sure all pathspecs participated in locating the paths * to be checked out. */ - /* TODO: audit for interaction with sparse-index. */ - ensure_full_index(&the_index); for (pos = 0; pos < active_nr; pos++) if (opts->overlay_mode) mark_ce_for_checkout_overlay(active_cache[pos], @@ -1593,6 +1588,9 @@ static int checkout_main(int argc, const char **argv, const char *prefix, git_config(git_checkout_config, opts); + prepare_repo_settings(the_repository); + the_repository->settings.command_requires_full_index = 0; + opts->track = BRANCH_TRACK_UNSPECIFIED; if (!opts->accept_pathspec && !opts->accept_ref) diff --git a/t/t1092-sparse-checkout-compatibility.sh b/t/t1092-sparse-checkout-compatibility.sh index 11c6dc8ec1f..1d7dd735d7d 100755 --- a/t/t1092-sparse-checkout-compatibility.sh +++ b/t/t1092-sparse-checkout-compatibility.sh @@ -581,7 +581,15 @@ test_expect_success 'sparse-index is not expanded' ' echo >>sparse-index/a && ensure_not_expanded commit --include a -m a && echo >>sparse-index/deep/deeper1/a && - ensure_not_expanded commit --include deep/deeper1/a -m deeper + ensure_not_expanded commit --include deep/deeper1/a -m deeper && + ensure_not_expanded checkout rename-out-to-out && + ensure_not_expanded checkout - && + ensure_not_expanded switch rename-out-to-out && + ensure_not_expanded switch - && + git -C sparse-index reset --hard && + ensure_not_expanded checkout rename-out-to-out -- deep/deeper1 && + git -C sparse-index reset --hard && + ensure_not_expanded restore -s rename-out-to-out -- deep/deeper1 ' test_expect_success 'reset mixed and checkout orphan' '