From patchwork Thu Apr 23 21:41:02 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11506603 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 3225C81 for ; Thu, 23 Apr 2020 21:41:06 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1B0D42077D for ; Thu, 23 Apr 2020 21:41:06 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="ZlapkSEG" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726532AbgDWVlF (ORCPT ); Thu, 23 Apr 2020 17:41:05 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55050 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725777AbgDWVlE (ORCPT ); Thu, 23 Apr 2020 17:41:04 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id DD648C09B042 for ; Thu, 23 Apr 2020 14:41:04 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id a7so3051947pju.2 for ; Thu, 23 Apr 2020 14:41:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=RTvKG1i9iSpQkXtkhMv9n7bFb2DzUwpr0ka+2ZBOaMc=; b=ZlapkSEGTQ1ogkjfrsD8ZM2RJBjU+DOEhSBipC84CC/u1STwnreY8//qqJJoLiFMnG wF1ZDzpJMW/1RncpCnEDtR1BtR3OQMul05bQdB2+rT+GA2v0M5Es4cw4rLcyL4F7MSdY EMb0AZwJp3H7wRYkmUX0zkOaOdU4ZhcHzKfoNbSeSAGgcfh0c18GjEg28NcRQx97o4+u xElNbtqbRzTxi4nnxmO30RGzjHh98kzKDJIa52Vpsjr3Byr9dBhRI5JMDavuMmPJkjHE pzyn482RVGEaYFQfYvFcHeGhJFm4gYl9hf+3n0KbLUwXGKERlVDTeTuty6HCvX48H4a/ mWYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=RTvKG1i9iSpQkXtkhMv9n7bFb2DzUwpr0ka+2ZBOaMc=; b=RtkWc9zNqRX4vxZ6cv0VetiyiO3enobWfIhAdJ/YrKPDYpHZEpOBNzR6Ewd3fkrUZS L4n9RnpfnqD8ICX0YrA1qQpet+Nf6IHcgkr8KuLPzWF9dHQeMquMizfLJkACF7xAyAZG JUFyQMpRsDpKxIPmqZD8Moguvmr9FgUHnAcfKCRcFV0dd71Uk32cUKTCFfg0Zw/o51pi 5OZMGGUxIULaSZ0NisNLfNr39po5K/yctw4j6ux9YApY8Z32F8IFrH0uSYXZsLifF1DY vEF7Uyej/S7owtuJ53U1s+0Vpn6Ck4/DuhM2mBgWD3Nesfjd6hgcYfzXiDLTO29vGFUL 6Xjg== X-Gm-Message-State: AGi0PuarjBA/qEQzq4JgY++vHCiZUFm0XCeAWg01pzZ/YCHM15bn0TBt j9oNHoXJo5jFkpKuoo0NhP27a7xtzYhS0Q== X-Google-Smtp-Source: APiQypIWj1VdGx/4rCyg1HOTAbUYBZwirMtTH4Vnjmv1J7rx/2dlW6XRqLkoOSVzsvqHRnUBRwM74Q== X-Received: by 2002:a17:902:fe03:: with SMTP id g3mr5727758plj.28.1587678063880; Thu, 23 Apr 2020 14:41:03 -0700 (PDT) Received: from localhost ([8.44.146.30]) by smtp.gmail.com with ESMTPSA id k1sm2887763pgh.72.2020.04.23.14.41.02 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Apr 2020 14:41:03 -0700 (PDT) Date: Thu, 23 Apr 2020 15:41:02 -0600 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, szeder.dev@gmail.com, dstolee@microsoft.com, gitster@pobox.com Subject: [PATCH 1/4] commit-graph.c: don't use discarded graph_name in error Message-ID: <17dff6142e62d202794986f0bd51e4a8604f8e49.1587677671.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When writing a commit-graph layer, we do so in a temporary file which is renamed into place. If we fail to create a temporary file, for e.g., because we have too many open files, then 'git_mkstemp_mode' sets the pattern to the empty string, in which case we get an error something along the lines of: error: unable to create '' It's not useful to show the pattern here at all, since we (1) know the pattern is well-formed, and (2) would have already shown the dirname when trying to create the leading directories. So, replace this error with something friendlier. Signed-off-by: Taylor Blau --- commit-graph.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/commit-graph.c b/commit-graph.c index b8737f0ce9..afde075962 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1397,7 +1397,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx) fd = git_mkstemp_mode(ctx->graph_name, 0444); if (fd < 0) { - error(_("unable to create '%s'"), ctx->graph_name); + error(_("unable to create temporary graph layer")); return -1; } From patchwork Thu Apr 23 21:41:06 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11506605 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 0072381 for ; Thu, 23 Apr 2020 21:41:10 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id DBD412077D for ; Thu, 23 Apr 2020 21:41:09 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="Y4EypW78" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726725AbgDWVlJ (ORCPT ); Thu, 23 Apr 2020 17:41:09 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55062 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726632AbgDWVlI (ORCPT ); Thu, 23 Apr 2020 17:41:08 -0400 Received: from mail-pf1-x442.google.com (mail-pf1-x442.google.com [IPv6:2607:f8b0:4864:20::442]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 9FF78C09B042 for ; Thu, 23 Apr 2020 14:41:08 -0700 (PDT) Received: by mail-pf1-x442.google.com with SMTP id x3so3641028pfp.7 for ; Thu, 23 Apr 2020 14:41:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=gzX68apARnZkmRg/kLbBj26JvbipzxwMjdqEGeeuZzY=; b=Y4EypW78KmYBOMRISHOO8oBjI2M0TbwwRETTaq7k7atEegN3YOZOmPPZd1BPSj2pHa XYBWURgt6bz9fY4htEynWIFkklbN4GIExzZiuH1rrmmOngyaMCcWHUyKwFdX9qkjDBp/ ZnQc1/MiAK4ENXivo4H98ri9YE1JWVJLfebFeULQ477ZmScT/diWleF7RXjxZbUJ65W3 6nYqvdRXg4eMFrU6tMbAWtFJpa8NGx8MlGs4SIbH26EgIjmqDo+jnASyltP0oA+Ag/S0 wR4R6d03fDixu4bjo0Szsz77h6aSoxmxMZMQnok9dLpYce2tkJIGFq5ns6pVsPPgSjMB nyXw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=gzX68apARnZkmRg/kLbBj26JvbipzxwMjdqEGeeuZzY=; b=pCEjzmSLGiuGrZ1NyQpPEm4Zlwfbdx0mp7HY39mENUC2RjaHPXd6jnZnWpUGM0qsRn PDWHreyaXgp+xtQh+oz9wGBZ9guJyICswUPtsKRWfSHF25/Beh19p7HCn/IKVbWgb2Ly 5XShYqVcHWND9zzMi97ZGKZ6+GEIixvyzwlZdc2FUCqMKUEkINcSIIfFHfIhE+7z8q/u BdLuDQWkl4BSSfSP+4loqWa9LjEaooGEzQu9fu0nKnuzxda/6hKn8l58206zJETMBDwi VBprHYLC/I8YOztR+aPtgHw7gyECQ4vEMyu/1gcoHg6D3PvHgRGZP9pk+2NihluSD1pA x1fA== X-Gm-Message-State: AGi0PuYUZwQhO4l7Ge1mTi1ffT/m54CezP7BILEtd7dshAPL2aA7/6bX 0i0OdDqPZ6a3KaoCFptpKE5tqilzUi0yEQ== X-Google-Smtp-Source: APiQypIEI6B9vkJ97mglvd/Q2g7cRdLxw3GrNqDj9uUGvO7/gjEa/UR7XCiz4IheY6sgbdiFZCFe3Q== X-Received: by 2002:a63:7d4:: with SMTP id 203mr2557121pgh.59.1587678067823; Thu, 23 Apr 2020 14:41:07 -0700 (PDT) Received: from localhost ([8.44.146.30]) by smtp.gmail.com with ESMTPSA id y184sm3549954pfg.127.2020.04.23.14.41.06 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Apr 2020 14:41:07 -0700 (PDT) Date: Thu, 23 Apr 2020 15:41:06 -0600 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, szeder.dev@gmail.com, dstolee@microsoft.com, gitster@pobox.com Subject: [PATCH 2/4] t/test-lib.sh: make ULIMIT_FILE_DESCRIPTORS available to tests Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org In t1400 the prerequisite 'ULIMIT_FILE_DESCRIPTORS' is defined and used to effectively guard the helper function 'run_with_limited_open_files' from being used on systems that do not satisfy this prerequisite. In the subsequent patch, we will introduce another test outside of t1400 that would benefit from using this prerequisite. So, move it to 'test-lib.sh' instead so that it can be used by multiple tests. Signed-off-by: Taylor Blau --- t/t1400-update-ref.sh | 9 --------- t/test-lib.sh | 9 +++++++++ 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh index a6224ef65f..1e7428a379 100755 --- a/t/t1400-update-ref.sh +++ b/t/t1400-update-ref.sh @@ -1354,15 +1354,6 @@ test_expect_success 'fails with duplicate ref update via symref' ' test_cmp expect actual ' -run_with_limited_open_files () { - (ulimit -n 32 && "$@") -} - -test_lazy_prereq ULIMIT_FILE_DESCRIPTORS ' - test_have_prereq !MINGW,!CYGWIN && - run_with_limited_open_files true -' - test_expect_success ULIMIT_FILE_DESCRIPTORS 'large transaction creating branches does not burst open file limit' ' ( for i in $(test_seq 33) diff --git a/t/test-lib.sh b/t/test-lib.sh index 9fe390bd5a..03bf2a4b93 100644 --- a/t/test-lib.sh +++ b/t/test-lib.sh @@ -1618,6 +1618,15 @@ test_lazy_prereq ULIMIT_STACK_SIZE ' run_with_limited_stack true ' +run_with_limited_open_files () { + (ulimit -n 32 && "$@") +} + +test_lazy_prereq ULIMIT_FILE_DESCRIPTORS ' + test_have_prereq !MINGW,!CYGWIN && + run_with_limited_open_files true +' + build_option () { git version --build-options | sed -ne "s/^$1: //p" From patchwork Thu Apr 23 21:41:09 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11506681 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1957281 for ; Thu, 23 Apr 2020 21:42:03 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id F1CBE2087E for ; Thu, 23 Apr 2020 21:42:02 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="WT2PJvrw" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728080AbgDWVmB (ORCPT ); Thu, 23 Apr 2020 17:42:01 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55074 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726970AbgDWVlN (ORCPT ); Thu, 23 Apr 2020 17:41:13 -0400 Received: from mail-pf1-x441.google.com (mail-pf1-x441.google.com [IPv6:2607:f8b0:4864:20::441]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7C2CAC09B042 for ; Thu, 23 Apr 2020 14:41:12 -0700 (PDT) Received: by mail-pf1-x441.google.com with SMTP id w65so3630742pfc.12 for ; Thu, 23 Apr 2020 14:41:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=W8kNugks4MXsfPxeACWeyHb7wmQzLURnexIYN7sejkI=; b=WT2PJvrwXgTsUInge30oMAV9HutXHcDoLKf9of2doiCPDBnQ12jYVlt4KaGc3Wr3DO L/PbjgimRVTE3C6ddYU3fTt0gh4Ty6Lr1fNR8z1HrIpA9+P44an99g9MeO6RJDzQzsK3 P4GtIN4HeK+1aVmC7iHJsWe9LrYhfAmRLjKlWdl/9q3ttwO8Ixq49jYMROocO+S/nPKW 5sdS0HxNez//oBRru7yPlSl3ahbQNNnbl7ms1fOPzIeUlIROehcyH6kBrlElh83Gqi0j 37+xmwvGzgF+Ll5sXplsCyc5Yg8NVq9esCrK7g1dyUoxvyaKiaH+0lUnfBtmGIpeCtn+ +RNg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=W8kNugks4MXsfPxeACWeyHb7wmQzLURnexIYN7sejkI=; b=LlMazn6cHnxcIsP9f4C+IGYK88MznKu+rLrJvpXeWS5mSXIHsr+TjUhPL5asQTJ2MD WLOVZycq2o8QewN5sJtJTSBXVZMz1Vbc7c9M5J/oQUdmoW9U39QYiUlgKVCg7iRFyP6W u/LuDUHKaAvEvwWEPdikqHhWHYl91e0MZlq2lIAhMrqpj40KJ483nkN+/Jaer3/0tY7Z djK/M037JIzj7tb5NqqnksPbE8JTWpjLDKIFRuanXuaYfmPksBbSIC4RtPdlJiZWCYTT m4LkQwszw7yCc1cnsC8/UEYJ3C6ugVsa3Qfl2UxLJ9Gh8IuFo05/As/3tV68rvgiLreg 2aAA== X-Gm-Message-State: AGi0PuakJmCJctFe7qZyuoh+T8HCgHuySTn0YTcZxx1eZl8/fRD9ar3B a44EUaWe2s9TIBDwuGegdl3g11uPg73/Xw== X-Google-Smtp-Source: APiQypKuyyrDWW9uORFXEiN7+7ms8XTOpx0d98L/+MjHUoKP5hUCRbJ5PWOW+Ax27KDXLr4EoBqrLQ== X-Received: by 2002:aa7:920d:: with SMTP id 13mr6108576pfo.112.1587678071305; Thu, 23 Apr 2020 14:41:11 -0700 (PDT) Received: from localhost ([8.44.146.30]) by smtp.gmail.com with ESMTPSA id q200sm3004866pgq.68.2020.04.23.14.41.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Apr 2020 14:41:10 -0700 (PDT) Date: Thu, 23 Apr 2020 15:41:09 -0600 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, szeder.dev@gmail.com, dstolee@microsoft.com, gitster@pobox.com Subject: [PATCH 3/4] commit-graph.c: gracefully handle file descriptor exhaustion Message-ID: <2b8ee726690861405f41adede5582b96749e98c5.1587677671.git.me@ttaylorr.com> References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org When writing a layered commit-graph, the commit-graph machinery uses 'commit_graph_filenames_after' and 'commit_graph_hash_after' to keep track of the layers in the chain that we are in the process of writing. When the number of commit-graph layers shrinks, we initialize all entries in the aforementioned arrays, because we know the structure of the new commit-graph chain immediately (since there are no new layers, there are no unknown hash values). But when the number of commit-graph layers grows (i.e., that 'num_commit_graphs_after > num_commit_graphs_before'), then we leave some entries in the filenames and hashes arrays as uninitialized, because we will fill them in later as those values become available. For instance, we rely on 'write_commit_graph_file's to store the filename and hash of the last layer in the new chain, which is the one that it is responsible for writing. But, it's possible that 'write_commit_graph_file' may fail, e.g., from file descriptor exhaustion. In this case it is possible that 'git_mkstemp_mode' will fail, and that function will return early *before* setting the values for the last commit-graph layer's filename and hash. This causes a number of upleasant side-effects. For instance, trying to 'free()' each entry in 'ctx->commit_graph_filenames_after' (and similarly for the hashes array) causes us to 'free()' uninitialized memory, since the area is allocated with 'malloc()' and is therefore subject to contain garbage (which is left alone when 'write_commit_graph_file' returns early). This can manifest in other issues, like a general protection fault, and/or leaving a stray 'commit-graph-chain.lock' around after the process dies. (The reasoning for this is still a mystery to me, since we'd otherwise usually expect the kernel to run tempfile.c's 'atexit()' handlers in the case of a normal death...) To resolve this, initialize the memory with 'CALLOC_ARRAY' so that uninitialized entries are filled with zeros, and can thus be 'free()'d as a noop instead of causing a fault. Helped-by: Jeff King Helped-by: Derrick Stolee Signed-off-by: Taylor Blau --- commit-graph.c | 4 ++-- t/t5324-split-commit-graph.sh | 13 +++++++++++++ 2 files changed, 15 insertions(+), 2 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index afde075962..b2d2fdfe3d 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -1602,8 +1602,8 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx) free(old_graph_name); } - ALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after); - ALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after); + CALLOC_ARRAY(ctx->commit_graph_filenames_after, ctx->num_commit_graphs_after); + CALLOC_ARRAY(ctx->commit_graph_hash_after, ctx->num_commit_graphs_after); for (i = 0; i < ctx->num_commit_graphs_after && i < ctx->num_commit_graphs_before; i++) diff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh index e5d8d64170..0d1db31b0a 100755 --- a/t/t5324-split-commit-graph.sh +++ b/t/t5324-split-commit-graph.sh @@ -381,4 +381,17 @@ test_expect_success '--split=replace replaces the chain' ' graph_read_expect 2 ' +test_expect_success ULIMIT_FILE_DESCRIPTORS 'handles file descriptor exhaustion' ' + git init ulimit && + ( + cd ulimit && + for i in $(test_seq 64) + do + test_commit $i && + test_might_fail run_with_limited_open_files git commit-graph write \ + --split=no-merge --reachable || return 1 + done + ) +' + test_done From patchwork Thu Apr 23 21:41:13 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Taylor Blau X-Patchwork-Id: 11506613 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 368C7912 for ; Thu, 23 Apr 2020 21:41:20 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 1CEFD214AF for ; Thu, 23 Apr 2020 21:41:20 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=ttaylorr-com.20150623.gappssmtp.com header.i=@ttaylorr-com.20150623.gappssmtp.com header.b="JaKutsk2" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727117AbgDWVlR (ORCPT ); Thu, 23 Apr 2020 17:41:17 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55086 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727072AbgDWVlQ (ORCPT ); Thu, 23 Apr 2020 17:41:16 -0400 Received: from mail-pj1-x1043.google.com (mail-pj1-x1043.google.com [IPv6:2607:f8b0:4864:20::1043]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 161EFC09B042 for ; Thu, 23 Apr 2020 14:41:16 -0700 (PDT) Received: by mail-pj1-x1043.google.com with SMTP id mq3so3042554pjb.1 for ; Thu, 23 Apr 2020 14:41:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ttaylorr-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=lynpfoXwYfpxnOFiOfS3Q3usZSFf+AAKP9sCVspmNxw=; b=JaKutsk2bNuc87mir++IjHpft1S6G8GIMdzRXkmeP5Ynr902P8QSo8/jMaafwaa4N+ co0iv4nq8U+zVYBGENa8CRo7gYvatR0UUaIh79O8nxgZGSezzhW/obr0a1Fn4M5tvNU9 FIkvoc0iWRS/vSynHHEZC4t4klBshbdh7dG2QhviJGATW7TmDvHjhvEB1mYnXz0wOEpE Zq7wTf3F9cvPq15IxOzrMKeeQJGmTMGhZEfPZeB+EYZG0XF0AGAXkpMSKfIWWb0hLbSz jTJ6S9XJCwuZLhRznuuJyWhDxfyUzhhhjyEeWQVS15xk7GgMEG+pvaA+dl/btoDIFEcE bIsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=lynpfoXwYfpxnOFiOfS3Q3usZSFf+AAKP9sCVspmNxw=; b=PESYtSP5IMyB1bLgVK9mg0iphLZ6cKEt06dCLpadsDZVv3GWi4pBUL8yDAkReMDffx RVrCwH000BnYLX8LA8z3PcyybNpVtO19CYlMAlU1vi+Bnn6JopxwyP3r3xfratPWE2cD Pnfz3wD3MRWFh8yHqCGqbOOKq1105Mazi1UBd/tETJA2sskCNv4xbLFN4CDiVv49XtHF 0qMSvCJ/WqAC+Y6eSuJVrdGzJNqq99H6Oxkf6iENmo2Lc08LdZQCp1QYWpxjiwL/1LQ8 uSTrp28OKhEIfE4W8UNdCTy0iCWc784hZTI2q1LtwHqGExqEiLdKT/Lill8Usnx5tEt/ QIqw== X-Gm-Message-State: AGi0Pubc5bDuwVwIAY8L/f+Al3R9THvxNwXrj43+354FpqOSobwpnsGS rB8N+zW6zXSV93JqYNNMr5G/2pX9CFi2zw== X-Google-Smtp-Source: APiQypIqxKkgNtgcTqAeQygZgK5ZoZTKGRBRJoQ/fBDYSS0CNhKXd7MfpwGTqxzbpsTYZn+fkBInDQ== X-Received: by 2002:a17:90a:1501:: with SMTP id l1mr2776485pja.82.1587678074955; Thu, 23 Apr 2020 14:41:14 -0700 (PDT) Received: from localhost ([8.44.146.30]) by smtp.gmail.com with ESMTPSA id h197sm3660437pfe.208.2020.04.23.14.41.13 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 23 Apr 2020 14:41:14 -0700 (PDT) Date: Thu, 23 Apr 2020 15:41:13 -0600 From: Taylor Blau To: git@vger.kernel.org Cc: peff@peff.net, szeder.dev@gmail.com, dstolee@microsoft.com, gitster@pobox.com Subject: [PATCH 4/4] commit-graph: close descriptors after mmap Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Jeff King We don't ever refer to the descriptor after mmap-ing it. And keeping it open means we can run out of descriptors in degenerate cases (e.g., thousands of split chain files). Let's close it as soon as possible. Signed-off-by: Jeff King Signed-off-by: Taylor Blau Signed-off-by: Derrick Stolee --- commit-graph.c | 15 +++++---------- commit-graph.h | 3 +-- fuzz-commit-graph.c | 5 ++--- 3 files changed, 8 insertions(+), 15 deletions(-) diff --git a/commit-graph.c b/commit-graph.c index b2d2fdfe3d..e9b458539f 100644 --- a/commit-graph.c +++ b/commit-graph.c @@ -69,7 +69,6 @@ static uint8_t oid_version(void) static struct commit_graph *alloc_commit_graph(void) { struct commit_graph *g = xcalloc(1, sizeof(*g)); - g->graph_fd = -1; return g; } @@ -123,14 +122,13 @@ struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st, return NULL; } graph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0); - ret = parse_commit_graph(graph_map, fd, graph_size); + close(fd); + ret = parse_commit_graph(graph_map, graph_size); if (ret) ret->odb = odb; - else { + else munmap(graph_map, graph_size); - close(fd); - } return ret; } @@ -165,8 +163,7 @@ static int verify_commit_graph_lite(struct commit_graph *g) return 0; } -struct commit_graph *parse_commit_graph(void *graph_map, int fd, - size_t graph_size) +struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size) { const unsigned char *data, *chunk_lookup; uint32_t i; @@ -209,7 +206,6 @@ struct commit_graph *parse_commit_graph(void *graph_map, int fd, graph->hash_len = the_hash_algo->rawsz; graph->num_chunks = *(unsigned char*)(data + 6); - graph->graph_fd = fd; graph->data = graph_map; graph->data_len = graph_size; @@ -2125,10 +2121,9 @@ void free_commit_graph(struct commit_graph *g) { if (!g) return; - if (g->graph_fd >= 0) { + if (g->data) { munmap((void *)g->data, g->data_len); g->data = NULL; - close(g->graph_fd); } free(g->filename); free(g); diff --git a/commit-graph.h b/commit-graph.h index 98ef121924..a0a2c4a1e5 100644 --- a/commit-graph.h +++ b/commit-graph.h @@ -66,8 +66,7 @@ struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st, struct object_directory *odb); struct commit_graph *read_commit_graph_one(struct repository *r, struct object_directory *odb); -struct commit_graph *parse_commit_graph(void *graph_map, int fd, - size_t graph_size); +struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size); /* * Return 1 if and only if the repository has a commit-graph diff --git a/fuzz-commit-graph.c b/fuzz-commit-graph.c index 0157acbf2e..9fd1c04edd 100644 --- a/fuzz-commit-graph.c +++ b/fuzz-commit-graph.c @@ -1,8 +1,7 @@ #include "commit-graph.h" #include "repository.h" -struct commit_graph *parse_commit_graph(void *graph_map, int fd, - size_t graph_size); +struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size); int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size); @@ -11,7 +10,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size) struct commit_graph *g; initialize_the_repository(); - g = parse_commit_graph((void *)data, -1, size); + g = parse_commit_graph((void *)data, size); repo_clear(the_repository); free(g);