From patchwork Mon Dec 3 23:37:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10710919 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C779A13BF for ; Mon, 3 Dec 2018 23:37:48 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B95A429267 for ; Mon, 3 Dec 2018 23:37:48 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id AD13C2B222; Mon, 3 Dec 2018 23:37:48 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 56AAD29267 for ; Mon, 3 Dec 2018 23:37:48 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726073AbeLCXhr (ORCPT ); Mon, 3 Dec 2018 18:37:47 -0500 Received: from mail-pf1-f202.google.com ([209.85.210.202]:46675 "EHLO mail-pf1-f202.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725938AbeLCXhq (ORCPT ); Mon, 3 Dec 2018 18:37:46 -0500 Received: by mail-pf1-f202.google.com with SMTP id h11so12481396pfj.13 for ; Mon, 03 Dec 2018 15:37:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=ur43xZWRloyiHiNXEcPnNN1YCs187IqiL32Jn2gt420=; b=HlZMIA0fc7WONlbQIk2Y81mHHrD12MMvVCQbvf4uBxjTfzgNbsmFIMnR2tiVAyoos6 P4nnml6IEC0YB6/9eiAjHqnbp4kkyABNrFhJV8InvhS5u1n5uolN0JHSd5134Ky0wmvH 8XxLpfqEy5ibfbOSlLM/1GN+fNI0/yyAJB4abmJDX6BjPUnij9MVUlXjDRebFQaVfYCi b+ObwN6z8mOhxmZo5okF1AvbU9W11H/w+EVK+loyQK9+za8hthAxg9vsXp0yovHcgMNS Xf/HMu7Hw4Vgit7Gv8FZnPEBJjHNVdp2bgHv4Qq5+SJMkwYkIeHNLuhnezqbSdwvDRh/ gSrg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=ur43xZWRloyiHiNXEcPnNN1YCs187IqiL32Jn2gt420=; b=FpHVx+gwHEqUeSvi71keH7kuMgUhsZ0n/qSkPJZ59DOfnJ/p26YbeXbSRWU6xJ0Y4e vDCWorA+4BZnv/CvCnBBJtd/+u3+IxuXcfqn3CBtKUkDcK4RWUhaGFheD5bCJ4RKkhnB r+1s73AxL5mW1cjAPhBvlxNZFVJisbOntlr0359D/ltqXUxNR2FGglH8/P2088qq5eTW 5HqDHrleqTpXRBZl6iLSSAaFx8qIkSWX7BPe0PIx+mLgZ11ZEWLcoEtvyGHILAP0SN0n /L/n536OKTZnk3ziTx4ux5tD+MAfa334ziUw1vL5qhi1cL3UwgoszLqkWs3Wurj9SSQD bYog== X-Gm-Message-State: AA+aEWYrk7n1WXAknBxTQFhps3hapKypPkWnX5GKdx48pSBXm1PIflmQ dY/1PJGumM2B9YyIiooi1iWhtb7ghA+Z9izIWH2QpwePrRDd5JHR1OXs+H/wjPmIWnlGkNSswTE FHF13I3Ri1uAF8kRQYttK2WZT186uQJ9hJ9PjkuUMSgcnVKb4Fm5NBpOj4xyMoLJMx9ji6epqFl s+ X-Google-Smtp-Source: AFSGD/XjjuTRTmC76J4KU3JSQTgieRJ0IibQqXl5UqffvMLzO11a323f3hRKzmcySFNc+bQxKXZuij6l5w4jyzVXzcrg X-Received: by 2002:a63:6dc4:: with SMTP id i187mr8119221pgc.105.1543880265782; Mon, 03 Dec 2018 15:37:45 -0800 (PST) Date: Mon, 3 Dec 2018 15:37:34 -0800 In-Reply-To: Message-Id: <061dacf51763bd2f72b95c382e08f44655e247a7.1543879256.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [WIP RFC 1/5] Documentation: order protocol v2 sections From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP The git command line expects Git servers to follow a specific order of sections when transmitting protocol v2 responses, but this is not explicit in the documentation. Make the order explicit. Signed-off-by: Jonathan Tan --- Documentation/technical/protocol-v2.txt | 18 ++++++++---------- 1 file changed, 8 insertions(+), 10 deletions(-) diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 09e4e0273f..345c00e08c 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -309,11 +309,11 @@ the 'wanted-refs' section in the server's response as explained below. The response of `fetch` is broken into a number of sections separated by delimiter packets (0001), with each section beginning with its section -header. +header. Most sections are sent only when the packfile is sent. - output = *section - section = (acknowledgments | shallow-info | wanted-refs | packfile) - (flush-pkt | delim-pkt) + output = acknowledgements flush-pkt | + [acknowledgments delim-pkt] [shallow-info delim-pkt] + [wanted-refs delim-pkt] packfile flush-pkt acknowledgments = PKT-LINE("acknowledgments" LF) (nak | *ack) @@ -335,9 +335,10 @@ header. *PKT-LINE(%x01-03 *%x00-ff) acknowledgments section - * If the client determines that it is finished with negotiations - by sending a "done" line, the acknowledgments sections MUST be - omitted from the server's response. + * If the client determines that it is finished with negotiations by + sending a "done" line (thus requiring the server to send a packfile), + the acknowledgments sections MUST be omitted from the server's + response. * Always begins with the section header "acknowledgments" @@ -388,9 +389,6 @@ header. which the client has not indicated was shallow as a part of its request. - * This section is only included if a packfile section is also - included in the response. - wanted-refs section * This section is only included if the client has requested a ref using a 'want-ref' line and if a packfile section is also From patchwork Mon Dec 3 23:37:35 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10710921 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EB91113BF for ; Mon, 3 Dec 2018 23:37:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DCFF629267 for ; Mon, 3 Dec 2018 23:37:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id D1C5F2B222; Mon, 3 Dec 2018 23:37:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4AB4E29267 for ; Mon, 3 Dec 2018 23:37:51 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726077AbeLCXhu (ORCPT ); Mon, 3 Dec 2018 18:37:50 -0500 Received: from mail-qk1-f201.google.com ([209.85.222.201]:37003 "EHLO mail-qk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725938AbeLCXht (ORCPT ); Mon, 3 Dec 2018 18:37:49 -0500 Received: by mail-qk1-f201.google.com with SMTP id s70so14945884qks.4 for ; Mon, 03 Dec 2018 15:37:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=BWwIOsLIF6mT399MbObuVLAM3Rq4NxNZHzPahQc6trM=; b=dcvhxUljjl1hskd3LhoUGEV6G1CvTE5JTurpLsQQrSU7RLFv6d0xGl5T/Icy8jDQ4a gHdrMbvj+PImbg92OFUdEAxzFgFRTP6qDYhX9fZzXxQAdIhtpjZ0A7MGvXoCwHLKB7P2 6dvsw4/1Yino3phbvdcvTnGIfNzb8BSY53/c2VD/RvJNxR+jH6xN1k9AQSi8GKpqc9kR jk7Fxq4isXyAkNGBcwoJ44sQ0g4Budms4+sx8WgD719R4CZDF29inJsJOvBM1mKCBgd3 EgDgusxigjg2gHUC0GmdN4qzqlseiWqlgX+Aokz6RaCk0XFZl3pN5UQZyWAYrind+Q66 RewA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=BWwIOsLIF6mT399MbObuVLAM3Rq4NxNZHzPahQc6trM=; b=NbYKJSJK8QCHsj7pndeG5wSGHgNNFzzAhIH7nTMe7y5jEAVd/vaPyWWluzZ/ZwXVQS Np4r/QyWhof8wgs5x4QFWSlEcJPVTzYNffzJIk+MMVwE5YZhaRrV55IHu1mBLUdpg073 mNgjzT8zdzsaaea1MV3N4t+x/2toSbQMU1Fy7rOC7eU3J/9MOGhPk2PsSal1j09LK6r7 n3II2JPwvCbyfq08UfYCsidC7xFmj8gU9dsmDdlh6IHyrUMOaa1pth2EZ0lq6vcyWZUe HA3rB8x/Qhbz9LAYENCOGEYyM1+LMVZRsl/vvXZTFTdiI6BcXFZgNqs4+iLUT/FRWrpS grsg== X-Gm-Message-State: AA+aEWbQwoVezOYUd/X48EVPWyiM+LS8fd/ew2VsmBPLVQfQvukmMMgC 8ePUkvK4ADzmPCIv/ZqPib+bkgf+gpiRw+2/wZ5QowtEoRbpe3yiE7ULV5XJ/3AQrsiN09XoGH8 zs2HYaygKroNcNbbz7d9sc+o+QUwfPLaC0b01pLkWgnKX513TS+UAorpdue/Ax67KsjOh9iiwlw Ry X-Google-Smtp-Source: AFSGD/VLeyacCCDhExkhXm9UyhApzslj7r9pUo/ILe0BgbwnGOH2Ab0Ag0VHdbANUoJK0H/iWxpPfc5+iGvBPfsAq3Fy X-Received: by 2002:a37:d448:: with SMTP id l69mr13545110qki.21.1543880268319; Mon, 03 Dec 2018 15:37:48 -0800 (PST) Date: Mon, 3 Dec 2018 15:37:35 -0800 In-Reply-To: Message-Id: <0461b362569362c6d0e73951469c547a03a1b59d.1543879256.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [WIP RFC 2/5] Documentation: add Packfile URIs design doc From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Signed-off-by: Jonathan Tan --- Documentation/technical/packfile-uri.txt | 83 ++++++++++++++++++++++++ Documentation/technical/protocol-v2.txt | 6 +- 2 files changed, 88 insertions(+), 1 deletion(-) create mode 100644 Documentation/technical/packfile-uri.txt diff --git a/Documentation/technical/packfile-uri.txt b/Documentation/technical/packfile-uri.txt new file mode 100644 index 0000000000..6535801486 --- /dev/null +++ b/Documentation/technical/packfile-uri.txt @@ -0,0 +1,83 @@ +Packfile URIs +============= + +This feature allows servers to serve part of their packfile response as URIs. +This allows server designs that improve scalability in bandwidth and CPU usage +(for example, by serving some data through a CDN), and (in the future) provides +some measure of resumability to clients. + +This feature is available only in protocol version 2. + +Protocol +-------- + +The server advertises `packfile-uris`. + +If the client replies with the following arguments: + + * packfile-uris + * thin-pack + * ofs-delta + +when the server sends the packfile, it MAY send a `packfile-uris` section +directly before the `packfile` section (right after `wanted-refs` if it is +sent) containing HTTP(S) URIs. See protocol-v2.txt for the documentation of +this section. + +Clients then should understand that the returned packfile could be incomplete, +and that it needs to download all the given URIs before the fetch or clone is +complete. Each URI should point to a Git packfile (which may be a thin pack and +which may contain offset deltas). + +Server design +------------- + +The server can be trivially made compatible with the proposed protocol by +having it advertise `packfile-uris`, tolerating the client sending +`packfile-uris`, and never sending any `packfile-uris` section. But we should +include some sort of non-trivial implementation in the Minimum Viable Product, +at least so that we can test the client. + +This is the implementation: a feature, marked experimental, that allows the +server to be configured by one or more `uploadpack.blobPackfileUri= +` entries. Whenever the list of objects to be sent is assembled, a blob +with the given sha1 can be replaced by the given URI. This allows, for example, +servers to delegate serving of large blobs to CDNs. + +Client design +------------- + +While fetching, the client needs to remember the list of URIs and cannot +declare that the fetch is complete until all URIs have been downloaded as +packfiles. + +The division of work (initial fetch + additional URIs) introduces convenient +points for resumption of an interrupted clone - such resumption can be done +after the Minimum Viable Product (see "Future work"). + +The client can inhibit this feature (i.e. refrain from sending the +`packfile-urls` parameter) by passing --no-packfile-urls to `git fetch`. + +Future work +----------- + +The protocol design allows some evolution of the server and client without any +need for protocol changes, so only a small-scoped design is included here to +form the MVP. For example, the following can be done: + + * On the server, a long-running process that takes in entire requests and + outputs a list of URIs and the corresponding inclusion and exclusion sets of + objects. This allows, e.g., signed URIs to be used and packfiles for common + requests to be cached. + * On the client, resumption of clone. If a clone is interrupted, information + could be recorded in the repository's config and a "clone-resume" command + can resume the clone in progress. (Resumption of subsequent fetches is more + difficult because that must deal with the user wanting to use the repository + even after the fetch was interrupted.) + +There are some possible features that will require a change in protocol: + + * Additional HTTP headers (e.g. authentication) + * Byte range support + * Different file formats referenced by URIs (e.g. raw object) + diff --git a/Documentation/technical/protocol-v2.txt b/Documentation/technical/protocol-v2.txt index 345c00e08c..2cb1c41742 100644 --- a/Documentation/technical/protocol-v2.txt +++ b/Documentation/technical/protocol-v2.txt @@ -313,7 +313,8 @@ header. Most sections are sent only when the packfile is sent. output = acknowledgements flush-pkt | [acknowledgments delim-pkt] [shallow-info delim-pkt] - [wanted-refs delim-pkt] packfile flush-pkt + [wanted-refs delim-pkt] [packfile-uris delim-pkt] + packfile flush-pkt acknowledgments = PKT-LINE("acknowledgments" LF) (nak | *ack) @@ -331,6 +332,9 @@ header. Most sections are sent only when the packfile is sent. *PKT-LINE(wanted-ref LF) wanted-ref = obj-id SP refname + packfile-uris = PKT-LINE("packfile-uris" LF) *packfile-uri + packfile-uri = PKT-LINE("uri" SP *%x20-ff LF) + packfile = PKT-LINE("packfile" LF) *PKT-LINE(%x01-03 *%x00-ff) From patchwork Mon Dec 3 23:37:36 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10710923 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id CAE3D14E2 for ; Mon, 3 Dec 2018 23:37:53 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id BC87C29267 for ; Mon, 3 Dec 2018 23:37:53 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id B10AA2B222; Mon, 3 Dec 2018 23:37:53 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4ABC729267 for ; Mon, 3 Dec 2018 23:37:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726081AbeLCXhw (ORCPT ); Mon, 3 Dec 2018 18:37:52 -0500 Received: from mail-it1-f201.google.com ([209.85.166.201]:43377 "EHLO mail-it1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725938AbeLCXhv (ORCPT ); Mon, 3 Dec 2018 18:37:51 -0500 Received: by mail-it1-f201.google.com with SMTP id 128so5531464itw.8 for ; Mon, 03 Dec 2018 15:37:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=q75k8oHP/P+PZ7h+Wn/kvgU2W+N4gR3oumZ6U473kyw=; b=V6HsUj3/aDo6n7OCajjQazMsaMsPh04NO8HJxs6ed4cNLpHao/UOvdcvPQClhQbpu6 9gLrkRNb9mUU+MD32nntFNNQ3vLKWTwDOw9hDpcTKY3CvJq6+Qs+4Xr8u/zmO/N9MyJW qXYJobXbj+2cENmENC9nFksYH+0fIjApv1ZnQz8vGwLcswKH6thbvNVzQWP+KbAzPxZP 2plhMwehO7zLoqaV/zCDDeORakp4MwqdvtI8+ADQEV1VF2MJpDKGFfEI5y0ZbOcZCnG8 auM5oFoaHehFDG0/50af7XK146JtPLFyRcSgYzo6EGgDbwIgsU4FKikSWE05o0NSjM1d 5law== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=q75k8oHP/P+PZ7h+Wn/kvgU2W+N4gR3oumZ6U473kyw=; b=GdhW1zkwuUXG1Bxv5OWmYsgHk5hYxIiR4bXT96NIN/Dxz0YRZeWQB1KXgtE73ewkdR BkLx13WEc3CmAb2sn31IWE/E5TZgFTx2gU8rlMkKzNlnomWtonRKLSkSMpfvrJ+keCdl NV1+IsZs19FAHzmCViiDXG2+d6nPjOtPrXhXq4LKYQgzGMqd3YJyyLDwLto/jwLQza0S BgszfDIKtQwmRhr+xEwAkaEKNCcvz9JFG7N2Ip+RhpJH2jwBgY66262ANbAY+0lNomwz y6ByCdWC7eXozIAzyuIX8J0ssIZxFxphN9mvu1vQYjaBBTyQxWgdIj9z1SpGvM7dA4sH NKEw== X-Gm-Message-State: AA+aEWZxOKOjUeZi5M8Z7ul0h//Hus3w3nvGptdghwiDJ2mC68XWBQQ6 1uMKmR8NFqe2tqCLvpI7wRlaEPSKypCvAifiFDnxbXFOfPOHudPZEucXsplAG00wS9i/UlfTwZ0 juTuskZIPzKoA5vm9YtB5Uy7r8tal4TPscpZI8W6NHQIT5uwDNM14IUnuacuE0g1nMbIqAO22QO Yp X-Google-Smtp-Source: AFSGD/WWUQiEYbgtS2r/6AuEeKTOAW0nFt/M9S8S2hC9f1Xo1CMeVbt4CCACC/iE7NAI+XslCuW+iG+ayTO8kapT8RKD X-Received: by 2002:a24:16d3:: with SMTP id a202mr8229288ita.21.1543880270540; Mon, 03 Dec 2018 15:37:50 -0800 (PST) Date: Mon, 3 Dec 2018 15:37:36 -0800 In-Reply-To: Message-Id: Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [WIP RFC 3/5] upload-pack: refactor reading of pack-objects out From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Subsequent patches will change how the output of pack-objects is processed, so extract that processing into its own function. Currently, at most 1 character can be buffered (in the "buffered" local variable). One of those patches will require a larger buffer, so replace that "buffered" local variable with a buffer array. Signed-off-by: Jonathan Tan --- upload-pack.c | 80 +++++++++++++++++++++++++++++---------------------- 1 file changed, 46 insertions(+), 34 deletions(-) diff --git a/upload-pack.c b/upload-pack.c index 5e81f1ff24..cec43e0a46 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -101,14 +101,51 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data) return 0; } +struct output_state { + char buffer[8193]; + int used; +}; + +static int read_pack_objects_stdout(int outfd, struct output_state *os) +{ + /* Data ready; we keep the last byte to ourselves + * in case we detect broken rev-list, so that we + * can leave the stream corrupted. This is + * unfortunate -- unpack-objects would happily + * accept a valid packdata with trailing garbage, + * so appending garbage after we pass all the + * pack data is not good enough to signal + * breakage to downstream. + */ + ssize_t readsz; + + readsz = xread(outfd, os->buffer + os->used, + sizeof(os->buffer) - os->used); + if (readsz < 0) { + return readsz; + } + os->used += readsz; + + if (os->used > 1) { + send_client_data(1, os->buffer, os->used - 1); + os->buffer[0] = os->buffer[os->used - 1]; + os->used = 1; + } else { + send_client_data(1, os->buffer, os->used); + os->used = 0; + } + + return readsz; +} + static void create_pack_file(const struct object_array *have_obj, const struct object_array *want_obj) { struct child_process pack_objects = CHILD_PROCESS_INIT; - char data[8193], progress[128]; + struct output_state output_state = {0}; + char progress[128]; char abort_msg[] = "aborting due to possible repository " "corruption on the remote side."; - int buffered = -1; ssize_t sz; int i; FILE *pipe_fd; @@ -235,39 +272,15 @@ static void create_pack_file(const struct object_array *have_obj, continue; } if (0 <= pu && (pfd[pu].revents & (POLLIN|POLLHUP))) { - /* Data ready; we keep the last byte to ourselves - * in case we detect broken rev-list, so that we - * can leave the stream corrupted. This is - * unfortunate -- unpack-objects would happily - * accept a valid packdata with trailing garbage, - * so appending garbage after we pass all the - * pack data is not good enough to signal - * breakage to downstream. - */ - char *cp = data; - ssize_t outsz = 0; - if (0 <= buffered) { - *cp++ = buffered; - outsz++; - } - sz = xread(pack_objects.out, cp, - sizeof(data) - outsz); - if (0 < sz) - ; - else if (sz == 0) { + int result = read_pack_objects_stdout(pack_objects.out, + &output_state); + + if (result == 0) { close(pack_objects.out); pack_objects.out = -1; - } - else + } else if (result < 0) { goto fail; - sz += outsz; - if (1 < sz) { - buffered = data[sz-1] & 0xFF; - sz--; } - else - buffered = -1; - send_client_data(1, data, sz); } /* @@ -292,9 +305,8 @@ static void create_pack_file(const struct object_array *have_obj, } /* flush the data */ - if (0 <= buffered) { - data[0] = buffered; - send_client_data(1, data, 1); + if (output_state.used > 0) { + send_client_data(1, output_state.buffer, output_state.used); fprintf(stderr, "flushed.\n"); } if (use_sideband) From patchwork Mon Dec 3 23:37:37 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10710925 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 12D4414E2 for ; Mon, 3 Dec 2018 23:37:56 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 0201329267 for ; Mon, 3 Dec 2018 23:37:56 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id E844F2B222; Mon, 3 Dec 2018 23:37:55 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 78C6B29267 for ; Mon, 3 Dec 2018 23:37:55 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726088AbeLCXhy (ORCPT ); Mon, 3 Dec 2018 18:37:54 -0500 Received: from mail-qk1-f201.google.com ([209.85.222.201]:33327 "EHLO mail-qk1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725938AbeLCXhx (ORCPT ); Mon, 3 Dec 2018 18:37:53 -0500 Received: by mail-qk1-f201.google.com with SMTP id 80so14994090qkd.0 for ; Mon, 03 Dec 2018 15:37:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=IZP3L7PtOa1HXrhgi5ZlMpI0/opEBv1MrL1tjcrgHoA=; b=VjT0NJz/FYY9O9Usg8VQhXTJ1ELFq4tmkc3rYm83dqvYK7VTSZPcVrijzgiy0qf8Y+ M08E1i6xEGpaY1Wb8N0PWGkz4rIttSlD9Db5B0zdRyahuplVGPWNdgozPAnxVarVSRti qKjx5yW6MNM3THvDrCtnovVP6icJHOdCNOX33uaEbesZF1zagVjwpAcqu59AnSA2ippQ SqQJlXuFHWygvrAfuV2liyLl8tVC34QAK33Pm6SOvgj8a2xYZCigkR1hngkWa9A5L2UZ /mEDE+DxMwZfm+d07XkG3Kix4MTf3ZkK1xBdz6DXnd8WfuEYa5Z1TniIre5SNv6l+dPv QR+Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=IZP3L7PtOa1HXrhgi5ZlMpI0/opEBv1MrL1tjcrgHoA=; b=nRJ7ySWHUV9jOWCUcnnDRB/RbfPX2aVO9bjaamZ1UgT6O421M5KJqyLT4ILY23r4nW H2uE5pvdEHcqW6POAf3FlYV4p9TfbawTLLsjiGrsn+GNnoGUqq1FnckCLbYJKC+BB2iF 6SERxi5JEJfEg9xcdxKyJ5KqupVwDxLcWjrQz4OYO31s6xjGaKKBP9Jf9y+vD8B5kMAo fM7A9Wjvjd18ZKZr07ZMZMaoBMbYaMsXuZvOET8AzZrNJDAPxf/v7sQaS5seT1FTeGnV 5jgxC7DvrH+YxTk1vX9Vx0Qj/shd413C+U9+oXAKgcWViGKx3mN1UIJiB6bdgYfeTwSG Rg6A== X-Gm-Message-State: AA+aEWaFa8HbBqrdnPjBfLT6Gcr4+WiNgT2+cPmz5PtmqZ+xUYuPlx07 AhX63ufAk0H5xrVju4Wqds13bnC/2uZNE8Ede5TdbtYKMeOuZOOceKEINWeyCFVhpmLwEYf/dmE +348dA/BeIiCN49L4mVD2bzJJexuxAnaxXKiQXe3gnG5vFuwMgzZYYmxBuAWjDWJ/R9cjr/UG8S t6 X-Google-Smtp-Source: AFSGD/U4dnu5qOeXRfSR11BiCxqeZjRgAOnaFSwxwOOAuwP9tFKQZlhQZkWr5Zf/mCMP7bdvu5hPegxjprOJLQLtXlgq X-Received: by 2002:a0c:981c:: with SMTP id c28mr14146631qvd.51.1543880272822; Mon, 03 Dec 2018 15:37:52 -0800 (PST) Date: Mon, 3 Dec 2018 15:37:37 -0800 In-Reply-To: Message-Id: <1d678a78a63b7e58988b52c8c0efab38c34a6848.1543879256.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [WIP RFC 4/5] upload-pack: refactor writing of "packfile" line From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP A subsequent patch allows pack-objects to output additional information (in addition to the packfile that it currently outputs). This means that we must hold off on writing the "packfile" section header to the client before we process the output of pack-objects, so move the writing of the "packfile" section header to read_pack_objects_stdout(). Unfortunately, this also means that we cannot send keepalive packets until pack-objects starts sending out the packfile, since the sideband is only established when the "packfile" section header is sent. Signed-off-by: Jonathan Tan --- upload-pack.c | 47 ++++++++++++++++++++++++++++++++++++----------- 1 file changed, 36 insertions(+), 11 deletions(-) diff --git a/upload-pack.c b/upload-pack.c index cec43e0a46..aa2589b858 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -104,9 +104,12 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data) struct output_state { char buffer[8193]; int used; + unsigned packfile_started : 1; + struct strbuf progress_buf; }; -static int read_pack_objects_stdout(int outfd, struct output_state *os) +static int read_pack_objects_stdout(int outfd, struct output_state *os, + int use_protocol_v2) { /* Data ready; we keep the last byte to ourselves * in case we detect broken rev-list, so that we @@ -126,6 +129,12 @@ static int read_pack_objects_stdout(int outfd, struct output_state *os) } os->used += readsz; + if (!os->packfile_started) { + os->packfile_started = 1; + if (use_protocol_v2) + packet_write_fmt(1, "packfile\n"); + } + if (os->used > 1) { send_client_data(1, os->buffer, os->used - 1); os->buffer[0] = os->buffer[os->used - 1]; @@ -138,8 +147,17 @@ static int read_pack_objects_stdout(int outfd, struct output_state *os) return readsz; } +static void flush_progress_buf(struct strbuf *progress_buf) +{ + if (!progress_buf->len) + return; + send_client_data(2, progress_buf->buf, progress_buf->len); + strbuf_reset(progress_buf); +} + static void create_pack_file(const struct object_array *have_obj, - const struct object_array *want_obj) + const struct object_array *want_obj, + int use_protocol_v2) { struct child_process pack_objects = CHILD_PROCESS_INIT; struct output_state output_state = {0}; @@ -260,9 +278,13 @@ static void create_pack_file(const struct object_array *have_obj, */ sz = xread(pack_objects.err, progress, sizeof(progress)); - if (0 < sz) - send_client_data(2, progress, sz); - else if (sz == 0) { + if (0 < sz) { + if (output_state.packfile_started) + send_client_data(2, progress, sz); + else + strbuf_add(&output_state.progress_buf, + progress, sz); + } else if (sz == 0) { close(pack_objects.err); pack_objects.err = -1; } @@ -273,8 +295,10 @@ static void create_pack_file(const struct object_array *have_obj, } if (0 <= pu && (pfd[pu].revents & (POLLIN|POLLHUP))) { int result = read_pack_objects_stdout(pack_objects.out, - &output_state); - + &output_state, + use_protocol_v2); + if (output_state.packfile_started) + flush_progress_buf(&output_state.progress_buf); if (result == 0) { close(pack_objects.out); pack_objects.out = -1; @@ -293,12 +317,14 @@ static void create_pack_file(const struct object_array *have_obj, * protocol to say anything, so those clients are just out of * luck. */ - if (!ret && use_sideband) { + if (!ret && use_sideband && output_state.packfile_started) { static const char buf[] = "0005\1"; write_or_die(1, buf, 5); } } + flush_progress_buf(&output_state.progress_buf); + if (finish_command(&pack_objects)) { error("git upload-pack: git-pack-objects died with error."); goto fail; @@ -1094,7 +1120,7 @@ void upload_pack(struct upload_pack_options *options) if (want_obj.nr) { struct object_array have_obj = OBJECT_ARRAY_INIT; get_common_commits(&have_obj, &want_obj); - create_pack_file(&have_obj, &want_obj); + create_pack_file(&have_obj, &want_obj, 0); } } @@ -1475,8 +1501,7 @@ int upload_pack_v2(struct repository *r, struct argv_array *keys, send_wanted_ref_info(&data); send_shallow_info(&data, &want_obj); - packet_write_fmt(1, "packfile\n"); - create_pack_file(&have_obj, &want_obj); + create_pack_file(&have_obj, &want_obj, 1); state = FETCH_DONE; break; case FETCH_DONE: From patchwork Mon Dec 3 23:37:38 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jonathan Tan X-Patchwork-Id: 10710927 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 5E70F14E2 for ; Mon, 3 Dec 2018 23:38:03 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 4F5A229267 for ; Mon, 3 Dec 2018 23:38:03 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 43A202B222; Mon, 3 Dec 2018 23:38:03 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.5 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_HI, USER_IN_DEF_DKIM_WL autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 8EBD529267 for ; Mon, 3 Dec 2018 23:38:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726098AbeLCXh4 (ORCPT ); Mon, 3 Dec 2018 18:37:56 -0500 Received: from mail-oi1-f201.google.com ([209.85.167.201]:53621 "EHLO mail-oi1-f201.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725938AbeLCXh4 (ORCPT ); Mon, 3 Dec 2018 18:37:56 -0500 Received: by mail-oi1-f201.google.com with SMTP id w128so9311380oie.20 for ; Mon, 03 Dec 2018 15:37:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:in-reply-to:message-id:mime-version:references:subject:from:to :cc; bh=zVdl3kFjBloHM1xxX+yiP/ndfBY95o+7TaWwx9wskf0=; b=Mn3j1WvOvpSHwcEqX1ekblQ58U3EaOiiMdtd17JeBQGdSQRMSCjJEi0e3QBeWEtcRj LtnTyrNEnIwDubwPSMIM4zgoviW496LwQyAdGYp4bRjYM7sbd6VPrQOxTvb0eiwxDtwo AMioVs8777gr8AqmpRN5VeXznY39/EB7KsZKqK4MWRD6b1nxu4PecvlOgAfMKvnn3Q8Y D7C44JelplJRj9arwPDxEOXhL9gPgwRG3bkb2J7MypYnft+2pYbaFkfhint/sTjLqUkC 15qKUdDXyS/zG5ujgaHtsqw4fh/sfGdi6dy7s88kOATjNSOg7lcnMSPal4ETstD35Rg4 /Zxg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:in-reply-to:message-id:mime-version :references:subject:from:to:cc; bh=zVdl3kFjBloHM1xxX+yiP/ndfBY95o+7TaWwx9wskf0=; b=qdpvvzbrDj9gTXX/z7QpyZ1W37fSEMLSwM2NMcVV9JXUQGe01f0kCznqia+n97UoyT uWTdGatHCHpfH/abWwy6AH1fSMHJUy3msKIykj15tfqQ15ohARIshBJEJwh2X67vNOqm T274UOyM5TwU+eWoZRXpb4zDmvWAB63zbQRXFD++uFAX60u/bUxv1a1FJk7v8LdYByZ7 6J+pFR863FZzCP7C44Zh7sJUAQ2H4mRVgqrevIG6fQyP4wNqz0b/TwwYLWh/KNFe/2nv 4TyoFc17bp7DaFXpsjMWuOScLnqHpUSSqh3Rx7O0lQGUstSJYJ0tln73f3yMMNXqp37k bqdg== X-Gm-Message-State: AA+aEWYTcjNhr2/JyIDj8pi6oDAq/AUIceWW+KNedLkQklBS8AUhA5L3 +/KLmr4w3iBfszYkpPJTsFS7AEtkt/TnGkuZepjCTlPrHAyD2tFBDknUxX/5CbrYsiQtxn/lAgf yMPBIEvrdXvqhKnivoHJs17mnefQuA69RoPOnrnlOFfb43csr6oJ1USlwphi397y7lgR4fhe3wS ou X-Google-Smtp-Source: AFSGD/Wqn42BNs3DS4QNaTKitknt0QDt0IjUA0dz/oFgVekI+ywUbLZJlVivc4uF3JlhPtLfrW0PHuYu5tAqhzAiJETg X-Received: by 2002:aca:ec82:: with SMTP id k124mr14642973oih.36.1543880275027; Mon, 03 Dec 2018 15:37:55 -0800 (PST) Date: Mon, 3 Dec 2018 15:37:38 -0800 In-Reply-To: Message-Id: <707a8181a0e8cd2e21989fa6da59cee38d9fadde.1543879256.git.jonathantanmy@google.com> Mime-Version: 1.0 References: X-Mailer: git-send-email 2.19.0.271.gfe8321ec05.dirty Subject: [WIP RFC 5/5] upload-pack: send part of packfile response as uri From: Jonathan Tan To: git@vger.kernel.org Cc: Jonathan Tan Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This is a partial implementation of upload-pack sending part of its packfile response as URIs. The client is not fully implemented - it knows to ignore the "packfile-uris" section, but because it does not actually fetch those URIs, the returned packfile is incomplete. A test is included to show that the appropriate URI is indeed transmitted, and that the returned packfile is lacking exactly the expected object. Signed-off-by: Jonathan Tan --- builtin/pack-objects.c | 48 ++++++++++++++++++++++++++++++++++++++++++ fetch-pack.c | 9 ++++++++ t/t5702-protocol-v2.sh | 25 ++++++++++++++++++++++ upload-pack.c | 37 ++++++++++++++++++++++++++++---- 4 files changed, 115 insertions(+), 4 deletions(-) diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c index e7ea206c08..2abbddd3cb 100644 --- a/builtin/pack-objects.c +++ b/builtin/pack-objects.c @@ -117,6 +117,15 @@ enum missing_action { static enum missing_action arg_missing_action; static show_object_fn fn_show_object; +struct configured_exclusion { + struct oidmap_entry e; + char *uri; +}; +static struct oidmap configured_exclusions; + +static int exclude_configured_blobs; +static struct oidset excluded_by_config; + /* * stats */ @@ -831,6 +840,23 @@ static off_t write_reused_pack(struct hashfile *f) return reuse_packfile_offset - sizeof(struct pack_header); } +static void write_excluded_by_configs(void) +{ + struct oidset_iter iter; + const struct object_id *oid; + + oidset_iter_init(&excluded_by_config, &iter); + while ((oid = oidset_iter_next(&iter))) { + struct configured_exclusion *ex = + oidmap_get(&configured_exclusions, oid); + + if (!ex) + BUG("configured exclusion wasn't configured"); + write_in_full(1, ex->uri, strlen(ex->uri)); + write_in_full(1, "\n", 1); + } +} + static const char no_split_warning[] = N_( "disabling bitmap writing, packs are split due to pack.packSizeLimit" ); @@ -1124,6 +1150,12 @@ static int want_object_in_pack(const struct object_id *oid, } } + if (exclude_configured_blobs && + oidmap_get(&configured_exclusions, oid)) { + oidset_insert(&excluded_by_config, oid); + return 0; + } + return 1; } @@ -2728,6 +2760,19 @@ static int git_pack_config(const char *k, const char *v, void *cb) pack_idx_opts.version); return 0; } + if (!strcmp(k, "uploadpack.blobpackfileuri")) { + struct configured_exclusion *ex = xmalloc(sizeof(*ex)); + const char *end; + + if (parse_oid_hex(v, &ex->e.oid, &end) || *end != ' ') + die(_("value of uploadpack.blobpackfileuri must be " + "of the form ' ' (got '%s')"), v); + if (oidmap_get(&configured_exclusions, &ex->e.oid)) + die(_("object already configured in another " + "uploadpack.blobpackfileuri (got '%s')"), v); + ex->uri = xstrdup(end + 1); + oidmap_put(&configured_exclusions, ex); + } return git_default_config(k, v, cb); } @@ -3314,6 +3359,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) N_("do not pack objects in promisor packfiles")), OPT_BOOL(0, "delta-islands", &use_delta_islands, N_("respect islands during delta compression")), + OPT_BOOL(0, "exclude-configured-blobs", &exclude_configured_blobs, + N_("respect uploadpack.blobpackfileuri")), OPT_END(), }; @@ -3487,6 +3534,7 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix) return 0; if (nr_result) prepare_pack(window, depth); + write_excluded_by_configs(); write_pack_file(); if (progress) fprintf_ln(stderr, diff --git a/fetch-pack.c b/fetch-pack.c index 9691046e64..6e1985ab55 100644 --- a/fetch-pack.c +++ b/fetch-pack.c @@ -1413,6 +1413,15 @@ static struct ref *do_fetch_pack_v2(struct fetch_pack_args *args, receive_wanted_refs(&reader, sought, nr_sought); /* get the pack */ + if (process_section_header(&reader, "packfile-uris", 1)) { + /* skip the whole section */ + process_section_header(&reader, "packfile-uris", 0); + while (packet_reader_read(&reader) == PACKET_READ_NORMAL) { + /* do nothing */ + } + if (reader.status != PACKET_READ_DELIM) + die("expected DELIM"); + } process_section_header(&reader, "packfile", 0); if (get_pack(args, fd, pack_lockfile)) die(_("git fetch-pack: fetch failed.")); diff --git a/t/t5702-protocol-v2.sh b/t/t5702-protocol-v2.sh index 0f2b09ebb8..ccb1fc510e 100755 --- a/t/t5702-protocol-v2.sh +++ b/t/t5702-protocol-v2.sh @@ -588,6 +588,31 @@ test_expect_success 'when server does not send "ready", expect FLUSH' ' test_i18ngrep "expected no other sections to be sent after no .ready." err ' +test_expect_success 'part of packfile response provided as URI' ' + rm -rf "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" http_child log && + + git init "$HTTPD_DOCUMENT_ROOT_PATH/http_parent" && + echo my-blob >"$HTTPD_DOCUMENT_ROOT_PATH/http_parent/my-blob" && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent/" add my-blob && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent/" commit -m x && + + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent/" hash-object my-blob >h && + git -C "$HTTPD_DOCUMENT_ROOT_PATH/http_parent/" config \ + "uploadpack.blobpackfileuri" \ + "$(cat h) https://example.com/a-uri" && + + # NEEDSWORK: "git clone" fails here because it ignores the URI provided + # instead of fetching it. + test_must_fail env GIT_TRACE_PACKET="$(pwd)/log" \ + git -c protocol.version=2 clone \ + "$HTTPD_URL/smart/http_parent" http_child 2>err && + # Although "git clone" fails, we can still check that the server + # provided the URI we requested and that the error message pinpoints + # the object that is missing. + grep "clone< uri https://example.com/a-uri" log && + test_i18ngrep "did not receive expected object $(cat h)" err +' + stop_httpd test_done diff --git a/upload-pack.c b/upload-pack.c index aa2589b858..a8eef697ec 100644 --- a/upload-pack.c +++ b/upload-pack.c @@ -104,6 +104,7 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data) struct output_state { char buffer[8193]; int used; + unsigned packfile_uris_started : 1; unsigned packfile_started : 1; struct strbuf progress_buf; }; @@ -129,10 +130,35 @@ static int read_pack_objects_stdout(int outfd, struct output_state *os, } os->used += readsz; - if (!os->packfile_started) { - os->packfile_started = 1; - if (use_protocol_v2) - packet_write_fmt(1, "packfile\n"); + while (!os->packfile_started) { + char *p; + if (os->used >= 4 && !memcmp(os->buffer, "PACK", 4)) { + os->packfile_started = 1; + if (use_protocol_v2) { + if (os->packfile_uris_started) + packet_delim(1); + packet_write_fmt(1, "packfile\n"); + } + break; + } + if ((p = memchr(os->buffer, '\n', os->used))) { + if (!os->packfile_uris_started) { + os->packfile_uris_started = 1; + if (!use_protocol_v2) + BUG("packfile_uris requires protocol v2"); + packet_write_fmt(1, "packfile-uris\n"); + } + *p = '\0'; + packet_write_fmt(1, "uri %s\n", os->buffer); + + os->used -= p - os->buffer + 1; + memmove(os->buffer, p, os->used); + } else { + /* + * Incomplete line. + */ + return readsz; + } } if (os->used > 1) { @@ -205,6 +231,9 @@ static void create_pack_file(const struct object_array *have_obj, filter_options.filter_spec); } } + if (use_protocol_v2) + argv_array_push(&pack_objects.args, + "--exclude-configured-blobs"); pack_objects.in = -1; pack_objects.out = -1;