From patchwork Fri Dec 13 23:52:35 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291967 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 173F3930 for ; Fri, 13 Dec 2019 23:53:23 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id EA55622527 for ; Fri, 13 Dec 2019 23:53:22 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="F94rruDa" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726642AbfLMXxV (ORCPT ); Fri, 13 Dec 2019 18:53:21 -0500 Received: from mail-pg1-f170.google.com ([209.85.215.170]:34486 "EHLO mail-pg1-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725747AbfLMXxV (ORCPT ); Fri, 13 Dec 2019 18:53:21 -0500 Received: by mail-pg1-f170.google.com with SMTP id r11so262556pgf.1 for ; Fri, 13 Dec 2019 15:53:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=DvNLEATWQXlU2p9oD9wdCuxAkVUEQK8+E+1dqDjbujM=; b=F94rruDaiecLIgZOUqOlC2WB2W8s4UJVGR6z1ENWUxs9GZD3wkOhvIGLTRhQk1Vx8O DrOD+Ty6pnu3x1Lts4s7oFtPrQc80MfePjEsp2lS91wKezsKhY5tIG0oc3qgse4e9OZ7 vcFxg09lptN4AhprJfCEtaTlLd5hB/TBZi4rRgNudfkFd1SSqtnypNVcMn6rJOEb30Bn j5cwzIY1m6IJ5FvPMn6i4TxxliziOZDXbswzuCOtfxksY2/UaeJEOFP8WWIJ0LrUULGW 2sIfKel4RnKUcnQAQLkH77Zdqtz1pOPhhvt+0GlZXYlZkYInV9tFaVRoSYuyGA6WtZi3 ncRQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=DvNLEATWQXlU2p9oD9wdCuxAkVUEQK8+E+1dqDjbujM=; b=RDimt2bp5tvQ6hkR5t7cD10dagmlPufnCkpyhFwi2/9TdVH+vEIKX5JI5fWlQXOcOC hSDYn3Cpx1tFIJ19vZdJolFBZVh2vJHacQmkLWDygiBAgjhOOAGmmJH2lGoz/NeUUcS1 MbyIEMJUsVamw7PY2dDUoVaIkby4mpodKxyivfbRyLfcwV9RPpnwECISaPbxlc8e/X9+ Ax1FkhMmYT2WCLxB4GHYs0PJZYjxTE6w4E2ggJQjtGFc7UDAZIIUpg8Vd+WZLFn0cHjF V47KwwfZ3LXNhrioq96n1ot7SI1eKBsGLLkZxkGgWT4mnky/AKCHwm+yScCMrLTONwfy HGMA== X-Gm-Message-State: APjAAAVXApa3lssOhLtCfHV30ca/fCmXIwzLxQ+30Jw3oRU0/dmqJl97 IS9yUwxj4dqpN4eDPNufCKpLqvMNqLXUqqOW X-Google-Smtp-Source: APXvYqwF6A2USGfZxod6h/tJVGK51j9+I3NR+gn6ZeWTdavTYEUOnFgcuVJcbDf3eURC0I737eD8ow== X-Received: by 2002:a62:154:: with SMTP id 81mr2466275pfb.126.1576281200053; Fri, 13 Dec 2019 15:53:20 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:19 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 01/14] git-p4: make python2.7 the oldest supported version Date: Fri, 13 Dec 2019 15:52:35 -0800 Message-Id: <20191213235247.23660-3-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Python2.6 and earlier have been end-of-life'd for many years now, and we actually already use 2.7-only features in the code. Make the version check reflect current realities. This also removes the need to explicitly define CalledProcessError if it's not available. Signed-off-by: Yang Zhao --- git-p4.py | 20 ++------------------ 1 file changed, 2 insertions(+), 18 deletions(-) diff --git a/git-p4.py b/git-p4.py index 60c73b6a37..37777bb9fd 100755 --- a/git-p4.py +++ b/git-p4.py @@ -8,9 +8,8 @@ # License: MIT # import sys -if sys.hexversion < 0x02040000: - # The limiter is the subprocess module - sys.stderr.write("git-p4: requires Python 2.4 or later.\n") +if sys.version_info.major < 3 and sys.version_info.minor < 7: + sys.stderr.write("git-p4: requires Python 2.7 or later.\n") sys.exit(1) import os import optparse @@ -43,21 +42,6 @@ bytes = str basestring = basestring -try: - from subprocess import CalledProcessError -except ImportError: - # from python2.7:subprocess.py - # Exception classes used by this module. - class CalledProcessError(Exception): - """This exception is raised when a process run by check_call() returns - a non-zero exit status. The exit status will be stored in the - returncode attribute.""" - def __init__(self, returncode, cmd): - self.returncode = returncode - self.cmd = cmd - def __str__(self): - return "Command '%s' returned non-zero exit status %d" % (self.cmd, self.returncode) - verbose = False # Only labels/tags matching this will be imported/exported From patchwork Fri Dec 13 23:52:36 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291971 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AA47D1593 for ; Fri, 13 Dec 2019 23:53:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7FADB20724 for ; Fri, 13 Dec 2019 23:53:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="s4nG4MyK" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726708AbfLMXxX (ORCPT ); Fri, 13 Dec 2019 18:53:23 -0500 Received: from mail-pl1-f196.google.com ([209.85.214.196]:42342 "EHLO mail-pl1-f196.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725747AbfLMXxW (ORCPT ); Fri, 13 Dec 2019 18:53:22 -0500 Received: by mail-pl1-f196.google.com with SMTP id x13so1882027plr.9 for ; Fri, 13 Dec 2019 15:53:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ecnvCqyfkZcd7TihBAMzcdaLrOm3d+1fccnRmDnk5Lk=; b=s4nG4MyKtIBQCSP/U+yeLk+kYJ8wj5hSlWJfOjhlv3Wt0P6tQeLzIJB2LkQZ8pWAtL tG0sN4dejCZqUQfnBOcaGdFdDgrJmKwDNxxvEcfhxfQ0ags8cgaCXCaNqbgshAS3pjpK KE08KOb4k0ThgHvmI8NdroGDRrSz/FYA2Z8Zxs0FfTHg0OF8i6eWyR0CIV7GETk6lg0q poLcbfdAj3thtd/ueuZFRbN53OWMXoW2ySBORUl/LVa1T1u3Y+4xwSDmy71mBVcNerGg UEz73XiEuZPCReKIC5KXBqB5C9RR8Cs4Sf++igXE28fBaZwW78HLAHHg8Z7X9MpfAD04 RpgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ecnvCqyfkZcd7TihBAMzcdaLrOm3d+1fccnRmDnk5Lk=; b=LW4sQaM5uVTIJ8rW6CaQXwqhk3GQje8CuEybICp+nEV6sfIOYLQwYvoa6wSgF5zVAl H9SO/ynhLmg+fEjnD8274NrZqGc8TWHyjBJNEbFDzoyp7bRFik6iRlFdigwKfRJARZUu cWswDN2ryKDNlJE++CNhSwtZ+ZPmZjby5ftejyp5nWocDcEyeMD6RADyFY/2WRX+630G TFVtfdx+Y3WfawotYtrdt4SoqBctEo4cVB7mSjbxvy3JOOfTyuH0xZKSnS8fu5yLTNDV IQu+DADOvp1AeBCcCvWAkJ2iZNap3gvDn1jHUCLwtIs0bhLQWGF6m9IdiIXZPsiTM22+ 4Izw== X-Gm-Message-State: APjAAAVQLAxGn7UsJFmeTLk0oY2aPqQMMX/PCKM9rAiEK2YXW51ceSOg VnNrs5cQ17DSbfrsn1P2Yvbzxz03RRkhmmbw X-Google-Smtp-Source: APXvYqyxY8J/vklTD+1PT94znKD6EsrRrXTPvsCLIAcMxJTMB9T4IrRVpslNla+lZqEev3aG64Cyuw== X-Received: by 2002:a17:90a:1b6b:: with SMTP id q98mr2509216pjq.106.1576281201183; Fri, 13 Dec 2019 15:53:21 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:20 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Ben Keene , luke@diamand.org, liu.denton@gmail.com, Yang Zhao Subject: [PATCH v2 02/14] git-p4: change the expansion test from basestring to list Date: Fri, 13 Dec 2019 15:52:36 -0800 Message-Id: <20191213235247.23660-4-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org From: Ben Keene Python 3 handles strings differently than Python 2.7. Since Python 2 is reaching it's end of life, a series of changes are being submitted to enable python 3.5 and following support. The current code fails basic tests under python 3.5. Some codepaths can represent a command line the program internally prepares to execute either as a single string (i.e. each token properly quoted, concatenated with $IFS) or as a list of argv[] elements, and there are 9 places where we say "if X is isinstance(_, basestring), then do this thing to handle X as a command line in a single string; if not, X is a command line in a list form". This does not work well with Python 3, as there is no basestring (everything is Unicode now), and even with Python 2, it was not an ideal way to tell the two cases apart, because an internally formed command line could have been in a single Unicode string. Flip the check to say "if X is not a list, then handle X as a command line in a single string; otherwise treat it as a command line in a list form". This will get rid of references to 'basestring', to migrate the code ready for Python 3. Thanks-to: Junio C Hamano Signed-off-by: Ben Keene Signed-off-by: Yang Zhao --- git-p4.py | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/git-p4.py b/git-p4.py index 37777bb9fd..2f177fb43b 100755 --- a/git-p4.py +++ b/git-p4.py @@ -89,7 +89,7 @@ def p4_build_cmd(cmd): # Provide a way to not pass this option by setting git-p4.retries to 0 real_cmd += ["-r", str(retries)] - if isinstance(cmd,basestring): + if not isinstance(cmd, list): real_cmd = ' '.join(real_cmd) + ' ' + cmd else: real_cmd += cmd @@ -155,7 +155,7 @@ def write_pipe(c, stdin): if verbose: sys.stderr.write('Writing pipe: %s\n' % str(c)) - expand = isinstance(c,basestring) + expand = not isinstance(c, list) p = subprocess.Popen(c, stdin=subprocess.PIPE, shell=expand) pipe = p.stdin val = pipe.write(stdin) @@ -177,7 +177,7 @@ def read_pipe_full(c): if verbose: sys.stderr.write('Reading pipe: %s\n' % str(c)) - expand = isinstance(c,basestring) + expand = not isinstance(c, list) p = subprocess.Popen(c, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=expand) (out, err) = p.communicate() return (p.returncode, out, err) @@ -213,7 +213,7 @@ def read_pipe_lines(c): if verbose: sys.stderr.write('Reading pipe: %s\n' % str(c)) - expand = isinstance(c, basestring) + expand = not isinstance(c, list) p = subprocess.Popen(c, stdout=subprocess.PIPE, shell=expand) pipe = p.stdout val = pipe.readlines() @@ -256,7 +256,7 @@ def p4_has_move_command(): return True def system(cmd, ignore_error=False): - expand = isinstance(cmd,basestring) + expand = not isinstance(cmd, list) if verbose: sys.stderr.write("executing %s\n" % str(cmd)) retcode = subprocess.call(cmd, shell=expand) @@ -268,7 +268,7 @@ def system(cmd, ignore_error=False): def p4_system(cmd): """Specifically invoke p4 as the system command. """ real_cmd = p4_build_cmd(cmd) - expand = isinstance(real_cmd, basestring) + expand = not isinstance(real_cmd, list) retcode = subprocess.call(real_cmd, shell=expand) if retcode: raise CalledProcessError(retcode, real_cmd) @@ -506,7 +506,7 @@ def getP4OpenedType(file): # Return the set of all p4 labels def getP4Labels(depotPaths): labels = set() - if isinstance(depotPaths,basestring): + if not isinstance(depotPaths, list): depotPaths = [depotPaths] for l in p4CmdList(["labels"] + ["%s..." % p for p in depotPaths]): @@ -593,7 +593,7 @@ def isModeExecChanged(src_mode, dst_mode): def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False, errors_as_exceptions=False): - if isinstance(cmd,basestring): + if not isinstance(cmd, list): cmd = "-G " + cmd expand = True else: @@ -610,7 +610,7 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False, stdin_file = None if stdin is not None: stdin_file = tempfile.TemporaryFile(prefix='p4-stdin', mode=stdin_mode) - if isinstance(stdin,basestring): + if not isinstance(stdin, list): stdin_file.write(stdin) else: for i in stdin: From patchwork Fri Dec 13 23:52:37 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291975 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DB31A930 for ; Fri, 13 Dec 2019 23:53:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B9D6522527 for ; Fri, 13 Dec 2019 23:53:27 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="P43ZhtSa" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726757AbfLMXxZ (ORCPT ); Fri, 13 Dec 2019 18:53:25 -0500 Received: from mail-pj1-f68.google.com ([209.85.216.68]:41695 "EHLO mail-pj1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726671AbfLMXxX (ORCPT ); Fri, 13 Dec 2019 18:53:23 -0500 Received: by mail-pj1-f68.google.com with SMTP id ca19so381515pjb.8 for ; Fri, 13 Dec 2019 15:53:22 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=T6hpro0bovGWTPkEBpi6ztIjb9/FgW3Wl1ARP6O6imc=; b=P43ZhtSaUZs8q8KtIAsRN0mdYgSv+vJOD7Va7j2HKhiigY8RkLNXCccFic1l/5a/Jq pNr2p+JwkmujYXuefhdWlG1CgQUT8NPjYFh76yYHjV74rxC+6Is4p0hFBFCc4TrgYOQ1 CIypuY6ET4QAV32RKInqsruC0+myTWoyeRzCQqM3cacA0NiSR3NXleb9z1Nuod2rnU+E QUsmZBCPpnGNeTIYVDj2Rd5JUyai7ioAw7Z05mH0hth4yBbKo3n/Spx5kFiZgu0ZY/Qx puyCNLdUPCKC5K/swXqVps2O5tzmO5sBg+1skLrDEagJixu8f1DSY75sE1DwTmzmgsWZ ULUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=T6hpro0bovGWTPkEBpi6ztIjb9/FgW3Wl1ARP6O6imc=; b=h9V3QvJPBlX6BskJmwJA4qjFixKJYEAOy0V8eE1MeDQfzpc/hlqJOvYrojSdxFYv7/ gVeKVAVYn7/76PV8fiwnoj4/N/l15hYznlIXqgWU6nSSGMDBKiQlbYXMB32AM89CMoe/ uPwQPNZSO2nDr1POjTmsYgGjoGrRs9Nt+k0wGeoLosf9ipTCry6bDi3QYeCaDU8iHuT6 UikCUfzQ8IuRYWHBT0I+K+x0E2m92HQDrogh3K4NX074y07PF8RDtoOH0406lcbmSCDp gaEOfJgl9vC9+IxUH+Rj+HIIBhVu5430LowhfmXoH6BI0uqw6WQiGhH8gqfETy0CZBEb zxAQ== X-Gm-Message-State: APjAAAUPqLVrTMk8EiSTGMa99C2vCLIQgEYbcECf9J4SkNd9BD5Es8HN HKwNqXs3AV9Ikswil4e9drx4np757uj6PlKs X-Google-Smtp-Source: APXvYqwouVJLr5GjNlTFD+olQQJKbIPxFN5IKHQxscAoAVRb9mgAhJ/v6ihbHyyl1T3QU8VUKZB1sA== X-Received: by 2002:a17:902:b083:: with SMTP id p3mr2225162plr.141.1576281202237; Fri, 13 Dec 2019 15:53:22 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.21 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:21 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 03/14] git-p4: remove string type aliasing Date: Fri, 13 Dec 2019 15:52:37 -0800 Message-Id: <20191213235247.23660-5-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Now that python2.7 is the minimum required version and we no longer use the basestring type, it is not necessary to use type aliasing to ensure python3 compatibility. Signed-off-by: Yang Zhao --- git-p4.py | 16 ---------------- 1 file changed, 16 deletions(-) diff --git a/git-p4.py b/git-p4.py index 2f177fb43b..153aff16f3 100755 --- a/git-p4.py +++ b/git-p4.py @@ -26,22 +26,6 @@ import ctypes import errno -# support basestring in python3 -try: - unicode = unicode -except NameError: - # 'unicode' is undefined, must be Python 3 - str = str - unicode = str - bytes = bytes - basestring = (str,bytes) -else: - # 'unicode' exists, must be Python 2 - str = str - unicode = unicode - bytes = str - basestring = basestring - verbose = False # Only labels/tags matching this will be imported/exported From patchwork Fri Dec 13 23:52:38 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291977 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 6D7B41593 for ; Fri, 13 Dec 2019 23:53:28 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 42C6A22527 for ; Fri, 13 Dec 2019 23:53:28 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="a5HdgvMx" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726744AbfLMXxZ (ORCPT ); Fri, 13 Dec 2019 18:53:25 -0500 Received: from mail-pl1-f193.google.com ([209.85.214.193]:39274 "EHLO mail-pl1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725747AbfLMXxY (ORCPT ); Fri, 13 Dec 2019 18:53:24 -0500 Received: by mail-pl1-f193.google.com with SMTP id z3so358909plk.6 for ; Fri, 13 Dec 2019 15:53:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=tvfWooet+YCIrq4gMPJWJqSVZ9mojouy3CgJCq/Kix0=; b=a5HdgvMxqBNVWb4T1VvSrjUeD/ifDxd/9rsyywNsiM1CfL3qenrjjY3y7juLnDMeSC m3dl6WqvXk06WaJJM0CPCUVb350bx+WlkCskFA6y/vo+R+6FEo1FrEYPlEiggTIZtmTP cxr+nrnoBW/peiAY8j1Gf0KYDwMMZQHOut7YiVnLVKKP7OuVAeTBG4bnYa40aIhWObwv t03u1MmzGkFVlMaAvvGn+I+QYQitQGq/vo8hXjpq0EPBlLt+LRnxSUZkHld+2JZsO43k ygu9SmFdPmYR3+mPBBCrMM5w/g1BIs/DNcphnMPC7yATFZbxi2tAWnJzjakJVRZTfSgM qwew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=tvfWooet+YCIrq4gMPJWJqSVZ9mojouy3CgJCq/Kix0=; b=c3pvQBffetTvArdS/gCiInE2WSPU/W3RGNw+WL/cNjkWT551DPoSQpa0SnG5qPFj4t 4cEokrdJwnbPrg524syRUh9g/p3ibgP/l9IA2q1/NEd6n1QafJsrEQZ6DNlHxwyHFtZN NQHYtFpNT85UNU5RXFE5msiImX5tui3RvmQPLDLDKvRRT8YCbHpUEY01QKySEQp0Ffx7 Q7CMo/1MRScx2ZtB1rIIUthS5Pi1l2Nm3s9Q1Cw1TovkHzKTwda2Yx3cUb1AzIXWAJrT sNR7Qji+59t3XDPswR0iwsqiJDuiEJLD5plp5iSflYXiXfpEOe1sO8HDBeCTSX4MmXkb waIQ== X-Gm-Message-State: APjAAAXWzuPdBw3h8t9zF9euIm3a6BfNIYSJPzhG/Mr7PD9feAQzO1DJ 4rJhQYQop7MbGXCv0F9wZN7QQFWWkUI99J0q X-Google-Smtp-Source: APXvYqx/nGuWolvvc7AwEzymTm29X/3Tb8g1RTTSQXqm1yxZdDTIkfvqZMvRZpBWYM8r8erMC5Ondg== X-Received: by 2002:a17:902:7c88:: with SMTP id y8mr2322667pll.104.1576281203346; Fri, 13 Dec 2019 15:53:23 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.22 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:22 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 04/14] git-p4: encode/decode communication with p4 for python3 Date: Fri, 13 Dec 2019 15:52:38 -0800 Message-Id: <20191213235247.23660-6-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org The marshalled dict in the response given on STDOUT by p4 uses `str` for keys and string values. When run using python3, these values are deserialized as `bytes`, leading to a whole host of problems as the rest of the code assumes `str` is used throughout. This patch changes the deserialization behaviour such that, as much as possible, text output from p4 is decoded to native unicode strings. Exceptions are made for the field `data` as it is usually arbitrary binary data. `depotFile[0-9]*`, `path`, and `clientFile` are also exempt as they contain path strings not encoded with UTF-8, and must survive survive round-trip back to p4. Conversely, text data being piped to p4 must always be encoded when running under python3. encode_text_stream() and decode_text_stream() were added to make these transformations more convenient. Signed-off-by: Yang Zhao --- git-p4.py | 59 +++++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 46 insertions(+), 13 deletions(-) diff --git a/git-p4.py b/git-p4.py index 153aff16f3..ca891e3d5d 100755 --- a/git-p4.py +++ b/git-p4.py @@ -135,6 +135,21 @@ def die(msg): sys.stderr.write(msg + "\n") sys.exit(1) +# We need different encoding/decoding strategies for text data being passed +# around in pipes depending on python version +if bytes is not str: + # For python3, always encode and decode as appropriate + def decode_text_stream(s): + return s.decode() if isinstance(s, bytes) else s + def encode_text_stream(s): + return s.encode() if isinstance(s, str) else s +else: + # For python2.7, pass read strings as-is, but also allow writing unicode + def decode_text_stream(s): + return s + def encode_text_stream(s): + return s.encode('utf_8') if isinstance(s, unicode) else s + def write_pipe(c, stdin): if verbose: sys.stderr.write('Writing pipe: %s\n' % str(c)) @@ -151,6 +166,8 @@ def write_pipe(c, stdin): def p4_write_pipe(c, stdin): real_cmd = p4_build_cmd(c) + if bytes is not str and isinstance(stdin, str): + stdin = encode_text_stream(stdin) return write_pipe(real_cmd, stdin) def read_pipe_full(c): @@ -164,7 +181,7 @@ def read_pipe_full(c): expand = not isinstance(c, list) p = subprocess.Popen(c, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=expand) (out, err) = p.communicate() - return (p.returncode, out, err) + return (p.returncode, out, decode_text_stream(err)) def read_pipe(c, ignore_error=False): """ Read output from command. Returns the output text on @@ -187,11 +204,11 @@ def read_pipe_text(c): if retcode != 0: return None else: - return out.rstrip() + return decode_text_stream(out).rstrip() -def p4_read_pipe(c, ignore_error=False): +def p4_read_pipe(c, ignore_error=False, raw=False): real_cmd = p4_build_cmd(c) - return read_pipe(real_cmd, ignore_error) + return read_pipe(real_cmd, ignore_error, raw=raw) def read_pipe_lines(c): if verbose: @@ -200,7 +217,7 @@ def read_pipe_lines(c): expand = not isinstance(c, list) p = subprocess.Popen(c, stdout=subprocess.PIPE, shell=expand) pipe = p.stdout - val = pipe.readlines() + val = [decode_text_stream(line) for line in pipe.readlines()] if pipe.close() or p.wait(): die('Command failed: %s' % str(c)) @@ -231,6 +248,7 @@ def p4_has_move_command(): cmd = p4_build_cmd(["move", "-k", "@from", "@to"]) p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) (out, err) = p.communicate() + err = decode_text_stream(err) # return code will be 1 in either case if err.find("Invalid option") >= 0: return False @@ -611,6 +629,20 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False, try: while True: entry = marshal.load(p4.stdout) + if bytes is not str: + # Decode unmarshalled dict to use str keys and values, except for: + # - `data` which may contain arbitrary binary data + # - `depotFile[0-9]*`, `path`, or `clientFile` which may contain non-UTF8 encoded text + decoded_entry = {} + for key, value in entry.items(): + key = key.decode() + if isinstance(value, bytes) and not (key in ('data', 'path', 'clientFile') or key.startswith('depotFile')): + value = value.decode() + decoded_entry[key] = value + # Parse out data if it's an error response + if decoded_entry.get('code') == 'error' and 'data' in decoded_entry: + decoded_entry['data'] = decoded_entry['data'].decode() + entry = decoded_entry if skip_info: if 'code' in entry and entry['code'] == 'info': continue @@ -828,6 +860,7 @@ def branch_exists(branch): cmd = [ "git", "rev-parse", "--symbolic", "--verify", branch ] p = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE) out, _ = p.communicate() + out = decode_text_stream(out) if p.returncode: return False # expect exactly one line of output: the branch name @@ -1971,7 +2004,7 @@ def applyCommit(self, id): tmpFile = os.fdopen(handle, "w+b") if self.isWindows: submitTemplate = submitTemplate.replace("\n", "\r\n") - tmpFile.write(submitTemplate) + tmpFile.write(encode_text_stream(submitTemplate)) tmpFile.close() if self.prepare_p4_only: @@ -2018,7 +2051,7 @@ def applyCommit(self, id): if self.edit_template(fileName): # read the edited message and submit tmpFile = open(fileName, "rb") - message = tmpFile.read() + message = decode_text_stream(tmpFile.read()) tmpFile.close() if self.isWindows: message = message.replace("\r\n", "\n") @@ -2707,7 +2740,7 @@ def splitFilesIntoBranches(self, commit): return branches def writeToGitStream(self, gitMode, relPath, contents): - self.gitStream.write('M %s inline %s\n' % (gitMode, relPath)) + self.gitStream.write(encode_text_stream(u'M {} inline {}\n'.format(gitMode, relPath))) self.gitStream.write('data %d\n' % sum(len(d) for d in contents)) for d in contents: self.gitStream.write(d) @@ -2748,7 +2781,7 @@ def streamOneP4File(self, file, contents): git_mode = "120000" # p4 print on a symlink sometimes contains "target\n"; # if it does, remove the newline - data = ''.join(contents) + data = ''.join(decode_text_stream(c) for c in contents) if not data: # Some version of p4 allowed creating a symlink that pointed # to nothing. This causes p4 errors when checking out such @@ -2802,7 +2835,7 @@ def streamOneP4File(self, file, contents): pattern = p4_keywords_regexp_for_type(type_base, type_mods) if pattern: regexp = re.compile(pattern, re.VERBOSE) - text = ''.join(contents) + text = ''.join(decode_text_stream(c) for c in contents) text = regexp.sub(r'$\1$', text) contents = [ text ] @@ -2817,7 +2850,7 @@ def streamOneP4Deletion(self, file): if verbose: sys.stdout.write("delete %s\n" % relPath) sys.stdout.flush() - self.gitStream.write("D %s\n" % relPath) + self.gitStream.write(encode_text_stream(u'D {}\n'.format(relPath))) if self.largeFileSystem and self.largeFileSystem.isLargeFile(relPath): self.largeFileSystem.removeLargeFile(relPath) @@ -2917,9 +2950,9 @@ def streamP4FilesCbSelf(entry): if 'shelved_cl' in f: # Handle shelved CLs using the "p4 print file@=N" syntax to print # the contents - fileArg = '%s@=%d' % (f['path'], f['shelved_cl']) + fileArg = f['path'] + encode_text_stream('@={}'.format(f['shelved_cl'])) else: - fileArg = '%s#%s' % (f['path'], f['rev']) + fileArg = f['path'] + encode_text_stream('#{}'.format(f['rev'])) fileArgs.append(fileArg) From patchwork Fri Dec 13 23:52:39 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291973 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 1E75B14DB for ; Fri, 13 Dec 2019 23:53:27 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id F0A7C20724 for ; Fri, 13 Dec 2019 23:53:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="N08wuF2C" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726767AbfLMXx0 (ORCPT ); Fri, 13 Dec 2019 18:53:26 -0500 Received: from mail-pl1-f172.google.com ([209.85.214.172]:40656 "EHLO mail-pl1-f172.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726741AbfLMXxZ (ORCPT ); Fri, 13 Dec 2019 18:53:25 -0500 Received: by mail-pl1-f172.google.com with SMTP id g6so1883010plp.7 for ; Fri, 13 Dec 2019 15:53:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=a4Lo37diy/qv5OQQYgH1+e/IZeMp2Ts13inSK6KPdzc=; b=N08wuF2C2m+jBHEqvXAdimPEDx768SatkZpMIT6JtqI2OvWOhgG9K/paE7o4mzK4m5 IKRc+mJdKIdGExwP70FYN70igPkamZMlEvBuv9JoboJSDq4ey1dzZR9GhnsnhRBFUbjH h5q5VKfS6rUJXPUS3AeGK12Zygs0xjICVhhZmb/1XRs1bGoeAKgrRqE6UvvHGdCbke6g 2K2pCc1nmnHQ4XUzD8PmUg9xxJ6UPjOKMBZuddmJt3QUOAx0Zl0ziAHSj+/9UKqmJheF cJh9NA0w+Cc8dMidyEBHCm5CaQVfZ8D3vOqSZHKdz/YG7lcLuhnE6g0grjcKX/YWbonb sQmg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=a4Lo37diy/qv5OQQYgH1+e/IZeMp2Ts13inSK6KPdzc=; b=QdkiN8QcI0WeUcTnTXgF3MP1oGNKAVRdbewpHOx1D5Tm//XYY4L+Q/uzhfx9/zYdXy 0gH7uV00T5YFqLQOAGc0c1ivVPjbEiF/nFMJq+ylF7o5KgyE4rcGOKi3Vhn2e91H3K8g QlPaevq1DfMmBumCOoSmMlYnWTKqYQHK/1lY8zUgvNvroetgxb5OGJ6l0BtUhz/pNjkC mTLb/Fjv9UYlJR1KBDevH3OjHKtH1R6tafeRgac3PevuUNXfCiEavu1/RdInZeAHxXG5 uSIAklOyBeX5u4pzXVsBq8oGS37eOgMEnZOB6ui9/QvccxyiFCNnz3OaRYPuBNtXrGNH 0iRQ== X-Gm-Message-State: APjAAAX42UiGxklkUO+XbVuWXexIv96ZOZXB4R1PqFyH0MphhyBGGSHW ReZUwTjxVqEwVgk5NjHfG2X/puiXVWylzCLI X-Google-Smtp-Source: APXvYqxpsOZ/IWnJ3WRj87ah1hqmwNN2Ei2AEX1jo1qrhFpOVt09Y7j1cfHpe1vJNg9BC1akAcVYvw== X-Received: by 2002:a17:902:8688:: with SMTP id g8mr2351304plo.132.1576281204470; Fri, 13 Dec 2019 15:53:24 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.23 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:23 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 05/14] git-p4: encode/decode communication with git for python3 Date: Fri, 13 Dec 2019 15:52:39 -0800 Message-Id: <20191213235247.23660-7-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Under python3, calls to write() on the stream to `git fast-import` must be encoded. This patch wraps the IO object such that this encoding is done transparently. Conversely, any text data read from subprocesses must also be decoded before running through the rest of the pipeline. Signed-off-by: Yang Zhao Reviewed-by: Ben Keene --- git-p4.py | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/git-p4.py b/git-p4.py index ca891e3d5d..d62fb05989 100755 --- a/git-p4.py +++ b/git-p4.py @@ -183,10 +183,12 @@ def read_pipe_full(c): (out, err) = p.communicate() return (p.returncode, out, decode_text_stream(err)) -def read_pipe(c, ignore_error=False): +def read_pipe(c, ignore_error=False, raw=False): """ Read output from command. Returns the output text on success. On failure, terminates execution, unless ignore_error is True, when it returns an empty string. + + If raw is True, do not attempt to decode output text. """ (retcode, out, err) = read_pipe_full(c) if retcode != 0: @@ -194,6 +196,8 @@ def read_pipe(c, ignore_error=False): out = "" else: die('Command failed: %s\nError: %s' % (str(c), err)) + if not raw: + out = decode_text_stream(out) return out def read_pipe_text(c): @@ -220,7 +224,6 @@ def read_pipe_lines(c): val = [decode_text_stream(line) for line in pipe.readlines()] if pipe.close() or p.wait(): die('Command failed: %s' % str(c)) - return val def p4_read_pipe_lines(c): @@ -616,7 +619,8 @@ def p4CmdList(cmd, stdin=None, stdin_mode='w+b', cb=None, skip_info=False, stdin_file.write(stdin) else: for i in stdin: - stdin_file.write(i + '\n') + stdin_file.write(encode_text_stream(i)) + stdin_file.write(b'\n') stdin_file.flush() stdin_file.seek(0) @@ -1245,7 +1249,7 @@ def generatePointer(self, contentFile): ['git', 'lfs', 'pointer', '--file=' + contentFile], stdout=subprocess.PIPE ) - pointerFile = pointerProcess.stdout.read() + pointerFile = decode_text_stream(pointerProcess.stdout.read()) if pointerProcess.wait(): os.remove(contentFile) die('git-lfs pointer command failed. Did you install the extension?') @@ -3538,6 +3542,15 @@ def openStreams(self): self.gitStream = self.importProcess.stdin self.gitError = self.importProcess.stderr + if bytes is not str: + # Wrap gitStream.write() so that it can be called using `str` arguments + def make_encoded_write(write): + def encoded_write(s): + return write(s.encode() if isinstance(s, str) else s) + return encoded_write + + self.gitStream.write = make_encoded_write(self.gitStream.write) + def closeStreams(self): self.gitStream.close() if self.importProcess.wait() != 0: From patchwork Fri Dec 13 23:52:40 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291983 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 92654930 for ; Fri, 13 Dec 2019 23:53:32 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 66EEF22527 for ; Fri, 13 Dec 2019 23:53:32 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="qdzU7D6O" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726801AbfLMXxb (ORCPT ); Fri, 13 Dec 2019 18:53:31 -0500 Received: from mail-pj1-f43.google.com ([209.85.216.43]:46674 "EHLO mail-pj1-f43.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726741AbfLMXx1 (ORCPT ); Fri, 13 Dec 2019 18:53:27 -0500 Received: by mail-pj1-f43.google.com with SMTP id z21so371785pjq.13 for ; Fri, 13 Dec 2019 15:53:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=WYjX5Mw55NWsjv24gVB0/RGNjf3ZTheyg2JsNFlgfwI=; b=qdzU7D6ONga+iiR4WIBI5ZVuGQrrlaGeShnus5c2NdrOtiB0w+fd3WD3sQN1G3k2Lk wbW7a22kfyEMaagFcBEI6RQj6edeR2ypjJ6jaHEaYEGg+cabqE4esNW04gyRhggHfD3l eCfh85kOqfRhG5bmKOPpnXgxgubRhvtJyYeAG0rH3SEuiiKvuR69V25F9nhHUDhOmSSI 6z3vqCKUwCstLlBx1LtIAfqo/AeEWsPSP0SY+y8IpQP6ge3NwgvdhPMVxyOgkaXJ+0vE 2Y1V/R3ssqI60nFkdfEGsImDiI4tu9RYc0M6Lm4THRHxTnLMkbDdoBPSSGuJ2iS0VW14 xxbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=WYjX5Mw55NWsjv24gVB0/RGNjf3ZTheyg2JsNFlgfwI=; b=DlBKKw7KvpKc2EyP1wZOkJX+3Utv1C74MrDrAkqCbnPNoVI6znxDxVEF38HBNFojWa YD24lt3ryoAlYtgcV/0b1Xv2sVQxKqx6GGeHBXuuHoacEp/ozS3SonWAeDegD9Ab08tu pUWchu2aSxESewlklokO1DCr7HeYp8Ny1iKF9tweAC6e5Krym6eZfJvuS8rY6pDZ+1bo 1XRvzdTXZgWr0lR9cu/dittLtSBGzwynQ0sPld/nNvvjIQ68ugximdhRS63Qifdw3+k1 zOI8sjMD9HNw6mhlc8on3aURPog3jgFvCeB31yNiAly4F9JMf9BI9JXqCg59SDEsyDTw wyvg== X-Gm-Message-State: APjAAAX8Wh3TvvcVd7OZEcXRtFKG/GYqUUh3Z0qEehgrLnYSi5fwXBZX lV9U5JoqTes54sszz2LgN2yjApsOmEjruv6h X-Google-Smtp-Source: APXvYqz9LlcSUg/m/I6wUlhteiiYWgXA3iQ7IkJJ48v1D7J+1TuN4aKlKNVuh1HcT69HVVo2AZsz/A== X-Received: by 2002:a17:902:8d91:: with SMTP id v17mr2401959plo.194.1576281205608; Fri, 13 Dec 2019 15:53:25 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:24 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 06/14] git-p4: convert path to unicode before processing them Date: Fri, 13 Dec 2019 15:52:40 -0800 Message-Id: <20191213235247.23660-8-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org P4 allows essentially arbitrary encoding for path data while we would perfer to be dealing only with unicode strings. Since path data need to survive round-trip back to p4, this patch implements the general policy that we store path data as-is, but decode them to unicode before doing any non-trivial processing. A new `decode_path()` method is provided that generally does the correct conversion, taking into account `git-p4.pathEncoding` configuration. For python2.7, path strings will be left as-is if it only contains ASCII characters. For python3, decoding is always done so that we have str objects. Signed-off-by: Yang Zhao Reviewed-by: Ben Keene --- git-p4.py | 69 +++++++++++++++++++++++++++++++++++-------------------- 1 file changed, 44 insertions(+), 25 deletions(-) diff --git a/git-p4.py b/git-p4.py index d62fb05989..d43f373b2d 100755 --- a/git-p4.py +++ b/git-p4.py @@ -150,6 +150,21 @@ def decode_text_stream(s): def encode_text_stream(s): return s.encode('utf_8') if isinstance(s, unicode) else s +def decode_path(path): + """Decode a given string (bytes or otherwise) using configured path encoding options + """ + encoding = gitConfig('git-p4.pathEncoding') or 'utf_8' + if bytes is not str: + return path.decode(encoding, errors='replace') if isinstance(path, bytes) else path + else: + try: + path.decode('ascii') + except: + path = path.decode(encoding, errors='replace') + if verbose: + print('Path with non-ASCII characters detected. Used {} to decode: {}'.format(encoding, path)) + return path + def write_pipe(c, stdin): if verbose: sys.stderr.write('Writing pipe: %s\n' % str(c)) @@ -697,7 +712,8 @@ def p4Where(depotPath): if "depotFile" in entry: # Search for the base client side depot path, as long as it starts with the branch's P4 path. # The base path always ends with "/...". - if entry["depotFile"].find(depotPath) == 0 and entry["depotFile"][-4:] == "/...": + entry_path = decode_path(entry['depotFile']) + if entry_path.find(depotPath) == 0 and entry_path[-4:] == "/...": output = entry break elif "data" in entry: @@ -712,11 +728,11 @@ def p4Where(depotPath): return "" clientPath = "" if "path" in output: - clientPath = output.get("path") + clientPath = decode_path(output['path']) elif "data" in output: data = output.get("data") - lastSpace = data.rfind(" ") - clientPath = data[lastSpace + 1:] + lastSpace = data.rfind(b" ") + clientPath = decode_path(data[lastSpace + 1:]) if clientPath.endswith("..."): clientPath = clientPath[:-3] @@ -2484,7 +2500,7 @@ def append(self, view_line): def convert_client_path(self, clientFile): # chop off //client/ part to make it relative - if not clientFile.startswith(self.client_prefix): + if not decode_path(clientFile).startswith(self.client_prefix): die("No prefix '%s' on clientFile '%s'" % (self.client_prefix, clientFile)) return clientFile[len(self.client_prefix):] @@ -2493,7 +2509,7 @@ def update_client_spec_path_cache(self, files): """ Caching file paths by "p4 where" batch query """ # List depot file paths exclude that already cached - fileArgs = [f['path'] for f in files if f['path'] not in self.client_spec_path_cache] + fileArgs = [f['path'] for f in files if decode_path(f['path']) not in self.client_spec_path_cache] if len(fileArgs) == 0: return # All files in cache @@ -2508,16 +2524,18 @@ def update_client_spec_path_cache(self, files): if "unmap" in res: # it will list all of them, but only one not unmap-ped continue + depot_path = decode_path(res['depotFile']) if gitConfigBool("core.ignorecase"): - res['depotFile'] = res['depotFile'].lower() - self.client_spec_path_cache[res['depotFile']] = self.convert_client_path(res["clientFile"]) + depot_path = depot_path.lower() + self.client_spec_path_cache[depot_path] = self.convert_client_path(res["clientFile"]) # not found files or unmap files set to "" for depotFile in fileArgs: + depotFile = decode_path(depotFile) if gitConfigBool("core.ignorecase"): depotFile = depotFile.lower() if depotFile not in self.client_spec_path_cache: - self.client_spec_path_cache[depotFile] = "" + self.client_spec_path_cache[depotFile] = b'' def map_in_client(self, depot_path): """Return the relative location in the client where this @@ -2635,7 +2653,7 @@ def isPathWanted(self, path): elif path.lower() == p.lower(): return False for p in self.depotPaths: - if p4PathStartsWith(path, p): + if p4PathStartsWith(path, decode_path(p)): return True return False @@ -2644,7 +2662,7 @@ def extractFilesFromCommit(self, commit, shelved=False, shelved_cl = 0): fnum = 0 while "depotFile%s" % fnum in commit: path = commit["depotFile%s" % fnum] - found = self.isPathWanted(path) + found = self.isPathWanted(decode_path(path)) if not found: fnum = fnum + 1 continue @@ -2678,7 +2696,7 @@ def stripRepoPath(self, path, prefixes): if self.useClientSpec: # branch detection moves files up a level (the branch name) # from what client spec interpretation gives - path = self.clientSpecDirs.map_in_client(path) + path = decode_path(self.clientSpecDirs.map_in_client(path)) if self.detectBranches: for b in self.knownBranches: if p4PathStartsWith(path, b + "/"): @@ -2712,14 +2730,15 @@ def splitFilesIntoBranches(self, commit): branches = {} fnum = 0 while "depotFile%s" % fnum in commit: - path = commit["depotFile%s" % fnum] + raw_path = commit["depotFile%s" % fnum] + path = decode_path(raw_path) found = self.isPathWanted(path) if not found: fnum = fnum + 1 continue file = {} - file["path"] = path + file["path"] = raw_path file["rev"] = commit["rev%s" % fnum] file["action"] = commit["action%s" % fnum] file["type"] = commit["type%s" % fnum] @@ -2728,7 +2747,7 @@ def splitFilesIntoBranches(self, commit): # start with the full relative path where this file would # go in a p4 client if self.useClientSpec: - relPath = self.clientSpecDirs.map_in_client(path) + relPath = decode_path(self.clientSpecDirs.map_in_client(path)) else: relPath = self.stripRepoPath(path, self.depotPaths) @@ -2766,14 +2785,15 @@ def encodeWithUTF8(self, path): # - helper for streamP4Files def streamOneP4File(self, file, contents): - relPath = self.stripRepoPath(file['depotFile'], self.branchPrefixes) - relPath = self.encodeWithUTF8(relPath) + file_path = file['depotFile'] + relPath = self.stripRepoPath(decode_path(file_path), self.branchPrefixes) + if verbose: if 'fileSize' in self.stream_file: size = int(self.stream_file['fileSize']) else: size = 0 # deleted files don't get a fileSize apparently - sys.stdout.write('\r%s --> %s (%i MB)\n' % (file['depotFile'], relPath, size/1024/1024)) + sys.stdout.write('\r%s --> %s (%i MB)\n' % (file_path, relPath, size/1024/1024)) sys.stdout.flush() (type_base, type_mods) = split_p4_type(file["type"]) @@ -2791,7 +2811,7 @@ def streamOneP4File(self, file, contents): # to nothing. This causes p4 errors when checking out such # a change, and errors here too. Work around it by ignoring # the bad symlink; hopefully a future change fixes it. - print("\nIgnoring empty symlink in %s" % file['depotFile']) + print("\nIgnoring empty symlink in %s" % file_path) return elif data[-1] == '\n': contents = [data[:-1]] @@ -2810,7 +2830,7 @@ def streamOneP4File(self, file, contents): # just the native "NT" type. # try: - text = p4_read_pipe(['print', '-q', '-o', '-', '%s@%s' % (file['depotFile'], file['change'])]) + text = p4_read_pipe(['print', '-q', '-o', '-', '%s@%s' % (decode_path(file['depotFile']), file['change'])], raw=True) except Exception as e: if 'Translation of file content failed' in str(e): type_base = 'binary' @@ -2818,7 +2838,7 @@ def streamOneP4File(self, file, contents): raise e else: if p4_version_string().find('/NT') >= 0: - text = text.replace('\r\n', '\n') + text = text.replace(b'\r\n', b'\n') contents = [ text ] if type_base == "apple": @@ -2849,8 +2869,7 @@ def streamOneP4File(self, file, contents): self.writeToGitStream(git_mode, relPath, contents) def streamOneP4Deletion(self, file): - relPath = self.stripRepoPath(file['path'], self.branchPrefixes) - relPath = self.encodeWithUTF8(relPath) + relPath = self.stripRepoPath(decode_path(file['path']), self.branchPrefixes) if verbose: sys.stdout.write("delete %s\n" % relPath) sys.stdout.flush() @@ -3037,8 +3056,8 @@ def commit(self, details, files, branch, parent = "", allow_empty=False): if self.clientSpecDirs: self.clientSpecDirs.update_client_spec_path_cache(files) - files = [f for f in files - if self.inClientSpec(f['path']) and self.hasBranchPrefix(f['path'])] + files = [f for (f, path) in ((f, decode_path(f['path'])) for f in files) + if self.inClientSpec(path) and self.hasBranchPrefix(path)] if gitConfigBool('git-p4.keepEmptyCommits'): allow_empty = True From patchwork Fri Dec 13 23:52:41 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291981 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id F0DAC14DB for ; Fri, 13 Dec 2019 23:53:31 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id CDBF7206EE for ; Fri, 13 Dec 2019 23:53:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="lFVMXnSD" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726786AbfLMXxa (ORCPT ); Fri, 13 Dec 2019 18:53:30 -0500 Received: from mail-pj1-f49.google.com ([209.85.216.49]:36757 "EHLO mail-pj1-f49.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726769AbfLMXx1 (ORCPT ); Fri, 13 Dec 2019 18:53:27 -0500 Received: by mail-pj1-f49.google.com with SMTP id n96so391064pjc.3 for ; Fri, 13 Dec 2019 15:53:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=voJ5PIZDOMnXJCToHfNX4qb2Z1KJm1SJx9O0w+hWOdk=; b=lFVMXnSDAcE/6vuQD3d7nxbFo+g3XStvF+Qk01q6flmNWJpltDuD4J3iEBEAfX7UoL Kl80gH1OGUhlsr6AeKPwTW7PbC2y1hphJd9WnABfF6eXXzZQiqfu3avwkuCjxk7/VXYA 4RTbgxIJZDEgQtXiqDYC30eJYzH4fEwHpf66QmKWKWLcEYzb9CYqsoxV7BPbdH3b+k1K bUiI5ZTOpCbS4ApWs9uP0oxMP6jbJ2mDPUD6fwelkYldoRn1ahOicCizzN9fO5N45tFn 4GvvhRjfcF5Mis0MR3VmZEHGSbZKBIEt4Ak1PVJgPIFJzqZkijb6Vdq/bDj6gJ2KWO5a aFVQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=voJ5PIZDOMnXJCToHfNX4qb2Z1KJm1SJx9O0w+hWOdk=; b=TDwpXK6Ui463wJsDt4yL/Cdfmj4oC0lWjy+/Eu5XU2J9ARqLau+jfa99x+JJ62SnUw 6hHLlik3nYUcRd8FyFyPhWdoWOiRVAC7b4A9J7jo7vDNqI34C5LbsolD57u08OOaTZUa COl947PO9dl00/z3qrVnv2HB93bqYwGsO3F+foA2ADLCgI4m7BptOQmbHWnnPbB8FBOL l+vc5iaAz7a5PcMeqB9Df7Gxqr93mtSZLyn9A/vPKnX7yNSxbjolLyZzXX41Yu5CMa4g 65iHWpQ7pjFmrrnJ9/QrrGp5pnbLI8U4DhOD+K9prgTSx9Wy4Xpl9UpKV85X2zXwEC6q CwJQ== X-Gm-Message-State: APjAAAUOeN3EgkBeVaFW+N5Cg5+1OqCFGlvAWYUvYZkh8qpcge3+hLsP 6Q1J/dHZetfEURZeocMdyvY3c+AZUJi4RioV X-Google-Smtp-Source: APXvYqwaiwhyyzq+e6T7EzvS0TpHdE+qnCX3Yt7cj5ktYr+NWKocOCluJ+RL86e0JuE8bLNXtUWCUg== X-Received: by 2002:a17:902:ab85:: with SMTP id f5mr2317543plr.102.1576281206801; Fri, 13 Dec 2019 15:53:26 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.25 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:26 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 07/14] git-p4: open .gitp4-usercache.txt in text mode Date: Fri, 13 Dec 2019 15:52:41 -0800 Message-Id: <20191213235247.23660-9-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Opening .gitp4-usercache.txt in text mode makes python 3 happy without explicitly adding encoding and decoding. Signed-off-by: Yang Zhao Reviewed-by: Ben Keene --- git-p4.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/git-p4.py b/git-p4.py index d43f373b2d..abcda60eee 100755 --- a/git-p4.py +++ b/git-p4.py @@ -1395,14 +1395,14 @@ def getUserMapFromPerforceServer(self): for (key, val) in self.users.items(): s += "%s\t%s\n" % (key.expandtabs(1), val.expandtabs(1)) - open(self.getUserCacheFilename(), "wb").write(s) + open(self.getUserCacheFilename(), 'w').write(s) self.userMapFromPerforceServer = True def loadUserMapFromCache(self): self.users = {} self.userMapFromPerforceServer = False try: - cache = open(self.getUserCacheFilename(), "rb") + cache = open(self.getUserCacheFilename(), 'r') lines = cache.readlines() cache.close() for line in lines: From patchwork Fri Dec 13 23:52:42 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291979 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id AED4814DB for ; Fri, 13 Dec 2019 23:53:30 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 8C44D206EE for ; Fri, 13 Dec 2019 23:53:30 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="1o/H7JES" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726777AbfLMXx3 (ORCPT ); Fri, 13 Dec 2019 18:53:29 -0500 Received: from mail-pg1-f194.google.com ([209.85.215.194]:34329 "EHLO mail-pg1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726771AbfLMXx2 (ORCPT ); Fri, 13 Dec 2019 18:53:28 -0500 Received: by mail-pg1-f194.google.com with SMTP id r11so262667pgf.1 for ; Fri, 13 Dec 2019 15:53:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=bO4jqekpdixoY18WHt+1rc/8RaVQvk0lNgf2uN6uRV8=; b=1o/H7JES0GMPYa29s4JLnJQGI62prhsZcSIN+EI0dUBLnxhr4P69984pE6hiX3LXXL 2iZQEsTXJPl9XWjZkCmheXb97lnsE4hD+8lu/vQcVfQa/ww2zA76H9YKffqn0M0Tcs4O 2GUhIXoY9jTqSoC4Y0Ff6nbClxxwR+sXAckK/s8Ru5bnnw5zVB4FaAPn8rpEE8b4IB/f 3GGFo6x8db98SaSjGK5pBvVBQl9ZjFHSTt2T0eIWX3BzYQ8ZsqcB1KRPcRibZYruBGsg k3tDbwIORE1Z8ljKar3bd7oH/l2m3vYLZRsgXHR0+qETHXPYM6p4M78sTgptblQvjm9n NByQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=bO4jqekpdixoY18WHt+1rc/8RaVQvk0lNgf2uN6uRV8=; b=kPnCm61Y59+KSGh67yZ4bnj5hhfcksmIjMkNMQ9qmnxt5YUjYfP5kPLiB3UbaCk3iC VVean+V3VviYhpzlYcgYoXqA8AbzN9b49CzSdEQrgPCxWlUZI0uRKTAa7igVBRrrEgCg WLu/KTBVWmcg3i1okQJKGZcVSNCV7fk9osaEFybpHqgxk46c2TOt2Uubn/UhIli7SKWd jAKqw9CvgaQ2OWXQEarrNB6d0IKSq75lSXASiU5ie005AhGYETm/JTbWMpnjKp2ItX+j pRkwPxPJ9nTYcQ30eot0db6hmSnIw0UxASCDFHdBu1FPWwb3+hu2oTOPGBGjOct57/sk VpyQ== X-Gm-Message-State: APjAAAXA1+SIIM5mX4qYZLvx5OZXxUEhxeJG9J73bAJw9xrLRlw7evN6 3qlx/EfmfklYrvMBld8eQkF/K5BCNc/BJtXw X-Google-Smtp-Source: APXvYqx/EKcbtMuOIVmUiC7DfgJe72jGFfgyDdvo6rVCsZM8Btgmwntr4oI7yl5D42f3ne/v3+yVEQ== X-Received: by 2002:aa7:9629:: with SMTP id r9mr2443201pfg.51.1576281207862; Fri, 13 Dec 2019 15:53:27 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:27 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 08/14] git-p4: use marshal format version 2 when sending to p4 Date: Fri, 13 Dec 2019 15:52:42 -0800 Message-Id: <20191213235247.23660-10-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org p4 does not appear to understand marshal format version 3 and above. Version 2 was the latest supported by python-2.7. Signed-off-by: Yang Zhao Reviewed-by: Ben Keene --- git-p4.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/git-p4.py b/git-p4.py index abcda60eee..c7170c9ae6 100755 --- a/git-p4.py +++ b/git-p4.py @@ -1679,7 +1679,8 @@ def modifyChangelistUser(self, changelist, newUser): c = changes[0] if c['User'] == newUser: return # nothing to do c['User'] = newUser - input = marshal.dumps(c) + # p4 does not understand format version 3 and above + input = marshal.dumps(c, 2) result = p4CmdList("change -f -i", stdin=input) for r in result: From patchwork Fri Dec 13 23:52:43 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291985 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 9D7A714DB for ; Fri, 13 Dec 2019 23:53:33 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 7D4B122527 for ; Fri, 13 Dec 2019 23:53:33 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="XP1tkT/+" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726590AbfLMXxc (ORCPT ); Fri, 13 Dec 2019 18:53:32 -0500 Received: from mail-pl1-f193.google.com ([209.85.214.193]:35406 "EHLO mail-pl1-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726784AbfLMXxa (ORCPT ); Fri, 13 Dec 2019 18:53:30 -0500 Received: by mail-pl1-f193.google.com with SMTP id s10so1897314plp.2 for ; Fri, 13 Dec 2019 15:53:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=oF6M98V6kq+d3Xs7wzwAw78C3NoRnbRxgo7xe43Its8=; b=XP1tkT/+DZdFuBj5Ptmob4oBWKPo9eb1VzyRLxuQjScS/IV5AgG5Q1vWpaOgFjk37p ChuyTK4HK0AxuBI1hkVkOvIKtlE4tD4WdW0UCDjykuUc/Jbr7IAmY8LCNpZFbeRm/2+O FgGY2Cl6v7FTRmjtBUEOSnTDF4p8vu4FH38xRJc7xA49QOXFDEgI4alNJDJ47PyyC99f VfHpXan/1rf1VkDzcV8SeIikjCYc7I2ha8H9PLjgJfiqLLe2Ls3XQVDOxj7HAVndt1Q5 J3ca8co34e9Mg6E5x+YT/d3KT1RA9zVRbEC3ere9zFJkV+BcU1sdsmu2yIkYeXvQu7Mj 7JsQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=oF6M98V6kq+d3Xs7wzwAw78C3NoRnbRxgo7xe43Its8=; b=HPTxA3A1zktrUWhWN2MD7ScQsAII2bD7W2WV0Ff+YJqDIYNXmnfHibMg8kn06Embb3 /Mc2QCbKcUkCb3DmKR4HeN7+9f3ff5cP3f83nWqc2tSegxShhM+0bldvPkbzqn2dBtfH cCapee6rffKYW9moAF4lXel0EMY5kGH7pR4SUof9uuhKfGIdFHRZu/LQlpcoTcHh7eCd 4TGzVoqJH7iBnck9B1jmYYyOmG8UyBTP5Qr0pa0RmjeGAdHS/DUejbPnxdfrJErHzGxf Dz3qa1uoZlsYlteWcZtx3kW9awqHaL0pAz62Kup/X++Y1wE+Z1uzUC0KPAkijdFb4zlq tb0Q== X-Gm-Message-State: APjAAAVEA9Bc+O+O7Ii2Bv/0O+kL/TZR7E15w+p6wkfnN31XA7zZSMCW 2bY9Gidenac6qD00gYCkm42Hd7lXpblrGoNy X-Google-Smtp-Source: APXvYqxinHrmsLlOnJq328moY4jN09nAelqI8yQT903oauzti/unURHFthizZfrKUbGIWDoDmaQpNg== X-Received: by 2002:a62:4e4e:: with SMTP id c75mr2432523pfb.144.1576281208808; Fri, 13 Dec 2019 15:53:28 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.27 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:28 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 09/14] git-p4: fix freezing while waiting for fast-import progress Date: Fri, 13 Dec 2019 15:52:43 -0800 Message-Id: <20191213235247.23660-11-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org As part of its importing process, git-p4 sends a `checkpoint` followed immediately by `progress` to fast-import to force synchronization. Due to buffering, it is possible for the `progress` command to not be flushed before git-p4 begins to wait for the corresponding response. This causes the script to freeze completely, and is consistently observable at least on python-3.6.9. Make sure this command sequence is completely flushed before waiting. Signed-off-by: Yang Zhao Reviewed-by: Ben Keene --- git-p4.py | 1 + 1 file changed, 1 insertion(+) diff --git a/git-p4.py b/git-p4.py index c7170c9ae6..68f9f4fdc6 100755 --- a/git-p4.py +++ b/git-p4.py @@ -2641,6 +2641,7 @@ def __init__(self): def checkpoint(self): self.gitStream.write("checkpoint\n\n") self.gitStream.write("progress checkpoint\n\n") + self.gitStream.flush() out = self.gitOutput.readline() if self.verbose: print("checkpoint finished: " + out) From patchwork Fri Dec 13 23:52:44 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291987 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id DDC6E14DB for ; Fri, 13 Dec 2019 23:53:34 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id BCF8F20724 for ; Fri, 13 Dec 2019 23:53:34 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="B4L6jYnW" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726820AbfLMXxe (ORCPT ); Fri, 13 Dec 2019 18:53:34 -0500 Received: from mail-pf1-f194.google.com ([209.85.210.194]:40456 "EHLO mail-pf1-f194.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726769AbfLMXxa (ORCPT ); Fri, 13 Dec 2019 18:53:30 -0500 Received: by mail-pf1-f194.google.com with SMTP id q8so2292949pfh.7 for ; Fri, 13 Dec 2019 15:53:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=APGGNcfznxbhrfNSjyLU0pW4ZC5c1b9GUWn09pwP3PA=; b=B4L6jYnWGyHKZo4v6tDnR9Z0MpCD7822QQnCTwm8WCxrAMVDhv/3LnvwfcoP//ZN5Y 774Rm5T5ZPBB6F6fCXvCoL47lFOwKq5WI7Mb6ftv1kf6/4ssC5+eWZEnsHsAH8cjxSVm CpL740f0nK5hN5EkU3HwP9yTTvnIO1WCp78olfnTHmtOXkbIZu3wJAEKBhKH+KFG9ZPM Xl5EjbAia/8TiKe/Y3NrtzOO155iW4IrXs54TwphxKG779rtp7CGWx3Vro0qb+nBIaI1 XPVpIY3jq/1ljjVdIuwn0pfjvI96RjREBr/lq07dQZetepftzfuUdfi/Qiri7Ln23g7w 5uZQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=APGGNcfznxbhrfNSjyLU0pW4ZC5c1b9GUWn09pwP3PA=; b=gezpKwdRnxXCW4jZxJ6MDVp5oFk9uzNmigL6QZslQse8acsGGiFVMlz2Kc8U/SHitI OBXJvjmF2Xy2W8I0VZIgXkk7bvwUqjS9aIp33JuZLzRvcuBSFO3TPhr1fmwj/AxXrGn4 LyPZVkEEY3cJIm7x0OK57Yw4tZwtsdx5XIM6N9yYfqt+z7eZFbNwt8K5uH89C2mgEP/I dDhMzPmY5lMF38ALpLEt01bC3cx0YLcNBdvUWohgASTnGQ4cAzzDhnJxcLbHQeE23ewm mQgGgmNXbLg4sRhH8DOwZojK+9W3NRNFjUzdoHbeteDJAuIa360AZbBcRlBA4hZn5R03 tLsw== X-Gm-Message-State: APjAAAVvAEpe5pQ9M25S+6ZBqI3Bq4VGUj45aPWsa7st52XY8p8cUAzs cjSjIzKXmnVj/SKGkPjqZe1uS2ks0eYtcw9P X-Google-Smtp-Source: APXvYqyqBillTFwLMuqSaoytTIYHhaFkW1yWyAZzFZi0JjPyQPAKq5sfFNwpis6AimAMlE7VG65z1Q== X-Received: by 2002:a63:647:: with SMTP id 68mr2449682pgg.202.1576281209880; Fri, 13 Dec 2019 15:53:29 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.28 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:29 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 10/14] git-p4: use functools.reduce instead of reduce Date: Fri, 13 Dec 2019 15:52:44 -0800 Message-Id: <20191213235247.23660-12-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org For python3, reduce() has been moved to functools.reduce(). This is also available in python2.7. Signed-off-by: Yang Zhao --- git-p4.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/git-p4.py b/git-p4.py index 68f9f4fdc6..7deee1b939 100755 --- a/git-p4.py +++ b/git-p4.py @@ -13,6 +13,7 @@ sys.exit(1) import os import optparse +import functools import marshal import subprocess import tempfile @@ -1158,7 +1159,7 @@ def pushFile(self, localLargeFile): assert False, "Method 'pushFile' required in " + self.__class__.__name__ def hasLargeFileExtension(self, relPath): - return reduce( + return functools.reduce( lambda a, b: a or b, [relPath.endswith('.' + e) for e in gitConfigList('git-p4.largeFileExtensions')], False From patchwork Fri Dec 13 23:52:45 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291991 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 79F19930 for ; Fri, 13 Dec 2019 23:53:36 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 582ED20724 for ; Fri, 13 Dec 2019 23:53:36 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="D6tNdfHT" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726833AbfLMXxf (ORCPT ); Fri, 13 Dec 2019 18:53:35 -0500 Received: from mail-pg1-f195.google.com ([209.85.215.195]:32983 "EHLO mail-pg1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726802AbfLMXxb (ORCPT ); Fri, 13 Dec 2019 18:53:31 -0500 Received: by mail-pg1-f195.google.com with SMTP id 6so265158pgk.0 for ; Fri, 13 Dec 2019 15:53:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=AsFx78VLSxJNAtG4kOBuFFJgV8dMv4lwgcP7TH725+c=; b=D6tNdfHTVlJsAsR1nREUWwAVRGCbmKtvUzB1DsEqTBJe7G7S9PQJwahBqHrFhKTPd4 t/yA9poEVA3lGSJN2BTtDVST9nZnw3BifOgcPgKRtDiq8BT1A8L6t1skb9HeponnoOQA 9oXt48wUdEJCigNT7Lg44bDzexcptbfnG7gaB0Qd4tQeUskEarN8ldsQSgHKkHETUiZ/ Z9scqEJYv9+gOV6thBqQfNwtnGt4IaszhJuP0KE9n6NjgGx33xcignuj7E1pQkEwuwE6 oeRUAFb37vw6dUkrPGX1SIjXd2PhUfdEGFnVDZdZgXtbQENyZA23mR3S0JwmBckx7wmk EoLw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=AsFx78VLSxJNAtG4kOBuFFJgV8dMv4lwgcP7TH725+c=; b=PUjvKZyvVl4w1jrLM9VbtCXtHumHrhqh5Y3nDbt2sN6dx/h3mIuIuAO6JKNOTnxl0a DG1DDWxi5Kmq5lvujolafUfXwWLQSesU9Sh+00UxMFSfXZo4y1NKy2rPpV9e4mCqJaXU qRx8ZD3Hxtsrfuw9wz13pN4glwZ++jMRvLpeXioorDwrWIM3R2e91wY4O99+3lb9UXBV GeaDN2Az3ECG213Xr6hNmK72orh2LzzRMRK/N91RxBTLo60WAVEir0jGZB4OZjveLh+3 TplfepMvE7ES1CIyXZyHaf7gDg1dxFAM5SeGHKZuJUdh7RzzOOw8vfP840xiZ9NKX0+y sDiA== X-Gm-Message-State: APjAAAX049DAg1Bxax3aUwxOB5cNj6+wtLRHHIxaoPp/RSNkSgR7dzkk RVuv7q5o5Kzs32WLfNAVtLXVCzUDfz2+DnYR X-Google-Smtp-Source: APXvYqyteDOP2DmNSiI652khGxCv/HSVwwPfZHLYriuOx8UTwaTErr6JfHDITweAyH1nYaHE6Mu58Q== X-Received: by 2002:aa7:8699:: with SMTP id d25mr2363813pfo.139.1576281210946; Fri, 13 Dec 2019 15:53:30 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.30 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:30 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 11/14] git-p4: use dict.items() iteration for python3 compatibility Date: Fri, 13 Dec 2019 15:52:45 -0800 Message-Id: <20191213235247.23660-13-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Python3 uses dict.items() instead of .iteritems() to provide iteratoration over dict. Although items() is technically less efficient for python2.7 (allocates a new list instead of simply iterating), the amount of data involved is very small and the penalty negligible. Signed-off-by: Yang Zhao Reviewed-by: Ben Keene --- git-p4.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/git-p4.py b/git-p4.py index 7deee1b939..b7e31d4738 100755 --- a/git-p4.py +++ b/git-p4.py @@ -1745,7 +1745,7 @@ def prepareSubmitTemplate(self, changelist=None): break if not change_entry: die('Failed to decode output of p4 change -o') - for key, value in change_entry.iteritems(): + for key, value in change_entry.items(): if key.startswith('File'): if 'depot-paths' in settings: if not [p for p in settings['depot-paths'] From patchwork Fri Dec 13 23:52:46 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291989 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 7793D930 for ; Fri, 13 Dec 2019 23:53:35 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 569FC20724 for ; Fri, 13 Dec 2019 23:53:35 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="diq3zjuo" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726828AbfLMXxe (ORCPT ); Fri, 13 Dec 2019 18:53:34 -0500 Received: from mail-pf1-f195.google.com ([209.85.210.195]:42543 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726813AbfLMXxc (ORCPT ); Fri, 13 Dec 2019 18:53:32 -0500 Received: by mail-pf1-f195.google.com with SMTP id 4so2286891pfz.9 for ; Fri, 13 Dec 2019 15:53:32 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=34T1a/IEmgMUS4B/nrxuttRxXmASlCVG0uMyRUBcR7I=; b=diq3zjuogGJE8egaMWF1eYiXU9cba8a1QAOy6XoI7GAls+yraNqk8fKN/wQ44cJfFk PZTYqcrpTuFFiF8qEIraRkMu5jAGIV250jWsLY5entXlfGT6cgSSdL0WYjVT/SaJfCc3 DFYzrH7tRly0/JWWA569LluRCjws2fp/6gsLbhUpDjFl1M508lPmiEdZd4l3Suxdjbbm lfMcb+T6NQqad46Ou6abWm2u4G6xmU4IcxFSAA51bqTHmfs9AcXJ5Kqkzn2nx+WXlmpo Y0YtIOVX8PmEpsPUnXS5n/QR60oWjPNN9aBN708sFRP9VhFRJEqoxkeKxqNOrLf34/H/ kXjw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=34T1a/IEmgMUS4B/nrxuttRxXmASlCVG0uMyRUBcR7I=; b=nZ1QfglmSM3c9ZFIRw0+Z+GPm1jH93RCXtokdiodFJkZLIDV9HX6/4YDUU5XQ9rI5R FII2+ztf65wKeyfKDBTTWMw9rbBWbLPei7ojef+hb0D95YMFIP6nZrycBoCimcPMIUSs UvpDXjWFfac7+1JeoU0WFnaPS5fwf2fJM4lwuhAebPasUFBud0tJgxQHjr4vTYf0Fbv1 0Iz10ZmPTrWjFRf+6rf8a1AN12+A1TCIPRo8JG9A3mPlu1BXqPtSlGzo5yzBHJdRM2yc fVnd32FxbwMSKfTGiqvC901EVkYaA3ZZLaRIDi7+KwELcpv4RFItZ/sLHvM0KN848ql7 1c3A== X-Gm-Message-State: APjAAAWzTmo1ulEdFHsQ6gVYesoi3AwcIfY8ATYX+Qf0Womuvh7b4mwt rgCb5e0tcuGfkmLQKwDufWOFP8g3MILXdxVa X-Google-Smtp-Source: APXvYqxBEBF6co8Hh6NA10xHg5utwrHXcUu3P3PYUw9BNw50nqXmNVFn2kdbl1ho88WsUfBw3GDmdw== X-Received: by 2002:a63:5442:: with SMTP id e2mr2412657pgm.18.1576281211931; Fri, 13 Dec 2019 15:53:31 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.31 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:31 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 12/14] git-p4: simplify regex pattern generation for parsing diff-tree Date: Fri, 13 Dec 2019 15:52:46 -0800 Message-Id: <20191213235247.23660-14-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org It is not clear why a generator was used to create the regex used to parse git-diff-tree output; I assume an early implementation required it, but is not part of the mainline change. Simply use a lazily initialized global instead. Signed-off-by: Yang Zhao Reviewed-by: Ben Keene --- git-p4.py | 13 ++++++------- 1 file changed, 6 insertions(+), 7 deletions(-) diff --git a/git-p4.py b/git-p4.py index b7e31d4738..3af8df9f83 100755 --- a/git-p4.py +++ b/git-p4.py @@ -544,12 +544,7 @@ def getGitTags(): gitTags.add(tag) return gitTags -def diffTreePattern(): - # This is a simple generator for the diff tree regex pattern. This could be - # a class variable if this and parseDiffTreeEntry were a part of a class. - pattern = re.compile(':(\d+) (\d+) (\w+) (\w+) ([A-Z])(\d+)?\t(.*?)((\t(.*))|$)') - while True: - yield pattern +_diff_tree_pattern = None def parseDiffTreeEntry(entry): """Parses a single diff tree entry into its component elements. @@ -570,7 +565,11 @@ def parseDiffTreeEntry(entry): If the pattern is not matched, None is returned.""" - match = diffTreePattern().next().match(entry) + global _diff_tree_pattern + if not _diff_tree_pattern: + _diff_tree_pattern = re.compile(':(\d+) (\d+) (\w+) (\w+) ([A-Z])(\d+)?\t(.*?)((\t(.*))|$)') + + match = _diff_tree_pattern.match(entry) if match: return { 'src_mode': match.group(1), From patchwork Fri Dec 13 23:52:47 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291995 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 2F57A14DB for ; Fri, 13 Dec 2019 23:53:40 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 0D7EA206EE for ; Fri, 13 Dec 2019 23:53:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="QIlWxUzB" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726842AbfLMXxj (ORCPT ); Fri, 13 Dec 2019 18:53:39 -0500 Received: from mail-pj1-f47.google.com ([209.85.216.47]:44328 "EHLO mail-pj1-f47.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726818AbfLMXxe (ORCPT ); Fri, 13 Dec 2019 18:53:34 -0500 Received: by mail-pj1-f47.google.com with SMTP id w5so373347pjh.11 for ; Fri, 13 Dec 2019 15:53:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=4jyP80vwwKnkuAU276zRpKjBo7AE7L+0OfAIr/aaiRM=; b=QIlWxUzBae0/WfdYjJaW+L6r2N6SUNyfHNBCimp23XIVW92Xl0OrcxhFbVStbbhsLG pDam7xEunLIJZhy/nL64YJhUhwe8VLLhpV707nKDzQ/G8nmknIl07XV8GEiTyQD5j5Wq L+rJnm9TFdHjMhV2n4Jo0Cw9PTF8fzFY5YYt0FrqmAnuTVZhyPvq5ueO5CSRBJRuEJk1 T2ReYAFoxhMMcgSgyXpoNMhEmxJU+6QWYaBqtqXd2fSh2oBXXOk4qfM1bJjWVxH4Ft76 w/Ouku2oZ+Fb9NATGlnWf8EtDNgGTABJwJXu3cokVhsNZjayPVJZKPsbuls6Jk88qN34 FVew== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4jyP80vwwKnkuAU276zRpKjBo7AE7L+0OfAIr/aaiRM=; b=IrKVCAE6eJDmOTVBoZWu5aFQhXnTuKHFfht7EOiCk5apKvCNFojYLrd9gggsTumVqX H1KW7exrBr3DLshVa2d51N6UyT8IxYGZL0Ces1pmN3n/x/M44sVMyH1DXoAXuXaA3Iz/ nzGTCh+80A/NhZBayBEHkv1TZBM0+HuBdZP0FBnRlYW7jbhdU7vv0n3cL2VknsGxTzGH KpvyI7RdeO75A64Sd4X6JpOXxoLCCAO4svrPc1Bmt8JZFAmJ2w/iZXXr2rIwY09Tnq2B U3wDqsaC8pgb51cB3RNwTgbFiE+ueFM2KdlF+PIlh8pxWAsThG/ow7MIjm1KRB+2lTSU mQ9w== X-Gm-Message-State: APjAAAWSfeGPxHW57kNhEhk7Jbl228ZKpgzUXEzeO14h8dsspnSI/ksu E+KyDqpyQUincxBCDKlZ/S1NpfPAnhlbkV3j X-Google-Smtp-Source: APXvYqwgUMOfiiGbMs2rMQ2NX+TSND/ukQekBkrUcTJyTBVJ6VZ+3JB0cIeDPEaUI9XiGmng4Oyhkg== X-Received: by 2002:a17:90a:d353:: with SMTP id i19mr2516746pjx.43.1576281212905; Fri, 13 Dec 2019 15:53:32 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:32 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com Subject: [PATCH v2 13/14] git-p4: use python3's input() everywhere Date: Fri, 13 Dec 2019 15:52:47 -0800 Message-Id: <20191213235247.23660-15-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org Python3 deprecates raw_input() from 2.7 and replaced it with input(). Since we do not need 2.7's input() semantics, `raw_input()` is aliased to `input()` for easy forward compatability. Signed-off-by: Yang Zhao --- git-p4.py | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/git-p4.py b/git-p4.py index 3af8df9f83..17f72f4309 100755 --- a/git-p4.py +++ b/git-p4.py @@ -27,6 +27,16 @@ import ctypes import errno +# On python2.7 where raw_input() and input() are both availble, +# we want raw_input's semantics, but aliased to input for python3 +# compatibility +# support basestring in python3 +try: + if raw_input and input: + input = raw_input +except: + pass + verbose = False # Only labels/tags matching this will be imported/exported @@ -1801,7 +1811,7 @@ def edit_template(self, template_file): return True while True: - response = raw_input("Submit template unchanged. Submit anyway? [y]es, [n]o (skip this patch) ") + response = input("Submit template unchanged. Submit anyway? [y]es, [n]o (skip this patch) ") if response == 'y': return True if response == 'n': @@ -2372,7 +2382,7 @@ def run(self, args): # prompt for what to do, or use the option/variable if self.conflict_behavior == "ask": print("What do you want to do?") - response = raw_input("[s]kip this commit but apply" + response = input("[s]kip this commit but apply" " the rest, or [q]uit? ") if not response: continue From patchwork Fri Dec 13 23:52:48 2019 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Yang Zhao X-Patchwork-Id: 11291993 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A954F14DB for ; Fri, 13 Dec 2019 23:53:37 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 88B2520724 for ; Fri, 13 Dec 2019 23:53:37 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=skyboxlabs-com.20150623.gappssmtp.com header.i=@skyboxlabs-com.20150623.gappssmtp.com header.b="K6YPs6Z6" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726840AbfLMXxg (ORCPT ); Fri, 13 Dec 2019 18:53:36 -0500 Received: from mail-pf1-f195.google.com ([209.85.210.195]:44668 "EHLO mail-pf1-f195.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726831AbfLMXxe (ORCPT ); Fri, 13 Dec 2019 18:53:34 -0500 Received: by mail-pf1-f195.google.com with SMTP id d199so2280643pfd.11 for ; Fri, 13 Dec 2019 15:53:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=skyboxlabs-com.20150623.gappssmtp.com; s=20150623; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=OEk0DorHiKYXJBKppQcBNfdw2rH99FD8zEg6SjEwr9I=; b=K6YPs6Z6Go8zbsd2T8J18nVNuGpQbZZyrAT3knPU4m/hCS5fYwRCaI69P1L8btyvvG s7bcOZt+Qsy6SFp8p2aJh/CM9czo2DBmnnhAETlZrAPpUY0kHNzEIjeJeCu5RC+SP++G Fj2ie1dWVB6ksCe10wppO7fQOzEJgAPVAhWBUzuSbgacePFF6HBx+HlGRRD1/d9L2Pza 3NVFT8LuKJlb55x+PoayIp5WPBP4ovrLrKqQwYKFuiJ+Ir1TV0YxSDD1xiFdA8PCdSBx rtQOJRYhqxe9Oq/s/cAl5giLQAAgFkJW0F9vzFDblxKBEvUtbiT5C1jvELJtC9qdnHIY nhuQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=OEk0DorHiKYXJBKppQcBNfdw2rH99FD8zEg6SjEwr9I=; b=S7nHm2/hED2tvCQ88AAxnufpusvpHlBM+3rFm+2mYZPZKdf/GS97s/AhkSLMUbH3Xb Ct130HnS7OMasXmu0P4XRtHt8fJo6fBYt53+EkFl8uVgitiHOmhv0KsbbKTvt5bd2eKs SwXN5b7//OYVwLKnTkkyki1WI4OZZCXub+qwqkoi13+TA4LHcSlwJgRX8GOu7C5Vs5Hb WVGwNLClbztiwxICoa7zjY7DB/uCCb1BjKF2+CNi2/0QQO9YgWoQJkCR3eoZS3k8QJv8 vQ1+ahcqcqjb19ScSp+93NZjC6NxphEvLQyZgy7/txX77gBBJs+hkM9o8IgPfvRpTAKb m6dA== X-Gm-Message-State: APjAAAUlwuF+u9nsqAaEsWuJNH1ss3VxRUJsXuTlykpdBBTaKE/gL/Mn uImlNy59tEX6cDNNym9sBhlcUYQGWvWa68q8 X-Google-Smtp-Source: APXvYqwx5aZJRbj+tIBEdxjBkQP5HcOZu0OH7/NPHD8GVWrZs6FAocoNeNZ7r2esCdGiNjRri30MRA== X-Received: by 2002:a63:e953:: with SMTP id q19mr2420825pgj.256.1576281213923; Fri, 13 Dec 2019 15:53:33 -0800 (PST) Received: from SBL-LP-YZHAO.skyboxlabs.local (d173-180-108-168.bchsia.telus.net. [173.180.108.168]) by smtp.gmail.com with ESMTPSA id r2sm11926036pgv.16.2019.12.13.15.53.32 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 13 Dec 2019 15:53:33 -0800 (PST) From: Yang Zhao To: git@vger.kernel.org Cc: Yang Zhao , luke@diamand.org, liu.denton@gmail.com, seraphire@gmail.com, szeder.dev@gmail.com Subject: [RFC PATCH v2 14/14] ci: also run linux-gcc pipeline with python3.5 environment Date: Fri, 13 Dec 2019 15:52:48 -0800 Message-Id: <20191213235247.23660-16-yang.zhao@skyboxlabs.com> X-Mailer: git-send-email 2.21.0.windows.1 In-Reply-To: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> References: <20191213235247.23660-1-yang.zhao@skyboxlabs.com> MIME-Version: 1.0 Sender: git-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: git@vger.kernel.org git-p4.py includes support for python3, but this was not previously validated in CI. Lets actually do that. As of writing, python-3.5 has reached end-of-life, but has been updated recently enough that it's reasonable to attemp to support it. We do not have a pressing need for features only available in 3.6 and later. Usage of python3 is limited to the linux-gcc pipeline on Azure. It is assumed that passing both python2 and python3 tests on one platform translates to doing the same on others. Travis-CI is unchanged, as no tests are run in those environments. Signed-off-by: Yang Zhao --- There has been some desire to make this more generally applicable instead of being Azure Pipelines specific. There may be some significant work required to make that work for all platforms where t98** tests are being run. I most likely won't have the bandwith to take on that task in the near future. If this patch is deemed insufficient, I would perfer it be dropped form the series rather than become a roadblock. Previous discussion thread here: http://public-inbox.org/git/20191210103014.GF6527@szeder.dev/ azure-pipelines.yml | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/azure-pipelines.yml b/azure-pipelines.yml index af2a5ea484..c473365812 100644 --- a/azure-pipelines.yml +++ b/azure-pipelines.yml @@ -331,7 +331,18 @@ jobs: displayName: linux-gcc condition: succeeded() pool: Hosted Ubuntu 1604 + strategy: + matrix: + python27: + python.version: '2.7' + python35: + python.version: '3.5' steps: + - task: UsePythonVersion@0 + inputs: + versionSpec: '$(python.version)' + - bash: | + echo "##vso[task.setvariable variable=python_path]$(which python)" - bash: | test "$GITFILESHAREPWD" = '$(gitfileshare.pwd)' || ci/mount-fileshare.sh //gitfileshare.file.core.windows.net/test-cache gitfileshare "$GITFILESHAREPWD" "$HOME/test-cache" || exit 1