From patchwork Tue Feb 12 08:37:45 2013 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Filipe Brandenburger X-Patchwork-Id: 2127711 Return-Path: X-Original-To: patchwork-linux-btrfs@patchwork.kernel.org Delivered-To: patchwork-process-083081@patchwork1.kernel.org Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by patchwork1.kernel.org (Postfix) with ESMTP id 59A5A3FCA4 for ; Tue, 12 Feb 2013 08:44:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758362Ab3BLIox (ORCPT ); Tue, 12 Feb 2013 03:44:53 -0500 Received: from mail-qa0-f73.google.com ([209.85.216.73]:58551 "EHLO mail-qa0-f73.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1758308Ab3BLIow (ORCPT ); Tue, 12 Feb 2013 03:44:52 -0500 Received: by mail-qa0-f73.google.com with SMTP id g10so391482qah.4 for ; Tue, 12 Feb 2013 00:44:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:from:to:cc:subject:date:message-id:x-mailer:in-reply-to :references; bh=V9Zub42OXPZKD/QIUl887X83nR7dyiNAtQ1JRLkIp9E=; b=jAdSjiu3zgn1BYzWkXfG/5HxCbUhDKbTaQiKNg5Lsv7/HEBQkQrNLFZ28mvpnL1Oqq Qoq4v8ib5BAjiBIO9QMnfYkicldeBRi0j4pgrNGPqS/uEauHxPkTb2dCXDYbQUGCqPp4 rkxa7Oz0DaBzO3pR4tfzkh/z0CmumHMvkQPE2k/9heABW+Qql9t3/UWM63Hi4mDuSySZ Bpe/heflzyXAXF6OWK17/6NHaR+Xti1CYy0WZN6To3sjtjsVsCq71vxuCSpcL4XtMXM4 oSnmr0WSgBlyHwSQILrT2Xt2cRKvKHVst4V7cel+WhqyTo3k+M5WAyM+ZRmxUA62M+Oh QH7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:from:to:cc:subject:date:message-id:x-mailer:in-reply-to :references:x-gm-message-state; bh=V9Zub42OXPZKD/QIUl887X83nR7dyiNAtQ1JRLkIp9E=; b=MEycfs+MeZeXlOqmB8MpaFIeT2U0uaMJdp4lEMqWcaNRiVHHIxCkci4Kqjqkd+TERY mGZ9S4bQhS+flxvXXJ0Sw5rgld6EeEFkggcaSopNIKp1Kk2pegwErLcxsruKftviCzGk FxaP4D6SYPYd554RU53EWoxemvPb0R0Sx9TFP8GIZTLfnLAylZYmhGuY9yFeI09FHlRF lEjacGnAyI8n62xAfUYXdj16nEHTy5MLawy/tOYr1uggTx3m/IefVcZwm+PPZdgGDE1/ HY4zOaRKBLMB2P3YDpkoeTchyCWsNIvCzc7MtEpdWAfPKYgzuFly3dJsB6Cp86z8yINo yELg== X-Received: by 10.236.159.100 with SMTP id r64mr8034962yhk.20.1360658281899; Tue, 12 Feb 2013 00:38:01 -0800 (PST) Received: from corp2gmr1-2.hot.corp.google.com (corp2gmr1-2.hot.corp.google.com [172.24.189.93]) by gmr-mx.google.com with ESMTPS id y44si1018960yhl.5.2013.02.12.00.38.01 (version=TLSv1.1 cipher=AES128-SHA bits=128/128); Tue, 12 Feb 2013 00:38:01 -0800 (PST) Received: from obelix.sbo.corp.google.com (obelix.sbo.corp.google.com [172.31.172.210]) by corp2gmr1-2.hot.corp.google.com (Postfix) with ESMTP id 97A985A41FF; Tue, 12 Feb 2013 00:38:01 -0800 (PST) Received: by obelix.sbo.corp.google.com (Postfix, from userid 180819) id 0DD99200DC3; Tue, 12 Feb 2013 00:38:01 -0800 (PST) From: Filipe Brandenburger To: linux-btrfs@vger.kernel.org Cc: Filipe Brandenburger Subject: [PATCH 2/2] btrfs-progs: introduce kernelsrc directory and script to update it Date: Tue, 12 Feb 2013 00:37:45 -0800 Message-Id: <1360658265-6987-3-git-send-email-filbranden@google.com> X-Mailer: git-send-email 1.8.1 In-Reply-To: <1360658265-6987-1-git-send-email-filbranden@google.com> References: <1360658265-6987-1-git-send-email-filbranden@google.com> X-Gm-Message-State: ALoCoQkAbCSmqUrk4tXcuhj27iuPg02idUfCF2pxbc4Ia6b6JgxY05vYJTxPNaTlVioQUj+wxAcR0vXVf5J+FXaxPVvtSdBtD8FpEOylsVp6gBEtXHarctvxDyLdFgV3glcHAKOWVEqB3DlmFzs4p5sN4rgvxQ4D3E6R8FGn9QGBgGTpiB0GM6CztOwz8W/XMGM4KqpiIMgU2tpMO0UFLzbi0ZawRR9pyQ== Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org The kernelsrc directory will contain a copy of the Btrfs-related files from the Linux kernel source tree (mainly the files in fs/btrfs/ directory.) This commit introduces an initial filelist (generated by comparing the list of files in both directories) and a script to fetch the files from a local clone of the kernel git directory into the kernelsrc directory. When updating the directory, a git tag needs to be specified, which is recorded in a local file, in a way that it is possible to track down to which kernel release the kernelsrc files originally belong. A README file was also added to explain the idea behind it and to document the procedure to update the kernelsrc directory. Signed-off-by: Filipe Brandenburger --- kernelsrc/README.kernelsrc | 108 ++++++++++++++++++++++++ kernelsrc/update_kernelsrc.filelist | 23 ++++++ kernelsrc/update_kernelsrc.py | 158 ++++++++++++++++++++++++++++++++++++ 3 files changed, 289 insertions(+) create mode 100644 kernelsrc/README.kernelsrc create mode 100644 kernelsrc/update_kernelsrc.filelist create mode 100755 kernelsrc/update_kernelsrc.py diff --git a/kernelsrc/README.kernelsrc b/kernelsrc/README.kernelsrc new file mode 100644 index 0000000..0aed4bb --- /dev/null +++ b/kernelsrc/README.kernelsrc @@ -0,0 +1,108 @@ + +The kernelsrc directory tree +============================ + +The `kernelsrc' directory in the btrfs-progs tree contains a copy of some of the +files in the Linux kernel tree, mainly the files in fs/btrfs/, that are shared +between the implementation of the filesystem kernel module and some of the +command-line utilities. + +These files should never be edited on the btrfs-progs tree. Changes to these +files should be committed and pushed to the Linux kernel git tree (optionally +using #define's and #ifdef's to make code compatible with both kernel and user +spaces) and then it should be updated here from the kernel tree. + +In order to update the `kernelsrc' directory, you need to have a checkout of +both the btrfs-progs git tree and the Linux kernel git tree. + +Additionally to the kernel source files, the `kernelsrc' directory tree also +contains the scripts and control files used to update this same tree. + + +Updating the kernelsrc tree with update_kernelsrc.py +==================================================== + +In order to update the `kernelsrc' tree to include the source code files from a +newer version of the Linux kernel tree, these are the steps that should be +followed: + + 1. Make sure that you are in the root of the btrfs-progs git tree and that + the tree is clean (i.e., run `git status' and check its output.) + + 2. Make sure you have a clone of the Linux kernel git tree. Let's assume + for this example that your kernel git tree is in directory ../linux/ + + 3. Make sure you have know to which tag of the Linux kernel you want to + upgrade the `kernelsrc' tree to (e.g. v3.7 or v3.8-rc7) + + 4. (Optional:) You may change the list of files that will be copied from + the Linux kernel tree. In order to do that, edit the file + `kernelsrc/update_kernelsrc.filelist' and either add or remove entries + from that file. + + 5. Run the following command from the root of the btrfs-progs git tree: + + $ kernelsrc/update_kernelsrc.py ../linux/ v3.7 + + (assuming Linux kernel git tree at ../linux/ and upgrading to tag v3.7) + + 6. Check whether the update matches your expectations. In particular, the + commands `git status' and `git diff --cached' may be helpful. + + 7. Build btrfs-progs (using `make') and test them to make sure they are + still compatible with the new kernel source tree. Any needed adjustments + may be done by editing files *OUTSIDE* of the `kernelsrc' tree. Once + everything seems to be in place and working correctly, you may add those + files to the same git commit using the `git add' command. + + 8. Commit the change, using the `git commit' command. You may use the + suggested commit command from the script output, with a standard commit + message. The editor will open anyways, so you can add more details + (particularly about files outside the `kernelsrc' tree that needed + changes.) + + +Migration to kernelsrc +====================== + +Using the `kernelsrc' tree allows for a gradual migration from the current +situation, where copies of the original source files from the kernel tree were +made in btrfs-progs, adjustments were made to allow them to compile in user +space, and they diverged further over time. + +First, it allows checking out the files from the kernel tree and adding them to +the btrfs-progs without actually using them initially. Then, it allows gradually +migrating the files one by one. One way to do it would be replacing the file in +the btrfs-progs root directory with a symbolic link to the same file in the +`kernelsrc' tree. Another way would be integrating the build of the `kernelsrc' +files into the Makefile (assuming all of them compile in userspace without +errors) so that they get built and can be linked from there. The integration may +be switched from the former to the latter at any point in a commit that +basically only touches the Makefiles and symlinks. + + +Alternatives to using a script to update kernelsrc +================================================== + +When coming up with the idea for `update_kernelsrc.py', my first idea was to use +`git submodule' instead. It seems to perfectly capture the idea of having a +repository with source code files that are used by a project in another +repository. So, in theory, it would be possible to have the Linux kernel git +tree as a submodule of the btrfs-progs source tree (it could even be placed +under a `kernelsrc' subdirectory.) Git would then track the revision of the +Linux git tree, and it would be possible to upgrade the tree by doing a checkout +on the Linux tree and then committing on the btrfs-progs tree, which would +simply record the git revision where the Linux tree was pointing to at the time +of the commit. + +The main problem with this approach is the size of the Linux git tree. A clone +of the btrfs-progs tree takes about 2 or 3 MB of diskspace, while the Linux tree +takes about 1 GB. I tried to look at options of doing a `sparse checkout' in +git, but it doesn't seem to be possible to do it for a submodule... Even if it +was, I'm not fully convinced that it would reduce the diskspace requirements, as +it would probably still need to fetch the objects into the .git tree anyways. + +So, the idea of a script that would update the tree by copying the files and +keeping the revision in control files seemed like a close enough solution that +would implement a similar concept without the diskspace problems. + diff --git a/kernelsrc/update_kernelsrc.filelist b/kernelsrc/update_kernelsrc.filelist new file mode 100644 index 0000000..50cdf95 --- /dev/null +++ b/kernelsrc/update_kernelsrc.filelist @@ -0,0 +1,23 @@ +fs/btrfs/ctree.c +fs/btrfs/ctree.h +fs/btrfs/dir-item.c +fs/btrfs/disk-io.c +fs/btrfs/disk-io.h +fs/btrfs/extent_io.c +fs/btrfs/extent_io.h +fs/btrfs/extent-tree.c +fs/btrfs/file-item.c +fs/btrfs/hash.h +fs/btrfs/inode-item.c +fs/btrfs/inode-map.c +fs/btrfs/ioctl.h +fs/btrfs/print-tree.c +fs/btrfs/print-tree.h +fs/btrfs/qgroup.c +fs/btrfs/root-tree.c +fs/btrfs/send.h +fs/btrfs/transaction.h +fs/btrfs/version.h +fs/btrfs/volumes.c +fs/btrfs/volumes.h +include/linux/list.h diff --git a/kernelsrc/update_kernelsrc.py b/kernelsrc/update_kernelsrc.py new file mode 100755 index 0000000..0608671 --- /dev/null +++ b/kernelsrc/update_kernelsrc.py @@ -0,0 +1,158 @@ +#! /usr/bin/python + +import os +import subprocess +import sys + +SCRIPT_NAME = 'kernelsrc/update_kernelsrc.py' +KERNELSRC_GIT_DIR = None +KERNELSRC_WORK_TREE = 'kernelsrc/' + +# read filelist stripping comments and blank lines +def read_filelist(): + with open('kernelsrc/update_kernelsrc.filelist') as f: + filelist = [l for l in (l.split('#')[0].strip() for l in f.readlines()) if l] + if filelist: + return filelist + else: + raise Exception('update_kernelsrc.filelist does not contain any files') + +# checkout files from kernel git tree +def checkout_kernelsrc(rev, filelist): + cmdline = ['git', '--git-dir=' + KERNELSRC_GIT_DIR, '--work-tree=' + KERNELSRC_WORK_TREE, + 'checkout', rev, '--'] + filelist + rc = subprocess.call(cmdline) + if rc != 0: + raise Exception('git checkout from kernel source git repo returned %d' % rc) + +# run a git add on the updated kernelsrc tree to add the changes to the index +def add_kernelsrc_to_index(): + # add them to git index of this git tree + rc = subprocess.call(['git', 'add', KERNELSRC_WORK_TREE]) + if rc != 0: + raise Exception('git add %s returned %d' % (KERNELSRC_WORK_TREE, rc)) + +# remove current files from kernelsrc to prepare to receive new ones +def cleanup_kernelsrc(): + for dirpath, dirnames, filenames in os.walk(KERNELSRC_WORK_TREE): + for f in filenames: + # ignore files named update_kernelsrc.* and README* on root of the kernelsrc/ tree: + if (dirpath == KERNELSRC_WORK_TREE) and (f.startswith('update_kernelsrc.') or f.startswith('README')): + continue + # otherwise, git rm it + filepath = os.path.join(dirpath, f) + rc = subprocess.call(['git', 'rm', '-q', filepath]) + if rc != 0: + raise Exception('git rm %s returned %d' % (filepath, rc)) + # second pass: remove directories, use a stack to remove them in the right order + stack = [] + for dirpath, dirnames, filenames in os.walk(KERNELSRC_WORK_TREE): + if dirpath != KERNELSRC_WORK_TREE: + stack.append(dirpath) + for d in reversed(stack): + os.rmdir(d) + +# check whether current tree is clean and ready to receive update +# only allowed change in index is to kernelsrc/update_kernelsrc.filelist +# changes not in index must be outside of kernelsrc tree +def check_for_clean_tree(): + status = subprocess.check_output(['git', 'status', '--porcelain']).splitlines() + for l in status: + worktree = l[0] + index = l[1] + path = l[3:] + if (path == os.path.join(KERNELSRC_WORK_TREE, 'update_kernelsrc.filelist')) and (worktree in ' M') and (index in ' M'): + # accept changes to the filelist in index and/or worktree + continue + if (not path.startswith(KERNELSRC_WORK_TREE)) and (worktree == '?') and (index == '?'): + # accept files with unknown status outside of kernelsrc + continue + # otherwise, abort + raise Exception('current tree is not clean! (e.g. %s)' % path) + +# resolve tag into revision +def resolve_tag(tag): + cmdline = ['git', '--git-dir=' + KERNELSRC_GIT_DIR, 'rev-parse', tag + '^0'] + rev = subprocess.check_output(cmdline).strip() + return rev + +# check whether the new tag is an upgrade from the previous one +# in other words, if the previous tag is an ancestor of the new one +def check_upgrade(rev): + # previous revision stored in kernelsrc/update_kernelsrc.rev + oldrevpath = os.path.join(KERNELSRC_WORK_TREE, 'update_kernelsrc.rev') + if not os.path.exists(oldrevpath): + # if it doesn't exist, then just assume it's OK for an upgrade + # but return None in order to signal the this is the first checkout + return None + with open(oldrevpath) as f: + oldrev = f.read().strip() + # use git merge-base to check what's the merge base between them + # if it's an upgrade, it will match oldrev + cmdline = ['git', '--git-dir=' + KERNELSRC_GIT_DIR, 'merge-base', oldrev, rev] + mergebase = subprocess.check_output(cmdline).strip() + if mergebase != oldrev: + raise Exception('new revision %s is not a descendent of previous revision %s' % (rev, oldrev)) + oldtagpath = os.path.join(KERNELSRC_WORK_TREE, 'update_kernelsrc.tag') + # return the old tag (for git commit sample message) + try: + with open(oldtagpath) as f: + oldtag = f.read().strip() + return oldtag + except OSError: + return oldrev + +# store tag (and revision) for future query/comparison +def store_tag(tag, rev): + with open(os.path.join(KERNELSRC_WORK_TREE, 'update_kernelsrc.tag'), 'w') as f: + print >>f, tag + with open(os.path.join(KERNELSRC_WORK_TREE, 'update_kernelsrc.rev'), 'w') as f: + print >>f, rev + +# do it all! +def main(): + global KERNELSRC_GIT_DIR + if len(sys.argv) != 3: + raise Exception('usage: %s ' % SCRIPT_NAME) + if sys.argv[0] != SCRIPT_NAME: + raise Exception('script must be called as %s from root directory in btrfs-progs tree' % SCRIPT_NAME) + if not os.path.isdir('.git'): + raise Exception('script must be called from root directory in btrfs-progs git tree') + if not os.path.isdir(KERNELSRC_WORK_TREE): + raise Exception('weird, this looks like a git tree but could not find directory %s' % KERNELSRC_WORK_TREE) + if not os.path.isdir(sys.argv[1]): + raise Exception('first parameter must be the path to the kernel source directory') + KERNELSRC_GIT_DIR = os.path.join(sys.argv[1], '.git') + if not os.path.isdir(KERNELSRC_GIT_DIR): + raise Exception('first parameter must be the path to the root of the kernel git tree') + # all checks so far, let's get the new tag and see if it's good: + tag = sys.argv[2] + rev = resolve_tag(tag) + oldtag = check_upgrade(tag) + # all good, let's check if the tree is clean + check_for_clean_tree() + # let's see if we can get the file list ok + filelist = read_filelist() + # all good! now let's do the update (remove followed by checkout) + cleanup_kernelsrc() + checkout_kernelsrc(rev, filelist) + # done upgrade, still not in index though... + # before adding it to the index, let's record tag and rev + store_tag(tag, rev) + # and finally, add to git index + add_kernelsrc_to_index() + # done! + print 'Done upgrading kernelsrc to %s.' % tag + print 'Please test the new tree and commit when ready.' + print '' + print 'Use the following command for commit:' + if oldtag is None: + print ' git commit -em "btrfs-progs: initial checkout of kernelsrc %s"' % tag + else: + print ' git commit -em "btrfs-progs: upgrade kernelsrc from %s to %s"' % (oldtag, tag) + print '' + +# if called as a script, just run main +if __name__ == '__main__': + main() +