From patchwork Fri Oct 27 16:33:41 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nick Desaulniers X-Patchwork-Id: 10030323 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 0EB206034B for ; Fri, 27 Oct 2017 16:34:41 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 1E1F7285AF for ; Fri, 27 Oct 2017 16:34:41 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 126B628607; Fri, 27 Oct 2017 16:34:41 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, DKIM_VALID, RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 804E8285AF for ; Fri, 27 Oct 2017 16:34:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:References: In-Reply-To:Message-Id:Date:Subject:To:From:Reply-To:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Owner; bh=1NDMA0yg+uLWSScCvSfNacPKO4g4SeFhvlRH2fIg6D0=; b=HrWLhcHLORq05xoMR33L6+e2QG VViIEeH7qdaJb5Ttrv3P9vf9WiLhzaxH4pkYVbpHRZQ2536Yd1LDKwX5DJwIWXWt6HQjpkUFYgbnw Zl+Vv78rZV+9WPM+MrhoEPiQggp+0mUDuFu0HvsDoTJrOgstkJCkKu9iox2Uw5EY45KrU1Jaw2sQ9 hK6iufUWbt6FB1Is2pgO1nWhlzDM+swbIYUfc5BigB7rvd966FE/QlcFxsCzZj+9vv83SvgKY+fNC zJvgqobHzG/e32pU6ucMNjF6EEHj2lOyD+LMpRXMHZaDaMVhDZpUiN2ILZER/XqA1631dUdiflNxZ X8S6riBw==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e87aZ-0001rC-0J; Fri, 27 Oct 2017 16:34:31 +0000 Received: from mail-io0-x244.google.com ([2607:f8b0:4001:c06::244]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1e87aU-0001px-DT for linux-arm-kernel@lists.infradead.org; Fri, 27 Oct 2017 16:34:28 +0000 Received: by mail-io0-x244.google.com with SMTP id h70so13920932ioi.4 for ; Fri, 27 Oct 2017 09:34:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=ZEecOEbG6+fzrdETDv/yFXxC3vx8f/0Ng0MFxaBzllQ=; b=dF6e4GKSxZSzUOChOVHy42hi17c+89iOqkzIFpx4am6GWG75g6vf6LTiKbcB0a4As1 W/73mSTd2M/AORL8c0h3sr0/2Hpy1MYqb0MvZcqS6T2ZptyOCRJxvL1ftuGsl+FFsZnD GM57Mq8LdVcZpHsk+zHRW35yX0hhxJzviiblAU+298CvF7JbEw6whzY8ef6xQ8y40Kx/ gj6/EjqTSYiMQK2R0/AU72wmSfU5ryuM3KXydf4nM7I1ZEbJXFdqsJJdsily1GxdxYUv 68v65D4YgdHqp/Do03ZFCljcSWhMwWLQG10H6yBC9A4hna+XBabatO8utNkt7dIn0YB7 ErcQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=ZEecOEbG6+fzrdETDv/yFXxC3vx8f/0Ng0MFxaBzllQ=; b=HzCKif2t4FRP5GMSeKgZnUk1fjVl2tx3j6e5CsCcCoN4MDJyouCWtEKI6r5MbPtpDk u+bka5cvPkqrbEAic0QS/xeZ5Q4FUwPobhlyEVM24KmwwWWE4Uk7rjFFA5fMzJHlMcsh oirwLzixj/W4JuA81auP4x4F/q8wg/W0/FKms4qKdWEcf/yKxIOrR6zL/4ZQ8KFm4gX5 hGa2InWyYrf6gxnRWgh5Hlvk+mayFj5LX+T5j/psjNb5x8Ayul/GLUwiHbkZSSrjEIwo sP4r4HxV29rzGZx+BRPRXghoFh0bH0pnev477lEPIWIzU8eZCiQXFctcEu6I+Y2/9SD4 OYYQ== X-Gm-Message-State: AMCzsaUKXjy9b75nHWbE2nhntevRRD5yHapViDVoxZ3ZWTv7ADkFRYLy mOgQcGZT6mvZHqqkQulnCt++/Q== X-Google-Smtp-Source: ABhQp+QSGSfxzsj6gmiM16cXDDOZnlQuhC1u/iqruaFvN9TRdJ++mK1dv7VyLwk0p/9foOENLq0kWg== X-Received: by 10.107.18.83 with SMTP id a80mr1320322ioj.302.1509122044599; Fri, 27 Oct 2017 09:34:04 -0700 (PDT) Received: from ndesaulniers0.mtv.corp.google.com ([100.98.120.97]) by smtp.gmail.com with ESMTPSA id 200sm1084292itm.13.2017.10.27.09.34.02 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 27 Oct 2017 09:34:03 -0700 (PDT) From: Nick Desaulniers To: Ard Biesheuvel Subject: [PATCH v3] arm64: prevent regressions in compressed kernel image size when upgrading to binutils 2.27 Date: Fri, 27 Oct 2017 09:33:41 -0700 Message-Id: <20171027163341.57550-1-ndesaulniers@google.com> X-Mailer: git-send-email 2.15.0.rc2.357.g7e34df9404-goog In-Reply-To: References: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171027_093426_500441_0E68DA3C X-CRM114-Status: GOOD ( 17.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Jiri Kosina , Kees Cook , Rahul Chaudhry , Catalin Marinas , Gopinath Elanchezhian , Nick Desaulniers , Sindhuri Pentyala , Will Deacon , linux-kernel@vger.kernel.org, Stephen Hines , Wei Wang , linux-arm-kernel@lists.infradead.org, Siqi Lin MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP Upon upgrading to binutils 2.27, we found that our lz4 and gzip compressed kernel images were significantly larger, resulting is 10ms boot time regressions. As noted by Rahul: "aarch64 binaries uses RELA relocations, where each relocation entry includes an addend value. This is similar to x86_64. On x86_64, the addend values are also stored at the relocation offset for relative relocations. This is an optimization: in the case where code does not need to be relocated, the loader can simply skip processing relative relocations. In binutils-2.25, both bfd and gold linkers did this for x86_64, but only the gold linker did this for aarch64. The kernel build here is using the bfd linker, which stored zeroes at the relocation offsets for relative relocations. Since a set of zeroes compresses better than a set of non-zero addend values, this behavior was resulting in much better lz4 compression. The bfd linker in binutils-2.27 is now storing the actual addend values at the relocation offsets. The behavior is now consistent with what it does for x86_64 and what gold linker does for both architectures. The change happened in this upstream commit: https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=1f56df9d0d5ad89806c24e71f296576d82344613 Since a bunch of zeroes got replaced by non-zero addend values, we see the side effect of lz4 compressed image being a bit bigger. To get the old behavior from the bfd linker, "--no-apply-dynamic-relocs" flag can be used: $ LDFLAGS="--no-apply-dynamic-relocs" make With this flag, the compressed image size is back to what it was with binutils-2.25. If the kernel is using ASLR, there aren't additional runtime costs to --no-apply-dynamic-relocs, as the relocations will need to be applied again anyway after the kernel is relocated to a random address. If the kernel is not using ASLR, then presumably the current default behavior of the linker is better. Since the static linker performed the dynamic relocs, and the kernel is not moved to a different address at load time, it can skip applying the relocations all over again." Some measurements: $ ld -v GNU ld (binutils-2.25-f3d35cf6) 2.25.51.20141117 ^ $ ls -l vmlinux -rwxr-x--- 1 ndesaulniers eng 300652760 Oct 26 11:57 vmlinux $ ls -l Image.lz4-dtb -rw-r----- 1 ndesaulniers eng 16932627 Oct 26 11:57 Image.lz4-dtb $ ld -v GNU ld (binutils-2.27-53dd00a1) 2.27.0.20170315 ^ pre patch: $ ls -l vmlinux -rwxr-x--- 1 ndesaulniers eng 300376208 Oct 26 11:43 vmlinux $ ls -l Image.lz4-dtb -rw-r----- 1 ndesaulniers eng 18159474 Oct 26 11:43 Image.lz4-dtb post patch: $ ls -l vmlinux -rwxr-x--- 1 ndesaulniers eng 300376208 Oct 26 12:06 vmlinux $ ls -l Image.lz4-dtb -rw-r----- 1 ndesaulniers eng 16932466 Oct 26 12:06 Image.lz4-dtb By Siqi's measurement w/ gzip: binutils 2.27 with this patch (with --no-apply-dynamic-relocs): Image 41535488 Image.gz 13404067 binutils 2.27 without this patch (without --no-apply-dynamic-relocs): Image 41535488 Image.gz 14125516 Any compression scheme should be able to get better results from the longer runs of zeros, not just GZIP and LZ4. 10ms boot time savings isn't anything to get excited about, but users of arm64+compression+bfd-2.27 should not have to pay a penalty for no runtime improvement. Reported-by: Gopinath Elanchezhian Reported-by: Sindhuri Pentyala Reported-by: Wei Wang Suggested-by: Ard Biesheuvel Suggested-by: Rahul Chaudhry Suggested-by: Siqi Lin Suggested-by: Stephen Hines Signed-off-by: Nick Desaulniers Reviewed-by: Ard Biesheuvel --- Changes since v2: * Combine this this LDFLAGS_vmlinux append to previous block, per Ard. * Add Siqi's measurements to commit message. * Change the conditional to check if set rather than if not empty, since this is similar to the rest of the Makefile, and more readable IMO, though that's subjective. That part can be reverted if we don't want to steal blame. arch/arm64/Makefile | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/arch/arm64/Makefile b/arch/arm64/Makefile index 939b310913cf..669378099dbe 100644 --- a/arch/arm64/Makefile +++ b/arch/arm64/Makefile @@ -14,8 +14,9 @@ LDFLAGS_vmlinux :=-p --no-undefined -X CPPFLAGS_vmlinux.lds = -DTEXT_OFFSET=$(TEXT_OFFSET) GZFLAGS :=-9 -ifneq ($(CONFIG_RELOCATABLE),) -LDFLAGS_vmlinux += -pie -shared -Bsymbolic +ifeq ($(CONFIG_RELOCATABLE), y) +LDFLAGS_vmlinux += -pie -shared -Bsymbolic \ + $(call ld-option, --no-apply-dynamic-relocs) endif ifeq ($(CONFIG_ARM64_ERRATUM_843419),y)