From patchwork Tue Oct 24 14:47:14 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Mark Salyzyn X-Patchwork-Id: 10024691 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 76BEB60245 for ; Tue, 24 Oct 2017 14:48:17 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 63B8A2884B for ; Tue, 24 Oct 2017 14:48:17 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 587D8288F6; Tue, 24 Oct 2017 14:48:17 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,RCVD_IN_DNSWL_MED autolearn=unavailable version=3.3.1 Received: from bombadil.infradead.org (bombadil.infradead.org [65.50.211.133]) (using TLSv1.2 with cipher AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id B54612884B for ; Tue, 24 Oct 2017 14:48:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:MIME-Version:Cc:List-Subscribe: List-Help:List-Post:List-Archive:List-Unsubscribe:List-Id:Message-Id:Date: Subject:To:From:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To: References:List-Owner; bh=2iRJ8SZ2z82YZIjH/w7Q3xpPmP10A8lxyVesa6Crtng=; b=hiS O+JeGmew2bPscUJJS2WVOfeuS8q491ig1HeAIqXb3CQoSPta8UmYizAxiiDhd5Q4Bmo75cZQGQkb3 AuGIhaF/oihEBftmwxTmvu/M6jcaM0BxJAziWbTDLT56t02q+RXp29JHpxR+IvljVKwK2jsQVQizg SGvkB0zaP3ybIa6yaEDDjMSWyEb3mb0B6hOF/twcDZ3xG3Ak3g1Dnx6KUmjrHVGZFLRvjG8Q8RMBo u1EQ8bXRxsGjB9+TG1BTq04+lL7MwDqECYUb4YdhEreq7fGpO8KZyyjLl6E3q2g7dj4vMhUQ1p+3y /MGOsrcfQP+mLFlf7q+yk5Bzb7eVl9A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.87 #1 (Red Hat Linux)) id 1e70V2-0001Hj-S3; Tue, 24 Oct 2017 14:48:12 +0000 Received: from mail-it0-x241.google.com ([2607:f8b0:4001:c0b::241]) by bombadil.infradead.org with esmtps (Exim 4.87 #1 (Red Hat Linux)) id 1e70Uz-00018l-Ed for linux-arm-kernel@lists.infradead.org; Tue, 24 Oct 2017 14:48:11 +0000 Received: by mail-it0-x241.google.com with SMTP id r127so10325592itb.5 for ; Tue, 24 Oct 2017 07:47:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=android.com; s=20161025; h=from:to:cc:subject:date:message-id; bh=gQ19vbRrltqNKhAwjC9vXuFEd84PqOBE218MJk9GyBM=; b=MwZ/uCItKvLp4XS2ARwyD0iGpPQ2b4PettwbENWrzkLNbGFMFbLRlrTr6U8x4xwrLf iBjyRYli71EkHZiuxNorMakug6HGmea+RW8c9LTA2AxeG8JwJdSyUcbK+SR7YIZbCcEk Uhf9GJKIA+JXDUMLGa3bzm5ncv9EMWR6sS/vZoR1yj5VrDS0ZRIu3hzXGPlwX+N8vP6v Sp3oapiJaBozKiXqtwjYx+9MGu4VSMtRZy+UuF1IYxMCpqoouA8m4HPnZ0lryP9zHip7 /UVodSJSJopvPvnZGpBg+nie7vgMTyaYyseR05JOATjuu9XT1et5f6KWPjHrofqxJcNf +yxQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id; bh=gQ19vbRrltqNKhAwjC9vXuFEd84PqOBE218MJk9GyBM=; b=tzfioeBdpG8SBqieMmqyKRGDaH/HKJSHS7vYfe4cj/hZPer8OvVuRtLvGx5nIJlz50 Ql0x8NnrbCILsGtrP5XJPoumehO5KNjFYA/phjJATvv7B5VNH8C5wD63J8yowZk5ECxE g8Tn6VAU8+c0CDSmmeaIjuyCDeV02bswBlMdKQtIlJTbfaWUP/0IonLhgpGGrv9naH3q k1eJRFA7f2EuslnRmi0ViXIMjoSlCYfh6mx/42lmNYHldbrlBy29IwgnWSJklKy85MT8 6OhvucQvAG3tizwIJQ9fxiS7SeyFPERuUiJpk/aU//wJZNvPsFYvKparuF/jy0Js7tog Z6WA== X-Gm-Message-State: AMCzsaV6lSgjEcwUi5WKGqFjF5E1Lf3tZV80kZt/+8rn1eXSoyhFj1gM h5/ByTxvr+MtxDpaS4IkRJIksw== X-Google-Smtp-Source: ABhQp+RkEf+KEeSBHmMsooV6Z55uPuVC/bqTorcsqq/WlN6KgWEiOmbnrlPh1Cwy/hHuyyV9qBy6kQ== X-Received: by 10.36.66.80 with SMTP id i77mr14749399itb.102.1508856467289; Tue, 24 Oct 2017 07:47:47 -0700 (PDT) Received: from nebulus.mtv.corp.google.com ([100.98.120.17]) by smtp.gmail.com with ESMTPSA id 200sm211056itm.13.2017.10.24.07.47.46 (version=TLS1_2 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 24 Oct 2017 07:47:46 -0700 (PDT) From: Mark Salyzyn To: linux-kernel@vger.kernel.org Subject: [PATCH v3] arm64: optimize __memcpy_fromio and __memcpy_toio Date: Tue, 24 Oct 2017 07:47:14 -0700 Message-Id: <20171024144726.12855-1-salyzyn@android.com> X-Mailer: git-send-email 2.15.0.rc0.271.g36b669edcc-goog X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20171024_074809_694106_6FDCED06 X-CRM114-Status: GOOD ( 13.89 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Tony Luck , Kees Cook , Catalin Marinas , Anton Vorontsov , Will Deacon , Mark Salyzyn , Mark Salyzyn , Colin Cross , robin.murphy@arm.com, linux-arm-kernel@lists.infradead.org MIME-Version: 1.0 Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org X-Virus-Scanned: ClamAV using ClamSMTP __memcpy_fromio and __memcpy_toio functions do not deal well with harmonically unaligned addresses unless they can ultimately be copied as quads (u64) to and from the destination. Without a harmonically aligned relationship, they perform byte operations over the entire buffer. Dropped the fragment that tried to align on the normal memory, placing a priority on using quad alignment on the io-side. Removed the volatile on the source for __memcpy_toio as it is unnecessary. This change was motivated by performance issues in the pstore driver. On a test platform, measuring probe time for pstore, console buffer size of 1/4MB and pmsg of 1/2MB, was in the 90-107ms region. Change managed to reduce it to 10-25ms, an improvement in boot time. Signed-off-by: Mark Salyzyn Cc: Kees Cook Cc: Anton Vorontsov Cc: Tony Luck Cc: Catalin Marinas Cc: Will Deacon Cc: Anton Vorontsov Cc: linux-arm-kernel@lists.infradead.org Cc: linux-kernel@vger.kernel.org Cc: robin.murphy@arm.com v2: - simplify, do not try so hard, or through steps, to align on the normal memory side, as it was a diminishing return. Dealing with any pathological short cases was unnecessary since there does not appear to be any. - drop similar __memset_io changes completely. v3: - do not replace 8 with sizeof(u64) --- arch/arm64/kernel/io.c | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/arch/arm64/kernel/io.c b/arch/arm64/kernel/io.c index 354be2a872ae..79b17384effa 100644 --- a/arch/arm64/kernel/io.c +++ b/arch/arm64/kernel/io.c @@ -25,8 +25,7 @@ */ void __memcpy_fromio(void *to, const volatile void __iomem *from, size_t count) { - while (count && (!IS_ALIGNED((unsigned long)from, 8) || - !IS_ALIGNED((unsigned long)to, 8))) { + while (count && !IS_ALIGNED((unsigned long)from, 8)) { *(u8 *)to = __raw_readb(from); from++; to++; @@ -54,23 +53,22 @@ EXPORT_SYMBOL(__memcpy_fromio); */ void __memcpy_toio(volatile void __iomem *to, const void *from, size_t count) { - while (count && (!IS_ALIGNED((unsigned long)to, 8) || - !IS_ALIGNED((unsigned long)from, 8))) { - __raw_writeb(*(volatile u8 *)from, to); + while (count && !IS_ALIGNED((unsigned long)to, 8)) { + __raw_writeb(*(u8 *)from, to); from++; to++; count--; } while (count >= 8) { - __raw_writeq(*(volatile u64 *)from, to); + __raw_writeq(*(u64 *)from, to); from += 8; to += 8; count -= 8; } while (count) { - __raw_writeb(*(volatile u8 *)from, to); + __raw_writeb(*(u8 *)from, to); from++; to++; count--;