From patchwork Sun Jul 28 20:29:58 2024 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Rong Xu X-Patchwork-Id: 13744052 Received: from mail-yw1-f201.google.com (mail-yw1-f201.google.com [209.85.128.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 248D07D3F5 for ; Sun, 28 Jul 2024 20:31:04 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.128.201 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722198665; cv=none; b=lwOFGYfyDDxWyNXxtuUhXpnx4crV6qEFQDgLwNNhh3ZcfRYFb3kM23s3wjQvEfmGn0BSBtIctMYvngunhK5NkSc1vBr+Mnq+OhW1eDjedn4gA/DEEnOFz7+D4D4gs4D34EUMvcQ7B7BsdaoLOLLM4BiFgBLaJdehemJxcyIujIM= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1722198665; c=relaxed/simple; bh=jgG0TSTo1c+hpUBT22yEfSilFpbm6otN2MQr5+RKpfs=; h=Date:In-Reply-To:Mime-Version:References:Message-ID:Subject:From: To:Cc:Content-Type; b=g30sXWn1DEd/dTuekplUlByvMB8IRjBpJUeJyZ+Srz2dmls2jje1nRqTbcFZzoCl7rUIDTSdoPI9rmu02dVWHDulj9efLZqnBIE1Z6c/O/CMS+8y/+CE0y3SurEGl8XtA2MdqY888SvwBuoFeOW9ql30aBRgEyRVrUHW7G6C6vY= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com; spf=pass smtp.mailfrom=flex--xur.bounces.google.com; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b=f6BjeUdH; arc=none smtp.client-ip=209.85.128.201 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=reject dis=none) header.from=google.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=flex--xur.bounces.google.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=google.com header.i=@google.com header.b="f6BjeUdH" Received: by mail-yw1-f201.google.com with SMTP id 00721157ae682-66890dbb7b8so46458847b3.0 for ; Sun, 28 Jul 2024 13:31:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722198663; x=1722803463; darn=vger.kernel.org; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:from:to:cc:subject:date:message-id:reply-to; bh=Tokg3sFCTTXy/Tg/kyuYTkBel/otDfHpuf9TI5sSKRc=; b=f6BjeUdHR36Kg9BILRErBF2TT4LyUUUeWe/2pIZ5RJ+8bUz3/NbWjd66ppYH8KLMcg wybYF/uZjne3GMXh1E+X1zAYra/WCpKEUBjZOVjjzz0/kOx4veBmncDp4c0wvtTsPXg5 Q7gXKDwUc91Ffz70gcy8siG+Z0Z8BHkr4mNeJMykTEEKau1hFbkhlp/9cmPjSSyWsQTe MpRwuxKr7a7JOtgdouDxBAK4YVsVaopo8toXBrxqtqBLzkmJ/gJ/volGOdGKRdngDgrw IAbShFBlEXNFgyUQgorAI/NfrnUVubUQo28Ed5qPYkmr/8FVyRU0slQDy9KU844a+4q/ YX4A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722198663; x=1722803463; h=cc:to:from:subject:message-id:references:mime-version:in-reply-to :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=Tokg3sFCTTXy/Tg/kyuYTkBel/otDfHpuf9TI5sSKRc=; b=iTmTozmudMI2sGt4K5Wb89KWb44Rv5fMFQLvtat6/p38GxpOpxegUkCTgczHCs2kBZ MAa3fqAGUqK3ig1i8/InIASPvyNeUyUSegWpDLmowKrftW3HDAWitQr04mv7Y8vla++7 zLuakPhPckNBvQcGHbNtL7RMO1/1dfdpdfZvFmQIs2oOTPJm4vVOUGV9HJLr9eR2gGAg TpyboAYXw8j0ULyGp7eeOnSnxYKn93Ud4Jq2NPbXrzYJ6zl8PRPVlV43CKEbxiuoTLLj /QlW2LurULbSQ5sCMtMLZ0wrnCInK7qYnb96YAIXrgB0tp5jVNRz0qnHfA73HRIFcrfy VrmQ== X-Forwarded-Encrypted: i=1; AJvYcCXNfY9qQ+iLsrx6VM9Gwl2uQTzUQ5tWZ/tEJJ480hbcFZBcpaOIGi0JAVHEXqROHEytPzgfXl4I50gOZWYu9NKFn5KYpkMOa0l2r9HN X-Gm-Message-State: AOJu0Yz6FprvZaZr4aI2/2a9w30g4SVy8MipkJK5gHwR6KFmB+4laL/j w19QCtCUSzT+7aUbYoH2Ut4TzIsRNQ7TaogeqI4zYMwlIgh2kTdv1LfIOpFud6KpcQ== X-Google-Smtp-Source: AGHT+IGFJCBpCGKio3GvIQMtPQU0FwuQLnvFXOWRFKgHtvK/N12utXtZSpDgbRkZ6jpM9dmXhNWYM7k= X-Received: from xur.c.googlers.com ([fda3:e722:ac3:cc00:20:ed76:c0a8:2330]) (user=xur job=sendgmr) by 2002:a05:690c:827:b0:667:8a45:d0f9 with SMTP id 00721157ae682-67a004a2775mr1250767b3.0.1722198663143; Sun, 28 Jul 2024 13:31:03 -0700 (PDT) Date: Sun, 28 Jul 2024 13:29:58 -0700 In-Reply-To: <20240728203001.2551083-1-xur@google.com> Precedence: bulk X-Mailing-List: linux-kbuild@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: Mime-Version: 1.0 References: <20240728203001.2551083-1-xur@google.com> X-Mailer: git-send-email 2.46.0.rc1.232.g9752f9e123-goog Message-ID: <20240728203001.2551083-6-xur@google.com> Subject: [PATCH 5/6] AutoFDO: Enable machine function split optimization for AutoFDO From: Rong Xu To: Rong Xu , Han Shen , Sriraman Tallam , David Li , Jonathan Corbet , Masahiro Yamada , Nathan Chancellor , Nicolas Schier , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , x86@kernel.org, "H . Peter Anvin" , Ard Biesheuvel , Arnd Bergmann , Josh Poimboeuf , Peter Zijlstra , Nick Desaulniers , Bill Wendling , Justin Stitt , Vegard Nossum , John Moon , Andrew Morton , Heiko Carstens , Luis Chamberlain , Samuel Holland , Mike Rapoport , "Paul E . McKenney" , Rafael Aquini , Petr Pavlu , Eric DeVolder , Bjorn Helgaas , Randy Dunlap , Benjamin Segall , Breno Leitao , Wei Yang , Brian Gerst , Juergen Gross , Palmer Dabbelt , Alexandre Ghiti , Kees Cook , Sami Tolvanen , Xiao Wang , Jan Kiszka Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-kbuild@vger.kernel.org, linux-efi@vger.kernel.org, linux-arch@vger.kernel.org, llvm@lists.linux.dev, Krzysztof Pszeniczny Enable the machine function split optimization for AutoFDO in Clang. Machine function split (MFS) is a pass in the Clang compiler that splits a function into hot and cold parts. The linker groups all cold blocks across functions together. This decreases hot code fragmentation and improves iCache and iTLB utilization. MFS requires a profile so this is enabled only for the AutoFDO builds. Co-developed-by: Han Shen Signed-off-by: Han Shen Signed-off-by: Rong Xu Suggested-by: Sriraman Tallam Suggested-by: Krzysztof Pszeniczny --- include/asm-generic/vmlinux.lds.h | 6 ++++++ scripts/Makefile.autofdo | 2 ++ 2 files changed, 8 insertions(+) diff --git a/include/asm-generic/vmlinux.lds.h b/include/asm-generic/vmlinux.lds.h index 97c8399e5532..7d9dc8a3c046 100644 --- a/include/asm-generic/vmlinux.lds.h +++ b/include/asm-generic/vmlinux.lds.h @@ -593,9 +593,14 @@ defined(CONFIG_AUTOFDO_CLANG) __unlikely_text_start = .; \ *(.text.unlikely .text.unlikely.*) \ __unlikely_text_end = .; +#define TEXT_SPLIT \ + __split_text_start = .; \ + *(.text.split .text.split.[0-9a-zA-Z_]*) \ + __split_text_end = .; #else #define TEXT_HOT *(.text.hot .text.hot.*) #define TEXT_UNLIKELY *(.text.unlikely .text.unlikely.*) +#define TEXT_SPLIT #endif /* @@ -611,6 +616,7 @@ defined(CONFIG_AUTOFDO_CLANG) #define TEXT_TEXT \ *(.text.asan.* .text.tsan.*) \ *(.text.unknown .text.unknown.*) \ + TEXT_SPLIT \ TEXT_UNLIKELY \ ALIGN_FUNCTION(); \ TEXT_HOT \ diff --git a/scripts/Makefile.autofdo b/scripts/Makefile.autofdo index f765bd9e81d7..80ad06689947 100644 --- a/scripts/Makefile.autofdo +++ b/scripts/Makefile.autofdo @@ -6,6 +6,7 @@ CFLAGS_AUTOFDO_CLANG := -fdebug-info-for-profiling -mllvm -enable-fs-discriminat ifdef CLANG_AUTOFDO_PROFILE CFLAGS_AUTOFDO_CLANG += -fprofile-sample-use=$(CLANG_AUTOFDO_PROFILE) -ffunction-sections +CFLAGS_AUTOFDO_CLANG += -fsplit-machine-functions endif ifdef CONFIG_LTO_CLANG @@ -14,6 +15,7 @@ ifdef CLANG_AUTOFDO_PROFILE KBUILD_LDFLAGS += --lto-sample-profile=$(CLANG_AUTOFDO_PROFILE) endif KBUILD_LDFLAGS += --mllvm=-enable-fs-discriminator=true --mllvm=-improved-fs-discriminator=true -plugin-opt=thinlto +KBUILD_LDFLAGS += -plugin-opt=-split-machine-functions endif endif