From patchwork Fri Oct 8 13:36:02 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hyeonggon Yoo <42.hyeyoo@gmail.com> X-Patchwork-Id: 12545357 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B4045C433EF for ; Fri, 8 Oct 2021 13:36:21 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3D3C761042 for ; Fri, 8 Oct 2021 13:36:21 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 3D3C761042 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=kvack.org Received: by kanga.kvack.org (Postfix) id A3D8D900002; Fri, 8 Oct 2021 09:36:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9C5BC6B0072; Fri, 8 Oct 2021 09:36:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 866B2900002; Fri, 8 Oct 2021 09:36:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0175.hostedemail.com [216.40.44.175]) by kanga.kvack.org (Postfix) with ESMTP id 7835A6B0071 for ; Fri, 8 Oct 2021 09:36:20 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 2CA3A27743 for ; Fri, 8 Oct 2021 13:36:20 +0000 (UTC) X-FDA: 78673369320.19.4F6FAE1 Received: from mail-pf1-f172.google.com (mail-pf1-f172.google.com [209.85.210.172]) by imf21.hostedemail.com (Postfix) with ESMTP id DD638D03BD61 for ; Fri, 8 Oct 2021 13:36:19 +0000 (UTC) Received: by mail-pf1-f172.google.com with SMTP id q19so7797721pfl.4 for ; Fri, 08 Oct 2021 06:36:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=lAa9I0nj7yA9YxZJQiDgv3qskMPgBaJlC94AloAQu6s=; b=SH/zWictbgR+u23oYMfUZA3ANPrWTpNhSxXfQKUxGULjXHqWglu9B0FDgIId8kmoX2 O0saEhkGXtupWgHlW4k5hMXNWZJxQSQk8wSeuvFTzEPd01O6hMh1dTrqDj5q6A4oJH3o GGlREyXWZMO8ZS5iXEfOn673L7wsBBHmlU8Mpgsqw7hdLnAATqf2G6uOZh/UhurllwCd 22d/RJ+vk8ibxJ1DC4uWl2QvtMb5LD8x1ua9EEmjEJ69XQa+FQ0sNsDKn2RgwE+GFSZL I1EZlKBeYyN3rc8b39U2tkQ6jAh2UZhWpeD72QDbqoMqspc24tIIIONun+m6pkPQXSEH wG5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=lAa9I0nj7yA9YxZJQiDgv3qskMPgBaJlC94AloAQu6s=; b=uiOpTaB/KBdHVOmltqK1EanX4FxgwY4fmI9mXNeT29r56A3+I25zjG7B6AHNEeSpWv PH+UpaaIc5L9ttOJ1g83go1+MJZmbeBynn2gsS2eCWQKaOkTuFxHdiHrqtOXjcCbN4q0 DZc4hN4bN0jYxoUm98/ko8P0SsGhVoyno9cJDZGkCzKDMW31y4PUK6dzgWLRHnbCzF64 A/KWuGtPx+lVIYvQMJtTxU573GO7Pb2VUXeJOYP9kAMj+7yOxqDEwQXqN4UMQqlvKjkI GLApw4p34jmr49nh+2updZb9gTKWlDMDpe70LDbBus9S6TmiPHuczBd+bK4D4pEEOxb9 jXlg== X-Gm-Message-State: AOAM533l9M563QReq3mMBtsEmfJDOmD501rwn13CrCwzNoJCh9FsKD1Y 4wlAZlLW03Y0a/7+QzYMclezWtos03Y= X-Google-Smtp-Source: ABdhPJwddVi2Idj4D1PKq2S3bOtrf8TL0aEdfBhVPeA8LLSxyTn/oxku4KZqxAE6LBgNmHbbKTZa1g== X-Received: by 2002:a05:6a00:181c:b0:44c:bf17:e8ad with SMTP id y28-20020a056a00181c00b0044cbf17e8admr9332555pfa.67.1633700178624; Fri, 08 Oct 2021 06:36:18 -0700 (PDT) Received: from kvm.asia-northeast3-a.c.our-ratio-313919.internal (24.151.64.34.bc.googleusercontent.com. [34.64.151.24]) by smtp.gmail.com with ESMTPSA id x7sm2944985pfj.164.2021.10.08.06.36.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 08 Oct 2021 06:36:17 -0700 (PDT) From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Hyeonggon Yoo <42.hyeyoo@gmail.com> Subject: [PATCH] mm, slub: Use prefetchw instead of prefetch Date: Fri, 8 Oct 2021 13:36:02 +0000 Message-Id: <20211008133602.4963-1-42.hyeyoo@gmail.com> X-Mailer: git-send-email 2.27.0 MIME-Version: 1.0 Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b="SH/zWict"; spf=pass (imf21.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.210.172 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: DD638D03BD61 X-Stat-Signature: qon314hn7zpqjwy4r3ky156wx4mgb6zg X-HE-Tag: 1633700179-281998 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: It's certain that an object will be not only read, but also written after allocation. Use prefetchw instead of prefetchw. On supported architecture like x86, it helps to invalidate cache line when the object exists in other processors' cache. Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> --- mm/slub.c | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-) diff --git a/mm/slub.c b/mm/slub.c index 3d2025f7163b..2aca7523165e 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -352,9 +352,9 @@ static inline void *get_freepointer(struct kmem_cache *s, void *object) return freelist_dereference(s, object + s->offset); } -static void prefetch_freepointer(const struct kmem_cache *s, void *object) +static void prefetchw_freepointer(const struct kmem_cache *s, void *object) { - prefetch(object + s->offset); + prefetchw(object + s->offset); } static inline void *get_freepointer_safe(struct kmem_cache *s, void *object) @@ -3195,10 +3195,9 @@ static __always_inline void *slab_alloc_node(struct kmem_cache *s, note_cmpxchg_failure("slab_alloc", s, tid); goto redo; } - prefetch_freepointer(s, next_object); + prefetchw_freepointer(s, next_object); stat(s, ALLOC_FASTPATH); } - maybe_wipe_obj_freeptr(s, object); init = slab_want_init_on_alloc(gfpflags, s);