From patchwork Thu Sep 3 07:01:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11752515 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id EFDF113B1 for ; Thu, 3 Sep 2020 07:01:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9E42120716 for ; Thu, 3 Sep 2020 07:01:44 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9E42120716 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 756F78E0001; Thu, 3 Sep 2020 03:01:43 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 6E08F6B005D; Thu, 3 Sep 2020 03:01:43 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5A9838E0001; Thu, 3 Sep 2020 03:01:43 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0169.hostedemail.com [216.40.44.169]) by kanga.kvack.org (Postfix) with ESMTP id 410606B005A for ; Thu, 3 Sep 2020 03:01:43 -0400 (EDT) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id F3D6E180AD807 for ; Thu, 3 Sep 2020 07:01:42 +0000 (UTC) X-FDA: 77220854886.04.knife80_27028a4270a7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id CAD208006433 for ; Thu, 3 Sep 2020 07:01:42 +0000 (UTC) X-Spam-Summary: 1,0,0,a20fc54c729570a9,d41d8cd98f00b204,alex.shi@linux.alibaba.com,,RULES_HIT:2:41:69:355:379:541:800:960:973:988:989:1260:1261:1345:1431:1437:1535:1605:1606:1730:1747:1777:1792:2393:2559:2562:2693:2742:3138:3139:3140:3141:3142:3865:3866:3867:3870:3871:3872:3874:4117:4321:4605:5007:6117:6261:7903:8531:8603:9036:10004:11026:11473:11658:11914:12043:12048:12296:12297:12438:12555:12683:12895:12986:13138:13231:13255:13846:14394:21060:21080:21324:21451:21627:21990:30054:30070,0,RBL:115.124.30.131:@linux.alibaba.com:.lbl8.mailshell.net-62.20.2.100 64.201.201.201;04ygpb4xjg5i5y5xaa61697zyyfd4ocw7i4ni4a48i39n4oxnq3sqhbxf8xj1ct.wuqkdf387ezyz7bq6aatt4gazxnuudm8qpiqtdwfbbfm3g7ff443hdxkts6ah56.r-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:24,LUA_SUMMARY:none X-HE-Tag: knife80_27028a4270a7 X-Filterd-Recvd-Size: 6584 Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) by imf30.hostedemail.com (Postfix) with ESMTP for ; Thu, 3 Sep 2020 07:01:40 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R201e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e04420;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=9;SR=0;TI=SMTPD_---0U7n-ifu_1599116494; Received: from aliy80.localdomain(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0U7n-ifu_1599116494) by smtp.aliyun-inc.com(127.0.0.1); Thu, 03 Sep 2020 15:01:35 +0800 From: Alex Shi To: Anshuman Khandual , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Alexander Duyck Cc: Andrew Morton , Mel Gorman , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v4 1/4] mm/pageblock: mitigation cmpxchg false sharing in pageblock flags Date: Thu, 3 Sep 2020 15:01:20 +0800 Message-Id: <1599116482-7410-1-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 X-Rspamd-Queue-Id: CAD208006433 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam03 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: pageblock_flags is used as long, since every pageblock_flags is just 4 bits, 'long' size will include 8(32bit machine) or 16 pageblocks' flags, that flag setting has to sync in cmpxchg with 7 or 15 other pageblock flags. It would cause long waiting for sync. If we could change the pageblock_flags variable as char, we could use char size cmpxchg, which just sync up with 2 pageblock flags. it could relief the false sharing in cmpxchg. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Mel Gorman Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Signed-off-by: Alex Shi --- include/linux/mmzone.h | 6 +++--- include/linux/pageblock-flags.h | 2 +- mm/page_alloc.c | 38 +++++++++++++++++++------------------- 3 files changed, 23 insertions(+), 23 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index 8379432f4f2f..be676e659fb7 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -437,7 +437,7 @@ struct zone { * Flags for a pageblock_nr_pages block. See pageblock-flags.h. * In SPARSEMEM, this map is stored in struct mem_section */ - unsigned long *pageblock_flags; + unsigned char *pageblock_flags; #endif /* CONFIG_SPARSEMEM */ /* zone_start_pfn == zone_start_paddr >> PAGE_SHIFT */ @@ -1159,7 +1159,7 @@ struct mem_section_usage { DECLARE_BITMAP(subsection_map, SUBSECTIONS_PER_SECTION); #endif /* See declaration of similar field in struct zone */ - unsigned long pageblock_flags[0]; + unsigned char pageblock_flags[0]; }; void subsection_map_init(unsigned long pfn, unsigned long nr_pages); @@ -1212,7 +1212,7 @@ struct mem_section { extern struct mem_section mem_section[NR_SECTION_ROOTS][SECTIONS_PER_ROOT]; #endif -static inline unsigned long *section_to_usemap(struct mem_section *ms) +static inline unsigned char *section_to_usemap(struct mem_section *ms) { return ms->usage->pageblock_flags; } diff --git a/include/linux/pageblock-flags.h b/include/linux/pageblock-flags.h index fff52ad370c1..d189441568eb 100644 --- a/include/linux/pageblock-flags.h +++ b/include/linux/pageblock-flags.h @@ -54,7 +54,7 @@ enum pageblock_bits { /* Forward declaration */ struct page; -unsigned long get_pfnblock_flags_mask(struct page *page, +unsigned char get_pfnblock_flags_mask(struct page *page, unsigned long pfn, unsigned long mask); diff --git a/mm/page_alloc.c b/mm/page_alloc.c index fab5e97dc9ca..81e96d4d9c42 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -445,7 +445,7 @@ static inline bool defer_init(int nid, unsigned long pfn, unsigned long end_pfn) #endif /* Return a pointer to the bitmap storing bits affecting a block of pages */ -static inline unsigned long *get_pageblock_bitmap(struct page *page, +static inline unsigned char *get_pageblock_bitmap(struct page *page, unsigned long pfn) { #ifdef CONFIG_SPARSEMEM @@ -474,24 +474,24 @@ static inline int pfn_to_bitidx(struct page *page, unsigned long pfn) * Return: pageblock_bits flags */ static __always_inline -unsigned long __get_pfnblock_flags_mask(struct page *page, +unsigned char __get_pfnblock_flags_mask(struct page *page, unsigned long pfn, unsigned long mask) { - unsigned long *bitmap; - unsigned long bitidx, word_bitidx; - unsigned long word; + unsigned char *bitmap; + unsigned long bitidx, byte_bitidx; + unsigned char byte; bitmap = get_pageblock_bitmap(page, pfn); bitidx = pfn_to_bitidx(page, pfn); - word_bitidx = bitidx / BITS_PER_LONG; - bitidx &= (BITS_PER_LONG-1); + byte_bitidx = bitidx / BITS_PER_BYTE; + bitidx &= (BITS_PER_BYTE-1); - word = bitmap[word_bitidx]; - return (word >> bitidx) & mask; + byte = bitmap[byte_bitidx]; + return (byte >> bitidx) & mask; } -unsigned long get_pfnblock_flags_mask(struct page *page, unsigned long pfn, +unsigned char get_pfnblock_flags_mask(struct page *page, unsigned long pfn, unsigned long mask) { return __get_pfnblock_flags_mask(page, pfn, mask); @@ -513,29 +513,29 @@ void set_pfnblock_flags_mask(struct page *page, unsigned long flags, unsigned long pfn, unsigned long mask) { - unsigned long *bitmap; - unsigned long bitidx, word_bitidx; - unsigned long old_word, word; + unsigned char *bitmap; + unsigned long bitidx, byte_bitidx; + unsigned char old_byte, byte; BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 4); BUILD_BUG_ON(MIGRATE_TYPES > (1 << PB_migratetype_bits)); bitmap = get_pageblock_bitmap(page, pfn); bitidx = pfn_to_bitidx(page, pfn); - word_bitidx = bitidx / BITS_PER_LONG; - bitidx &= (BITS_PER_LONG-1); + byte_bitidx = bitidx / BITS_PER_BYTE; + bitidx &= (BITS_PER_BYTE-1); VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page); mask <<= bitidx; flags <<= bitidx; - word = READ_ONCE(bitmap[word_bitidx]); + byte = READ_ONCE(bitmap[byte_bitidx]); for (;;) { - old_word = cmpxchg(&bitmap[word_bitidx], word, (word & ~mask) | flags); - if (word == old_word) + old_byte = cmpxchg(&bitmap[byte_bitidx], byte, (byte & ~mask) | flags); + if (byte == old_byte) break; - word = old_word; + byte = old_byte; } } From patchwork Thu Sep 3 07:01:21 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11752517 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id C9E90109B for ; Thu, 3 Sep 2020 07:01:46 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 9F94120716 for ; Thu, 3 Sep 2020 07:01:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 9F94120716 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 316A66B005A; Thu, 3 Sep 2020 03:01:44 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 24EB56B005D; Thu, 3 Sep 2020 03:01:44 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0C7296B0062; Thu, 3 Sep 2020 03:01:44 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0221.hostedemail.com [216.40.44.221]) by kanga.kvack.org (Postfix) with ESMTP id E214C6B005A for ; Thu, 3 Sep 2020 03:01:43 -0400 (EDT) Received: from smtpin26.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id A3A8B180AD80F for ; Thu, 3 Sep 2020 07:01:43 +0000 (UTC) X-FDA: 77220854886.26.lake13_0b1336e270a7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin26.hostedemail.com (Postfix) with ESMTP id 76A7C1804B655 for ; Thu, 3 Sep 2020 07:01:43 +0000 (UTC) X-Spam-Summary: 1,0,0,e6f3ea9e61318d78,d41d8cd98f00b204,alex.shi@linux.alibaba.com,,RULES_HIT:41:355:379:421:541:800:960:966:973:988:989:1260:1261:1345:1359:1431:1437:1534:1543:1711:1730:1747:1777:1792:2196:2199:2393:2559:2562:2693:2740:3138:3139:3140:3141:3142:3354:3865:3867:3868:3870:3871:3872:4321:4385:4605:5007:6261:6630:8531:8603:9592:10004:11026:11232:11473:11658:11914:12043:12048:12296:12297:12438:12555:12895:12986:13161:13229:13846:14181:14394:14721:21060:21080:21222:21451:21627:21966:30054,0,RBL:115.124.30.130:@linux.alibaba.com:.lbl8.mailshell.net-64.201.201.201 62.20.2.100;04yru666tongwrcjmfh8ymj6zis95ocoaqrb5m4p6rxe7qxydb4f8q67t6m1qop.o3dhd4unbyytyngkphtagcjmomrk3p6mbiwe3d4pryn7da77fpibm439d4qwh7x.a-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:0:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: lake13_0b1336e270a7 X-Filterd-Recvd-Size: 4636 Received: from out30-130.freemail.mail.aliyun.com (out30-130.freemail.mail.aliyun.com [115.124.30.130]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Thu, 3 Sep 2020 07:01:41 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R211e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e01355;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=9;SR=0;TI=SMTPD_---0U7n-ifu_1599116494; Received: from aliy80.localdomain(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0U7n-ifu_1599116494) by smtp.aliyun-inc.com(127.0.0.1); Thu, 03 Sep 2020 15:01:35 +0800 From: Alex Shi To: Anshuman Khandual , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Alexander Duyck Cc: Andrew Morton , Mel Gorman , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: [PATCH v4 2/4] mm/pageblock: remove false sharing in pageblock_flags Date: Thu, 3 Sep 2020 15:01:21 +0800 Message-Id: <1599116482-7410-2-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1599116482-7410-1-git-send-email-alex.shi@linux.alibaba.com> References: <1599116482-7410-1-git-send-email-alex.shi@linux.alibaba.com> X-Rspamd-Queue-Id: 76A7C1804B655 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Current pageblock_flags is only 4 bits, so it has to share a char size in cmpxchg when get set, the false sharing cause perf drop. If we incrase the bits up to 8, false sharing would gone in cmpxchg. and the only cost is half char per pageblock, which is half char per 128MB on x86, 4 chars in 1 GB. Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Mel Gorman Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- include/linux/page-isolation.h | 2 +- include/linux/pageblock-flags.h | 2 +- mm/page_alloc.c | 10 +++------- 3 files changed, 5 insertions(+), 9 deletions(-) diff --git a/include/linux/page-isolation.h b/include/linux/page-isolation.h index 572458016331..baa2e1ecc134 100644 --- a/include/linux/page-isolation.h +++ b/include/linux/page-isolation.h @@ -35,7 +35,7 @@ static inline bool is_migrate_isolate(int migratetype) struct page *has_unmovable_pages(struct zone *zone, struct page *page, int migratetype, int flags); -void set_pageblock_migratetype(struct page *page, int migratetype); +void set_pageblock_migratetype(struct page *page, unsigned char migratetype); int move_freepages_block(struct zone *zone, struct page *page, int migratetype, int *num_movable); diff --git a/include/linux/pageblock-flags.h b/include/linux/pageblock-flags.h index d189441568eb..f785c9d6d68c 100644 --- a/include/linux/pageblock-flags.h +++ b/include/linux/pageblock-flags.h @@ -25,7 +25,7 @@ enum pageblock_bits { * Assume the bits will always align on a word. If this assumption * changes then get/set pageblock needs updating. */ - NR_PAGEBLOCK_BITS + NR_PAGEBLOCK_BITS = BITS_PER_BYTE }; #ifdef CONFIG_HUGETLB_PAGE diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 81e96d4d9c42..3688e6b83318 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -517,19 +517,15 @@ void set_pfnblock_flags_mask(struct page *page, unsigned long flags, unsigned long bitidx, byte_bitidx; unsigned char old_byte, byte; - BUILD_BUG_ON(NR_PAGEBLOCK_BITS != 4); + BUILD_BUG_ON(NR_PAGEBLOCK_BITS != BITS_PER_BYTE); BUILD_BUG_ON(MIGRATE_TYPES > (1 << PB_migratetype_bits)); bitmap = get_pageblock_bitmap(page, pfn); bitidx = pfn_to_bitidx(page, pfn); byte_bitidx = bitidx / BITS_PER_BYTE; - bitidx &= (BITS_PER_BYTE-1); VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page); - mask <<= bitidx; - flags <<= bitidx; - byte = READ_ONCE(bitmap[byte_bitidx]); for (;;) { old_byte = cmpxchg(&bitmap[byte_bitidx], byte, (byte & ~mask) | flags); @@ -539,13 +535,13 @@ void set_pfnblock_flags_mask(struct page *page, unsigned long flags, } } -void set_pageblock_migratetype(struct page *page, int migratetype) +void set_pageblock_migratetype(struct page *page, unsigned char migratetype) { if (unlikely(page_group_by_mobility_disabled && migratetype < MIGRATE_PCPTYPES)) migratetype = MIGRATE_UNMOVABLE; - set_pfnblock_flags_mask(page, (unsigned long)migratetype, + set_pfnblock_flags_mask(page, migratetype, page_to_pfn(page), MIGRATETYPE_MASK); } From patchwork Thu Sep 3 07:01:22 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Shi X-Patchwork-Id: 11752519 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id A294214E5 for ; Thu, 3 Sep 2020 07:01:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 6AD7D20716 for ; Thu, 3 Sep 2020 07:01:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 6AD7D20716 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6E327900002; Thu, 3 Sep 2020 03:01:51 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id 66BD36B0062; Thu, 3 Sep 2020 03:01:51 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 534CD900002; Thu, 3 Sep 2020 03:01:51 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0055.hostedemail.com [216.40.44.55]) by kanga.kvack.org (Postfix) with ESMTP id 2CE936B005D for ; Thu, 3 Sep 2020 03:01:51 -0400 (EDT) Received: from smtpin07.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 120AF2485 for ; Thu, 3 Sep 2020 07:01:50 +0000 (UTC) X-FDA: 77220855180.07.stage06_2717271270a7 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin07.hostedemail.com (Postfix) with ESMTP id D52101803F9AA for ; Thu, 3 Sep 2020 07:01:49 +0000 (UTC) X-Spam-Summary: 1,0,0,020213bc26c44730,d41d8cd98f00b204,alex.shi@linux.alibaba.com,,RULES_HIT:2:41:69:355:379:541:800:960:973:988:989:1260:1261:1345:1359:1431:1437:1535:1605:1730:1747:1777:1792:2194:2199:2393:2559:2562:3138:3139:3140:3141:3142:3622:3865:3866:3867:3868:3870:3871:3872:4049:4119:4250:4321:4605:5007:6119:6261:6742:7514:7903:9036:9707:11026:11232:11473:11657:11658:11914:12043:12048:12296:12297:12438:12555:12895:12986:13138:13221:13229:13231:13846:14096:14394:21060:21080:21451:21611:21627:21987:21990:30054:30064:30070,0,RBL:115.124.30.56:@linux.alibaba.com:.lbl8.mailshell.net-64.201.201.201 62.20.2.100;04y8b8aaxfah9a5f1fys9fc5qs4zzyc7mjcex3zh1j1fmnjpzxmpy1zrubkin8x.r9n7rhupp6gj8uk1ybby9qaew6353zxt8sk71xc1cbkqc94sox1dtogmmsoouej.r-lbl8.mailshell.net-223.238.255.100,CacheIP:none,Bayesian:0.5,0.5,0.5,Netcheck:none,DomainCache:0,MSF:not bulk,SPF:fp,MSBL:0,DNSBL:none,Custom_rules:0:1:0,LFtime:23,LUA_SUMMARY:none X-HE-Tag: stage06_2717271270a7 X-Filterd-Recvd-Size: 8734 Received: from out30-56.freemail.mail.aliyun.com (out30-56.freemail.mail.aliyun.com [115.124.30.56]) by imf40.hostedemail.com (Postfix) with ESMTP for ; Thu, 3 Sep 2020 07:01:48 +0000 (UTC) X-Alimail-AntiSpam: AC=PASS;BC=-1|-1;BR=01201311R121e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01e07425;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=19;SR=0;TI=SMTPD_---0U7n-ifu_1599116494; Received: from aliy80.localdomain(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0U7n-ifu_1599116494) by smtp.aliyun-inc.com(127.0.0.1); Thu, 03 Sep 2020 15:01:36 +0800 From: Alex Shi To: Anshuman Khandual , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , Alexander Duyck Cc: Andrew Morton , Baolin Wang , Russell King , Yoshinori Sato , Rich Felker , "David S. Miller" , Chris Zankel , Max Filippov , linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-xtensa@linux-xtensa.org, linux-mm@kvack.org Subject: [PATCH v4 3/4] mm/pageblock: work around multiple arch's cmpxchg support issue Date: Thu, 3 Sep 2020 15:01:22 +0800 Message-Id: <1599116482-7410-3-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1599116482-7410-1-git-send-email-alex.shi@linux.alibaba.com> References: <1599116482-7410-1-git-send-email-alex.shi@linux.alibaba.com> X-Rspamd-Queue-Id: D52101803F9AA X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Armv6, sh2, sparc32 and xtensa can not do cmpxchg1, so we have to use cmpxchg4 on it. Here we mark above 4 arch's NO_CMPXCHG_BYTE, and would add more if we found. This is the first usages of cmpxchg flase sharing change. We'd better check more cmpxchg usages in current kernel... Reported-by: kernel test robot Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Baolin Wang Cc: Russell King Cc: Yoshinori Sato Cc: Rich Felker Cc: "David S. Miller" Cc: Chris Zankel Cc: Max Filippov Cc: Andrew Morton Cc: linux-kernel@vger.kernel.org Cc: linux-arm-kernel@lists.infradead.org Cc: linux-sh@vger.kernel.org Cc: sparclinux@vger.kernel.org Cc: linux-xtensa@linux-xtensa.org Cc: linux-mm@kvack.org --- arch/Kconfig | 3 +++ arch/arm/Kconfig | 1 + arch/sh/Kconfig | 1 + arch/sparc/Kconfig | 1 + arch/xtensa/Kconfig | 1 + include/linux/mmzone.h | 15 ++++++++++++--- mm/page_alloc.c | 22 +++++++++++----------- 7 files changed, 30 insertions(+), 14 deletions(-) diff --git a/arch/Kconfig b/arch/Kconfig index af14a567b493..3514570c0f5f 100644 --- a/arch/Kconfig +++ b/arch/Kconfig @@ -431,6 +431,9 @@ config HAVE_CMPXCHG_LOCAL config HAVE_CMPXCHG_DOUBLE bool +config NO_CMPXCHG_BYTE + bool + config ARCH_WEAK_RELEASE_ACQUIRE bool diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig index e00d94b16658..03a6c7fd999d 100644 --- a/arch/arm/Kconfig +++ b/arch/arm/Kconfig @@ -48,6 +48,7 @@ config ARM select GENERIC_ALLOCATOR select GENERIC_ARCH_TOPOLOGY if ARM_CPU_TOPOLOGY select GENERIC_ATOMIC64 if CPU_V7M || CPU_V6 || !CPU_32v6K || !AEABI + select NO_CMPXCHG_BYTE if CPU_V6 select GENERIC_CLOCKEVENTS_BROADCAST if SMP select GENERIC_CPU_AUTOPROBE select GENERIC_EARLY_IOREMAP diff --git a/arch/sh/Kconfig b/arch/sh/Kconfig index d20927128fce..4c7f0ad5b93f 100644 --- a/arch/sh/Kconfig +++ b/arch/sh/Kconfig @@ -155,6 +155,7 @@ menu "System type" config CPU_SH2 bool select SH_INTC + select NO_CMPXCHG_BYTE config CPU_SH2A bool diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig index efeff2c896a5..51ae5c8ede87 100644 --- a/arch/sparc/Kconfig +++ b/arch/sparc/Kconfig @@ -58,6 +58,7 @@ config SPARC32 select CLZ_TAB select HAVE_UID16 select OLD_SIGACTION + select NO_CMPXCHG_BYTE config SPARC64 def_bool 64BIT diff --git a/arch/xtensa/Kconfig b/arch/xtensa/Kconfig index e997e0119c02..862b008ab09e 100644 --- a/arch/xtensa/Kconfig +++ b/arch/xtensa/Kconfig @@ -42,6 +42,7 @@ config XTENSA select MODULES_USE_ELF_RELA select PERF_USE_VMALLOC select VIRT_TO_BUS + select NO_CMPXCHG_BYTE help Xtensa processors are 32-bit RISC machines designed by Tensilica primarily for embedded systems. These processors are both diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index be676e659fb7..0bc5ac0f8cd7 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -406,6 +406,15 @@ enum zone_type { #ifndef __GENERATING_BOUNDS_H +/* cmpxchg only support 32-bits operands on ARMv6, SPARC32, sh2, XTENSA.*/ +#ifdef CONFIG_NO_CMPXCHG_BYTE +#define BITS_PER_FLAGS BITS_PER_LONG +typedef unsigned long pageblockflags_t; +#else +#define BITS_PER_FLAGS BITS_PER_BYTE +typedef unsigned char pageblockflags_t; +#endif + struct zone { /* Read-mostly fields */ @@ -437,7 +446,7 @@ struct zone { * Flags for a pageblock_nr_pages block. See pageblock-flags.h. * In SPARSEMEM, this map is stored in struct mem_section */ - unsigned char *pageblock_flags; + pageblockflags_t *pageblock_flags; #endif /* CONFIG_SPARSEMEM */ /* zone_start_pfn == zone_start_paddr >> PAGE_SHIFT */ @@ -1159,7 +1168,7 @@ struct mem_section_usage { DECLARE_BITMAP(subsection_map, SUBSECTIONS_PER_SECTION); #endif /* See declaration of similar field in struct zone */ - unsigned char pageblock_flags[0]; + pageblockflags_t pageblock_flags[0]; }; void subsection_map_init(unsigned long pfn, unsigned long nr_pages); @@ -1212,7 +1221,7 @@ struct mem_section { extern struct mem_section mem_section[NR_SECTION_ROOTS][SECTIONS_PER_ROOT]; #endif -static inline unsigned char *section_to_usemap(struct mem_section *ms) +static inline pageblockflags_t *section_to_usemap(struct mem_section *ms) { return ms->usage->pageblock_flags; } diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3688e6b83318..8b65d83d8be6 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -445,7 +445,7 @@ static inline bool defer_init(int nid, unsigned long pfn, unsigned long end_pfn) #endif /* Return a pointer to the bitmap storing bits affecting a block of pages */ -static inline unsigned char *get_pageblock_bitmap(struct page *page, +static inline pageblockflags_t *get_pageblock_bitmap(struct page *page, unsigned long pfn) { #ifdef CONFIG_SPARSEMEM @@ -474,24 +474,24 @@ static inline int pfn_to_bitidx(struct page *page, unsigned long pfn) * Return: pageblock_bits flags */ static __always_inline -unsigned char __get_pfnblock_flags_mask(struct page *page, +pageblockflags_t __get_pfnblock_flags_mask(struct page *page, unsigned long pfn, unsigned long mask) { - unsigned char *bitmap; + pageblockflags_t *bitmap; unsigned long bitidx, byte_bitidx; - unsigned char byte; + pageblockflags_t byte; bitmap = get_pageblock_bitmap(page, pfn); bitidx = pfn_to_bitidx(page, pfn); - byte_bitidx = bitidx / BITS_PER_BYTE; - bitidx &= (BITS_PER_BYTE-1); + byte_bitidx = bitidx / BITS_PER_FLAGS; + bitidx &= (BITS_PER_FLAGS - 1); byte = bitmap[byte_bitidx]; return (byte >> bitidx) & mask; } -unsigned char get_pfnblock_flags_mask(struct page *page, unsigned long pfn, +pageblockflags_t get_pfnblock_flags_mask(struct page *page, unsigned long pfn, unsigned long mask) { return __get_pfnblock_flags_mask(page, pfn, mask); @@ -513,16 +513,16 @@ void set_pfnblock_flags_mask(struct page *page, unsigned long flags, unsigned long pfn, unsigned long mask) { - unsigned char *bitmap; + pageblockflags_t *bitmap; unsigned long bitidx, byte_bitidx; - unsigned char old_byte, byte; + pageblockflags_t old_byte, byte; - BUILD_BUG_ON(NR_PAGEBLOCK_BITS != BITS_PER_BYTE); + BUILD_BUG_ON(NR_PAGEBLOCK_BITS != BITS_PER_FLAGS); BUILD_BUG_ON(MIGRATE_TYPES > (1 << PB_migratetype_bits)); bitmap = get_pageblock_bitmap(page, pfn); bitidx = pfn_to_bitidx(page, pfn); - byte_bitidx = bitidx / BITS_PER_BYTE; + byte_bitidx = bitidx / BITS_PER_FLAGS; VM_BUG_ON_PAGE(!zone_spans_pfn(page_zone(page), pfn), page);