From patchwork Wed Dec 9 14:51:49 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Wool X-Patchwork-Id: 11961709 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B2A9AC4361B for ; Wed, 9 Dec 2020 14:52:28 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 39799233EF for ; Wed, 9 Dec 2020 14:52:27 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 39799233EF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9CD4D6B00E6; Wed, 9 Dec 2020 09:52:27 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 97BCE6B00E7; Wed, 9 Dec 2020 09:52:27 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 86B2C6B00E8; Wed, 9 Dec 2020 09:52:27 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0042.hostedemail.com [216.40.44.42]) by kanga.kvack.org (Postfix) with ESMTP id 6E6956B00E6 for ; Wed, 9 Dec 2020 09:52:27 -0500 (EST) Received: from smtpin25.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 3A5B11EE6 for ; Wed, 9 Dec 2020 14:52:27 +0000 (UTC) X-FDA: 77574034734.25.beds58_1415b92273f0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin25.hostedemail.com (Postfix) with ESMTP id 14C121804E3A0 for ; Wed, 9 Dec 2020 14:52:27 +0000 (UTC) X-HE-Tag: beds58_1415b92273f0 X-Filterd-Recvd-Size: 7694 Received: from mail-wr1-f68.google.com (mail-wr1-f68.google.com [209.85.221.68]) by imf28.hostedemail.com (Postfix) with ESMTP for ; Wed, 9 Dec 2020 14:52:26 +0000 (UTC) Received: by mail-wr1-f68.google.com with SMTP id c1so2038197wrq.6 for ; Wed, 09 Dec 2020 06:52:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=J2hkHNzCDEGB0J9sgU5PXkAy+/sgZMRXXXGzM8sHpFc=; b=aBTkU87uGdVCZaCoNz7qeW5WKZ1AlBZKjV71BA10HzW0BnLo09qiTXT270YHzU/7rq +HhpphftS+8XYqug0QA7oT83TxR4Mq3Uxx2Gd6p8xmG1fUB72IgQ0Hw/1OcbKHVuzLcV xXU8O5z/YZBYDSwSLj8gjlmRVhyhjQQ3ODqkQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=J2hkHNzCDEGB0J9sgU5PXkAy+/sgZMRXXXGzM8sHpFc=; b=iQYUAm8DIdgPO2C6GE/zWSvGUw+jZzu6zkqb/1bAK4HaPkBHcDUtNIaYi/n269Ceuc RAU5mFT3eelGwlF8opsNJDEKQmhYQcTRqz5maAP//Aw6h/+eet0QeATkg7x86r/QDJuG yOm/vr+Vv406zaQnmMU/sPJbD51a+HSleb3vETqv9GgXlazOfW3URiw76zJI/eN0YfZ2 tNR4IAInchK4T02HLdSCbddfhf7p1kFL65wXx2gyEVMB/Pyjrtsm+OOLvwvGGeti5vHP hkXaISkCf6bB5IXadiMWR+0CvTUxfZyVsfLNVmMgA/xC0NxOgEAldLMxifMf6SBVsDBb QMOA== X-Gm-Message-State: AOAM530PTCKV4lY2ESQ6UP0ui8NG3Oi+9+6vzDKg0K8+0mMk5a+Ps9j2 uVv/pwMzS84CRJIAAZBQM6yuXCcJaSzbv2SdMO8= X-Google-Smtp-Source: ABdhPJzipTU+hiVIinfES/sQ4fz736DXvse1IQaCV4h73aNNUxx/Yt/FOMFgFm5Z/BlSVFysn1df7w== X-Received: by 2002:adf:e802:: with SMTP id o2mr3204533wrm.251.1607525545292; Wed, 09 Dec 2020 06:52:25 -0800 (PST) Received: from taos.konsulko.bg (lan.nucleusys.com. [92.247.61.126]) by smtp.gmail.com with ESMTPSA id 189sm3831957wma.22.2020.12.09.06.52.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 06:52:24 -0800 (PST) From: Vitaly Wool To: linux-mm@kvack.org Cc: lkml@vger.kernel.org, linux-rt-users@vger.kernel.org, Sebastian Andrzej Siewior , Mike Galbraith , akpm@linux-foundation.org, Vitaly Wool , stable@kernel.org Subject: [PATCH 1/3] z3fold: simplify freeing slots Date: Wed, 9 Dec 2020 16:51:49 +0200 Message-Id: <20201209145151.18994-2-vitaly.wool@konsulko.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20201209145151.18994-1-vitaly.wool@konsulko.com> References: <20201209145151.18994-1-vitaly.wool@konsulko.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: There used to be two places in the code where slots could be freed, namely when freeing the last allocated handle from the slots and when releasing the z3fold header these slots aree linked to. The logic to decide on whether to free certain slots was complicated and error prone in both functions and it led to failures in RT case. To fix that, make free_handle() the single point of freeing slots. Signed-off-by: Vitaly Wool Tested-by: Mike Galbraith Cc: stable@kernel.org --- mm/z3fold.c | 55 +++++++++++++---------------------------------------- 1 file changed, 13 insertions(+), 42 deletions(-) diff --git a/mm/z3fold.c b/mm/z3fold.c index 18feaa0bc537..6c2325cd3fba 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -90,7 +90,7 @@ struct z3fold_buddy_slots { * be enough slots to hold all possible variants */ unsigned long slot[BUDDY_MASK + 1]; - unsigned long pool; /* back link + flags */ + unsigned long pool; /* back link */ rwlock_t lock; }; #define HANDLE_FLAG_MASK (0x03) @@ -181,13 +181,6 @@ enum z3fold_page_flags { PAGE_CLAIMED, /* by either reclaim or free */ }; -/* - * handle flags, go under HANDLE_FLAG_MASK - */ -enum z3fold_handle_flags { - HANDLES_ORPHANED = 0, -}; - /* * Forward declarations */ @@ -303,10 +296,9 @@ static inline void put_z3fold_header(struct z3fold_header *zhdr) z3fold_page_unlock(zhdr); } -static inline void free_handle(unsigned long handle) +static inline void free_handle(unsigned long handle, struct z3fold_header *zhdr) { struct z3fold_buddy_slots *slots; - struct z3fold_header *zhdr; int i; bool is_free; @@ -316,22 +308,13 @@ static inline void free_handle(unsigned long handle) if (WARN_ON(*(unsigned long *)handle == 0)) return; - zhdr = handle_to_z3fold_header(handle); slots = handle_to_slots(handle); write_lock(&slots->lock); *(unsigned long *)handle = 0; - if (zhdr->slots == slots) { - write_unlock(&slots->lock); - return; /* simple case, nothing else to do */ - } + if (zhdr->slots != slots) + zhdr->foreign_handles--; - /* we are freeing a foreign handle if we are here */ - zhdr->foreign_handles--; is_free = true; - if (!test_bit(HANDLES_ORPHANED, &slots->pool)) { - write_unlock(&slots->lock); - return; - } for (i = 0; i <= BUDDY_MASK; i++) { if (slots->slot[i]) { is_free = false; @@ -343,6 +326,8 @@ static inline void free_handle(unsigned long handle) if (is_free) { struct z3fold_pool *pool = slots_to_pool(slots); + if (zhdr->slots == slots) + zhdr->slots = NULL; kmem_cache_free(pool->c_handle, slots); } } @@ -525,8 +510,6 @@ static void __release_z3fold_page(struct z3fold_header *zhdr, bool locked) { struct page *page = virt_to_page(zhdr); struct z3fold_pool *pool = zhdr_to_pool(zhdr); - bool is_free = true; - int i; WARN_ON(!list_empty(&zhdr->buddy)); set_bit(PAGE_STALE, &page->private); @@ -536,21 +519,6 @@ static void __release_z3fold_page(struct z3fold_header *zhdr, bool locked) list_del_init(&page->lru); spin_unlock(&pool->lock); - /* If there are no foreign handles, free the handles array */ - read_lock(&zhdr->slots->lock); - for (i = 0; i <= BUDDY_MASK; i++) { - if (zhdr->slots->slot[i]) { - is_free = false; - break; - } - } - if (!is_free) - set_bit(HANDLES_ORPHANED, &zhdr->slots->pool); - read_unlock(&zhdr->slots->lock); - - if (is_free) - kmem_cache_free(pool->c_handle, zhdr->slots); - if (locked) z3fold_page_unlock(zhdr); @@ -973,6 +941,9 @@ static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *pool, } } + if (zhdr && !zhdr->slots) + zhdr->slots = alloc_slots(pool, + can_sleep ? GFP_NOIO : GFP_ATOMIC); return zhdr; } @@ -1270,7 +1241,7 @@ static void z3fold_free(struct z3fold_pool *pool, unsigned long handle) } if (!page_claimed) - free_handle(handle); + free_handle(handle, zhdr); if (kref_put(&zhdr->refcount, release_z3fold_page_locked_list)) { atomic64_dec(&pool->pages_nr); return; @@ -1429,19 +1400,19 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) ret = pool->ops->evict(pool, middle_handle); if (ret) goto next; - free_handle(middle_handle); + free_handle(middle_handle, zhdr); } if (first_handle) { ret = pool->ops->evict(pool, first_handle); if (ret) goto next; - free_handle(first_handle); + free_handle(first_handle, zhdr); } if (last_handle) { ret = pool->ops->evict(pool, last_handle); if (ret) goto next; - free_handle(last_handle); + free_handle(last_handle, zhdr); } next: if (test_bit(PAGE_HEADLESS, &page->private)) { From patchwork Wed Dec 9 14:51:50 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Wool X-Patchwork-Id: 11961711 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 017C7C433FE for ; Wed, 9 Dec 2020 14:52:47 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 956CA233EF for ; Wed, 9 Dec 2020 14:52:46 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 956CA233EF Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 140526B00E8; Wed, 9 Dec 2020 09:52:46 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0F08A6B00E9; Wed, 9 Dec 2020 09:52:46 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 005EF6B00EA; Wed, 9 Dec 2020 09:52:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0162.hostedemail.com [216.40.44.162]) by kanga.kvack.org (Postfix) with ESMTP id DF0396B00E8 for ; Wed, 9 Dec 2020 09:52:45 -0500 (EST) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id A1505181AC9CC for ; Wed, 9 Dec 2020 14:52:45 +0000 (UTC) X-FDA: 77574035490.19.size04_3913c25273f0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin19.hostedemail.com (Postfix) with ESMTP id 7C5991AD1B1 for ; Wed, 9 Dec 2020 14:52:45 +0000 (UTC) X-HE-Tag: size04_3913c25273f0 X-Filterd-Recvd-Size: 5165 Received: from mail-wr1-f66.google.com (mail-wr1-f66.google.com [209.85.221.66]) by imf49.hostedemail.com (Postfix) with ESMTP for ; Wed, 9 Dec 2020 14:52:45 +0000 (UTC) Received: by mail-wr1-f66.google.com with SMTP id 91so2041716wrj.7 for ; Wed, 09 Dec 2020 06:52:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=QO+mDdc0W+GDe87CR6+J/pH9JlgkXQ9+W7Jahz1zoF4=; b=g9Xue5OG7EgSbVCscVZfnyXDCXaL+U6x8BVxb8qWNccgIVM5Pno4SkL9aGGc006+qz jcErJyaSq+DL/OcldXEsQ7PTMFZNy7C8HI8lbpfRJeNbj+CY9ni9f0YgyVKw+FFnh5aa yCiKBlDYja7hs0IuChBIxdI24YraSbk2BqOFY= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=QO+mDdc0W+GDe87CR6+J/pH9JlgkXQ9+W7Jahz1zoF4=; b=n9aD2isvO4WAqaPW3Ru6Jl4bgk6gBRR6fsKLmXWDLX+1LPWLTA6td5SMyitHwobTme 7NQxqErBpBVKbWgz5Q4X36w1a1iklJOZCv0up6hd3+hEMDm5QGb4WFNzILOVW6xfBAbu 03MXBn7KPMeGlMzJAS3yHZCJ7wSm1oLcX7TJmI7i8fu4jrbUsYylmRk1eLy4XT+qCBuj YFxwm2+gScW5fnPX0RDL8W807v9kgS4DArQjQ9w6rM7dfTlTMrI1TAGHQccBHWx54ui1 +Mg4Y1ZxAFP1r+6LyqCe/R+2jKCiA8uMWOIOSJ3zwwxYfC+NNlHick+RnNunAxixhJrH qrIw== X-Gm-Message-State: AOAM533Bruc9W2mFm0jxjybMIVG2CxIT5/x9bTOJbtme5r+zCM7qFd8J jqd5u8PzAXLSBevv5yumHdMdDOtyzuUWwnuWUaU= X-Google-Smtp-Source: ABdhPJyhALIcSeelDT7f6z1tke+0gzPilS0oVWeaT6VDdfYT0A6zqk0G9/PHg4pkylf1EdTY7kw2Bg== X-Received: by 2002:adf:f98b:: with SMTP id f11mr3207719wrr.235.1607525563914; Wed, 09 Dec 2020 06:52:43 -0800 (PST) Received: from taos.konsulko.bg (lan.nucleusys.com. [92.247.61.126]) by smtp.gmail.com with ESMTPSA id 189sm3831957wma.22.2020.12.09.06.52.42 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 06:52:43 -0800 (PST) From: Vitaly Wool To: linux-mm@kvack.org Cc: lkml@vger.kernel.org, linux-rt-users@vger.kernel.org, Sebastian Andrzej Siewior , Mike Galbraith , akpm@linux-foundation.org, Vitaly Wool Subject: [PATCH 2/3] z3fold: Remove preempt disabled sections for RT Date: Wed, 9 Dec 2020 16:51:50 +0200 Message-Id: <20201209145151.18994-3-vitaly.wool@konsulko.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20201209145151.18994-1-vitaly.wool@konsulko.com> References: <20201209145151.18994-1-vitaly.wool@konsulko.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Replace get_cpu_ptr() with migrate_disable()+this_cpu_ptr() so RT can take spinlocks that become sleeping locks. Signed-off-by Mike Galbraith Signed-off-by: Vitaly Wool --- mm/z3fold.c | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) diff --git a/mm/z3fold.c b/mm/z3fold.c index 6c2325cd3fba..9fc1cc9630fe 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -610,14 +610,16 @@ static inline void add_to_unbuddied(struct z3fold_pool *pool, { if (zhdr->first_chunks == 0 || zhdr->last_chunks == 0 || zhdr->middle_chunks == 0) { - struct list_head *unbuddied = get_cpu_ptr(pool->unbuddied); - + struct list_head *unbuddied; int freechunks = num_free_chunks(zhdr); + + migrate_disable(); + unbuddied = this_cpu_ptr(pool->unbuddied); spin_lock(&pool->lock); list_add(&zhdr->buddy, &unbuddied[freechunks]); spin_unlock(&pool->lock); zhdr->cpu = smp_processor_id(); - put_cpu_ptr(pool->unbuddied); + migrate_enable(); } } @@ -854,8 +856,9 @@ static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *pool, int chunks = size_to_chunks(size), i; lookup: + migrate_disable(); /* First, try to find an unbuddied z3fold page. */ - unbuddied = get_cpu_ptr(pool->unbuddied); + unbuddied = this_cpu_ptr(pool->unbuddied); for_each_unbuddied_list(i, chunks) { struct list_head *l = &unbuddied[i]; @@ -873,7 +876,7 @@ static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *pool, !z3fold_page_trylock(zhdr)) { spin_unlock(&pool->lock); zhdr = NULL; - put_cpu_ptr(pool->unbuddied); + migrate_enable(); if (can_sleep) cond_resched(); goto lookup; @@ -887,7 +890,7 @@ static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *pool, test_bit(PAGE_CLAIMED, &page->private)) { z3fold_page_unlock(zhdr); zhdr = NULL; - put_cpu_ptr(pool->unbuddied); + migrate_enable(); if (can_sleep) cond_resched(); goto lookup; @@ -902,7 +905,7 @@ static inline struct z3fold_header *__z3fold_alloc(struct z3fold_pool *pool, kref_get(&zhdr->refcount); break; } - put_cpu_ptr(pool->unbuddied); + migrate_enable(); if (!zhdr) { int cpu; From patchwork Wed Dec 9 14:51:51 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Vitaly Wool X-Patchwork-Id: 11961713 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 16A19C433FE for ; Wed, 9 Dec 2020 14:52:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 97BDF233EA for ; Wed, 9 Dec 2020 14:52:51 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 97BDF233EA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=konsulko.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 3D7E96B00E9; Wed, 9 Dec 2020 09:52:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 389CF6B00EA; Wed, 9 Dec 2020 09:52:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 24FAD6B00EB; Wed, 9 Dec 2020 09:52:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0019.hostedemail.com [216.40.44.19]) by kanga.kvack.org (Postfix) with ESMTP id 10EEA6B00E9 for ; Wed, 9 Dec 2020 09:52:51 -0500 (EST) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D056A8249980 for ; Wed, 9 Dec 2020 14:52:50 +0000 (UTC) X-FDA: 77574035700.12.hair87_3813066273f0 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id B3DA91801530A for ; Wed, 9 Dec 2020 14:52:50 +0000 (UTC) X-HE-Tag: hair87_3813066273f0 X-Filterd-Recvd-Size: 13498 Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by imf14.hostedemail.com (Postfix) with ESMTP for ; Wed, 9 Dec 2020 14:52:49 +0000 (UTC) Received: by mail-wr1-f65.google.com with SMTP id t4so2032538wrr.12 for ; Wed, 09 Dec 2020 06:52:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=konsulko.com; s=google; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=P4ADCprBh51UPTi4qQxtPSZoVYMu6eRk2ZmPIDnYt18=; b=s7bpjjaM+he+DQ0FOJk5vfQ3eaUUaiKXW9uRqTL32jLoq0l5qwhPR2LSbXjPtM9jCo Ndajx5eH7/hFD0L5jsM0qiMS5CkFJOEnjvl8aH6rPjEm3qgUj0mNPGjSAU3b3lQ/ljjp RcFX7fXqiBKXPb1BetTi67nowt4N3tpLmpxK4= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=P4ADCprBh51UPTi4qQxtPSZoVYMu6eRk2ZmPIDnYt18=; b=uCcTFtdzDjETEwPevRYwriUf/sZEPTmKma7DCur1T4IN3LWuPxi/3t6lBHkKnncAG0 0Af2WD4H8+s7R+9e6Quo7owj5pbXUXtu10jE2wfvpbT/GZ+Nr+XsDjfd5RaJIQAtVb0k Qa/ePC1ueRqr8/fu35NRzsd6MFwPyG7Js2gj/ro7pjEUTXb4vcrErOph7bB4gii9lG4I Rnet8dj1IrWgL/LhdSeLkRKLBNULkIRJzdZJYmvOLmuX7aXR152vIdsTc0mR+k11lAlt lfJ0g2pYGvf+DjCAoJGK8SeMTtFfGh6ey+QShUYx+d8RT6X6HJl87kMXoL5eVzWHmmPz QzBg== X-Gm-Message-State: AOAM531FPuCYNfs2Jj4EO6C1cV92tkITYz+OVbm0atM+ERIe9m4vYmhj z8afU8yfoF0a/EoO8q3ykjBtCtejAqrjq8ilGus= X-Google-Smtp-Source: ABdhPJzAV8h8z52C1fB1QxdKxxe0h/jX9fYk+kwacv+kji+UrtAAiYbtgGQJoUpd8tO6t+2NNJhiqg== X-Received: by 2002:a5d:474d:: with SMTP id o13mr3078391wrs.379.1607525568708; Wed, 09 Dec 2020 06:52:48 -0800 (PST) Received: from taos.konsulko.bg (lan.nucleusys.com. [92.247.61.126]) by smtp.gmail.com with ESMTPSA id 189sm3831957wma.22.2020.12.09.06.52.47 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 09 Dec 2020 06:52:48 -0800 (PST) From: Vitaly Wool To: linux-mm@kvack.org Cc: lkml@vger.kernel.org, linux-rt-users@vger.kernel.org, Sebastian Andrzej Siewior , Mike Galbraith , akpm@linux-foundation.org, Vitaly Wool , stable@kernel.org Subject: [PATCH 3/3] z3fold: stricter locking and more careful reclaim Date: Wed, 9 Dec 2020 16:51:51 +0200 Message-Id: <20201209145151.18994-4-vitaly.wool@konsulko.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20201209145151.18994-1-vitaly.wool@konsulko.com> References: <20201209145151.18994-1-vitaly.wool@konsulko.com> MIME-Version: 1.0 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Use temporary slots in reclaim function to avoid possible race when freeing those. While at it, make sure we check CLAIMED flag under page lock in the reclaim function to make sure we are not racing with z3fold_alloc(). Signed-off-by: Vitaly Wool Cc: stable@kernel.org --- mm/z3fold.c | 143 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 85 insertions(+), 58 deletions(-) diff --git a/mm/z3fold.c b/mm/z3fold.c index 9fc1cc9630fe..dacb0d70fa61 100644 --- a/mm/z3fold.c +++ b/mm/z3fold.c @@ -181,6 +181,13 @@ enum z3fold_page_flags { PAGE_CLAIMED, /* by either reclaim or free */ }; +/* + * handle flags, go under HANDLE_FLAG_MASK + */ +enum z3fold_handle_flags { + HANDLES_NOFREE = 0, +}; + /* * Forward declarations */ @@ -311,6 +318,12 @@ static inline void free_handle(unsigned long handle, struct z3fold_header *zhdr) slots = handle_to_slots(handle); write_lock(&slots->lock); *(unsigned long *)handle = 0; + + if (test_bit(HANDLES_NOFREE, &slots->pool)) { + write_unlock(&slots->lock); + return; /* simple case, nothing else to do */ + } + if (zhdr->slots != slots) zhdr->foreign_handles--; @@ -623,6 +636,28 @@ static inline void add_to_unbuddied(struct z3fold_pool *pool, } } +static inline enum buddy get_free_buddy(struct z3fold_header *zhdr, int chunks) +{ + enum buddy bud = HEADLESS; + + if (zhdr->middle_chunks) { + if (!zhdr->first_chunks && + chunks <= zhdr->start_middle - ZHDR_CHUNKS) + bud = FIRST; + else if (!zhdr->last_chunks) + bud = LAST; + } else { + if (!zhdr->first_chunks) + bud = FIRST; + else if (!zhdr->last_chunks) + bud = LAST; + else + bud = MIDDLE; + } + + return bud; +} + static inline void *mchunk_memmove(struct z3fold_header *zhdr, unsigned short dst_chunk) { @@ -684,18 +719,7 @@ static struct z3fold_header *compact_single_buddy(struct z3fold_header *zhdr) if (WARN_ON(new_zhdr == zhdr)) goto out_fail; - if (new_zhdr->first_chunks == 0) { - if (new_zhdr->middle_chunks != 0 && - chunks >= new_zhdr->start_middle) { - new_bud = LAST; - } else { - new_bud = FIRST; - } - } else if (new_zhdr->last_chunks == 0) { - new_bud = LAST; - } else if (new_zhdr->middle_chunks == 0) { - new_bud = MIDDLE; - } + new_bud = get_free_buddy(new_zhdr, chunks); q = new_zhdr; switch (new_bud) { case FIRST: @@ -817,9 +841,8 @@ static void do_compact_page(struct z3fold_header *zhdr, bool locked) return; } - if (unlikely(PageIsolated(page) || - test_bit(PAGE_CLAIMED, &page->private) || - test_bit(PAGE_STALE, &page->private))) { + if (test_bit(PAGE_STALE, &page->private) || + test_and_set_bit(PAGE_CLAIMED, &page->private)) { z3fold_page_unlock(zhdr); return; } @@ -828,13 +851,16 @@ static void do_compact_page(struct z3fold_header *zhdr, bool locked) zhdr->mapped_count == 0 && compact_single_buddy(zhdr)) { if (kref_put(&zhdr->refcount, release_z3fold_page_locked)) atomic64_dec(&pool->pages_nr); - else + else { + clear_bit(PAGE_CLAIMED, &page->private); z3fold_page_unlock(zhdr); + } return; } z3fold_compact_page(zhdr); add_to_unbuddied(pool, zhdr); + clear_bit(PAGE_CLAIMED, &page->private); z3fold_page_unlock(zhdr); } @@ -1083,17 +1109,8 @@ static int z3fold_alloc(struct z3fold_pool *pool, size_t size, gfp_t gfp, retry: zhdr = __z3fold_alloc(pool, size, can_sleep); if (zhdr) { - if (zhdr->first_chunks == 0) { - if (zhdr->middle_chunks != 0 && - chunks >= zhdr->start_middle) - bud = LAST; - else - bud = FIRST; - } else if (zhdr->last_chunks == 0) - bud = LAST; - else if (zhdr->middle_chunks == 0) - bud = MIDDLE; - else { + bud = get_free_buddy(zhdr, chunks); + if (bud == HEADLESS) { if (kref_put(&zhdr->refcount, release_z3fold_page_locked)) atomic64_dec(&pool->pages_nr); @@ -1239,7 +1256,6 @@ static void z3fold_free(struct z3fold_pool *pool, unsigned long handle) pr_err("%s: unknown bud %d\n", __func__, bud); WARN_ON(1); put_z3fold_header(zhdr); - clear_bit(PAGE_CLAIMED, &page->private); return; } @@ -1254,8 +1270,7 @@ static void z3fold_free(struct z3fold_pool *pool, unsigned long handle) z3fold_page_unlock(zhdr); return; } - if (unlikely(PageIsolated(page)) || - test_and_set_bit(NEEDS_COMPACTING, &page->private)) { + if (test_and_set_bit(NEEDS_COMPACTING, &page->private)) { put_z3fold_header(zhdr); clear_bit(PAGE_CLAIMED, &page->private); return; @@ -1319,6 +1334,10 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) struct page *page = NULL; struct list_head *pos; unsigned long first_handle = 0, middle_handle = 0, last_handle = 0; + struct z3fold_buddy_slots slots __attribute__((aligned(SLOTS_ALIGN))); + + rwlock_init(&slots.lock); + slots.pool = (unsigned long)pool | (1 << HANDLES_NOFREE); spin_lock(&pool->lock); if (!pool->ops || !pool->ops->evict || retries == 0) { @@ -1333,35 +1352,36 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) list_for_each_prev(pos, &pool->lru) { page = list_entry(pos, struct page, lru); - /* this bit could have been set by free, in which case - * we pass over to the next page in the pool. - */ - if (test_and_set_bit(PAGE_CLAIMED, &page->private)) { - page = NULL; - continue; - } - - if (unlikely(PageIsolated(page))) { - clear_bit(PAGE_CLAIMED, &page->private); - page = NULL; - continue; - } zhdr = page_address(page); if (test_bit(PAGE_HEADLESS, &page->private)) break; + if (kref_get_unless_zero(&zhdr->refcount) == 0) { + zhdr = NULL; + break; + } if (!z3fold_page_trylock(zhdr)) { - clear_bit(PAGE_CLAIMED, &page->private); + if (kref_put(&zhdr->refcount, + release_z3fold_page)) + atomic64_dec(&pool->pages_nr); zhdr = NULL; continue; /* can't evict at this point */ } - if (zhdr->foreign_handles) { - clear_bit(PAGE_CLAIMED, &page->private); - z3fold_page_unlock(zhdr); + + /* test_and_set_bit is of course atomic, but we still + * need to do it under page lock, otherwise checking + * that bit in __z3fold_alloc wouldn't make sense + */ + if (zhdr->foreign_handles || + test_and_set_bit(PAGE_CLAIMED, &page->private)) { + if (kref_put(&zhdr->refcount, + release_z3fold_page)) + atomic64_dec(&pool->pages_nr); + else + z3fold_page_unlock(zhdr); zhdr = NULL; continue; /* can't evict such page */ } - kref_get(&zhdr->refcount); list_del_init(&zhdr->buddy); zhdr->cpu = -1; break; @@ -1383,12 +1403,16 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) first_handle = 0; last_handle = 0; middle_handle = 0; + memset(slots.slot, 0, sizeof(slots.slot)); if (zhdr->first_chunks) - first_handle = encode_handle(zhdr, FIRST); + first_handle = __encode_handle(zhdr, &slots, + FIRST); if (zhdr->middle_chunks) - middle_handle = encode_handle(zhdr, MIDDLE); + middle_handle = __encode_handle(zhdr, &slots, + MIDDLE); if (zhdr->last_chunks) - last_handle = encode_handle(zhdr, LAST); + last_handle = __encode_handle(zhdr, &slots, + LAST); /* * it's safe to unlock here because we hold a * reference to this page @@ -1403,19 +1427,16 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) ret = pool->ops->evict(pool, middle_handle); if (ret) goto next; - free_handle(middle_handle, zhdr); } if (first_handle) { ret = pool->ops->evict(pool, first_handle); if (ret) goto next; - free_handle(first_handle, zhdr); } if (last_handle) { ret = pool->ops->evict(pool, last_handle); if (ret) goto next; - free_handle(last_handle, zhdr); } next: if (test_bit(PAGE_HEADLESS, &page->private)) { @@ -1429,9 +1450,11 @@ static int z3fold_reclaim_page(struct z3fold_pool *pool, unsigned int retries) spin_unlock(&pool->lock); clear_bit(PAGE_CLAIMED, &page->private); } else { + struct z3fold_buddy_slots *slots = zhdr->slots; z3fold_page_lock(zhdr); if (kref_put(&zhdr->refcount, release_z3fold_page_locked)) { + kmem_cache_free(pool->c_handle, slots); atomic64_dec(&pool->pages_nr); return 0; } @@ -1547,8 +1570,7 @@ static bool z3fold_page_isolate(struct page *page, isolate_mode_t mode) VM_BUG_ON_PAGE(!PageMovable(page), page); VM_BUG_ON_PAGE(PageIsolated(page), page); - if (test_bit(PAGE_HEADLESS, &page->private) || - test_bit(PAGE_CLAIMED, &page->private)) + if (test_bit(PAGE_HEADLESS, &page->private)) return false; zhdr = page_address(page); @@ -1560,6 +1582,8 @@ static bool z3fold_page_isolate(struct page *page, isolate_mode_t mode) if (zhdr->mapped_count != 0 || zhdr->foreign_handles != 0) goto out; + if (test_and_set_bit(PAGE_CLAIMED, &page->private)) + goto out; pool = zhdr_to_pool(zhdr); spin_lock(&pool->lock); if (!list_empty(&zhdr->buddy)) @@ -1586,16 +1610,17 @@ static int z3fold_page_migrate(struct address_space *mapping, struct page *newpa VM_BUG_ON_PAGE(!PageMovable(page), page); VM_BUG_ON_PAGE(!PageIsolated(page), page); + VM_BUG_ON_PAGE(!test_bit(PAGE_CLAIMED, &page->private), page); VM_BUG_ON_PAGE(!PageLocked(newpage), newpage); zhdr = page_address(page); pool = zhdr_to_pool(zhdr); - if (!z3fold_page_trylock(zhdr)) { + if (!z3fold_page_trylock(zhdr)) return -EAGAIN; - } if (zhdr->mapped_count != 0 || zhdr->foreign_handles != 0) { z3fold_page_unlock(zhdr); + clear_bit(PAGE_CLAIMED, &page->private); return -EBUSY; } if (work_pending(&zhdr->work)) { @@ -1637,6 +1662,7 @@ static int z3fold_page_migrate(struct address_space *mapping, struct page *newpa queue_work_on(new_zhdr->cpu, pool->compact_wq, &new_zhdr->work); page_mapcount_reset(page); + clear_bit(PAGE_CLAIMED, &page->private); put_page(page); return 0; } @@ -1660,6 +1686,7 @@ static void z3fold_page_putback(struct page *page) spin_lock(&pool->lock); list_add(&page->lru, &pool->lru); spin_unlock(&pool->lock); + clear_bit(PAGE_CLAIMED, &page->private); z3fold_page_unlock(zhdr); }