From patchwork Wed Jun 9 04:05:21 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12308825 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C63ECC47095 for ; Wed, 9 Jun 2021 04:05:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 59D4E6124C for ; Wed, 9 Jun 2021 04:05:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 59D4E6124C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C33DA6B006E; Wed, 9 Jun 2021 00:05:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id BE3FD6B0070; Wed, 9 Jun 2021 00:05:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id A84BA6B0071; Wed, 9 Jun 2021 00:05:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0250.hostedemail.com [216.40.44.250]) by kanga.kvack.org (Postfix) with ESMTP id 75D166B006E for ; Wed, 9 Jun 2021 00:05:25 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 11542ABE8 for ; Wed, 9 Jun 2021 04:05:25 +0000 (UTC) X-FDA: 78232845810.19.AA39F93 Received: from mail-oi1-f182.google.com (mail-oi1-f182.google.com [209.85.167.182]) by imf10.hostedemail.com (Postfix) with ESMTP id 6F7724202A07 for ; Wed, 9 Jun 2021 04:05:21 +0000 (UTC) Received: by mail-oi1-f182.google.com with SMTP id c13so18297820oib.13 for ; Tue, 08 Jun 2021 21:05:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:subject:message-id:mime-version; bh=GvD7glKDmqy0C8wgTkrBDLR7pK27K361Vqkf6hOXhYA=; b=SAi4kOfFlJAobWQC5FfaMehHga/M3Wa0Hmdh+mQcnYGc/JRs6pihYdMMtSS5M/Bk98 3ALaK8Gue0Us9N3LxFDvt0a9llHAQ9YTg5EhaDN8CsijPeD1+ZGyUm1qvQJu+ix7phVA 43TJUe9702cUG2c0X6jV4mLwZFPhNX8aUnOE1Y2ME19AQT8qXf3qtZ4W34dBW2K4b/KX 13gbj7KSeaxJsLFzRDWNwPFNAjnh1AUTTZJdHQiWoVFHAXvlusvOQvqy9w3B9peDqN6w cLP+BSiKw0K0Szi5lXIWHaekLUwG9hI1jI0eMG6Hcc06ZiGEO/GjWNnik7/M3Uxvh2Bs tBng== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:subject:message-id:mime-version; bh=GvD7glKDmqy0C8wgTkrBDLR7pK27K361Vqkf6hOXhYA=; b=BmL49OjeBJFBoEbIcIkRGk1Qb/uNdWaqGgdWlrKvNQ4m4WnYddPae/F0AOw6lPoTfd FBF9Ml3sOMZt5RFOuzP3zKKJvs0viWOR7tG2h0DmA70PPpdqjuymbye8FhW48LdCCsBw dwZDxAPW4D46JKo/6Y/e75irSKdmM8AZsf2X9hV0ZnIGhxWn+zGbNJ2iQdcvkFeuQLgo /oFUJ2hQF8ltYmiANAvnoDU1ebQlliaBynM6vwgVRnKyeuyeYZ7TiNDBj8Z3VSCQ6oIE I3eLVjNkWBA4ftze5rWaOX40ZfoZQvIZsfwHKsGptGp2uJsMPRY48F1vOe0I2JBTJnE4 DCjw== X-Gm-Message-State: AOAM5309U9B9EkIpxqhYh6BiZQzd0eqvQ44EfpH7D05UXVDTzCHMEUy6 2PNyDyrWW0I254vYcxKDYxptWXUNUVX04g== X-Google-Smtp-Source: ABdhPJz2ZyojP1wY65WXX6sa5VGixFsfCgnnjUYEZuWwujSze0Qk1mzybTYSjorkuWNbORFSP8JjDQ== X-Received: by 2002:aca:af90:: with SMTP id y138mr5150511oie.92.1623211523605; Tue, 08 Jun 2021 21:05:23 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id q22sm3416834otl.11.2021.06.08.21.05.22 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 21:05:23 -0700 (PDT) Date: Tue, 8 Jun 2021 21:05:21 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 01/10] mm/thp: fix __split_huge_pmd_locked() on shmem migration entry (fwd) Message-ID: <59d94b4-c0dd-310-894-be99416f3c92@google.com> MIME-Version: 1.0 Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=SAi4kOfF; spf=pass (imf10.hostedemail.com: domain of hughd@google.com designates 209.85.167.182 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 6F7724202A07 X-Stat-Signature: rfyr5tehtrtmehe7w47khq7rc1utqxfg X-HE-Tag: 1623211521-637405 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: ---------- Forwarded message ---------- Date: Tue, 8 Jun 2021 21:00:12 -0700 (PDT) From: Hugh Dickins To: Andrew Morton Cc: Hugh Dickins , Kirill A. Shutemov , Yang Shi , Wang Yugui , Matthew Wilcox , Naoya Horiguchi , Alistair Popple , Ralph Campbell , Zi Yan , Miaohe Lin , Minchan Kim , Jue Wang , Peter Xu , Jan Kara , Shakeel Butt , Oscar Salvador Subject: [PATCH v2 01/10] mm/thp: fix __split_huge_pmd_locked() on shmem migration entry Stressing huge tmpfs page migration racing hole punch often crashed on the VM_BUG_ON(!pmd_present) in pmdp_huge_clear_flush(), with DEBUG_VM=y kernel; or shortly afterwards, on a bad dereference in __split_huge_pmd_locked() when DEBUG_VM=n. They forgot to allow for pmd migration entries in the non-anonymous case. Full disclosure: those particular experiments were on a kernel with more relaxed mmap_lock and i_mmap_rwsem locking, and were not repeated on the vanilla kernel: it is conceivable that stricter locking happens to avoid those cases, or makes them less likely; but __split_huge_pmd_locked() already allowed for pmd migration entries when handling anonymous THPs, so this commit brings the shmem and file THP handling into line. And while there: use old_pmd rather than _pmd, as in the following blocks; and make it clearer to the eye that the !vma_is_anonymous() block is self-contained, making an early return after accounting for unmapping. Fixes: e71769ae5260 ("mm: enable thp migration for shmem thp") Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov Reviewed-by: Yang Shi --- v2: omit is_huge_zero_pmd() mods (done differently in next), per Kirill mm/huge_memory.c | 27 ++++++++++++++++++--------- mm/pgtable-generic.c | 5 ++--- 2 files changed, 20 insertions(+), 12 deletions(-) diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 63ed6b25deaa..42cfefc6e66e 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2044,7 +2044,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, count_vm_event(THP_SPLIT_PMD); if (!vma_is_anonymous(vma)) { - _pmd = pmdp_huge_clear_flush_notify(vma, haddr, pmd); + old_pmd = pmdp_huge_clear_flush_notify(vma, haddr, pmd); /* * We are going to unmap this huge page. So * just go ahead and zap it @@ -2053,16 +2053,25 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, zap_deposited_table(mm, pmd); if (vma_is_special_huge(vma)) return; - page = pmd_page(_pmd); - if (!PageDirty(page) && pmd_dirty(_pmd)) - set_page_dirty(page); - if (!PageReferenced(page) && pmd_young(_pmd)) - SetPageReferenced(page); - page_remove_rmap(page, true); - put_page(page); + if (unlikely(is_pmd_migration_entry(old_pmd))) { + swp_entry_t entry; + + entry = pmd_to_swp_entry(old_pmd); + page = migration_entry_to_page(entry); + } else { + page = pmd_page(old_pmd); + if (!PageDirty(page) && pmd_dirty(old_pmd)) + set_page_dirty(page); + if (!PageReferenced(page) && pmd_young(old_pmd)) + SetPageReferenced(page); + page_remove_rmap(page, true); + put_page(page); + } add_mm_counter(mm, mm_counter_file(page), -HPAGE_PMD_NR); return; - } else if (pmd_trans_huge(*pmd) && is_huge_zero_pmd(*pmd)) { + } + + if (pmd_trans_huge(*pmd) && is_huge_zero_pmd(*pmd)) { /* * FIXME: Do we want to invalidate secondary mmu by calling * mmu_notifier_invalidate_range() see comments below inside diff --git a/mm/pgtable-generic.c b/mm/pgtable-generic.c index c2210e1cdb51..4e640baf9794 100644 --- a/mm/pgtable-generic.c +++ b/mm/pgtable-generic.c @@ -135,9 +135,8 @@ pmd_t pmdp_huge_clear_flush(struct vm_area_struct *vma, unsigned long address, { pmd_t pmd; VM_BUG_ON(address & ~HPAGE_PMD_MASK); - VM_BUG_ON(!pmd_present(*pmdp)); - /* Below assumes pmd_present() is true */ - VM_BUG_ON(!pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp)); + VM_BUG_ON(pmd_present(*pmdp) && !pmd_trans_huge(*pmdp) && + !pmd_devmap(*pmdp)); pmd = pmdp_huge_get_and_clear(vma->vm_mm, address, pmdp); flush_pmd_tlb_range(vma, address, address + HPAGE_PMD_SIZE); return pmd; From patchwork Wed Jun 9 04:08:09 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Hugh Dickins X-Patchwork-Id: 12308827 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-23.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 85237C47095 for ; Wed, 9 Jun 2021 04:08:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 1EE196124C for ; Wed, 9 Jun 2021 04:08:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1EE196124C Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 74E5C6B0036; Wed, 9 Jun 2021 00:08:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 6FE526B006E; Wed, 9 Jun 2021 00:08:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5505E6B0070; Wed, 9 Jun 2021 00:08:14 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0158.hostedemail.com [216.40.44.158]) by kanga.kvack.org (Postfix) with ESMTP id 24A5B6B0036 for ; Wed, 9 Jun 2021 00:08:14 -0400 (EDT) Received: from smtpin39.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id B980C8249980 for ; Wed, 9 Jun 2021 04:08:13 +0000 (UTC) X-FDA: 78232852866.39.ABE5904 Received: from mail-ot1-f50.google.com (mail-ot1-f50.google.com [209.85.210.50]) by imf16.hostedemail.com (Postfix) with ESMTP id 260F080192EE for ; Wed, 9 Jun 2021 04:08:10 +0000 (UTC) Received: by mail-ot1-f50.google.com with SMTP id 6-20020a9d07860000b02903e83bf8f8fcso9611729oto.12 for ; Tue, 08 Jun 2021 21:08:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=date:from:to:cc:subject:in-reply-to:message-id:references :mime-version; bh=QmK8FBhttGUph4KR+UOsWqjeoyXcMjszBFQUOkXXU7c=; b=VcnMtrzlBE+CY5kEaMTWz55SvEBEWCwvrikYxeBoFoxZErVCk3TtL6h4J1h59LMz0V JxeEdVA1ksuzwAK+iAk47JDOZx4YTCcDk8JdBOAv1OnQoonbr2kwMMJkI7tqqaZ0HtAD EoYvnIkY+oC5Pliif6xm5c6BUyv919tY4ctZiE0bHgx6LtTvRDoS6GoLmnMYegZRmJdd 8n1xn99no8iBELP0GxuZV4OAyDWURL5m+kzMRZgQ0J1DtAWGUa9zrXthMKmClikBJGiI pKqfQCP/56PEp2oca7jKizlV7SONU1T2DUmxS1+yTVM3o+MqdJC/pC4yozqUb/rNbhEV 9BDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:in-reply-to:message-id :references:mime-version; bh=QmK8FBhttGUph4KR+UOsWqjeoyXcMjszBFQUOkXXU7c=; b=UH1gE+s22Y1nTt9FSb92qnWEixmK0cX0uEyxSQI0Wyk8e63qJrxSZtBvTzy83wSs3t jNqxYoe4/2E+tfVs+9gr2kDLQW8m6BBVpr2aeS5QSNQOT4bUjdSPbg3DltL3N6sU0aff Z9VqTSWtiy0m807MNOxKkkUly//irvg7bbjcPwv0LZ9NbMar6u5HrEXRtaJT4dYGiHiJ Le96xJ+whVwcrcPmInseZK5IN9IUV8LWO3OlO+4iYJHAiGjCUJML9ET8IFCOC1oGVf3i V6zxecQyNbmoG4HKtAE3f9WK47Uz2kiidoWWUXXXUf/vBYdVZdDAUEhVT0FPuXfjqYuI 4sDQ== X-Gm-Message-State: AOAM531fArMf7alPEFhiISW3nWk+pXHNB7Ep7JJNxcjRWFQGGvx2Jv66 uXlkknMV4vUjrMpj4et0DL3TyQ== X-Google-Smtp-Source: ABdhPJzHa+Ru6vQBkltZYZhJjYyaVUL60oRVZQeRKT3d8Ottzz1rXugi84dq+noUyrtFItweMCApXQ== X-Received: by 2002:a9d:2cf:: with SMTP id 73mr11534662otl.204.1623211692702; Tue, 08 Jun 2021 21:08:12 -0700 (PDT) Received: from ripple.attlocal.net (172-10-233-147.lightspeed.sntcca.sbcglobal.net. [172.10.233.147]) by smtp.gmail.com with ESMTPSA id x13sm3481856ote.70.2021.06.08.21.08.10 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 08 Jun 2021 21:08:12 -0700 (PDT) Date: Tue, 8 Jun 2021 21:08:09 -0700 (PDT) From: Hugh Dickins X-X-Sender: hugh@ripple.anvils To: Andrew Morton cc: Hugh Dickins , "Kirill A. Shutemov" , Yang Shi , Wang Yugui , Matthew Wilcox , Naoya Horiguchi , Alistair Popple , Ralph Campbell , Zi Yan , Miaohe Lin , Minchan Kim , Jue Wang , Peter Xu , Jan Kara , Shakeel Butt , Oscar Salvador , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 02/10] mm/thp: make is_huge_zero_pmd() safe and quicker In-Reply-To: Message-ID: <21ea9ca-a1f5-8b90-5e88-95fb1c49bbfa@google.com> References: MIME-Version: 1.0 X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 260F080192EE Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=VcnMtrzl; spf=pass (imf16.hostedemail.com: domain of hughd@google.com designates 209.85.210.50 as permitted sender) smtp.mailfrom=hughd@google.com; dmarc=pass (policy=reject) header.from=google.com X-Stat-Signature: qog9j75a9yz9nmmobhbq6cp5pqfqp9si X-HE-Tag: 1623211690-813417 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Most callers of is_huge_zero_pmd() supply a pmd already verified present; but a few (notably zap_huge_pmd()) do not - it might be a pmd migration entry, in which the pfn is encoded differently from a present pmd: which might pass the is_huge_zero_pmd() test (though not on x86, since L1TF forced us to protect against that); or perhaps even crash in pmd_page() applied to a swap-like entry. Make it safe by adding pmd_present() check into is_huge_zero_pmd() itself; and make it quicker by saving huge_zero_pfn, so that is_huge_zero_pmd() will not need to do that pmd_page() lookup each time. __split_huge_pmd_locked() checked pmd_trans_huge() before: that worked, but is unnecessary now that is_huge_zero_pmd() checks present. Fixes: e71769ae5260 ("mm: enable thp migration for shmem thp") Signed-off-by: Hugh Dickins Cc: Acked-by: Kirill A. Shutemov Reviewed-by: Yang Shi --- Patch added (replacing part of first) since the v1 series was posted. include/linux/huge_mm.h | 8 +++++++- mm/huge_memory.c | 5 ++++- 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index 9626fda5efce..2a8ebe6c222e 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -286,6 +286,7 @@ struct page *follow_devmap_pud(struct vm_area_struct *vma, unsigned long addr, vm_fault_t do_huge_pmd_numa_page(struct vm_fault *vmf, pmd_t orig_pmd); extern struct page *huge_zero_page; +extern unsigned long huge_zero_pfn; static inline bool is_huge_zero_page(struct page *page) { @@ -294,7 +295,7 @@ static inline bool is_huge_zero_page(struct page *page) static inline bool is_huge_zero_pmd(pmd_t pmd) { - return is_huge_zero_page(pmd_page(pmd)); + return READ_ONCE(huge_zero_pfn) == pmd_pfn(pmd) && pmd_present(pmd); } static inline bool is_huge_zero_pud(pud_t pud) @@ -440,6 +441,11 @@ static inline bool is_huge_zero_page(struct page *page) return false; } +static inline bool is_huge_zero_pmd(pmd_t pmd) +{ + return false; +} + static inline bool is_huge_zero_pud(pud_t pud) { return false; diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 42cfefc6e66e..5885c5f5836f 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -62,6 +62,7 @@ static struct shrinker deferred_split_shrinker; static atomic_t huge_zero_refcount; struct page *huge_zero_page __read_mostly; +unsigned long huge_zero_pfn __read_mostly = ~0UL; bool transparent_hugepage_enabled(struct vm_area_struct *vma) { @@ -98,6 +99,7 @@ static bool get_huge_zero_page(void) __free_pages(zero_page, compound_order(zero_page)); goto retry; } + WRITE_ONCE(huge_zero_pfn, page_to_pfn(zero_page)); /* We take additional reference here. It will be put back by shrinker */ atomic_set(&huge_zero_refcount, 2); @@ -147,6 +149,7 @@ static unsigned long shrink_huge_zero_page_scan(struct shrinker *shrink, if (atomic_cmpxchg(&huge_zero_refcount, 1, 0) == 1) { struct page *zero_page = xchg(&huge_zero_page, NULL); BUG_ON(zero_page == NULL); + WRITE_ONCE(huge_zero_pfn, ~0UL); __free_pages(zero_page, compound_order(zero_page)); return HPAGE_PMD_NR; } @@ -2071,7 +2074,7 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd, return; } - if (pmd_trans_huge(*pmd) && is_huge_zero_pmd(*pmd)) { + if (is_huge_zero_pmd(*pmd)) { /* * FIXME: Do we want to invalidate secondary mmu by calling * mmu_notifier_invalidate_range() see comments below inside