From patchwork Sun Feb 23 19:25:18 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Arcangeli X-Patchwork-Id: 11399011 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 29D30138D for ; Sun, 23 Feb 2020 19:25:49 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 0807E206ED for ; Sun, 23 Feb 2020 19:25:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="g6lSvm5v"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="KWIm41Dr" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0807E206ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yXepzAvvp5dzbB2CemfO/IEWNYjjt5y4G0AiGRO8l6c=; b=g6lSvm5vlrEH+R Jkze18nV+/YRoV8nv3TuVmNTmm/FG5OEY0WOVV0XHoTvIYSS1/8Pl8DGWIQw62dJLKDEMZihcqn6L 42ywmhM5mOnNWjNz2gpsQLcQFk3KbKWbAO63LS4DJa2JvCxT7kU2CgTyknoqKR3mXrDG5z+l9ai0E ChMrj/i1IawSLeB0OwbNZBK7x02XyuBlKYLCIZCf8TiWmsYYIL0ESeCG+IZSBgTZgLbZitwlJmU1H 9c7FDFHwT7WlPZyGSF4ovLtd4/lQCdfCDnt3vWs42W+2pIVZHDGyzfjBqwl23HA40ANfeFWHNJ7dv 86H7h4ZNKm3O5c1v5ONA==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1j5wsu-0000I7-6H; Sun, 23 Feb 2020 19:25:48 +0000 Received: from us-smtp-2.mimecast.com ([207.211.31.81] helo=us-smtp-delivery-1.mimecast.com) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1j5wsh-000060-7X for linux-arm-kernel@lists.infradead.org; Sun, 23 Feb 2020 19:25:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582485932; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=WdAkDw0jQFrUByye8xgaI8zjWL8waIYULZhRupYTANc=; b=KWIm41Dr4PCgJ8lKtHQdx2eGcYTMTuK5CBWtUya2fOCR42OTJVGpDnELB6YqjxS+kIjf6l dHQWxDQWQe/bwnCu26HRTaGGcvB+50zbbB1FgNT/aF/0cp+VoYfU7Gbx+o9tPJaSdK4DU4 Is45a8tXCbrgR03caurIFcznAhZXWA0= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-383-2r8xQ-VeOtOrXPcYfZce8A-1; Sun, 23 Feb 2020 14:25:26 -0500 X-MC-Unique: 2r8xQ-VeOtOrXPcYfZce8A-1 Received: from smtp.corp.redhat.com (int-mx06.intmail.prod.int.phx2.redhat.com [10.5.11.16]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 3F1BF107ACC5; Sun, 23 Feb 2020 19:25:24 +0000 (UTC) Received: from mail (ovpn-120-118.rdu2.redhat.com [10.10.120.118]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3C33A5C1B2; Sun, 23 Feb 2020 19:25:21 +0000 (UTC) From: Andrea Arcangeli To: Will Deacon , Catalin Marinas , Rafael Aquini , Mark Salter Subject: [PATCH 1/3] mm: use_mm: fix for arches checking mm_users to optimize TLB flushes Date: Sun, 23 Feb 2020 14:25:18 -0500 Message-Id: <20200223192520.20808-2-aarcange@redhat.com> In-Reply-To: <20200223192520.20808-1-aarcange@redhat.com> References: <20200223192520.20808-1-aarcange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.16 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200223_112535_345549_18DD51D0 X-CRM114-Status: UNSURE ( 9.87 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [207.211.31.81 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: QI Fuli , Jon Masters , linux-kernel@vger.kernel.org, Michal Hocko , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org alpha, ia64, mips, powerpc, sh, sparc are relying on a check on mm->mm_users to know if they can skip some remote TLB flushes for single threaded processes. Most callers of use_mm() tend to invoke mmget_not_zero() or get_task_mm() before use_mm() to ensure the mm will remain alive in between use_mm() and unuse_mm(). Some callers however don't increase mm_users and they instead rely on serialization in __mmput() to ensure the mm will remain alive in between use_mm() and unuse_mm(). Not increasing mm_users during use_mm() is however unsafe for aforementioned arch TLB flushes optimizations. So either mmget()/mmput() should be added to the problematic callers of use_mm()/unuse_mm() or we can embed them in use_mm()/unuse_mm() which is more robust. Signed-off-by: Andrea Arcangeli Acked-by: Rafael Aquini --- mm/mmu_context.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/mm/mmu_context.c b/mm/mmu_context.c index 3e612ae748e9..ced0e1218c0f 100644 --- a/mm/mmu_context.c +++ b/mm/mmu_context.c @@ -30,6 +30,7 @@ void use_mm(struct mm_struct *mm) mmgrab(mm); tsk->active_mm = mm; } + mmget(mm); tsk->mm = mm; switch_mm(active_mm, mm, tsk); task_unlock(tsk); @@ -57,6 +58,7 @@ void unuse_mm(struct mm_struct *mm) task_lock(tsk); sync_mm_rss(mm); tsk->mm = NULL; + mmput(mm); /* active_mm is still 'mm' */ enter_lazy_tlb(mm, tsk); task_unlock(tsk); From patchwork Sun Feb 23 19:25:19 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Arcangeli X-Patchwork-Id: 11399007 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id 72FB814E3 for ; Sun, 23 Feb 2020 19:25:40 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 514FD206ED for ; Sun, 23 Feb 2020 19:25:40 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="bpCs0eLW"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="E3gpa2eW" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 514FD206ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=qDw6n/EDGKMH/k69y4O69lBXp/b+EZ8a7IQvudyjq/U=; b=bpCs0eLWYYHO41 Qv751qqTp4YdPXonIhhPK9UxPJzmVz2oeE0ZdRy7gJOifUhkbxgANaFlQfFQuN5tFyzfmG5j8mBf+ O5GY07pb9BOPLGp0vnaxLEDz0uSreHlt2fnghdHyWDozEhCabgqqHSp4/qYZMFbytWlwYqHZCQh46 DF3FE7ce8oMIJ+QJQRdIX8/zzd/BxRiNwaf+8bZFdcFnNfRQfE9Lpb2sMERLchab2VNBxZ/Jy2cLi MK/QBVQ0BFznsq5aDkznSt0SIYbdFzrejIOSGeC+RLLBDiUTOqmGL0atFquTDyXQddF+sv/LAr3th 9uG7tLPzqmjIsO+iWNLg==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1j5wsk-00008M-0y; Sun, 23 Feb 2020 19:25:38 +0000 Received: from us-smtp-delivery-1.mimecast.com ([205.139.110.120] helo=us-smtp-1.mimecast.com) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1j5wsg-00005i-Ud for linux-arm-kernel@lists.infradead.org; Sun, 23 Feb 2020 19:25:36 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582485929; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=UU+BZ7vU0vNDbayPRMPpX1ixOqnikP7N9LBd4qlvCto=; b=E3gpa2eWonfZ8wXHmPjW9pmZadPecWfzhrQBOEVZbPcpMlZmof6n0VD2IXHWbfrLmfHXsX WtpbPKOgOKcVrDy+x5y1QL8uY4TR5Wk8LxsjsdNld7PSrxN0lHU310RvJwU0OMAJXrjml7 ZITAgn9yj5Q0/XsIVjn43Y/jO3DOQ18= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-358-UwRa0Mr-NV2TRQOgVtHpPw-1; Sun, 23 Feb 2020 14:25:25 -0500 X-MC-Unique: UwRa0Mr-NV2TRQOgVtHpPw-1 Received: from smtp.corp.redhat.com (int-mx02.intmail.prod.int.phx2.redhat.com [10.5.11.12]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 1F85E184B121; Sun, 23 Feb 2020 19:25:24 +0000 (UTC) Received: from mail (ovpn-120-118.rdu2.redhat.com [10.10.120.118]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 3A42F60BF1; Sun, 23 Feb 2020 19:25:21 +0000 (UTC) From: Andrea Arcangeli To: Will Deacon , Catalin Marinas , Rafael Aquini , Mark Salter Subject: [PATCH 2/3] arm64: select CPUMASK_OFFSTACK if NUMA Date: Sun, 23 Feb 2020 14:25:19 -0500 Message-Id: <20200223192520.20808-3-aarcange@redhat.com> In-Reply-To: <20200223192520.20808-1-aarcange@redhat.com> References: <20200223192520.20808-1-aarcange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.12 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200223_112535_070244_3A178616 X-CRM114-Status: UNSURE ( 9.99 ) X-CRM114-Notice: Please train this message. X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [205.139.110.120 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: QI Fuli , Jon Masters , linux-kernel@vger.kernel.org, Michal Hocko , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org It's unclear why normally CPUMASK_OFFSTACK can only be manually configured "if DEBUG_PER_CPU_MAPS" which is not an option meant to be enabled on enterprise arm64 kernels. The default enterprise kernels NR_CPUS is 4096 which is fairly large. So it'll save some RAM and it'll increase reliability to select CPUMASK_OFFSET at least when NUMA is selected and a large NR_CPUS is to be expected. Signed-off-by: Andrea Arcangeli Acked-by: Rafael Aquini --- arch/arm64/Kconfig | 1 + 1 file changed, 1 insertion(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 0b30e884e088..882887e65394 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -955,6 +955,7 @@ config NUMA bool "Numa Memory Allocation and Scheduler Support" select ACPI_NUMA if ACPI select OF_NUMA + select CPUMASK_OFFSTACK help Enable NUMA (Non Uniform Memory Access) support. From patchwork Sun Feb 23 19:25:20 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Andrea Arcangeli X-Patchwork-Id: 11399015 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id D3A8214E3 for ; Sun, 23 Feb 2020 19:26:24 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id A9DAD206ED for ; Sun, 23 Feb 2020 19:26:24 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="O5pXHyBI"; dkim=fail reason="signature verification failed" (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="SYezrZJf" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org A9DAD206ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/YyWJb1aMdVlgFBfrb1feijY/5TktStMtzJp/BLglS0=; b=O5pXHyBIkWGcrG xZzGIf9rKvL5jjcTxiJHheC1X6sX3g2jgkLRz6UNu5WOF/jYIb+aA0qgPQFWDr+5Q1Zj/oN6V10V2 5Sxo05Zs98e2PLYwz7fdlLnImTYWV3cSuvwwKzT5ra9X3b/7cGxE2ahHs69JL1/6szCS4kO13bn76 Z6Uv2WmqC4vqXRftyvBA9j/7TAaK87/kvkk+Tmm9YOAkaiUlw3WAkKytpomXUeZ+Ax1yIszB+Beoy CNBrXJrCvQxUVPwCj8ntB2/izv+Y7kxPo0YjZe6EL87IOnS15X8Nj0CD0MRx1lddaZansG8+JYsZi w8Q7sxJ0bumDrLCdp+0g==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1j5wtS-0000oV-HM; Sun, 23 Feb 2020 19:26:22 +0000 Received: from us-smtp-delivery-1.mimecast.com ([207.211.31.120] helo=us-smtp-1.mimecast.com) by bombadil.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1j5wst-0000DJ-PS for linux-arm-kernel@lists.infradead.org; Sun, 23 Feb 2020 19:25:50 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1582485941; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=i/2wg9LCl0SB70jh/SKKr4EkDW2T5q3maIDLvIOj7Es=; b=SYezrZJfJgap4s6s/uTmFk2/HyQb203Xd59gLwvYeuK1LnjF4iW+PtOc1fPnIRHL+uzmzn 6m/hRV+e4cbN8xxDQglQ0SUywrCrBmSBxk0NgwL4XKx02pUnK58E4Lqbgfvl3M/FjgSvg7 yFfGdWQXkKwtYhFozWEILeGveu7ZjMU= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-73-K7Mug57cNKCBvKO9UlWLLw-1; Sun, 23 Feb 2020 14:25:25 -0500 X-MC-Unique: K7Mug57cNKCBvKO9UlWLLw-1 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.13]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 02278107B26F; Sun, 23 Feb 2020 19:25:24 +0000 (UTC) Received: from mail (ovpn-120-118.rdu2.redhat.com [10.10.120.118]) by smtp.corp.redhat.com (Postfix) with ESMTPS id 4AAB990772; Sun, 23 Feb 2020 19:25:21 +0000 (UTC) From: Andrea Arcangeli To: Will Deacon , Catalin Marinas , Rafael Aquini , Mark Salter Subject: [PATCH 3/3] arm64: tlb: skip tlbi broadcast Date: Sun, 23 Feb 2020 14:25:20 -0500 Message-Id: <20200223192520.20808-4-aarcange@redhat.com> In-Reply-To: <20200223192520.20808-1-aarcange@redhat.com> References: <20200223192520.20808-1-aarcange@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.79 on 10.5.11.13 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200223_112548_054898_0F09A80E X-CRM114-Status: GOOD ( 20.98 ) X-Spam-Score: -0.2 (/) X-Spam-Report: SpamAssassin version 3.4.3 on bombadil.infradead.org summary: Content analysis details: (-0.2 points) pts rule name description ---- ---------------------- -------------------------------------------------- -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at https://www.dnswl.org/, no trust [207.211.31.120 listed in list.dnswl.org] 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record -0.0 SPF_PASS SPF: sender matches SPF record -0.1 DKIM_VALID_AU Message has a valid DKIM or DK signature from author's domain -0.1 DKIM_VALID Message has at least one valid DKIM or DK signature 0.1 DKIM_SIGNED Message has a DKIM or DK signature, not necessarily valid -0.1 DKIM_VALID_EF Message has a valid DKIM or DK signature from envelope-from domain -0.0 DKIMWL_WL_HIGH DKIMwl.org - Whitelisted High sender X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: QI Fuli , Jon Masters , linux-kernel@vger.kernel.org, Michal Hocko , linux-mm@kvack.org, linux-arm-kernel@lists.infradead.org Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+patchwork-linux-arm=patchwork.kernel.org@lists.infradead.org With multiple NUMA nodes and multiple sockets, the tlbi broadcast shall be delivered through the interconnects in turn increasing the CPU interconnect traffic and the latency of the tlbi broadcast instruction. To avoid the synchronous delivery of the tlbi broadcast before the tlbi instruction can be retired, the hardware would need to implement a replicated mm_cpumask bitflag for each ASID and every CPU would need to tell every other CPU which ASID is being loaded. Exactly what x86 does with mm_cpumask in software. Even within a single NUMA node the latency of the tlbi broadcast instruction increases almost linearly with the number of CPUs trying to send tlbi broadcasts at the same time. If a single thread of the process is running and it's also running in the CPU issuing the TLB flush, or if no thread of the process are running, we can achieve full SMP scalability in the arm64 TLB flushng by skipping the tlbi broadcasting. After the local TLB flush this means the ASID context goes out of sync in all CPUs except the local one. This can be tracked on the per-mm cpumask: if the bit is set it means the ASID context is stale for that CPU. This results in an extra local ASID TLB flush only when threads are running in new CPUs after a TLB flush. Skipping the tlbi instruction broadcasting is already implemented in local_flush_tlb_all(), this patch only extends it to flush_tlb_mm(), flush_tlb_range() and flush_tlb_page() too. The below benchmarks are measured on a non-NUMA 32 CPUs system (ARMv8 Ampere), so it should be far from a worst case scenario: the enterprise kernel config allows multiple NUMA nodes with NR_CPUS set by default to 4096. === stock === # cat for-each-cpu.sh #!/bin/bash for i in $(seq `nproc`); do "$@" &>/dev/null & done wait # perf stat -r 10 -e dummy ./for-each-cpu.sh ./mprotect-threaded 10000 [..] 2.1696 +- 0.0122 seconds time elapsed ( +- 0.56% ) # perf stat -r 10 -e dummy ./for-each-cpu.sh ./gperftools/tcmalloc_large_heap_fragmentation_unittest [..] 0.99018 +- 0.00360 seconds time elapsed ( +- 0.36% ) # cat sort-compute #!/bin/bash for x in `seq 256`; do for i in `seq 32`; do /usr/bin/sort /dev/null; done & done wait # perf stat -r 10 -e dummy ./sort-compute [..] 1.8094 +- 0.0139 seconds time elapsed ( +- 0.77% ) === patch applied === # perf stat -r 10 -e dummy ./for-each-cpu.sh ./mprotect-threaded 10000 [..] 0.13941 +- 0.00449 seconds time elapsed ( +- 3.22% ) # perf stat -r 10 -e dummy ./for-each-cpu.sh ./gperftools/tcmalloc_large_heap_fragmentation_unittest [..] 0.90510 +- 0.00262 seconds time elapsed ( +- 0.29% ) # perf stat -r 10 -e dummy ./sort-compute [..] 1.64025 +- 0.00618 seconds time elapsed ( +- 0.38% ) Signed-off-by: Andrea Arcangeli Signed-off-by: Andrea Arcangeli --- arch/arm64/include/asm/efi.h | 2 +- arch/arm64/include/asm/mmu.h | 4 +- arch/arm64/include/asm/mmu_context.h | 33 ++++++++-- arch/arm64/include/asm/tlbflush.h | 95 +++++++++++++++++++++++++++- arch/arm64/mm/context.c | 54 ++++++++++++++++ 5 files changed, 177 insertions(+), 11 deletions(-) diff --git a/arch/arm64/include/asm/efi.h b/arch/arm64/include/asm/efi.h index 44531a69d32b..5d9a1433d918 100644 --- a/arch/arm64/include/asm/efi.h +++ b/arch/arm64/include/asm/efi.h @@ -131,7 +131,7 @@ static inline void efifb_setup_from_dmi(struct screen_info *si, const char *opt) static inline void efi_set_pgd(struct mm_struct *mm) { - __switch_mm(mm); + __switch_mm(mm, smp_processor_id()); if (system_uses_ttbr0_pan()) { if (mm != current->active_mm) { diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h index e4d862420bb4..9072fd7bc5f8 100644 --- a/arch/arm64/include/asm/mmu.h +++ b/arch/arm64/include/asm/mmu.h @@ -20,6 +20,7 @@ typedef struct { atomic64_t id; void *vdso; unsigned long flags; + atomic_t nr_active_mm; } mm_context_t; /* @@ -27,7 +28,8 @@ typedef struct { * ASID change and therefore doesn't need to reload the counter using * atomic64_read. */ -#define ASID(mm) ((mm)->context.id.counter & 0xffff) +#define __ASID(asid) ((asid) & 0xffff) +#define ASID(mm) __ASID((mm)->context.id.counter) extern bool arm64_use_ng_mappings; diff --git a/arch/arm64/include/asm/mmu_context.h b/arch/arm64/include/asm/mmu_context.h index 3827ff4040a3..9c66fe317e2f 100644 --- a/arch/arm64/include/asm/mmu_context.h +++ b/arch/arm64/include/asm/mmu_context.h @@ -175,7 +175,10 @@ static inline void cpu_replace_ttbr1(pgd_t *pgdp) #define destroy_context(mm) do { } while(0) void check_and_switch_context(struct mm_struct *mm, unsigned int cpu); -#define init_new_context(tsk,mm) ({ atomic64_set(&(mm)->context.id, 0); 0; }) +#define init_new_context(tsk,mm) \ + ({ atomic64_set(&(mm)->context.id, 0); \ + atomic_set(&(mm)->context.nr_active_mm, 0); \ + 0; }) #ifdef CONFIG_ARM64_SW_TTBR0_PAN static inline void update_saved_ttbr0(struct task_struct *tsk, @@ -203,6 +206,15 @@ static inline void update_saved_ttbr0(struct task_struct *tsk, static inline void enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) { + unsigned int cpu = smp_processor_id(); + if (per_cpu(cpu_not_lazy_tlb, cpu) && + is_idle_task(tsk)) { + per_cpu(cpu_not_lazy_tlb, cpu) = false; + if (!system_uses_ttbr0_pan()) + cpu_set_reserved_ttbr0(); + atomic_dec(&mm->context.nr_active_mm); + } + VM_WARN_ON(atomic_read(&mm->context.nr_active_mm) < 0); /* * We don't actually care about the ttbr0 mapping, so point it at the * zero page. @@ -210,10 +222,8 @@ enter_lazy_tlb(struct mm_struct *mm, struct task_struct *tsk) update_saved_ttbr0(tsk, &init_mm); } -static inline void __switch_mm(struct mm_struct *next) +static inline void __switch_mm(struct mm_struct *next, unsigned int cpu) { - unsigned int cpu = smp_processor_id(); - /* * init_mm.pgd does not contain any user mappings and it is always * active for kernel addresses in TTBR1. Just set the reserved TTBR0. @@ -230,8 +240,19 @@ static inline void switch_mm(struct mm_struct *prev, struct mm_struct *next, struct task_struct *tsk) { - if (prev != next) - __switch_mm(next); + unsigned int cpu = smp_processor_id(); + + if (!per_cpu(cpu_not_lazy_tlb, cpu)) { + per_cpu(cpu_not_lazy_tlb, cpu) = true; + atomic_inc(&next->context.nr_active_mm); + __switch_mm(next, cpu); + } else if (prev != next) { + atomic_inc(&next->context.nr_active_mm); + __switch_mm(next, cpu); + atomic_dec(&prev->context.nr_active_mm); + } + VM_WARN_ON(!atomic_read(&next->context.nr_active_mm)); + VM_WARN_ON(atomic_read(&prev->context.nr_active_mm) < 0); /* * Update the saved TTBR0_EL1 of the scheduled-in task as the previous diff --git a/arch/arm64/include/asm/tlbflush.h b/arch/arm64/include/asm/tlbflush.h index bc3949064725..0bd987ff9cbd 100644 --- a/arch/arm64/include/asm/tlbflush.h +++ b/arch/arm64/include/asm/tlbflush.h @@ -136,6 +136,15 @@ static inline void local_flush_tlb_all(void) isb(); } +static inline void local_flush_tlb_asid(unsigned long asid) +{ + asid = __TLBI_VADDR(0, __ASID(asid)); + dsb(nshst); + __tlbi(aside1, asid); + __tlbi_user(aside1, asid); + dsb(nsh); +} + static inline void flush_tlb_all(void) { dsb(ishst); @@ -144,9 +153,38 @@ static inline void flush_tlb_all(void) isb(); } +DECLARE_PER_CPU(bool, cpu_not_lazy_tlb); + +enum tlb_flush_types { + TLB_FLUSH_NO, + TLB_FLUSH_LOCAL, + TLB_FLUSH_BROADCAST, +}; +extern enum tlb_flush_types tlb_flush_check(struct mm_struct *mm, + unsigned int cpu); + static inline void flush_tlb_mm(struct mm_struct *mm) { unsigned long asid = __TLBI_VADDR(0, ASID(mm)); + enum tlb_flush_types flush; + + flush = tlb_flush_check(mm, get_cpu()); + switch (flush) { + case TLB_FLUSH_LOCAL: + + dsb(nshst); + __tlbi(aside1, asid); + __tlbi_user(aside1, asid); + dsb(nsh); + + /* fall through */ + case TLB_FLUSH_NO: + put_cpu(); + return; + case TLB_FLUSH_BROADCAST: + break; + } + put_cpu(); dsb(ishst); __tlbi(aside1is, asid); @@ -167,7 +205,31 @@ static inline void flush_tlb_page_nosync(struct vm_area_struct *vma, static inline void flush_tlb_page(struct vm_area_struct *vma, unsigned long uaddr) { - flush_tlb_page_nosync(vma, uaddr); + struct mm_struct *mm = vma->vm_mm; + unsigned long addr = __TLBI_VADDR(uaddr, ASID(mm)); + enum tlb_flush_types flush; + + flush = tlb_flush_check(mm, get_cpu()); + switch (flush) { + case TLB_FLUSH_LOCAL: + + dsb(nshst); + __tlbi(vale1, addr); + __tlbi_user(vale1, addr); + dsb(nsh); + + /* fall through */ + case TLB_FLUSH_NO: + put_cpu(); + return; + case TLB_FLUSH_BROADCAST: + break; + } + put_cpu(); + + dsb(ishst); + __tlbi(vale1is, addr); + __tlbi_user(vale1is, addr); dsb(ish); } @@ -181,14 +243,16 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, unsigned long start, unsigned long end, unsigned long stride, bool last_level) { - unsigned long asid = ASID(vma->vm_mm); + struct mm_struct *mm = vma->vm_mm; + unsigned long asid = ASID(mm); unsigned long addr; + enum tlb_flush_types flush; start = round_down(start, stride); end = round_up(end, stride); if ((end - start) >= (MAX_TLBI_OPS * stride)) { - flush_tlb_mm(vma->vm_mm); + flush_tlb_mm(mm); return; } @@ -198,6 +262,31 @@ static inline void __flush_tlb_range(struct vm_area_struct *vma, start = __TLBI_VADDR(start, asid); end = __TLBI_VADDR(end, asid); + flush = tlb_flush_check(mm, get_cpu()); + switch (flush) { + case TLB_FLUSH_LOCAL: + + dsb(nshst); + for (addr = start; addr < end; addr += stride) { + if (last_level) { + __tlbi(vale1, addr); + __tlbi_user(vale1, addr); + } else { + __tlbi(vae1, addr); + __tlbi_user(vae1, addr); + } + } + dsb(nsh); + + /* fall through */ + case TLB_FLUSH_NO: + put_cpu(); + return; + case TLB_FLUSH_BROADCAST: + break; + } + put_cpu(); + dsb(ishst); for (addr = start; addr < end; addr += stride) { if (last_level) { diff --git a/arch/arm64/mm/context.c b/arch/arm64/mm/context.c index 8ef73e89d514..3152b7f7da12 100644 --- a/arch/arm64/mm/context.c +++ b/arch/arm64/mm/context.c @@ -25,6 +25,7 @@ static unsigned long *asid_map; static DEFINE_PER_CPU(atomic64_t, active_asids); static DEFINE_PER_CPU(u64, reserved_asids); static cpumask_t tlb_flush_pending; +DEFINE_PER_CPU(bool, cpu_not_lazy_tlb); #define ASID_MASK (~GENMASK(asid_bits - 1, 0)) #define ASID_FIRST_VERSION (1UL << asid_bits) @@ -191,6 +192,12 @@ static u64 new_context(struct mm_struct *mm) set_asid: __set_bit(asid, asid_map); cur_idx = asid; + /* + * check_and_switch_context() will change the ASID of this mm + * so no need of extra ASID local TLB flushes: the new ASID + * isn't stale anymore after the tlb_flush_pending was set. + */ + cpumask_clear(mm_cpumask(mm)); return idx2asid(asid) | generation; } @@ -240,6 +247,15 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) raw_spin_unlock_irqrestore(&cpu_asid_lock, flags); switch_mm_fastpath: + /* + * Enforce CPU ordering between the atomic_inc(nr_active_mm) + * in switch_mm() and the below cpumask_test_cpu(mm_cpumask). + */ + smp_mb(); + if (cpumask_test_cpu(cpu, mm_cpumask(mm))) { + cpumask_clear_cpu(cpu, mm_cpumask(mm)); + local_flush_tlb_asid(asid); + } arm64_apply_bp_hardening(); @@ -251,6 +267,44 @@ void check_and_switch_context(struct mm_struct *mm, unsigned int cpu) cpu_switch_mm(mm->pgd, mm); } +enum tlb_flush_types tlb_flush_check(struct mm_struct *mm, unsigned int cpu) +{ + if (atomic_read(&mm->context.nr_active_mm) <= 1) { + bool is_local = current->active_mm == mm && + per_cpu(cpu_not_lazy_tlb, cpu); + cpumask_t *stale_cpumask = mm_cpumask(mm); + unsigned int next_zero = cpumask_next_zero(-1, stale_cpumask); + bool local_is_clear = false; + if (next_zero < nr_cpu_ids && + (is_local && next_zero == cpu)) { + next_zero = cpumask_next_zero(next_zero, stale_cpumask); + local_is_clear = true; + } + if (next_zero < nr_cpu_ids) { + cpumask_setall(stale_cpumask); + local_is_clear = false; + } + + /* + * Enforce CPU ordering between the above + * cpumask_setall(mm_cpumask) and the below + * atomic_read(nr_active_mm). + */ + smp_mb(); + + if (likely(atomic_read(&mm->context.nr_active_mm)) <= 1) { + if (is_local) { + if (!local_is_clear) + cpumask_clear_cpu(cpu, stale_cpumask); + return TLB_FLUSH_LOCAL; + } + if (atomic_read(&mm->context.nr_active_mm) == 0) + return TLB_FLUSH_NO; + } + } + return TLB_FLUSH_BROADCAST; +} + /* Errata workaround post TTBRx_EL1 update. */ asmlinkage void post_ttbr_update_workaround(void) {