From patchwork Fri Aug 24 19:25:48 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Jerome Glisse X-Patchwork-Id: 10575709 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id E0AA5921 for ; Fri, 24 Aug 2018 19:26:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id CF4B22BA85 for ; Fri, 24 Aug 2018 19:26:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C39192BAE7; Fri, 24 Aug 2018 19:26:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 408722BA85 for ; Fri, 24 Aug 2018 19:26:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 046AF6B3131; Fri, 24 Aug 2018 15:25:57 -0400 (EDT) Delivered-To: linux-mm-outgoing@kvack.org Received: by kanga.kvack.org (Postfix, from userid 40) id EE66D6B3133; Fri, 24 Aug 2018 15:25:56 -0400 (EDT) X-Original-To: int-list-linux-mm@kvack.org X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC7596B3135; Fri, 24 Aug 2018 15:25:56 -0400 (EDT) X-Original-To: linux-mm@kvack.org X-Delivered-To: linux-mm@kvack.org Received: from mail-qt0-f200.google.com (mail-qt0-f200.google.com [209.85.216.200]) by kanga.kvack.org (Postfix) with ESMTP id 9B6AB6B3131 for ; Fri, 24 Aug 2018 15:25:56 -0400 (EDT) Received: by mail-qt0-f200.google.com with SMTP id b5-v6so7855328qtk.4 for ; Fri, 24 Aug 2018 12:25:56 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-original-authentication-results:x-gm-message-state:from:to:cc :subject:date:message-id:in-reply-to:references:mime-version :content-transfer-encoding; bh=SK4WUpORe+//Dy7cTe6PZth2LRYQLWVWcWj9UEiAPZ4=; b=PolFRjXEQb0Eft9vtDBgbkHjc24IL7j+cHxKATgPKMkNYIQwBz0v9Q7p8lLE/uMT7B QnOZT0cY67bbBguhMGVAw/KSFN4uaA/acWS+9PNlhOsy6wP9HUuR/PkwvtxDk6fCbsxv BMVUf//UXpYbWEk5Ncwg+cNZD5eIbbQvp45qouKrTxBtumdtRKhSyHRi7NozUYf5sCo5 mhQdteL99RX/IbgBJG0PM5isWKGQJwDIlnBEMZ9KTfokyM0eE+LHGEMAw77JqHB7tBYg sMlK9pNYUlEXRsX1ivDCVWMey1338lYPfbXNPcXdvYX/sx8/GCzOKe0J5ZkMVp0Maekj w7UA== X-Original-Authentication-Results: mx.google.com; spf=pass (google.com: domain of jglisse@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=jglisse@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com X-Gm-Message-State: APzg51CRPhYbqex/Q56XUCdpnJXnCdMppHm5NVsj15mV8N5ASo/YEVjr K3ZpsXSBLkqtGoe8+srNoL1DPJVDomgDhD2CAV9d/DPVvP0b6jDER4PNhfsxGlQjWqW20FQRrkc Wi4+KkG0o+qlUdok8Czj+UY7HF8CK6LDvJa1Pe7eVwfJD2TG0oVcuzAHlhnP9WWnqHA== X-Received: by 2002:a37:d282:: with SMTP id f124-v6mr3225050qkj.281.1535138756428; Fri, 24 Aug 2018 12:25:56 -0700 (PDT) X-Google-Smtp-Source: ANB0VdaAlo8JV1qLs8WlLIK2neuMIyMWIfv3imzvIMWlFSqx93qaVWNyGft/+IGUs2BcUwrBD1Ez X-Received: by 2002:a37:d282:: with SMTP id f124-v6mr3225006qkj.281.1535138755821; Fri, 24 Aug 2018 12:25:55 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1535138755; cv=none; d=google.com; s=arc-20160816; b=pq7fldEVGX9MwGF9qTtDHG7yAR4DV79+yzaHul5XImqBfMQjyMBoltAHeNpSclK+FM n0mYjQDaDpqbdqp/9vnKobTGlIrB5UzQlWiDBYGiW9+U/toR86ndUKcN21EIdhYRD0+3 sr6YHpc/A2izpe1jYGRR/+VTmzW/AqBCLAFXaHzWCGD1VlbbPAGARI9P8St4/Gv+MWis hi+yfyog2v6c9AlcR6v2hmgaWvUk9zVmExzaUntye/6yvYFFDkiP8P3VDzcJddoXjH9D onpdZtXPgX8LdchdudvIiW09LUmZhfbb9eRMSdiPUuizuH2MAaMDhBiSHzdUSNq1f3gA vSlw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:arc-authentication-results; bh=SK4WUpORe+//Dy7cTe6PZth2LRYQLWVWcWj9UEiAPZ4=; b=J57813wE/oiFjy98GzmwWlNipBlpqfXzDubi51WbyPnIkqu8lwfw7B3AjJLRo7/xaq 9U//yQjwDBF36WIvloltCV1mmJRudkW7CtWFX37TYw2Nf4z7zyiqss9zUMSNHWknlgYe lfTuPClgswepAJpGKMfLTMF3sZPcrBjOyhbLgy1nOo1P2sv01w1d9azUJh/BLVPQ4i9T KkfyzENgoUh3xl0GLLC8u17fVl5xIeVzjxLqDRLVj7ZWocg0eBxC7Hu1cAnIqFxALiTI hA2pbYrBnvQbEEur1s3FC1L6PARFBT0jOCmbjnaCjMnqEcPooM9nbdtUlwKzyZPLjBNi 0zIg== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of jglisse@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=jglisse@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from mx1.redhat.com (mx3-rdu2.redhat.com. [66.187.233.73]) by mx.google.com with ESMTPS id n57-v6si7926198qtk.212.2018.08.24.12.25.55 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 24 Aug 2018 12:25:55 -0700 (PDT) Received-SPF: pass (google.com: domain of jglisse@redhat.com designates 66.187.233.73 as permitted sender) client-ip=66.187.233.73; Authentication-Results: mx.google.com; spf=pass (google.com: domain of jglisse@redhat.com designates 66.187.233.73 as permitted sender) smtp.mailfrom=jglisse@redhat.com; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.rdu2.redhat.com [10.11.54.4]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 76972402332F; Fri, 24 Aug 2018 19:25:55 +0000 (UTC) Received: from localhost.localdomain.com (ovpn-122-125.rdu2.redhat.com [10.10.122.125]) by smtp.corp.redhat.com (Postfix) with ESMTP id 0D8572026D6D; Fri, 24 Aug 2018 19:25:55 +0000 (UTC) From: jglisse@redhat.com To: linux-mm@kvack.org Cc: Andrew Morton , linux-kernel@vger.kernel.org, =?utf-8?b?SsOpcsO0bWUgR2xpc3Nl?= , Ralph Campbell , John Hubbard Subject: [PATCH 6/7] mm/hmm: invalidate device page table at start of invalidation Date: Fri, 24 Aug 2018 15:25:48 -0400 Message-Id: <20180824192549.30844-7-jglisse@redhat.com> In-Reply-To: <20180824192549.30844-1-jglisse@redhat.com> References: <20180824192549.30844-1-jglisse@redhat.com> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 2.78 on 10.11.54.4 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Fri, 24 Aug 2018 19:25:55 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Fri, 24 Aug 2018 19:25:55 +0000 (UTC) for IP:'10.11.54.4' DOMAIN:'int-mx04.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'jglisse@redhat.com' RCPT:'' X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: X-Virus-Scanned: ClamAV using ClamSMTP From: Jérôme Glisse Invalidate device page table at start of invalidation and invalidate in progress CPU page table snapshooting at both start and end of any invalidation. This is helpful when device need to dirty page because the device page table report the page as dirty. Dirtying page must happen in the start mmu notifier callback and not in the end one. Signed-off-by: Jérôme Glisse Cc: Ralph Campbell Cc: John Hubbard Cc: Andrew Morton --- include/linux/hmm.h | 2 +- mm/hmm.c | 21 ++++++++++++++------- 2 files changed, 15 insertions(+), 8 deletions(-) diff --git a/include/linux/hmm.h b/include/linux/hmm.h index a7f7600b6bb0..064924bce75c 100644 --- a/include/linux/hmm.h +++ b/include/linux/hmm.h @@ -325,7 +325,7 @@ struct hmm_mirror_ops { * synchronous call. */ void (*sync_cpu_device_pagetables)(struct hmm_mirror *mirror, - const struct hmm_update *update); + const struct hmm_update *update); }; /* diff --git a/mm/hmm.c b/mm/hmm.c index debd2f734ab5..6fe31e2bfa1e 100644 --- a/mm/hmm.c +++ b/mm/hmm.c @@ -43,7 +43,6 @@ static const struct mmu_notifier_ops hmm_mmu_notifier_ops; * * @mm: mm struct this HMM struct is bound to * @lock: lock protecting ranges list - * @sequence: we track updates to the CPU page table with a sequence number * @ranges: list of range being snapshotted * @mirrors: list of mirrors for this mm * @mmu_notifier: mmu notifier to track updates to CPU page table @@ -52,7 +51,6 @@ static const struct mmu_notifier_ops hmm_mmu_notifier_ops; struct hmm { struct mm_struct *mm; spinlock_t lock; - atomic_t sequence; struct list_head ranges; struct list_head mirrors; struct mmu_notifier mmu_notifier; @@ -85,7 +83,6 @@ static struct hmm *hmm_register(struct mm_struct *mm) return NULL; INIT_LIST_HEAD(&hmm->mirrors); init_rwsem(&hmm->mirrors_sem); - atomic_set(&hmm->sequence, 0); hmm->mmu_notifier.ops = NULL; INIT_LIST_HEAD(&hmm->ranges); spin_lock_init(&hmm->lock); @@ -126,8 +123,8 @@ void hmm_mm_destroy(struct mm_struct *mm) kfree(mm->hmm); } -static void hmm_invalidate_range(struct hmm *hmm, - const struct hmm_update *update) +static void hmm_invalidate_range(struct hmm *hmm, bool device, + const struct hmm_update *update) { struct hmm_mirror *mirror; struct hmm_range *range; @@ -147,6 +144,9 @@ static void hmm_invalidate_range(struct hmm *hmm, } spin_unlock(&hmm->lock); + if (!device) + return; + down_read(&hmm->mirrors_sem); list_for_each_entry(mirror, &hmm->mirrors, list) mirror->ops->sync_cpu_device_pagetables(mirror, update); @@ -185,11 +185,18 @@ static int hmm_invalidate_range_start(struct mmu_notifier *mn, unsigned long end, bool blockable) { + struct hmm_update update; struct hmm *hmm = mm->hmm; + if (!blockable) + return -EAGAIN; + VM_BUG_ON(!hmm); - atomic_inc(&hmm->sequence); + update.start = start; + update.end = end; + update.event = HMM_UPDATE_INVALIDATE; + hmm_invalidate_range(hmm, true, &update); return 0; } @@ -207,7 +214,7 @@ static void hmm_invalidate_range_end(struct mmu_notifier *mn, update.start = start; update.end = end; update.event = HMM_UPDATE_INVALIDATE; - hmm_invalidate_range(hmm, &update); + hmm_invalidate_range(hmm, false, &update); } static const struct mmu_notifier_ops hmm_mmu_notifier_ops = {