From patchwork Thu Feb 25 07:29:04 2021 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Nadav Amit X-Patchwork-Id: 12103555 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-8.5 required=3.0 tests=BAYES_00,DKIM_ADSP_CUSTOM_MED, DKIM_INVALID,DKIM_SIGNED,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9442EC433DB for ; Thu, 25 Feb 2021 07:33:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0C8E064ED9 for ; Thu, 25 Feb 2021 07:33:50 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0C8E064ED9 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 488236B0071; Thu, 25 Feb 2021 02:33:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 438D36B0072; Thu, 25 Feb 2021 02:33:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 327888D0001; Thu, 25 Feb 2021 02:33:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0022.hostedemail.com [216.40.44.22]) by kanga.kvack.org (Postfix) with ESMTP id 1DDA46B0071 for ; Thu, 25 Feb 2021 02:33:50 -0500 (EST) Received: from smtpin24.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id DD7C510F78 for ; Thu, 25 Feb 2021 07:33:49 +0000 (UTC) X-FDA: 77855975778.24.E6B02A5 Received: from mail-pf1-f176.google.com (mail-pf1-f176.google.com [209.85.210.176]) by imf03.hostedemail.com (Postfix) with ESMTP id 50FC2C0007C8 for ; Thu, 25 Feb 2021 07:33:46 +0000 (UTC) Received: by mail-pf1-f176.google.com with SMTP id z5so3056523pfe.3 for ; Wed, 24 Feb 2021 23:33:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=7uPn+gulMqhjaE42kJBeaOP3zpjHqeQ12o0+1jYccOE=; b=MclzpfaLfKRRoTmtHKOR+QpD7iRU7gZnLM4D29YRc09BXBn6SZA0QA16+njtTz4Yoy Ks5SD7jL/4HV6tcNHIZGTQB/J5tjAasaxzcJKnNe++qadDyDLJsL+yTi5JRr1SjCiqjj +4rDvsM1JQ2ZdaXT1z3K+8BIIaoLhvM8lW+FM/GhEdA/H6PHn6mjp3nt89vBEEvBumIo 4L3Dd5EaOQx+AS8CYcb9m+ZLDGmA3gDJSayp1sQgAqOAi8K7heOgqUP+72auHRfBMeBY zCo7eXCAVdBTaxOnQw2OGRilwbn6RwA+6D2Uv7fPK7pUJzJ4JSaEn2UvPlUgdQGuRzfE sp9w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:mime-version :content-transfer-encoding; bh=7uPn+gulMqhjaE42kJBeaOP3zpjHqeQ12o0+1jYccOE=; b=dRJ7qnsPhS9fvhkXX2yeDx8tu07Zu4tuPxBS4YlBQcqkT/jz0dcVavewrmsfxBN498 avgNhYRKh0CZFrwtuITEaewRVeq3NjQN0v2cZRlfZo2mUwefD8383F2YTSnaALTtHYmS Q+yB1CkJRoYjNkvGJMCiKTNYpvqjTU29K0ueee4KODCR604gbrSyQ2VkWKHgiqlTvUev E6wkiMSWEy/Sl58FLSoJ+GSgmoKeapD1qAiMQzZaLpQVnnxhqzKxQKjhMQpZ/k8ud2Z5 /wzkOeQ46sDwTRb/N+q6KRstcjUq8AcOVs0xP50NaPYnPTaTYgb++yifb8LmCio91v7Z Vdbw== X-Gm-Message-State: AOAM533oRwWMdTMjeglsEie9z/omJn7eEeIa6W6bAUxzcUz4PPV2guRA bBOu46tZkXAa8MfaJjSsUmDyBPLM3CG9qw== X-Google-Smtp-Source: ABdhPJwiWsHFEuH/7othAr6zlobxWGfqE+W8XfT/aMywpYl+JLK2CPN8bEN880NqGtcE7kVM1JG7GQ== X-Received: by 2002:aa7:83cc:0:b029:1ed:446b:1ae2 with SMTP id j12-20020aa783cc0000b02901ed446b1ae2mr1938968pfn.47.1614238427898; Wed, 24 Feb 2021 23:33:47 -0800 (PST) Received: from sc2-haas01-esx0118.eng.vmware.com ([66.170.99.1]) by smtp.gmail.com with ESMTPSA id w3sm4917561pjt.24.2021.02.24.23.33.46 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 24 Feb 2021 23:33:47 -0800 (PST) From: Nadav Amit X-Google-Original-From: Nadav Amit To: linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Hugh Dickins , Andy Lutomirski , Thomas Gleixner , Peter Zijlstra , Ingo Molnar , Borislav Petkov , Nadav Amit , Sean Christopherson , Andrew Morton , x86@kernel.org Subject: [RFC 0/6] x86: prefetch_page() vDSO call Date: Wed, 24 Feb 2021 23:29:04 -0800 Message-Id: <20210225072910.2811795-1-namit@vmware.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-Stat-Signature: egot5zaua7d5hxyywicw7143wy3brxy4 X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 50FC2C0007C8 Received-SPF: none (<>: No applicable sender policy available) receiver=imf03; identity=mailfrom; envelope-from="<>"; helo=mail-pf1-f176.google.com; client-ip=209.85.210.176 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1614238426-493692 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Nadav Amit Just as applications can use prefetch instructions to overlap computations and memory accesses, applications may want to overlap the page-faults and compute or overlap the I/O accesses that are required for page-faults of different pages. Applications can use multiple threads and cores for this matter, by running one thread that prefetches the data (i.e., faults in the data) and another that does the compute, but this scheme is inefficient. Using mincore() can tell whether a page is mapped, but might not tell whether the page is in the page-cache and does not fault in the data. Introduce prefetch_page() vDSO-call to prefetch, i.e. fault-in memory asynchronously. The semantic of this call is: try to prefetch a page of in a given address and return zero if the page is accessible following the call. Start I/O operations to retrieve the page if such operations are required and there is no high memory pressure that might introduce slowdowns. Note that as usual the page might be paged-out at any point and therefore, similarly to mincore(), there is no guarantee that the page will be present at the time that the user application uses the data that resides on the page. Nevertheless, it is expected that in the vast majority of the cases this would not happen, since prefetch_page() accesses the page and therefore sets the PTE access-bit (if it is clear). The implementation is as follows. The vDSO code accesses the data, triggering a page-fault it is not present. The handler detects based on the instruction pointer that this is an asynchronous-#PF, using the recently introduce vDSO exception tables. If the page can be brought without waiting (e.g., the page is already in the page-cache), the kernel handles the fault and returns success (zero). If there is memory pressure that prevents the proper handling of the fault (i.e., requires heavy-weight reclamation) it returns a failure. Otherwise, it starts an I/O to bring the page and returns failure. Compilers can be extended to issue the prefetch_page() calls when needed. Cc: Andy Lutomirski Cc: Peter Zijlstra Cc: Sean Christopherson Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Borislav Petkov Cc: Andrew Morton Cc: x86@kernel.org Nadav Amit (6): vdso/extable: fix calculation of base x86/vdso: add mask and flags to extable x86/vdso: introduce page_prefetch() mm/swap_state: respect FAULT_FLAG_RETRY_NOWAIT mm: use lightweight reclaim on FAULT_FLAG_RETRY_NOWAIT testing/selftest: test vDSO prefetch_page() arch/x86/Kconfig | 1 + arch/x86/entry/vdso/Makefile | 1 + arch/x86/entry/vdso/extable.c | 70 +++-- arch/x86/entry/vdso/extable.h | 21 +- arch/x86/entry/vdso/vdso.lds.S | 1 + arch/x86/entry/vdso/vprefetch.S | 39 +++ arch/x86/entry/vdso/vsgx.S | 9 +- arch/x86/include/asm/vdso.h | 38 ++- arch/x86/mm/fault.c | 11 +- lib/vdso/Kconfig | 5 + mm/memory.c | 47 +++- mm/shmem.c | 1 + mm/swap_state.c | 12 +- tools/testing/selftests/vDSO/Makefile | 2 + .../selftests/vDSO/vdso_test_prefetch_page.c | 265 ++++++++++++++++++ 15 files changed, 470 insertions(+), 53 deletions(-) create mode 100644 arch/x86/entry/vdso/vprefetch.S create mode 100644 tools/testing/selftests/vDSO/vdso_test_prefetch_page.c