From patchwork Mon Feb 3 16:51:16 2020 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Luis Henriques X-Patchwork-Id: 11363109 Return-Path: Received: from mail.kernel.org (pdx-korg-mail-1.web.codeaurora.org [172.30.200.123]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id BCC7F1398 for ; Mon, 3 Feb 2020 16:51:17 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id A50C92051A for ; Mon, 3 Feb 2020 16:51:17 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728384AbgBCQvO (ORCPT ); Mon, 3 Feb 2020 11:51:14 -0500 Received: from mx2.suse.de ([195.135.220.15]:49954 "EHLO mx2.suse.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727319AbgBCQvO (ORCPT ); Mon, 3 Feb 2020 11:51:14 -0500 X-Virus-Scanned: by amavisd-new at test-mx.suse.de Received: from relay2.suse.de (unknown [195.135.220.254]) by mx2.suse.de (Postfix) with ESMTP id B93DCAC44; Mon, 3 Feb 2020 16:51:12 +0000 (UTC) From: Luis Henriques To: Jeff Layton , Sage Weil , Ilya Dryomov , "Yan, Zheng" , Gregory Farnum Cc: ceph-devel@vger.kernel.org, linux-kernel@vger.kernel.org, Luis Henriques Subject: [PATCH v3 0/1] parallel 'copy-from' Ops in copy_file_range Date: Mon, 3 Feb 2020 16:51:16 +0000 Message-Id: <20200203165117.5701-1-lhenriques@suse.com> MIME-Version: 1.0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org Hi, Here's another re-spin of the patch to improve copy_file_range performance by parallelizing the OSD requests. Changelog since v2: - Refactor copy loop to ensure 'ret' is updated with the number of bytes already copied only after the ceph_osdc_wait_requests() returns. This meant to change this helper function to also return the number of successful requests. Also fixed error handling. - Use counter for # copies instead of re-calculating ncopies (now copy_count) everytime time it reaches zero. - Dropped 'dout' in ceph_osdc_wait_requests - Have ceph_osdc_wait_requests and ceph_osdc_wait_request functions close Changelog since v1 (the RFC): - Dropped synchronous version of ceph_osdc_copy_from * This was the reason for merging the patchset into a single patch, as changing ceph_osdc_copy_from definition would break bisectability - Moved wait_copy_from_reqs into libceph, renaming it to ceph_osdc_wait_requests - Use ->r_private_item instead of adding a new list_head into struct ceph_osd_request - Dropped the module parameter (used for testing) and added the throttling mechanism using the formula suggested by Ilya. For reference, here's the original RFC cover letter (fixed typo in results table: 'throttle=5' should be 'throttle=0'). -------------------------------------------------------------------------- As discussed here[1] I'm sending an RFC patchset that does the parallelization of the requests sent to the OSDs during a copy_file_range syscall in CephFS. [1] https://lore.kernel.org/lkml/20200108100353.23770-1-lhenriques@suse.com/ I've also some performance numbers that I wanted to share. Here's a description of the very simple tests I've run: - create a file with 200 objects in it * i.e. tests with different object sizes mean different file sizes - drop all caches and umount the filesystem - Measure: * mount filesystem * full file copy (with copy_file_range) * umount filesystem Tests were repeated several times and the average value was used for comparison. DISCLAIMER: These numbers are only indicative, and different clusters and client configs will for sure show different performance! More rigorous tests would be require to validate these results. Having as baseline a full read+write (basically, a copy_file_range operation within a filesystem mounted without the 'copyfrom' option), here's some values for different object sizes: 8M 4M 1M 65k read+write 100% 100% 100% 100% sequential 51% 52% 83% >100% parallel (throttle=1) 51% 52% 83% >100% parallel (throttle=0) 17% 17% 83% >100% Notes: - 'parallel (throttle=0)' was a test where *all* the requests (i.e. 200 requests to copy the 200 objects in the file) were sent to the OSDs and the wait for requests completion is done at the end only. - 'parallel (throttle=1)' was just a control test, where the wait for completion is done immediately after a request is sent. It was expected to be very similar to the non-optimized ('sequential') tests. - These tests were executed on a cluster with 40 OSDs, spread across 5 (bare-metal) nodes. - The tests with object size of 65k show that copy_file_range definitely doesn't scale to files with small object sizes. '> 100%' actually means more than 10x slower. Measuring the mount+copy+umount masks the actual difference between different throttle values due to the time spent in mount+umount. Thus, there was no real difference between throttle=0 (send all and wait) and throttle=20 (send 20, wait, send 20, ...). But here's what I observed when measuring only the copy operation (4M object size): read+write 100% parallel (throttle=1) 56% parallel (throttle=5) 23% parallel (throttle=10) 14% parallel (throttle=20) 9% parallel (throttle=0) 5% Anyway, I'll still need to revisit patch 0003 as it doesn't follow the suggestion done by Jeff to *not* add another knob to fine-tune the throttle value -- this patch adds a kernel parameter for a knob that I wanted to use in my testing to observe different values of this throttle limit. The goal is to probably to drop this patch and do the throttling in patch 0002. I just need to come up with a decent heuristic. Jeff's suggestion was to use rsize/wsize, which are set to 64M by default IIRC. Somehow I feel that it should be related to the number of OSDs in the cluster instead, but I'm not sure how. And testing these sort of heuristics would require different clusters, which isn't particularly easy to get. Anyway, comments are welcome! Cheers, --- Luis Luis Henriques (1): ceph: parallelize all copy-from requests in copy_file_range fs/ceph/file.c | 45 +++++++++++++++++++++----- include/linux/ceph/osd_client.h | 6 +++- net/ceph/osd_client.c | 56 +++++++++++++++++++++++++-------- 3 files changed, 85 insertions(+), 22 deletions(-)