From patchwork Mon Dec 17 16:29:34 2018 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Simmons X-Patchwork-Id: 10733823 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork-2.web.codeaurora.org (Postfix) with ESMTP id B6E521399 for ; Mon, 17 Dec 2018 16:30:10 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id A84EB2A1E6 for ; Mon, 17 Dec 2018 16:30:10 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 927062A206; Mon, 17 Dec 2018 16:30:10 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.9 required=2.0 tests=BAYES_00,MAILING_LIST_MULTI, RCVD_IN_DNSWL_NONE autolearn=ham version=3.3.1 Received: from pdx1-mailman02.dreamhost.com (pdx1-mailman02.dreamhost.com [64.90.62.194]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id DEE572A1EF for ; Mon, 17 Dec 2018 16:30:08 +0000 (UTC) Received: from pdx1-mailman02.dreamhost.com (localhost [IPv6:::1]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 524D621F426; Mon, 17 Dec 2018 08:30:08 -0800 (PST) X-Original-To: lustre-devel@lists.lustre.org Delivered-To: lustre-devel-lustre.org@pdx1-mailman02.dreamhost.com Received: from smtp3.ccs.ornl.gov (smtp3.ccs.ornl.gov [160.91.203.39]) by pdx1-mailman02.dreamhost.com (Postfix) with ESMTP id 7E7BD21F426 for ; Mon, 17 Dec 2018 08:30:06 -0800 (PST) Received: from star.ccs.ornl.gov (star.ccs.ornl.gov [160.91.202.134]) by smtp3.ccs.ornl.gov (Postfix) with ESMTP id 1C37025F; Mon, 17 Dec 2018 11:30:05 -0500 (EST) Received: by star.ccs.ornl.gov (Postfix, from userid 2004) id 162921E8; Mon, 17 Dec 2018 11:30:05 -0500 (EST) From: James Simmons To: Andreas Dilger , Oleg Drokin , Bobi Jam , Jinshan Xiong , NeilBrown Date: Mon, 17 Dec 2018 11:29:34 -0500 Message-Id: <1545064202-22483-1-git-send-email-jsimmons@infradead.org> X-Mailer: git-send-email 1.8.3.1 Subject: [lustre-devel] [PATCH RFC 00/28] lustre: PFL port to linux client X-BeenThere: lustre-devel@lists.lustre.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: "For discussing Lustre software development." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Lustre Development List MIME-Version: 1.0 Errors-To: lustre-devel-bounces@lists.lustre.org Sender: "lustre-devel" X-Virus-Scanned: ClamAV using ClamSMTP This is the initial PFL port to the linux lustre client. This opens up feed back on the port so far. Currently sanity passes but the test for sanity-pfl fail as below. I have been tracking downing various bugs but this one remains and I haven't found out why its failing. So far from what I can tell is lov_io_setattr_iter_init() it returning -ENODATA due to lsm_entry_inited() is not initialized. Hoping that sending this out more eyes might help to see where this last problem is. Lustre: DEBUG MARKER: == sanity-pfl test 0: Create full components file, no reused OSTs ======= ============================= 10:53:08 (1545061988) Lustre: DEBUG MARKER: create directory /lustre/lustre/d0.sanity-pfl Lustre: DEBUG MARKER: create comp_file Lustre: DEBUG MARKER: instantiate components LustreError: 19350:0:(cl_io.c:439:cl_io_iter_fini()) ASSERTION( io->ci_state == CIS_UNLOCKED ) failed: LustreError: 19350:0:(cl_io.c:439:cl_io_iter_fini()) LBUG Pid: 19350, comm: dd 4.20.0-rc6+ #1 SMP PREEMPT Sat Dec 15 11:22:06 EST 2018 Call Trace: libcfs_call_trace+0x8b/0xc0 [libcfs] lbug_with_loc+0x41/0x90 [libcfs] cl_io_iter_fini+0x10c/0x110 [obdclass] cl_io_loop+0x46/0x220 [obdclass] cl_setattr_ost+0x1ed/0x2a0 [lustre] ll_setattr_raw+0x797/0x980 [lustre] notify_change+0x1dc/0x430 do_truncate+0x72/0xc0 do_sys_ftruncate+0xf5/0x160 do_syscall_64+0x68/0x38f Bobi Jam (20): lustre: lov: move code for PFL work lustre: lov: merge lov_mds_md_v3 and lov_mds_md_v1 handling lustre: lov: fold lmm_verify() handling into lmm_unpackmd() lustre: lov: create struct lov_stripe_md_entry lustre: lov: add composite layout unpacking lustre: lov: embedded raid0 in struct lov_layout_composite lustre: lov: migrate lov raid0 to future PFL component handling lustre: lov: reduce code indentation lustre: lov: change lo_entries to array. lustre: lov: move around PFL code and cleanups lustre: lov: remove lsm_stripe_by_[index|offset]_plain lustre: lov: add looping lsm_entry_count times lustre: lov: create lov_comp_* wrappers lustre: clio: client side implementation for PFL lustre: pfl: dynamic layout modification with write/truncate lustre: pfl: calculate PFL file LOVEA correctly lustre: lov: keep minimum LOVEA size lustre: pfl: fix hang with grouplocks lustre: pfl: fix ost pool op->size handling lustre: llite: restore ll_file_getstripe in ll_lov_setstripe Fan Yong (1): lustre: pfl: enhance PFID EA for PFL Jinshan Xiong (3): lustre: pfl: Read should not trigger layout write intent lustre: lov: readahead shouldn't exceed component boundary lustre: lov: do not split IO for single striped file Niu Yawei (4): lustre: pfl: Basic data structures for composite layout lustre: clio: getstripe support comp layout lustre: uapi: support negative flags lustre: llite: return v1/v3 layout for legacy app .../lustre/include/uapi/linux/lustre/lustre_idl.h | 36 +- .../lustre/include/uapi/linux/lustre/lustre_user.h | 88 ++- drivers/staging/lustre/lustre/include/cl_object.h | 12 +- drivers/staging/lustre/lustre/include/lustre_sec.h | 4 +- .../staging/lustre/lustre/include/lustre_swab.h | 1 + drivers/staging/lustre/lustre/include/obd.h | 4 - drivers/staging/lustre/lustre/llite/dir.c | 38 +- drivers/staging/lustre/lustre/llite/file.c | 185 +++-- .../staging/lustre/lustre/llite/llite_internal.h | 3 + drivers/staging/lustre/lustre/llite/vvp_io.c | 44 +- drivers/staging/lustre/lustre/llite/xattr.c | 70 +- .../staging/lustre/lustre/lov/lov_cl_internal.h | 191 ++--- drivers/staging/lustre/lustre/lov/lov_ea.c | 570 ++++++++++---- drivers/staging/lustre/lustre/lov/lov_internal.h | 175 +++-- drivers/staging/lustre/lustre/lov/lov_io.c | 651 +++++++++------- drivers/staging/lustre/lustre/lov/lov_lock.c | 94 ++- drivers/staging/lustre/lustre/lov/lov_merge.c | 12 +- drivers/staging/lustre/lustre/lov/lov_object.c | 833 ++++++++++++--------- drivers/staging/lustre/lustre/lov/lov_offset.c | 65 +- drivers/staging/lustre/lustre/lov/lov_pack.c | 364 +++++---- drivers/staging/lustre/lustre/lov/lov_page.c | 42 +- drivers/staging/lustre/lustre/lov/lov_pool.c | 20 +- drivers/staging/lustre/lustre/lov/lovsub_object.c | 23 +- drivers/staging/lustre/lustre/mdc/mdc_locks.c | 79 +- drivers/staging/lustre/lustre/obdclass/cl_object.c | 5 +- drivers/staging/lustre/lustre/obdclass/genops.c | 16 +- drivers/staging/lustre/lustre/osc/osc_io.c | 4 +- drivers/staging/lustre/lustre/ptlrpc/layout.c | 6 +- .../staging/lustre/lustre/ptlrpc/pack_generic.c | 84 ++- .../staging/lustre/lustre/ptlrpc/ptlrpc_internal.h | 7 +- drivers/staging/lustre/lustre/ptlrpc/sec.c | 5 +- drivers/staging/lustre/lustre/ptlrpc/wiretest.c | 125 +++- 32 files changed, 2483 insertions(+), 1373 deletions(-)