From patchwork Tue Sep 26 13:24:49 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Jeff Layton X-Patchwork-Id: 9972019 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 2EE4D60365 for ; Tue, 26 Sep 2017 13:24:58 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 29FF6205A8 for ; Tue, 26 Sep 2017 13:24:58 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 1CA4A284B5; Tue, 26 Sep 2017 13:24:58 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.4 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_TVD_MIME_EPI autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 79E6C205A8 for ; Tue, 26 Sep 2017 13:24:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S935929AbdIZNYy (ORCPT ); Tue, 26 Sep 2017 09:24:54 -0400 Received: from mail-qk0-f174.google.com ([209.85.220.174]:47400 "EHLO mail-qk0-f174.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934627AbdIZNYx (ORCPT ); Tue, 26 Sep 2017 09:24:53 -0400 Received: by mail-qk0-f174.google.com with SMTP id b82so9993850qkc.4 for ; Tue, 26 Sep 2017 06:24:52 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:date:mime-version; bh=GtMdyMyZWHZfOsJ6kLqrG+Id35RvQ6NHiIOKBxiEpUQ=; b=J4at21znE6Ogf8V66hwsFH2nXsghIz3AKHd77R2KpoiUJID7O1o77qfsRTmX8f+mxo WMWX3aQdPjAmJcRb68MrtEin+1CbajdoWbCTK8tGsbezuxvQfHIrElUoOWQhUsBqFJLM qgPlY+etjvOHvxQppnJFIuGRjWikujAOpTsI2LtIKdsOUehqwTV4d1pNaLrjvosKQO4p m9lIBhQq894aM5pxJe6qvYnlyeZ9smFC9jekC1/PpQaK9/k00NCcp1W9zeTY5NG8+oGA X/GTLoPjSqy+LItS60WQQyxB0/LLcZwgKntXkMTDoc5JgfHIwO4BSLbJQURrMf4QAjRl YpDQ== X-Gm-Message-State: AHPjjUghMtv7onZFjLIFB5jJ90kxeB1rndaFUUTQjG8riPox4Oibf8TS RI/DndV7sQMaSJ5YIUAEFWLZNbU9g0k= X-Google-Smtp-Source: AOwi7QCqP7R5qwy8Hi4yuzxaDURSOFGE0lrhHZSpTpKjXYCXtV/1OR3G38nDyxXYWKDMzt5K3wxmgw== X-Received: by 10.55.27.161 with SMTP id m33mr14151824qkh.225.1506432291770; Tue, 26 Sep 2017 06:24:51 -0700 (PDT) Received: from redhat-78.nfsv4bat.org ([66.187.232.65]) by smtp.gmail.com with ESMTPSA id s16sm6708195qks.63.2017.09.26.06.24.50 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 26 Sep 2017 06:24:51 -0700 (PDT) Message-ID: <1506432289.3182.16.camel@redhat.com> Subject: lingering caps outstanding after client shutdown? From: Jeff Layton To: "open list:CEPH DISTRIBUTED..." Date: Tue, 26 Sep 2017 09:24:49 -0400 X-Mailer: Evolution 3.24.5 (3.24.5-1.fc26) Mime-Version: 1.0 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP In order to get the correct semantics for a delegation or oplock, we need to be able to break delegations on open. Cephfs' open routine is pretty light with the caps though. One way to fix that is to request a enough caps to cause a delegation break if any are outstanding. So, I've been playing with the attached patch. It may be wrong and not what we want, but it seems like it ought to work. This patch however causes the LibCephFS.Fchmod test to hang, waiting on caps, iff the DoubleChmod test is run before it. If you run just Fchmod test, it works fine. They both use the same filename and hence the same inode. The test hangs at the first Fchmod test ceph_open call, waiting for the caps. At that point though, there should be no other clients accessing the mount. Is it possible that outstanding caps granted to the client from the earlier test could be surviving a ceph_shutdown, such that it blocks the next client from getting them? To reproduce I've been running this on a build with the attached patch vs. a vstart cluster: $ ./bin/ceph_test_libcephfs --gtest_filter=LibCephFS.DoubleChmod:LibCephFS.Fchmod I'll continue to dig into this, but I'm starting to wonder if this is exposing a bug in the mds. Should I open a tracker bug on this? From 6f2d562f06529631235bea02b821f83a1a3d0e73 Mon Sep 17 00:00:00 2001 From: Jeffrey Layton Date: Mon, 25 Sep 2017 15:09:48 -0400 Subject: [PATCH] client: request caps on open Signed-off-by: Jeff Layton --- src/client/Client.cc | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/src/client/Client.cc b/src/client/Client.cc index 78517c808a..ffe877fdf8 100644 --- a/src/client/Client.cc +++ b/src/client/Client.cc @@ -8425,11 +8425,22 @@ int Client::_open(Inode *in, int flags, mode_t mode, Fh **fhp, in->get_open_ref(cmode); // make note of pending open, since it effects _wanted_ caps. - if ((flags & O_TRUNC) == 0 && - in->caps_issued_mask(want)) { + if ((flags & O_TRUNC) == 0 && in->caps_issued_mask(want)) { // update wanted? check_caps(in, CHECK_CAPS_NODELAY); } else { + int need = 0, have; + + /* Wait on minimal caps, just to break delegations */ + if (cmode & CEPH_FILE_MODE_WR) + need |= CEPH_CAP_FILE_WR; + if (cmode & CEPH_FILE_MODE_RD) + need |= CEPH_CAP_FILE_RD; + + result = get_caps(in, need, want, &have, -1); + if (result < 0) + return result; + MetaRequest *req = new MetaRequest(CEPH_MDS_OP_OPEN); filepath path; in->make_nosnap_relative_path(path); @@ -8444,6 +8455,7 @@ int Client::_open(Inode *in, int flags, mode_t mode, Fh **fhp, req->head.args.open.old_size = in->size; // for O_TRUNC req->set_inode(in); result = make_request(req, perms); + put_cap_ref(in, need); } // success? -- 2.13.5