From patchwork Fri May 20 17:52:43 2011 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Trond Myklebust X-Patchwork-Id: 804402 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by demeter1.kernel.org (8.14.4/8.14.3) with ESMTP id p4KHqq4T007097 for ; Fri, 20 May 2011 17:52:52 GMT Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1757228Ab1ETRwq (ORCPT ); Fri, 20 May 2011 13:52:46 -0400 Received: from mx2.netapp.com ([216.240.18.37]:7608 "EHLO mx2.netapp.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751049Ab1ETRwo convert rfc822-to-8bit (ORCPT ); Fri, 20 May 2011 13:52:44 -0400 X-IronPort-AV: E=Sophos;i="4.65,243,1304319600"; d="scan'208";a="549605499" Received: from smtp1.corp.netapp.com ([10.57.156.124]) by mx2-out.netapp.com with ESMTP; 20 May 2011 10:52:44 -0700 Received: from sacrsexc2-prd.hq.netapp.com (sacrsexc2-prd.hq.netapp.com [10.99.115.28]) by smtp1.corp.netapp.com (8.13.1/8.13.1/NTAP-1.6) with ESMTP id p4KHqiDe029127; Fri, 20 May 2011 10:52:44 -0700 (PDT) Received: from SACMVEXC2-PRD.hq.netapp.com ([10.99.115.18]) by sacrsexc2-prd.hq.netapp.com with Microsoft SMTPSVC(6.0.3790.3959); Fri, 20 May 2011 10:52:43 -0700 Received: from 10.58.56.180 ([10.58.56.180]) by SACMVEXC2-PRD.hq.netapp.com ([10.99.115.16]) with Microsoft Exchange Server HTTP-DAV ; Fri, 20 May 2011 17:52:43 +0000 Received: from lade.trondhjem.org by SACMVEXC2-PRD.hq.netapp.com; 20 May 2011 13:52:43 -0400 Subject: Re: 2.6.38.6 - state manager constantly respawns From: Trond Myklebust To: "Dr. J. Bruce Fields" Cc: Harry Edmon , Chuck Lever , linux-nfs@vger.kernel.org Date: Fri, 20 May 2011 13:52:43 -0400 In-Reply-To: <20110520172639.GA11670@fieldses.org> References: <05D08339-888C-4A64-BDC5-8667B3901E7A@oracle.com> <4DD1772E.9010609@uw.edu> <6A6FB1C3-D4C3-40BE-810A-B4551FA9E591@oracle.com> <4DD17CB5.7010009@uw.edu> <1305575007.19725.3.camel@lade.trondhjem.org> <4DD17F79.305@uw.edu> <1305575656.19725.9.camel@lade.trondhjem.org> <20110516202059.GC1680@fieldses.org> <20110516205351.GD1680@fieldses.org> <4DD694DF.2060302@uw.edu> <20110520172639.GA11670@fieldses.org> Organization: NetApp Inc X-Mailer: Evolution 3.0.1 (3.0.1-1.fc15) Message-ID: <1305913963.12712.6.camel@lade.trondhjem.org> Mime-Version: 1.0 X-OriginalArrivalTime: 20 May 2011 17:52:43.0731 (UTC) FILETIME=[B865F230:01CC1716] Sender: linux-nfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-nfs@vger.kernel.org X-Greylist: IP, sender and recipient auto-whitelisted, not delayed by milter-greylist-4.2.6 (demeter1.kernel.org [140.211.167.41]); Fri, 20 May 2011 17:52:52 +0000 (UTC) On Fri, 2011-05-20 at 13:26 -0400, Dr. J. Bruce Fields wrote: > On Fri, May 20, 2011 at 09:20:47AM -0700, Harry Edmon wrote: > > On 05/16/11 13:53, Dr. J. Bruce Fields wrote: > > >Hm, so the renews all have clid 465ccc4d09000000, and the reads all have > > >a stateid (0, 465ccc4dc24c0a0000000000). > > > > > >So the first 4 bytes matching just tells me both were handed out by the > > >same server instance (so there was no server reboot in between); there's > > >no way for me to tell whether they really belong to the same client. > > > > > >The server does assume that any stateid from the current server instance > > >that no longer exists in its table is expired. I believe that's > > >correct, given a correctly functioning client, but perhaps I'm missing a > > >case. > > > > > >--b. > > I am very appreciative of the quick initial comments I receive from > > all of you on my NFS problem. I notice that there has been silence > > on the problem since the 16th, so I assume that either this is a > > hard bug to track down or you have been busy with higher priority > > tasks. Is there anything I can do to help develop a solution to > > this problem? > > Well, the only candidate explanation for the problem is that my > assumption--that any time the server gets a stateid from the current > boot instance that it doesn't recognize as an active stateid, it is safe > for the server to return EXPIRED--is wrong. > > I don't immediately see why it's wrong, and based on the silence nobody > else does either, but I'm not 100% convinced I'm right either. > > So one approach might be to add server code that makes a better effort > to return EXPIRED only when we're sure it's a stateid from an expired > client, and see if that solves your problem. > > Remind me, did you have an easy way to reproduce your problem? My silence is simply because I'm mystified as to how this can happen. Patching for it is trivial (see below). When the server tells us that our lease is expired, the normal behaviour for the client is to re-establish the lease, and then proceed to recover all known stateids. I don't see how we can 'miss' a stateid that then needs to be recovered afterwards... Cheers Trond 8<---------------------------------------------------------------------------- From 920ddb153f28717be363f6e87dde24ef2a8d0ce2 Mon Sep 17 00:00:00 2001 From: Trond Myklebust Date: Fri, 20 May 2011 13:44:02 -0400 Subject: [PATCH] NFSv4: Handle expired stateids when the lease is still valid Currently, if the server returns NFS4ERR_EXPIRED in reply to a READ or WRITE, but the RENEW test determines that the lease is still active, we fail to recover and end up looping forever in a READ/WRITE + RENEW death spiral. Signed-off-by: Trond Myklebust --- fs/nfs/nfs4proc.c | 9 +++++++-- 1 files changed, 7 insertions(+), 2 deletions(-) diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index cf1b339..d0e15db 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -267,9 +267,11 @@ static int nfs4_handle_exception(struct nfs_server *server, int errorcode, struc break; nfs4_schedule_stateid_recovery(server, state); goto wait_on_recovery; + case -NFS4ERR_EXPIRED: + if (state != NULL) + nfs4_schedule_stateid_recovery(server, state); case -NFS4ERR_STALE_STATEID: case -NFS4ERR_STALE_CLIENTID: - case -NFS4ERR_EXPIRED: nfs4_schedule_lease_recovery(clp); goto wait_on_recovery; #if defined(CONFIG_NFS_V4_1) @@ -3670,9 +3672,11 @@ nfs4_async_handle_error(struct rpc_task *task, const struct nfs_server *server, break; nfs4_schedule_stateid_recovery(server, state); goto wait_on_recovery; + case -NFS4ERR_EXPIRED: + if (state != NULL) + nfs4_schedule_stateid_recovery(server, state); case -NFS4ERR_STALE_STATEID: case -NFS4ERR_STALE_CLIENTID: - case -NFS4ERR_EXPIRED: nfs4_schedule_lease_recovery(clp); goto wait_on_recovery; #if defined(CONFIG_NFS_V4_1) @@ -4543,6 +4547,7 @@ int nfs4_lock_delegation_recall(struct nfs4_state *state, struct file_lock *fl) case -ESTALE: goto out; case -NFS4ERR_EXPIRED: + nfs4_schedule_stateid_recovery(server, state); case -NFS4ERR_STALE_CLIENTID: case -NFS4ERR_STALE_STATEID: nfs4_schedule_lease_recovery(server->nfs_client);