From patchwork Wed May 13 14:57:31 2015 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Ilya Dryomov X-Patchwork-Id: 6397471 Return-Path: X-Original-To: patchwork-ceph-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.136]) by patchwork2.web.kernel.org (Postfix) with ESMTP id E36EBBEEE1 for ; Wed, 13 May 2015 14:58:02 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 05A3720377 for ; Wed, 13 May 2015 14:58:02 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 14F06203B0 for ; Wed, 13 May 2015 14:58:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S934521AbbEMO57 (ORCPT ); Wed, 13 May 2015 10:57:59 -0400 Received: from mail-wg0-f49.google.com ([74.125.82.49]:34086 "EHLO mail-wg0-f49.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S934303AbbEMO5z (ORCPT ); Wed, 13 May 2015 10:57:55 -0400 Received: by wgic8 with SMTP id c8so46534814wgi.1 for ; Wed, 13 May 2015 07:57:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=QsuZoIvN9MviNbdM77Cy+Ni+2kOzixjjhAOedoTl4X0=; b=danfcmOAAaUqGWqfotPyNAd9pJfbAIinQbIdu53O9ZgN2HPoMHpvvSnT7UWalvf7hA lR7QQs563M+7NR9lKAHB0Tx5ufKgAXVD8MkzZU4kt2uP7+LX484acJHlu9QjjlmyTL5D ucmWcbOlt5LAKLNSE/Mv5xIQ6QWnr4DDt8bKsc1aVZnR/ogtEsI6DrhHYcVGzc7l4dMS y4ZtVlVuep0EDLi53/e2dtmW81MbUV+D7NdJcw0p5LBIfPX7HVpDNadU1woAYrYdCrM3 O2Mw69S2FyiHNySANP14jrF6Y3GoPePclVdbe2DEe7oU4rYGGKDUPKPLtzTUy1P0VxAL kfTQ== X-Received: by 10.180.101.105 with SMTP id ff9mr10892633wib.64.1431529073902; Wed, 13 May 2015 07:57:53 -0700 (PDT) Received: from localhost.localdomain ([109.110.66.238]) by mx.google.com with ESMTPSA id i13sm8492547wic.13.2015.05.13.07.57.52 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 13 May 2015 07:57:53 -0700 (PDT) From: Ilya Dryomov To: ceph-devel@vger.kernel.org Cc: Alex Elder Subject: [PATCH 1/2] libceph: request a new osdmap if lingering request maps to no osd Date: Wed, 13 May 2015 17:57:31 +0300 Message-Id: <1431529052-13932-2-git-send-email-idryomov@gmail.com> X-Mailer: git-send-email 1.9.3 In-Reply-To: <1431529052-13932-1-git-send-email-idryomov@gmail.com> References: <1431529052-13932-1-git-send-email-idryomov@gmail.com> Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Spam-Status: No, score=-6.8 required=5.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, T_DKIM_INVALID, T_RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP This commit does two things. First, if there are any homeless lingering requests, we now request a new osdmap even if the osdmap that is being processed brought no changes, i.e. if a given lingering request turned homeless in one of the previous epochs and remained homeless in the current epoch. Not doing so leaves us with a stale osdmap and as a result we may miss our window for reestablishing the watch and lose notifies. MON=1 OSD=1: # cat linger-needmap.sh #!/bin/bash rbd create --size 1 test DEV=$(rbd map test) ceph osd out 0 rbd map dne/dne # obtain a new osdmap as a side effect (!) sleep 1 ceph osd in 0 rbd resize --size 2 test # rbd info test | grep size -> 2M # blockdev --getsize $DEV -> 1M N.B.: Not obtaining a new osdmap in between "osd out" and "osd in" above is enough to make it miss that resize notify, but that is a bug^Wlimitation of ceph watch/notify v1. Second, homeless lingering requests are now kicked just like those lingering requests whose mapping has changed. This is mainly to recognize that a homeless lingering request makes no sense and to preserve the invariant that a registered lingering request is not sitting on any of r_req_lru_item lists. This spares us a WARN_ON, which commit ba9d114ec557 ("libceph: clear r_req_lru_item in __unregister_linger_request()") tried to fix the _wrong_ way. Cc: stable@vger.kernel.org # 3.10+ Signed-off-by: Ilya Dryomov Reviewed-by: Sage Weil --- net/ceph/osd_client.c | 31 ++++++++++++++++++++----------- 1 file changed, 20 insertions(+), 11 deletions(-) diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c index 41a4abc7e98e..31d4b1ebff01 100644 --- a/net/ceph/osd_client.c +++ b/net/ceph/osd_client.c @@ -2017,20 +2017,29 @@ static void kick_requests(struct ceph_osd_client *osdc, bool force_resend, err = __map_request(osdc, req, force_resend || force_resend_writes); dout("__map_request returned %d\n", err); - if (err == 0) - continue; /* no change and no osd was specified */ if (err < 0) continue; /* hrm! */ - if (req->r_osd == NULL) { - dout("tid %llu maps to no valid osd\n", req->r_tid); - needmap++; /* request a newer map */ - continue; - } + if (req->r_osd == NULL || err > 0) { + if (req->r_osd == NULL) { + dout("lingering %p tid %llu maps to no osd\n", + req, req->r_tid); + /* + * A homeless lingering request makes + * no sense, as it's job is to keep + * a particular OSD connection open. + * Request a newer map and kick the + * request, knowing that it won't be + * resent until we actually get a map + * that can tell us where to send it. + */ + needmap++; + } - dout("kicking lingering %p tid %llu osd%d\n", req, req->r_tid, - req->r_osd ? req->r_osd->o_osd : -1); - __register_request(osdc, req); - __unregister_linger_request(osdc, req); + dout("kicking lingering %p tid %llu osd%d\n", req, + req->r_tid, req->r_osd ? req->r_osd->o_osd : -1); + __register_request(osdc, req); + __unregister_linger_request(osdc, req); + } } reset_changed_osds(osdc); mutex_unlock(&osdc->request_mutex);