From patchwork Tue Mar 25 17:43:57 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Alex Elder X-Patchwork-Id: 3889301 Return-Path: X-Original-To: patchwork-ceph-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork1.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork1.web.kernel.org (Postfix) with ESMTP id 24AFF9F334 for ; Tue, 25 Mar 2014 17:44:08 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id 474E720179 for ; Tue, 25 Mar 2014 17:44:07 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 595CB2017B for ; Tue, 25 Mar 2014 17:44:06 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756009AbaCYRnl (ORCPT ); Tue, 25 Mar 2014 13:43:41 -0400 Received: from mail-ve0-f170.google.com ([209.85.128.170]:51813 "EHLO mail-ve0-f170.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1756005AbaCYRni (ORCPT ); Tue, 25 Mar 2014 13:43:38 -0400 Received: by mail-ve0-f170.google.com with SMTP id pa12so985617veb.1 for ; Tue, 25 Mar 2014 10:43:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-type :content-transfer-encoding; bh=uCH/zlYVfRVG/amXcZGy+bnjMLlWsGZFMe5vzB5esqE=; b=M92rWKixTTkywNTEgXbezWYt3ptK07+Y6uLhswPkGAInCktiIe01uHu8SjQo/0w5cA rDzXArEYIYJI6P4d1yrwwD2rMaqsNfsq7VyBPBRaiXdoY8dkkk9CNbCMRQJufNGKZBPW KZIDGCPYnoZag1aAsN043DSJYI3aKJEH3019a8qn8GBnyE+sCZBujL5ov7ZCeVwhAbDm SDGEDoBeVr6ZPmGcRPjzfEBktKBC89TEzGobYhks8B5l0qOG8Xw++gIVvM6H06FUR6Im joqdKg9CI3BXQvOKcjEAgYzouaIUMKdj2Isz8//4hf8+nZN3g2DxzU/zVmYM12GlSFg2 h9ag== X-Gm-Message-State: ALoCoQk0DUoe8t9/teyjkL+chjqKadHoAYL7QtWDfvIZg0C6lp4shKz77n8nnPFftsA0rxd22TA3 X-Received: by 10.58.31.136 with SMTP id a8mr11923736vei.20.1395769418068; Tue, 25 Mar 2014 10:43:38 -0700 (PDT) Received: from [172.22.22.4] (c-71-195-31-37.hsd1.mn.comcast.net. [71.195.31.37]) by mx.google.com with ESMTPSA id am5sm33480389vdc.15.2014.03.25.10.43.37 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 25 Mar 2014 10:43:37 -0700 (PDT) Message-ID: <5331C05D.1060008@ieee.org> Date: Tue, 25 Mar 2014 12:43:57 -0500 From: Alex Elder User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.3.0 MIME-Version: 1.0 To: Olivier Bonvalet CC: Ilya Dryomov , Ceph Development Subject: Re: Issue #5876 : assertion failure in rbd_img_obj_callback() References: <1395736765.2823.29.camel@localhost> <53316D18.7040103@ieee.org> <53317BC2.9010700@ieee.org> <1395753516.2823.37.camel@localhost> <533184AF.9050101@ieee.org> <5331853D.40408@ieee.org> <1395767705.9967.5.camel@localhost> In-Reply-To: <1395767705.9967.5.camel@localhost> Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Spam-Status: No, score=-7.3 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, RP_MATCHES_RCVD, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Please try applying this, on top of the previous patch. If you can then reproduce the problem we'll have a bunch of new information about the particular request that's leading to the failure. That might tell us what more we can do to find the root cause. Thank you. -Alex PS I hope my mailer doesn't botch the long lines. It might. --- drivers/block/rbd.c | 25 ++++++++++++++++++++++++- 1 file changed, 24 insertions(+), 1 deletion(-) -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Index: b/drivers/block/rbd.c =================================================================== --- a/drivers/block/rbd.c +++ b/drivers/block/rbd.c @@ -2132,7 +2132,30 @@ static void rbd_img_obj_callback(struct spin_lock_irq(&img_request->completion_lock); if (which > img_request->next_completion) goto out; - rbd_assert(which == img_request->next_completion); + if (which != img_request->next_completion) { + printk("%s: bad image object request information:\n", __func__); + printk("obj_request %p\n", obj_request); + printk(" ->object_name <%s>\n", obj_request->object_name); + printk(" ->offset %llu\n", obj_request->offset); + printk(" ->length %llu\n", obj_request->length); + printk(" ->type 0x%x\n", (u32)obj_request->type); + printk(" ->flags 0x%lx\n", obj_request->flags); + printk(" ->which %u\n", obj_request->which); + printk(" ->xferred %llu\n", obj_request->xferred); + printk(" ->result %d\n", obj_request->result); + + printk("img_request %p\n", img_request); + printk(" ->snap 0x%016llx\n", img_request->snap_id); + printk(" ->offset %llu\n", img_request->offset); + printk(" ->length %llu\n", img_request->length); + printk(" ->flags 0x%lx\n", img_request->flags); + printk(" ->obj_request_count %u\n", + img_request->obj_request_count); + printk(" ->next_completion %u\n", + img_request->next_completion); + printk(" ->xferred %llu\n", img_request->xferred); + printk(" ->result %d\n", img_request->result); + } for_each_obj_request_from(img_request, obj_request) { rbd_assert(more);