From patchwork Wed Jun 18 15:30:20 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Patchwork-Submitter: Loic Dachary X-Patchwork-Id: 4377541 Return-Path: X-Original-To: patchwork-ceph-devel@patchwork.kernel.org Delivered-To: patchwork-parsemail@patchwork2.web.kernel.org Received: from mail.kernel.org (mail.kernel.org [198.145.19.201]) by patchwork2.web.kernel.org (Postfix) with ESMTP id ACBA6BEEAA for ; Wed, 18 Jun 2014 15:30:47 +0000 (UTC) Received: from mail.kernel.org (localhost [127.0.0.1]) by mail.kernel.org (Postfix) with ESMTP id ACCD32034F for ; Wed, 18 Jun 2014 15:30:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 9F8E9202FE for ; Wed, 18 Jun 2014 15:30:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753303AbaFRPan (ORCPT ); Wed, 18 Jun 2014 11:30:43 -0400 Received: from smtp.dmail.dachary.org ([91.121.254.229]:48420 "EHLO smtp.dmail.dachary.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751942AbaFRPal (ORCPT ); Wed, 18 Jun 2014 11:30:41 -0400 Received: by smtp.dmail.dachary.org (Postfix, from userid 65534) id 9001D42B48; Wed, 18 Jun 2014 17:30:38 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on mail.kernel.org X-Spam-Level: X-Spam-Status: No, score=-6.9 required=5.0 tests=BAYES_00, RCVD_IN_DNSWL_HI, T_RP_MATCHES_RCVD, T_TVD_MIME_EPI, UNPARSEABLE_RELAY autolearn=ham version=3.3.1 Received: from [10.9.0.6] (unknown [10.0.2.28]) by smtp.dmail.dachary.org (Postfix) with ESMTP id E40694209E; Wed, 18 Jun 2014 17:30:23 +0200 (CEST) Message-ID: <53A1B08C.10800@dachary.org> Date: Wed, 18 Jun 2014 17:30:20 +0200 From: Loic Dachary Organization: Artisan Logiciel Libre User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101 Thunderbird/24.4.0 MIME-Version: 1.0 To: =?UTF-8?B?TGx1w61zIFDDoG1pZXMgaSBKdcOhcmV6?= , Michael Nelson CC: Ceph Development Subject: Re: [ceph-users] EC pool errors with some k/m combinations References: <5337FFD7.4080102@dachary.org> In-Reply-To: X-Enigmail-Version: 1.5.2 Sender: ceph-devel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: ceph-devel@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Hi Lluís, Here is a minimal test that verifies your patch works (and will prevent regression). It fails without your patch. Feel free to cherry-pick it in your pull request. erasure-code: verify that rados put enforces alignment http://tracker.ceph.com/issues/8622 refs: #8622 Signed-off-by: Loic Dachary --- src/test/erasure-code/test-erasure-code.sh | 11 +++++++++++ 1 file changed, 11 insertions(+) Cheers On 17/06/2014 20:23, Lluís Pàmies i Juárez wrote: > Michael, Loic, > > I'm having similar issues with certain (k,m) combinations when storing > objects larger than 4MB. Since this thread is relatively old I was > wondering if anyone had fixed it or at least has identified this bug. > Any news? > > Lluis > > On Mon, Mar 31, 2014 at 5:08 PM, Michael Nelson wrote: >> >> >> On Mon, 31 Mar 2014, Michael Nelson wrote: >> >>> Hi Loic, >>> >>> On Sun, 30 Mar 2014, Loic Dachary wrote: >>> >>>> Hi Michael, >>>> >>>> I'm trying to reproduce the problem from sources (today's instead of >>>> yesterday's but there is no difference that could explain the behaviour you >>>> have): >>>> >>>> cd src >>>> rm -fr /tmp/dev /tmp/out ; mkdir -p /tmp/dev ; CEPH_DIR=/tmp LC_ALL=C >>>> MON=1 OSD=6 bash -x ./vstart.sh -d -n -X -l mon osd >>>> ceph osd erasure-code-profile set profile33 ruleset-failure-domain=osd >>>> k=3 m=3 >>>> ceph osd crush rule create-erasure ecruleset33 profile33 >>>> ceph osd pool create testec-33 20 20 erasure profile33 ecruleset33 >>>> ./rados --pool testec-33 put SOMETHING /etc/group >>>> >>>> but it succeeds. Could you please script a minimal set of commands I >>>> could run to repeat the problem you're seeing ? >>> >>> >>> The file that I put into the pool was 15MB in size and the error occurred >>> after the first 4MB chunk. That might be the difference. >>> >>> I will try to come up with a list of commands in case that doesn't trigger >>> it for you. >> >> >> Here is a concise version of what I am deploying with (limited to 6 OSDs in >> this example). >> >> ceph-deploy new c1 c2 c3 >> pdsh -wc[1-3] yum -y install ceph >> ceph-deploy mon create c1 c2 c3 >> sleep 30 >> >> cd /etc/ceph >> ceph-deploy gatherkeys c1 c2 c3 >> >> machines=(c1 c2 c3) >> >> for machine in ${machines[@]}; do >> for n in {1..2}; do >> ssh root@$machine 'rm -rf /data$n/osd$n && mkdir -p >> /data$n/osd$n' >> ssh root@$machine 'rm -rf /journals/journal-osd$n' >> ceph-deploy osd prepare >> $machine:/data$n/osd$n:/journals/journal-osd$n >> ceph-deploy osd activate >> $machine:/data$n/osd$n:/journals/journal-osd$n >> done >> done >> >> # Wait for OSDs to settle >> sleep 60 >> >> >> ceph osd erasure-code-profile set profile33 ruleset-failure-domain=osd k=3 >> m=3 >> ceph osd crush rule create-erasure ecruleset33 profile33 >> ceph osd pool create ectest-33 20 20 erasure profile33 ecruleset33 >> >> root@c1:~/ceph# rados -p ectest-33 put xyz sampledata >> error putting ectest-33/xyz: Operation not supported >> >> root@c1:~/ceph# du -h sampledata >> 15M sampledata >> >> >> _______________________________________________ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > diff --git a/src/test/erasure-code/test-erasure-code.sh b/src/test/erasure-code/test-erasure-code.sh index b779f68..ba0d43a 100755 --- a/src/test/erasure-code/test-erasure-code.sh +++ b/src/test/erasure-code/test-erasure-code.sh @@ -60,6 +60,17 @@ function TEST_rados_put() { diff $dir/ORIGINAL $dir/COPY || return 1 rm $dir/ORIGINAL $dir/COPY + + # + # Verify that the rados command enforces alignment constraints + # imposed by the stripe width + # See http://tracker.ceph.com/issues/8622 + # + local stripe_width=$(./ceph-conf --show-config-value osd_pool_erasure_code_stripe_width) + local block_size=$((stripe_width - 1)) + dd if=/dev/zero of=$dir/ORIGINAL bs=$block_size count=2 + ./rados --block-size=$block_size \ + --pool ecpool put UNALIGNED $dir/ORIGINAL || return 1 } main test-erasure-code