From patchwork Tue Dec 13 05:29:25 2016 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Zhen Ren X-Patchwork-Id: 9471789 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id E9BDB607F0 for ; Tue, 13 Dec 2016 05:31:51 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id DB10D284DC for ; Tue, 13 Dec 2016 05:31:51 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id C98EC28547; Tue, 13 Dec 2016 05:31:51 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.2 required=2.0 tests=BAYES_00, RCVD_IN_DNSWL_MED autolearn=ham version=3.3.1 Received: from userp1040.oracle.com (userp1040.oracle.com [156.151.31.81]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id 35E63284DC for ; Tue, 13 Dec 2016 05:31:50 +0000 (UTC) Received: from aserv0021.oracle.com (aserv0021.oracle.com [141.146.126.233]) by userp1040.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id uBD5VGtI012831 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 13 Dec 2016 05:31:16 GMT Received: from oss.oracle.com (oss-old-reserved.oracle.com [137.254.22.2]) by aserv0021.oracle.com (8.13.8/8.14.4) with ESMTP id uBD5VGLR009927 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 13 Dec 2016 05:31:16 GMT Received: from localhost ([127.0.0.1] helo=lb-oss.oracle.com) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1cGfgK-0007xV-7j; Mon, 12 Dec 2016 21:31:16 -0800 Received: from aserv0021.oracle.com ([141.146.126.233]) by oss.oracle.com with esmtp (Exim 4.63) (envelope-from ) id 1cGffs-0007qh-CQ; Mon, 12 Dec 2016 21:30:48 -0800 Received: from aserp1060.oracle.com (aserp1060.oracle.com [141.146.126.71]) by aserv0021.oracle.com (8.13.8/8.14.4) with ESMTP id uBD5UmbN009019 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 13 Dec 2016 05:30:48 GMT Received: from userp2040.oracle.com (userp2040.oracle.com [156.151.31.90]) by aserp1060.oracle.com (Sentrion-MTA-4.3.2/Sentrion-MTA-4.3.2) with ESMTP id uBD5UlEm028660 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 13 Dec 2016 05:30:48 GMT Received: from pps.filterd (userp2040.oracle.com [127.0.0.1]) by userp2040.oracle.com (8.16.0.17/8.16.0.17) with SMTP id uBD5S1bU046827; Tue, 13 Dec 2016 05:30:47 GMT Authentication-Results: oracle.com; spf=pass smtp.mailfrom=zren@suse.com Received: from prv3-mh.provo.novell.com (prv3-mh.provo.novell.com [137.65.250.26]) by userp2040.oracle.com with ESMTP id 2788bae312-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 13 Dec 2016 05:30:47 +0000 Received: from laptop.apac.novell.com (prv-ext-foundry1int.gns.novell.com [137.65.251.240]) by prv3-mh.provo.novell.com with ESMTP (TLS encrypted); Mon, 12 Dec 2016 22:30:30 -0700 From: Eric Ren To: ocfs2-test-devel@oss.oracle.com Date: Tue, 13 Dec 2016 13:29:25 +0800 Message-Id: <1481606975-13186-8-git-send-email-zren@suse.com> X-Mailer: git-send-email 2.6.6 In-Reply-To: <1481606975-13186-1-git-send-email-zren@suse.com> References: <1481606975-13186-1-git-send-email-zren@suse.com> X-PDR: PASS X-ServerName: prv3-mh.provo.novell.com X-Proofpoint-SPF-Result: pass X-Proofpoint-SPF-Record: v=spf1 include:microfocus.com ~all X-Proofpoint-Virus-Version: vendor=nai engine=5800 definitions=8377 signatures=670775 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 suspectscore=1 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1609300000 definitions=main-1612130094 Cc: ocfs2-devel@oss.oracle.com Subject: [Ocfs2-devel] [PATCH 07/17] lvb_torture: failed when pcmk is used as cluster stack X-BeenThere: ocfs2-devel@oss.oracle.com X-Mailman-Version: 2.1.9 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , MIME-Version: 1.0 Sender: ocfs2-devel-bounces@oss.oracle.com Errors-To: ocfs2-devel-bounces@oss.oracle.com X-Source-IP: aserv0021.oracle.com [141.146.126.233] X-Virus-Scanned: ClamAV using ClamSMTP This test case failed with "pcmk" stack. Output errors like: "rank 1: /dlm/ has no write permission." "rank 1: o2dlm_initialize failed: -1485330936". Signed-off-by: Eric Ren --- programs/lvb_torture/lvb_torture.c | 110 ++++++++++++++++++++++++++++++++- programs/python_common/multiple_run.sh | 12 +++- 2 files changed, 117 insertions(+), 5 deletions(-) diff --git a/programs/lvb_torture/lvb_torture.c b/programs/lvb_torture/lvb_torture.c index 1459849..93a29ec 100644 --- a/programs/lvb_torture/lvb_torture.c +++ b/programs/lvb_torture/lvb_torture.c @@ -180,6 +180,101 @@ static void run_test(struct o2dlm_ctxt *dlm, char *lockid) } } +/* + * Copied from run_test(), this is a ugly but straightforward workaround. + * "fsdlm" is used when using pcmk as cluster stack, which only supports + * 32-bits lvb so far. + */ +static void run_test_fsdlm(struct o2dlm_ctxt *dlm, char *lockid) +{ + unsigned long long iter = 0; + unsigned long long expected, to_write = 0; + int ret; + unsigned int read, written; + errcode_t err; + enum o2dlm_lock_level level; + __u32 lvb; + + while (iter < max_iter && !caught_sig) { + expected = iter; + + if ((iter % num_procs) == rank) + level = O2DLM_LEVEL_EXMODE; + else + level = O2DLM_LEVEL_PRMODE; + + if (level == O2DLM_LEVEL_PRMODE) { + ret = MPI_Barrier(MPI_COMM_WORLD); + if (ret != MPI_SUCCESS) + rprintf(rank, "read MPI_Barrier failed: %d\n", ret); + err = o2dlm_lock(dlm, lockid, 0, level); + if (err) + rprintf(rank, "o2dlm_lock failed: %d\n", err); + + expected++; + } else { + err = o2dlm_lock(dlm, lockid, 0, level); + if (err) + rprintf(rank, "o2dlm_lock failed: %d\n", err); + + ret = MPI_Barrier(MPI_COMM_WORLD); + if (ret != MPI_SUCCESS) + rprintf(rank, "read MPI_Barrier failed: %d\n", ret); + to_write = iter + 1; + } + + err = o2dlm_read_lvb(dlm, lockid, (char *)&lvb, sizeof(lvb), + &read); + if (err) + rprintf(rank, "o2dlm_read_lvb failed: %d\n", err); + + lvb = be32_to_cpu(lvb); + + if (level == O2DLM_LEVEL_PRMODE) + printf("%s: read iter: %llu, lvb: %llu exp: %llu\n", + hostname, (unsigned long long)iter, + (unsigned long long)lvb, + (unsigned long long)expected); + else + printf("%s: write iter: %llu, lvb: %llu wri: %llu\n", + hostname, (unsigned long long)iter, + (unsigned long long)lvb, + (unsigned long long)to_write); + + fflush(stdout); + + if (lvb != expected) { + printf("Test failed! %s: rank %d, read lvb %llu, expected %llu\n", + hostname, rank, (unsigned long long) lvb, + (unsigned long long) expected); + MPI_Abort(MPI_COMM_WORLD, 1); + } + + if (level == O2DLM_LEVEL_EXMODE) { + lvb = cpu_to_be32(to_write); + + err = o2dlm_write_lvb(dlm, lockid, (char *)&lvb, + sizeof(lvb), &written); + if (err) + rprintf(rank, "o2dlm_write_lvb failed: %d\n", err); + if (written != sizeof(lvb)) + rprintf(rank, "o2dlm_write_lvb() wrote %d, we asked for %d\n", written, sizeof(lvb)); + } + + err = o2dlm_unlock(dlm, lockid); + if (err) + rprintf(rank, "o2dlm_unlock failed: %d\n", err); + + /* This second barrier is not necessary and can be + * commented out to ramp the test up */ + ret = MPI_Barrier(MPI_COMM_WORLD); + if (ret != MPI_SUCCESS) + rprintf(rank, "unlock MPI_Barrier failed: %d\n", ret); + + iter++; + } +} + static void clear_lock(struct o2dlm_ctxt *dlm, char *lockid) { char empty[O2DLM_LOCK_ID_MAX_LEN]; @@ -363,8 +458,7 @@ int main(int argc, char *argv[]) printf("%s: rank: %d, nodes: %d, dlm: %s, dom: %s, lock: %s, iter: %llu\n", hostname, rank, num_procs, dlmfs_path, domain, lockid, (unsigned long long) max_iter); - - if (access(dlmfs_path, W_OK) < 0) { + if (strcmp(dlmfs_path, "NULL") != 0 && access(dlmfs_path, W_OK) < 0) { sleep(2); rprintf(rank, "%s has no write permission.\n", dlmfs_path); return EACCES; @@ -380,6 +474,13 @@ int main(int argc, char *argv[]) rprintf(rank, "start_heartbeat failed\n"); } + /* + * "pcmk" is used as cluster stack if "NULL" + * is passed here + */ + if (strcmp(dlmfs_path, "NULL") == 0) + dlmfs_path = NULL; + error = o2dlm_initialize(dlmfs_path, domain, &dlm); if (error) rprintf(rank, "o2dlm_initialize failed: %d\n", error); @@ -391,7 +492,10 @@ int main(int argc, char *argv[]) if (ret != MPI_SUCCESS) rprintf(rank, "prep MPI_Barrier failed: %d\n", ret); - run_test(dlm, lockid); + if (dlmfs_path) + run_test(dlm, lockid); + else + run_test_fsdlm(dlm, lockid); error = o2dlm_destroy(dlm); if (error) diff --git a/programs/python_common/multiple_run.sh b/programs/python_common/multiple_run.sh index 2e0ec72..4340c40 100755 --- a/programs/python_common/multiple_run.sh +++ b/programs/python_common/multiple_run.sh @@ -31,6 +31,7 @@ DD=`which dd` GREP=`which grep` AWK=`which awk` ECHO="`which echo` -e" +O2CLUSTER="`which o2cluster`" SUDO="`which sudo` -u root" IFCONFIG_BIN="`which ifconfig`" @@ -398,6 +399,7 @@ run_lvb_torture_test() local logfile=${logdir}/lvb_torture_${DATE}.log local workplace=${MOUNT_POINT} local testfile=${workplace}/lvb_torture_test_file + local dlmfs_path="/dlm" ${MKDIR_BIN} -p ${logdir} ${CHMOD_BIN} -R 777 ${logdir} @@ -427,9 +429,15 @@ ${DEVICE} ${FEATURES} ${JOURNALSIZE} ${BLOCKS} local UUID="`${DEBUGFS_BIN} -R stats ${DEVICE} |grep UUID|cut -d: -f 2`" local LOCK="`${DEBUGFS_BIN} -R 'encode lvb_torture_test_file' ${DEVICE}`" - LogMsg "Run lvb_torture, CMD: ${BINDIR}/run_lvb_torture.py -d /dlm/ -i 60000 \ + #dlmfs_path should be "NULL" if cluster stack is "pcmk" + local stack="`${SUDO} ${O2CLUSTER} -r | ${AWK} -F',' '{printf $1}'`" + if [ "$stack" = "pcmk" ];then + dlmfs_path="NULL" + fi + + LogMsg "Run lvb_torture, CMD: ${BINDIR}/run_lvb_torture.py -d ${dlmfs_path} -i 60000 \ -H ${DEVICE} -l ${logfile} -n ${NODE_LIST} "${UUID}" "${LOCK}"" - ${SUDO} ${BINDIR}/run_lvb_torture.py -d /dlm/ -c 60000 -i ${INTERFACE} -l \ + ${SUDO} ${BINDIR}/run_lvb_torture.py -d ${dlmfs_path} -c 60000 -i ${INTERFACE} -l \ ${logfile} -n ${NODE_LIST} "${UUID}" "${LOCK}" >> ${LOGFILE} 2>&1 LogRC $?