From patchwork Thu Jun 1 15:14:22 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Anton Nefedov X-Patchwork-Id: 9759813 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id 6D471602BF for ; Thu, 1 Jun 2017 15:16:25 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 5D7D12849A for ; Thu, 1 Jun 2017 15:16:25 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id 528EC284F8; Thu, 1 Jun 2017 15:16:25 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=2.0 tests=BAD_ENC_HEADER,BAYES_00, DKIM_SIGNED, RCVD_IN_DNSWL_HI, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from lists.gnu.org (lists.gnu.org [208.118.235.17]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.wl.linuxfoundation.org (Postfix) with ESMTPS id BF8512849A for ; Thu, 1 Jun 2017 15:16:24 +0000 (UTC) Received: from localhost ([::1]:45183 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dGRpn-0000DV-Rc for patchwork-qemu-devel@patchwork.kernel.org; Thu, 01 Jun 2017 11:16:23 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:51517) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1dGRoW-00007z-Bb for qemu-devel@nongnu.org; Thu, 01 Jun 2017 11:15:05 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1dGRoV-0004eg-Dw for qemu-devel@nongnu.org; Thu, 01 Jun 2017 11:15:04 -0400 Received: from mail-eopbgr40133.outbound.protection.outlook.com ([40.107.4.133]:34349 helo=EUR03-DB5-obe.outbound.protection.outlook.com) by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1dGRoV-0004d9-2O for qemu-devel@nongnu.org; Thu, 01 Jun 2017 11:15:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=virtuozzo.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=jYg+2OtHP+T2Fn1Wr8t/YsHZ9YWDeNIn50bxQCklhIo=; b=H0NdawuWrENqU3kEhO/AyaVl6q1y0cdEX1HEQWThvhpnv3q0PTUi788REOzT5ch4v/6BmV++PhyNi0iN1d6xFoYqciKrZy9zR4qE4LhOyOkERcV2rEqwX2JBWQGZXIexgz3OTrOSQwQuaKutBgzHJJmAnk8oBaTWlr6N8jrPqn0= Authentication-Results: nongnu.org; dkim=none (message not signed) header.d=none; nongnu.org; dmarc=none action=none header.from=virtuozzo.com; Received: from xantnef-ws.sw.ru (195.214.232.6) by VI1PR0801MB1999.eurprd08.prod.outlook.com (2603:10a6:800:8a::20) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1124.9; Thu, 1 Jun 2017 15:15:01 +0000 From: Anton Nefedov To: Date: Thu, 1 Jun 2017 18:14:22 +0300 Message-ID: <1496330073-51338-5-git-send-email-anton.nefedov@virtuozzo.com> X-Mailer: git-send-email 2.7.4 In-Reply-To: <1496330073-51338-1-git-send-email-anton.nefedov@virtuozzo.com> References: <1496330073-51338-1-git-send-email-anton.nefedov@virtuozzo.com> MIME-Version: 1.0 X-Originating-IP: [195.214.232.6] X-ClientProxiedBy: DB6PR1001CA0005.EURPRD10.PROD.OUTLOOK.COM (2603:10a6:4:b7::15) To VI1PR0801MB1999.eurprd08.prod.outlook.com (2603:10a6:800:8a::20) X-MS-PublicTrafficType: Email X-MS-TrafficTypeDiagnostic: VI1PR0801MB1999: X-MS-Office365-Filtering-Correlation-Id: a3de84ed-3d93-4220-fe04-08d4a900f9bd X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201703131423075)(201703031133081); SRVR:VI1PR0801MB1999; X-Microsoft-Exchange-Diagnostics: 1; VI1PR0801MB1999; 3:BrU+RfuDm0hpUm/RAhYNoiftKmDXYioGW3RrHMXiGvyTC2l7q/1vnjdro0d+1xlBD0d8jMry+v37EpbNdVPhJcZ49HyTQpe2b8GeCYn697k16Pqpcq0NY55R37z5dbiXU733uHfIakmGQbRTbhhC1tBfZoGh1C8bHtkeH8eQrW3haKx+p6+FTuFJgmsO336Yn2qiHk9UHEChuINwcNwCVcUCa1CAtaob3mHrghM4wgk5jmGFXdPaXuwSXKZapfSGhhXTVT6ts96fZgTFPxMls0rTXXIqKHE5VVrTSz+xPOnHRPBEd9aGRWicD4M/XCgi2D4zpX15RAXRd6rXrPcMYQ==; 25:vxJXvhW0r6/djOc+0EQw4Yu1vpf8R1nXh6YnzETOsf9YiLsLGJleLRkvYBE99F5HU1Vb6mDnRMDy/KViLcwXEeIA0Tfz+sc9ZgR5lrXwsNYuJIRrmLjYSFGaoarJJRkcLtZnyjmaIQzWhtYDJG37Kw9I1TRI61thFmPGeLHY27KeF1bspRlazM9Kg0gyRF4zCeAWJ7iUAZn3kgbrwfRpt7yb5P0IJKBae//nHFfnWcsM7q+q8bavnFb3oZwARpB6cefgkpS/oLTA42jz/G/fWzae0WBdWEzHZH6GyPhWCtVkgsNDNVCvZy3Y8e8sEdWsXIRt5/cno8XnDq/soayNh8M8N0djv/3x/oAHvVRUh/uNOIsjVqMmJgiuaokGjwZiE4RjesQdDrdeyd/JdHmToUIFpBt3sM5sOhi6yBNBe7ZrCgwOSabNd0i233QDcrhhr55g/rg2hFKmah80WABnxpOMPGinvUKUmmMlzo0ZYwQ= X-Microsoft-Exchange-Diagnostics: 1; VI1PR0801MB1999; 31:sbVC7iQ+U5M4RseUR9IjNjRISLJUjXIPOxo3CUOcgwtMC8w14nhXSbznpDCIAva8QAbL1at/i9XXukZNbGkNzMGSSR/yACSS9xDXO5ejrIdL0bUFV7YQeEJFwIu6rIYUHmNAGckqDuiCjSKY7wDB3Wjtc+Z4ifWt2CgVTUzHpSnMG/7qSf/GbCvo81MuI3i2WUgBEwi3AupA4RMv0Ag94ruVGSf6g1mmYo1n5LmrutY=; 20:bQH37/FPhbqoh7HYp5F7TMl2uyGAqraxHSTod6pHDzAf5AfP9aL8BpNk2LLE8nwiTr+34MGUSbrTx2C3J8akXMSdAV+qSGRl8YI9hrSJV61XHELRUH+IALtLWYP3Hu6+VzoH8mwDcAGgAEZlmL8Tunk7pxW44kdb3RGNhfmCeFfdoePUx6eIvkj1iI5Y4dZJxCk0XuWFj8k83tj4ZyXFVmGPD5hnNWzLD4ZCu60WMohOzx6JCtR4KLyLMFqSKE4jHZiuKf/zpGbSB0FRGdEelNKXe62n5I/+Q6GADFGN7SIr1jPlSIYCH5nnLf0lRivHscS/2XXbAgiWi4c5n+1Xn4dFDsvon3XtmoeCU+26WKrhRc7CMGrVME/V/n4no+vINZu839xzH6JPttNcsJL0NANquC1Hzb2xOMsmP6se1vA= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:; X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(93006095)(93001095)(10201501046)(100000703101)(100105400095)(3002001)(6041248)(20161123560025)(20161123558100)(20161123555025)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123564025)(6072148)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:VI1PR0801MB1999; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:VI1PR0801MB1999; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; VI1PR0801MB1999; 4:rQm9XwGFl9PGs5BBLqXFXB5IDA4QssQMIbRyIxlX?= =?us-ascii?Q?yh3RAVGn7IU+CvxKRcWo3NAjEQLps8kAccpp3f9163gls83vlwckiTrhws5V?= =?us-ascii?Q?6rS1VWNjRCaSF+IVKOgFUX/dUJRnOhzvlzT+52aAJnsqCsIGq9sJkHlz7/Y9?= =?us-ascii?Q?7qvi3hsNy8I7lnc/M0f1RKnmnmJMx6siN8YWym/FSIXzY0xaaP3MInex1KAo?= =?us-ascii?Q?EDRrw9dNS4qudLjdnJHTHGAI5eImn2RBV41PptVU4WPHBXFK7x1dO8NsPdL1?= =?us-ascii?Q?72f41GWi+6Ckgft+UK6HaNzFxDnE8KA0aRnJe+HkcIdaUlpTLNWb5/Pwakuh?= =?us-ascii?Q?CpELPGEa7C5QXqegN1fRq658JszQYm4vptSL21mXwPbv6UTqMU4sK2eyMyDO?= =?us-ascii?Q?UM8WgogZwNGrmycORRAFCc26ll3tq9MQOu5b9bTCJd2X84x0jhwILM0+ZSwL?= =?us-ascii?Q?4WvK3rYG0nKsYU3Wy3cVC3H614rhc2wkuqgsIuCunX6eZxVImpW+Xx3PTYK8?= =?us-ascii?Q?tRSDcFQt8rV3hRyumCWNgMbq8BLQ2o4hfAdgI9rXPgjH/FA99/mENJHdTQM1?= =?us-ascii?Q?bFlD9DJLmtjoNfhuV7VJVD/p2PZxPZiBDV1zbT1uJUH4KdDMfMHtAP2JbEoT?= =?us-ascii?Q?fJYJ3kgh+rojWT0WjsdBMAOsN18qIfVrHnRrT2zSQT2dbEHVXxOWh8H3sikY?= =?us-ascii?Q?XxqFtYYgVxTKkGwEBHoB9B9NNsYUz1THGgMyjwQNcQ3fpKJFZmEPzzHcQsaQ?= =?us-ascii?Q?NQ3VFs99WQ3b9xAlqJJc+eU4vt9aSVFl54Q/S2lQlsrFURnubQ8NCHJyox1r?= =?us-ascii?Q?euMvJbxDO2AFLC2UxI6QwtUp0jOoy2ALTr+8d4DQ/32p8kTYVW1vVmEZUIds?= =?us-ascii?Q?TzEPoMesT3P28VAja0j6n2eJj71GwXXgl1d2Zl74jIS0PIjihAGSIM3XykOW?= =?us-ascii?Q?zEtg1927r1sG5ofLOCFPMyp230e3gNan3ypfs9f1rH28TIPeo5AdgoG38Ck1?= =?us-ascii?Q?5zNvgJSnPX9EUKWKHVFPY2ag3mUKAps03Gmc+BrDE2IkZFSgkNSwjhZ9SSRw?= =?us-ascii?Q?7+lBbj2Sr/kOMrCEd2xA4v1fpADd7DstDzyhNH3ylYNO24rTJadZtwp3r42b?= =?us-ascii?Q?XByeYdR2zp7kBb2Jx+0n2101FHELiWX8?= X-Forefront-PRVS: 0325F6C77B X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10019020)(4630300001)(6009001)(39830400002)(39450400003)(39400400002)(6916009)(6666003)(6486002)(7736002)(478600001)(5660300001)(305945005)(54906002)(53936002)(4326008)(6512007)(6506006)(2950100002)(110136004)(107886003)(3846002)(50226002)(8676002)(66066001)(42186005)(81166006)(38730400002)(25786009)(6116002)(76176999)(50986999)(47776003)(86362001)(5003940100001)(2906002)(2351001)(36756003)(48376002)(50466002)(33646002)(53416004)(189998001); DIR:OUT; SFP:1102; SCL:1; SRVR:VI1PR0801MB1999; H:xantnef-ws.sw.ru; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; VI1PR0801MB1999; 23:aUUoSTM51oQd30EE1cGe8axqCCsHG5Ll5GggBwz?= =?us-ascii?Q?6nb+RHGm8u6MIo1HSui+n4ZQtJsfkkwUEWSKbISCkyk/yEaUoVE5UVu0vJcp?= =?us-ascii?Q?1+RgsAjE858YRgjszzAqDSdHwi88FMiPbj2/U4cPLDtJoRBSmirKo3YXP6qH?= =?us-ascii?Q?3SPuYz7UNdBnJC8X+r3p2MMGQkwru4idX03YqgFXhy/0i++EEyPUljjJOQad?= =?us-ascii?Q?bRXe0S3J7luHEqb5Y4R9UEtuTiGLB3b4SlUhXhB+Bc0516UV9Esior1KtE74?= =?us-ascii?Q?8WPq44/MNW/SybSkL2fvvpurOTQoUECXUax2B0a791pe7zXP3V2lDYYGa0X5?= =?us-ascii?Q?G1LyN+URBL2xsFr09Tq44dXhSB2iFx+AwjIOZmCMmkR+MRizGjY6DW1dRjlH?= =?us-ascii?Q?qhTXpjhHTVBTqiZ0NwJdcSCs5z0nBLT+sBGqxQwAHWNWyWY06O25hn9fMPZ9?= =?us-ascii?Q?Fv4krOclBGFzAKfYjeff9uA0pRgbk6OdsXt16wXE8iiSESc2GKSv+AciMmmT?= =?us-ascii?Q?SNYcOALmShT4MIg7MZXoAe4negEBBlI37P8kXKooy2p9yI97UQfMlLB47SsX?= =?us-ascii?Q?bwrbNY7u2dl9c1sbGhHY2vlC4rsCESgDtftt2mgi1fyBcbmMjoYmc90mNpMf?= =?us-ascii?Q?9G/DBChfl4FDvCRHJ3AFP3J7cZOOJtVjuPx1plex/2thpb65Hl+YRkEmveDa?= =?us-ascii?Q?OZsdGLHlI8TKv/ejxpSswLbogkHNv2LeyUQ9juPxoo4VjSsYVL6Zr6LRD4ab?= =?us-ascii?Q?EMHZEy5RHNN01hwUf/3otwshOiAAz8lqq8eLo9RaC81WPKXbf2Yd6JR4Z/8g?= =?us-ascii?Q?4mM/GbkATBCTDP+J8YgX18ljcjewTNLxnmWtXtUC7z/HyKbJcUqIWnjX7/a0?= =?us-ascii?Q?yjhg/saTxHjr1UvJYntd0KPWW7EhYkfOTnFdJZoQtAcvlkLrZLPlcGoQ88MO?= =?us-ascii?Q?K5IldQmVsIFwr63Kua1wkSHAytWYcLkTWfznZcZNnLVdw/k6qjV51vqm+VnI?= =?us-ascii?Q?0lzLr2UyDYFPbb5cSmyx9Rp1hTI4Y5aL1KA+81M4lICbtE0/F2nAucsuJgC+?= =?us-ascii?Q?KV+3L0HU=3D?= X-Microsoft-Exchange-Diagnostics: 1; VI1PR0801MB1999; 6:FviGbkbb3vVAkYCgZl4RAag/+Zq7Mw+/vYgrZeFh6mjQXuy21PCNjalHAwsI92PYji6ElR/3OrqfFyBUMG2/qqin81M3CbspakT7Kb7ekgwum7hw8xBF343Fy+psC+mEfNJXQwXWOjmosR//rU8/c7FKX0P4sk/QU+60nAwK990kjFu+sa2f/B1Qwz/3mBQ07qIivX7mXOm/G4wy3bK2vlOp/Qm1EPgxd6vgpRCTxWS1z0lEyvg2uNqP+4ISw8kktLJrY35ln0KN7e4ApcGhXrlA/DNokGWDuh/VZCORPxtRTgVyVPWSgqjryqCmdwAY2p6/kOdbGmPGUmKDOEWhlIuDEtw2ioma/HYNP+yL4ShcYSqFolWIjLb6B4Y53ycdsoH2qR0ZPz5g+7kusyBem7a+WyP9XZ3l6eR36J/ATuPbxx0pEK0buGCE9OEJANbfayb1Xb1qzZ1h/LgBy6DnM3WMgCAtGFRivLzXvIq8uqDJE3Hpny4h15eWiRCITYWz/bLFCnEt789YoS7OHkoOiw== X-Microsoft-Exchange-Diagnostics: 1; VI1PR0801MB1999; 5:D/pqRSGwYoy23cciLhypPNlEdVfo+FpaIA/e/kBQFbHxZwmnBmhG3mAY6Wh0Hflp6RGku3byNLQo/2mqteWy0UaSh0F2X6PawyvwGZ4NAKUrJMnnmp7IDlsn3Iw3gxDuTy1Zr5Yrq+/h8jxtj9C9kPQ1GyAs23OG01LoM3IlKKpSms0QOs2W+Clz+cZOwlwFf/QhVpL/tO6C+Fw0SW2hNLtbgxsEByPATY6IzewYViZ0HACyoR+qOKuP2G4x4M1vg8Vy9hKSXBt4YHOCbyMJYBM5lmxmshZM7JC2V5I0a9Vp8TaXO0BE+pS69LEXdnlj+pPmaVoi7DbxlY32w1qkYs5siIq6bFUjgGNngEARv4Q6VnatrAGNeu4DqY1OpaFLvfZ8SqWifAOUasZh0pMZif06ELCG0RkwU1cTL6prDrmgPPXGwiOYkq1hh1YOmPSV9ksPMXipCvSMpK0m3yzAPKhiHj5+69zYthG8tKGkq2HDzCgWYkqahAM6TbATmg5l; 24:dq1Gza1AApphoa1SXqGLWYR49VYrQDNZ5rfh698n7vdkbhhWo9ht7AUJn1vlhivVh26An27uFMlgpPrKUVn6/7BxSf/NpS/ONmrltT7lD8I= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; VI1PR0801MB1999; 7:KIju5Mrw8YflIDJwfQr0PwkOoqdUqE6WaTK51AiOxa4jdAyqybV00vnjK9P7As3XvOJl3nX4iS3Cvvr6On0byBqRs5d+qfRiJQIaD0j5r5/7vb6TCCxdJlX5q8ucXZW7URZC9/Pz59QPCnKwUXhsUq0hpB84XEggklbtrjTzRXQygdaypydgMFLnvlaVQVKWzCX1H8js0u2zk1fOHBqTU90OJkZPPUjJUacTGFkWRHsrSpjwrqSb0G75H5urgN51+xl28wYGye0hwVYKJG2kdGl/hW/QK3ATkmzPvKmFvLr6Wa0HpxtRKX3QploDpVaEJlBXjrL9v6iSxZL0LGW6dg==; 20:93eL0jMdh5SHJVB7K00xgSq5icFCG5YtM4h2irLzZM5VMbyHkxijoIAxQEaC5KD7YxoYbd7SoZ51ZDBq/EqT26BcpA5pdMVokYzJHzCl1bUbR+V3ESZeY/k3dZzynpMfWbOaWPb35llSY3HythrubwEPQYjyHN3/iVfjbxlP1eY= X-OriginatorOrg: virtuozzo.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 01 Jun 2017 15:15:01.2404 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0801MB1999 X-detected-operating-system: by eggs.gnu.org: Windows 7 or 8 [fuzzy] X-Received-From: 40.107.4.133 Subject: [Qemu-devel] [PATCH v2 04/15] qcow2: alloc space for COW in one chunk X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: kwolf@redhat.com, Anton Nefedov , den@virtuozzo.com, mreitz@redhat.com, "Denis V. Lunev" Errors-To: qemu-devel-bounces+patchwork-qemu-devel=patchwork.kernel.org@nongnu.org Sender: "Qemu-devel" X-Virus-Scanned: ClamAV using ClamSMTP From: "Denis V. Lunev" Currently each single write operation can result in 3 write operations if guest offsets are not cluster aligned. One write is performed for the real payload and two for COW-ed areas. Thus the data possibly lays non-contiguously on the host filesystem. This will reduce further sequential read performance significantly. The patch allocates the space in the file with cluster granularity, ensuring 1. better host offset locality 2. less space allocation operations (which can be expensive on distributed storage) Signed-off-by: Denis V. Lunev Signed-off-by: Anton Nefedov --- block/qcow2.c | 24 +++++++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/block/qcow2.c b/block/qcow2.c index b3ba5da..cd5efba 100644 --- a/block/qcow2.c +++ b/block/qcow2.c @@ -1575,6 +1575,24 @@ fail: return ret; } +static void handle_alloc_space(BlockDriverState *bs, QCowL2Meta *l2meta) +{ + BDRVQcow2State *s = bs->opaque; + QCowL2Meta *m; + + for (m = l2meta; m != NULL; m = m->next) { + uint64_t bytes = m->nb_clusters << s->cluster_bits; + + if (m->cow_start.nb_bytes == 0 && m->cow_end.nb_bytes == 0) { + continue; + } + + /* try to alloc host space in one chunk for better locality */ + bdrv_co_pwrite_zeroes(bs->file, m->alloc_offset, bytes, + BDRV_REQ_ALLOCATE); + } +} + static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t offset, uint64_t bytes, QEMUIOVector *qiov, int flags) @@ -1656,8 +1674,12 @@ static coroutine_fn int qcow2_co_pwritev(BlockDriverState *bs, uint64_t offset, if (ret < 0) { goto fail; } - qemu_co_mutex_unlock(&s->lock); + + if (bs->file->bs->supported_zero_flags & BDRV_REQ_ALLOCATE) { + handle_alloc_space(bs, l2meta); + } + BLKDBG_EVENT(bs->file, BLKDBG_WRITE_AIO); trace_qcow2_writev_data(qemu_coroutine_self(), cluster_offset + offset_in_cluster);