From patchwork Thu Sep 28 14:33:40 2017 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: Timofey Titovets X-Patchwork-Id: 9976255 Return-Path: Received: from mail.wl.linuxfoundation.org (pdx-wl-mail.web.codeaurora.org [172.30.200.125]) by pdx-korg-patchwork.web.codeaurora.org (Postfix) with ESMTP id BFFEB60568 for ; Thu, 28 Sep 2017 14:34:07 +0000 (UTC) Received: from mail.wl.linuxfoundation.org (localhost [127.0.0.1]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id B2B9F2951A for ; Thu, 28 Sep 2017 14:34:07 +0000 (UTC) Received: by mail.wl.linuxfoundation.org (Postfix, from userid 486) id A743C2957C; Thu, 28 Sep 2017 14:34:07 +0000 (UTC) X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on pdx-wl-mail.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.3 required=2.0 tests=BAYES_00, DKIM_ADSP_CUSTOM_MED, DKIM_SIGNED, FREEMAIL_FROM, RCVD_IN_DNSWL_HI, RCVD_IN_SORBS_SPAM, T_DKIM_INVALID autolearn=ham version=3.3.1 Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.wl.linuxfoundation.org (Postfix) with ESMTP id 51D3C2951A for ; Thu, 28 Sep 2017 14:34:07 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932125AbdI1OeG (ORCPT ); Thu, 28 Sep 2017 10:34:06 -0400 Received: from mail-wr0-f193.google.com ([209.85.128.193]:35125 "EHLO mail-wr0-f193.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753344AbdI1Odz (ORCPT ); Thu, 28 Sep 2017 10:33:55 -0400 Received: by mail-wr0-f193.google.com with SMTP id n64so2244480wrb.2 for ; Thu, 28 Sep 2017 07:33:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references; bh=JmfAKlTIaEQU61/30KQiD2mMTVlkjiyuC8NqxWcdTKo=; b=C67dke9DQN0XydR2aPBqYp2PEPJF/7MAqsWiNV59eMC+YCmpc0URTl4mbweC6Up1VQ +Dg9qqlLo9CNtW8q74+WDn/ZEtEWkMHCfT3FoTFWslR5Exffyw6KLMfsLWJ+p+V/+3oM w5UYzsWVNKg/ehtlgqM1wCqLqCV5aU4AVPYlqU2MhORFCFZPf0I/MvNJ9/lCPEOrdl3P W+dTW9p0goumVpn+7SO2Qn+BEXs1IpdnOqhvt9uA2nUW1nJro19Xq2h96MU19VprmvXT H+s9uWpl6wtJTzvmRu0nIhNeJWEJ3aB8oEzdYbe2JPRNBAVuTtbcdgNZ4k8yWSsjBuNs E39g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references; bh=JmfAKlTIaEQU61/30KQiD2mMTVlkjiyuC8NqxWcdTKo=; b=c17q1ZwiNBhHlcyYDZTw3/NZBhg53++kryw53mB/wxCvzdFlm8spMtkwoEm8g5Xnfs MenmOzqezdS4jsIOspVno0ZLIrHuaPTDfUSIOAnwuc5F8JT4gxNnXVhdojAiuYsPWnu3 1zcU73cEu2EpG7YKDWNcqlHKy8qBZ6ZXe+xqdXPrph2pXcrZ3pPCKrhDW8FPWEe8Czko Qg3c40mRpDJDgZUWGWo0b7PfKZduQiDSvMc8RKZeSuBuQXeS7uojZHcCNvvx94JHZPbJ efvtvGW13KOwcJ6dHQiZ1szkmz+PJEr9WTo0hxrZ18I5iwjmAKvHxbGdGqAEAPUnXS0L Dd3w== X-Gm-Message-State: AHPjjUgMv+QNYhGKNbbz201G6bfEbGV2O21hIE+KrFB/C0vHXn6tIhTK nHK2B2i4hvfFYaBRBBuR2bx78A== X-Google-Smtp-Source: AOwi7QBeAPCHbDLAo6cQ7n+5Yu6XucTNUMsi03pv9GC+kaTjNJOKDseTnOo18E6xjCUzIcKacm0S5g== X-Received: by 10.223.144.71 with SMTP id h65mr4990261wrh.41.1506609233662; Thu, 28 Sep 2017 07:33:53 -0700 (PDT) Received: from titovetst-l.itransition.corp ([93.171.6.182]) by smtp.gmail.com with ESMTPSA id k82sm460633wmf.19.2017.09.28.07.33.52 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 28 Sep 2017 07:33:53 -0700 (PDT) From: Timofey Titovets To: linux-btrfs@vger.kernel.org Cc: Timofey Titovets Subject: [PATCH v8 5/6] Btrfs: heuristic add byte set calculation Date: Thu, 28 Sep 2017 17:33:40 +0300 Message-Id: <20170928143341.24491-6-nefelim4ag@gmail.com> X-Mailer: git-send-email 2.14.2 In-Reply-To: <20170928143341.24491-1-nefelim4ag@gmail.com> References: <20170928143341.24491-1-nefelim4ag@gmail.com> Sender: linux-btrfs-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-btrfs@vger.kernel.org X-Virus-Scanned: ClamAV using ClamSMTP Calculate byte set size for data sample: Calculate how many unique bytes has been in sample By count all bytes in bucket with count > 0 If byte set low (~25%), data are easily compressible Otherwise need additional analize Signed-off-by: Timofey Titovets --- fs/btrfs/compression.c | 48 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) -- 2.14.2 -- To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in the body of a message to majordomo@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c index 1cb4df023d5e..83daf2f9d051 100644 --- a/fs/btrfs/compression.c +++ b/fs/btrfs/compression.c @@ -1201,6 +1201,48 @@ int btrfs_decompress_buf2page(const char *buf, unsigned long buf_start, } +/* + * Count byte types in bucket + * That heuristic can detect text like data (configs, xml, json, html & etc) + * Because in most text like data byte set are restricted to limit number + * of possible characters, and that restriction in most cases + * make data easy compressible. + * + * @BYTE_SET_THRESHOLD - assume that all data with that byte set size: + * less - compressible + * more - need additional analize + */ + +#define BYTE_SET_THRESHOLD 64 + +static u32 byte_set_size(const struct heuristic_ws *ws) +{ + u32 i; + u32 byte_set_size = 0; + + for (i = 0; i < BYTE_SET_THRESHOLD; i++) { + if (ws->bucket[i].count > 0) + byte_set_size++; + } + + /* + * Continue collecting count of byte types in bucket + * If byte set size bigger then threshold + * That useless to continue, because for that data type + * detection technique fail + */ + for (; i < BUCKET_SIZE; i++) { + if (ws->bucket[i].count > 0) { + byte_set_size++; + if (byte_set_size > BYTE_SET_THRESHOLD) + return byte_set_size; + } + } + + return byte_set_size; +} + + static bool sample_repeated_patterns(struct heuristic_ws *ws) { u32 half_of_sample = ws->sample_size / 2; @@ -1300,6 +1342,12 @@ int btrfs_compress_heuristic(struct inode *inode, u64 start, u64 end) ws->bucket[byte].count++; } + i = byte_set_size(ws); + if (i < BYTE_SET_THRESHOLD) { + ret = 2; + goto out; + } + out: __free_workspace(0, ws_list, true); return ret;