From: Patrick Steinhardt <ps@pks.im>
To: git@vger.kernel.org
Cc: Karthik Nayak <karthik.188@gmail.com>,
Justin Tobler <jltobler@gmail.com>
Subject: [PATCH v2 06/11] refs/reftable: allow configuring block size
Date: Fri, 10 May 2024 12:29:45 +0200 [thread overview]
Message-ID: <be5bdc6dc13510890dbc9ed06e2de6dd866c29e5.1715336798.git.ps@pks.im> (raw)
In-Reply-To: <cover.1715336797.git.ps@pks.im>
[-- Attachment #1: Type: text/plain, Size: 6231 bytes --]
Add a new option `reftable.blockSize` that allows the user to control
the block size used by the reftable library.
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
Documentation/config.txt | 2 +
Documentation/config/reftable.txt | 14 ++++++
refs/reftable-backend.c | 31 ++++++++++++-
t/t0613-reftable-write-options.sh | 72 +++++++++++++++++++++++++++++++
4 files changed, 118 insertions(+), 1 deletion(-)
create mode 100644 Documentation/config/reftable.txt
diff --git a/Documentation/config.txt b/Documentation/config.txt
index 70b448b132..fa1469e5e7 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -497,6 +497,8 @@ include::config/rebase.txt[]
include::config/receive.txt[]
+include::config/reftable.txt[]
+
include::config/remote.txt[]
include::config/remotes.txt[]
diff --git a/Documentation/config/reftable.txt b/Documentation/config/reftable.txt
new file mode 100644
index 0000000000..fa7c4be014
--- /dev/null
+++ b/Documentation/config/reftable.txt
@@ -0,0 +1,14 @@
+reftable.blockSize::
+ The size in bytes used by the reftable backend when writing blocks.
+ The block size is determined by the writer, and does not have to be a
+ power of 2. The block size must be larger than the longest reference
+ name or log entry used in the repository, as references cannot span
+ blocks.
++
+Powers of two that are friendly to the virtual memory system or
+filesystem (such as 4kB or 8kB) are recommended. Larger sizes (64kB) can
+yield better compression, with a possible increased cost incurred by
+readers during access.
++
+The largest block size is `16777215` bytes (15.99 MiB). The default value is
+`4096` bytes (4kB). A value of `0` will use the default value.
diff --git a/refs/reftable-backend.c b/refs/reftable-backend.c
index 1cda48c504..bd9999cefc 100644
--- a/refs/reftable-backend.c
+++ b/refs/reftable-backend.c
@@ -1,6 +1,7 @@
#include "../git-compat-util.h"
#include "../abspath.h"
#include "../chdir-notify.h"
+#include "../config.h"
#include "../environment.h"
#include "../gettext.h"
#include "../hash.h"
@@ -230,6 +231,22 @@ static int read_ref_without_reload(struct reftable_stack *stack,
return ret;
}
+static int reftable_be_config(const char *var, const char *value,
+ const struct config_context *ctx,
+ void *_opts)
+{
+ struct reftable_write_options *opts = _opts;
+
+ if (!strcmp(var, "reftable.blocksize")) {
+ unsigned long block_size = git_config_ulong(var, value, ctx->kvi);
+ if (block_size > 16777215)
+ die("reftable block size cannot exceed 16MB");
+ opts->block_size = block_size;
+ }
+
+ return 0;
+}
+
static struct ref_store *reftable_be_init(struct repository *repo,
const char *gitdir,
unsigned int store_flags)
@@ -245,12 +262,24 @@ static struct ref_store *reftable_be_init(struct repository *repo,
base_ref_store_init(&refs->base, repo, gitdir, &refs_be_reftable);
strmap_init(&refs->worktree_stacks);
refs->store_flags = store_flags;
- refs->write_options.block_size = 4096;
+
refs->write_options.hash_id = repo->hash_algo->format_id;
refs->write_options.default_permissions = calc_shared_perm(0666 & ~mask);
refs->write_options.disable_auto_compact =
!git_env_bool("GIT_TEST_REFTABLE_AUTOCOMPACTION", 1);
+ git_config(reftable_be_config, &refs->write_options);
+
+ /*
+ * It is somewhat unfortunate that we have to mirror the default block
+ * size of the reftable library here. But given that the write options
+ * wouldn't be updated by the library here, and given that we require
+ * the proper block size to trim reflog message so that they fit, we
+ * must set up a proper value here.
+ */
+ if (!refs->write_options.block_size)
+ refs->write_options.block_size = 4096;
+
/*
* Set up the main reftable stack that is hosted in GIT_COMMON_DIR.
* This stack contains both the shared and the main worktree refs.
diff --git a/t/t0613-reftable-write-options.sh b/t/t0613-reftable-write-options.sh
index 462980c37c..8bdbc6ec70 100755
--- a/t/t0613-reftable-write-options.sh
+++ b/t/t0613-reftable-write-options.sh
@@ -99,4 +99,76 @@ test_expect_success 'many refs results in multiple blocks' '
)
'
+test_expect_success 'tiny block size leads to error' '
+ test_when_finished "rm -rf repo" &&
+ git init repo &&
+ (
+ cd repo &&
+ test_commit initial &&
+ cat >expect <<-EOF &&
+ error: unable to compact stack: entry too large
+ EOF
+ test_must_fail git -c reftable.blockSize=50 pack-refs 2>err &&
+ test_cmp expect err
+ )
+'
+
+test_expect_success 'small block size leads to multiple ref blocks' '
+ test_config_global core.logAllRefUpdates false &&
+ test_when_finished "rm -rf repo" &&
+ git init repo &&
+ (
+ cd repo &&
+ test_commit A &&
+ test_commit B &&
+ git -c reftable.blockSize=100 pack-refs &&
+
+ cat >expect <<-EOF &&
+ header:
+ block_size: 100
+ ref:
+ - length: 53
+ restarts: 1
+ - length: 74
+ restarts: 1
+ - length: 38
+ restarts: 1
+ EOF
+ test-tool dump-reftable -b .git/reftable/*.ref >actual &&
+ test_cmp expect actual
+ )
+'
+
+test_expect_success 'small block size fails with large reflog message' '
+ test_when_finished "rm -rf repo" &&
+ git init repo &&
+ (
+ cd repo &&
+ test_commit A &&
+ perl -e "print \"a\" x 500" >logmsg &&
+ cat >expect <<-EOF &&
+ fatal: update_ref failed for ref ${SQ}refs/heads/logme${SQ}: reftable: transaction failure: entry too large
+ EOF
+ test_must_fail git -c reftable.blockSize=100 \
+ update-ref -m "$(cat logmsg)" refs/heads/logme HEAD 2>err &&
+ test_cmp expect err
+ )
+'
+
+test_expect_success 'block size exceeding maximum supported size' '
+ test_config_global core.logAllRefUpdates false &&
+ test_when_finished "rm -rf repo" &&
+ git init repo &&
+ (
+ cd repo &&
+ test_commit A &&
+ test_commit B &&
+ cat >expect <<-EOF &&
+ fatal: reftable block size cannot exceed 16MB
+ EOF
+ test_must_fail git -c reftable.blockSize=16777216 pack-refs 2>err &&
+ test_cmp expect err
+ )
+'
+
test_done
--
2.45.0
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
next prev parent reply other threads:[~2024-05-10 10:29 UTC|newest]
Thread overview: 78+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-05-02 6:51 [PATCH 00/11] reftable: expose write options as config Patrick Steinhardt
2024-05-02 6:51 ` [PATCH 01/11] reftable: consistently refer to `reftable_write_options` as `opts` Patrick Steinhardt
2024-05-10 9:00 ` Karthik Nayak
2024-05-10 10:13 ` Patrick Steinhardt
2024-05-02 6:51 ` [PATCH 02/11] reftable: consistently pass write opts as value Patrick Steinhardt
2024-05-02 6:51 ` [PATCH 03/11] reftable/writer: drop static variable used to initialize strbuf Patrick Steinhardt
2024-05-02 6:51 ` [PATCH 04/11] reftable/writer: improve error when passed an invalid block size Patrick Steinhardt
2024-05-02 6:51 ` [PATCH 05/11] reftable/dump: support dumping a table's block structure Patrick Steinhardt
2024-05-02 6:51 ` [PATCH 06/11] refs/reftable: allow configuring block size Patrick Steinhardt
2024-05-10 9:29 ` Karthik Nayak
2024-05-10 10:13 ` Patrick Steinhardt
2024-05-02 6:52 ` [PATCH 07/11] reftable: use `uint16_t` to track restart interval Patrick Steinhardt
2024-05-02 6:52 ` [PATCH 08/11] refs/reftable: allow configuring " Patrick Steinhardt
2024-05-02 6:52 ` [PATCH 09/11] refs/reftable: allow disabling writing the object index Patrick Steinhardt
2024-05-02 6:52 ` [PATCH 10/11] reftable: make the compaction factor configurable Patrick Steinhardt
2024-05-10 9:55 ` Karthik Nayak
2024-05-10 10:13 ` Patrick Steinhardt
2024-05-02 6:52 ` [PATCH 11/11] refs/reftable: allow configuring geometric factor Patrick Steinhardt
2024-05-10 9:58 ` Karthik Nayak
2024-05-10 10:13 ` Patrick Steinhardt
2024-05-02 7:29 ` [PATCH 00/11] reftable: expose write options as config Patrick Steinhardt
2024-05-03 20:38 ` Junio C Hamano
2024-05-06 6:51 ` Patrick Steinhardt
2024-05-06 21:29 ` Justin Tobler
2024-05-10 10:00 ` Karthik Nayak
2024-05-10 10:14 ` Patrick Steinhardt
2024-05-10 10:29 ` [PATCH v2 " Patrick Steinhardt
2024-05-10 10:29 ` [PATCH v2 01/11] reftable: consistently refer to `reftable_write_options` as `opts` Patrick Steinhardt
2024-05-10 21:03 ` Junio C Hamano
2024-05-10 10:29 ` [PATCH v2 02/11] reftable: consistently pass write opts as value Patrick Steinhardt
2024-05-10 21:11 ` Junio C Hamano
2024-05-13 7:53 ` Patrick Steinhardt
2024-05-10 10:29 ` [PATCH v2 03/11] reftable/writer: drop static variable used to initialize strbuf Patrick Steinhardt
2024-05-10 21:19 ` Junio C Hamano
2024-05-10 10:29 ` [PATCH v2 04/11] reftable/writer: improve error when passed an invalid block size Patrick Steinhardt
2024-05-10 21:25 ` Junio C Hamano
2024-05-13 7:53 ` Patrick Steinhardt
2024-05-10 10:29 ` [PATCH v2 05/11] reftable/dump: support dumping a table's block structure Patrick Steinhardt
2024-05-13 22:42 ` Junio C Hamano
2024-05-10 10:29 ` Patrick Steinhardt [this message]
2024-05-10 10:29 ` [PATCH v2 07/11] reftable: use `uint16_t` to track restart interval Patrick Steinhardt
2024-05-13 22:42 ` Junio C Hamano
2024-05-14 4:54 ` Patrick Steinhardt
2024-05-10 10:29 ` [PATCH v2 08/11] refs/reftable: allow configuring " Patrick Steinhardt
2024-05-10 21:57 ` Junio C Hamano
2024-05-13 7:54 ` Patrick Steinhardt
2024-05-10 10:30 ` [PATCH v2 09/11] refs/reftable: allow disabling writing the object index Patrick Steinhardt
2024-05-10 10:30 ` [PATCH v2 10/11] reftable: make the compaction factor configurable Patrick Steinhardt
2024-05-10 22:12 ` Junio C Hamano
2024-05-13 7:54 ` Patrick Steinhardt
2024-05-13 16:22 ` Junio C Hamano
2024-05-14 4:54 ` Patrick Steinhardt
2024-05-10 10:30 ` [PATCH v2 11/11] refs/reftable: allow configuring geometric factor Patrick Steinhardt
2024-05-10 11:43 ` [PATCH v2 00/11] reftable: expose write options as config Karthik Nayak
2024-05-13 8:17 ` [PATCH v3 " Patrick Steinhardt
2024-05-13 8:17 ` [PATCH v3 01/11] reftable: consistently refer to `reftable_write_options` as `opts` Patrick Steinhardt
2024-05-13 8:17 ` [PATCH v3 02/11] reftable: pass opts as constant pointer Patrick Steinhardt
2024-05-17 8:02 ` Karthik Nayak
2024-05-21 23:22 ` Justin Tobler
2024-05-22 7:19 ` Patrick Steinhardt
2024-05-13 8:18 ` [PATCH v3 03/11] reftable/writer: drop static variable used to initialize strbuf Patrick Steinhardt
2024-05-13 8:18 ` [PATCH v3 04/11] reftable/writer: improve error when passed an invalid block size Patrick Steinhardt
2024-05-13 8:18 ` [PATCH v3 05/11] reftable/dump: support dumping a table's block structure Patrick Steinhardt
2024-05-21 23:35 ` Justin Tobler
2024-05-22 7:19 ` Patrick Steinhardt
2024-05-13 8:18 ` [PATCH v3 06/11] refs/reftable: allow configuring block size Patrick Steinhardt
2024-05-17 8:09 ` Karthik Nayak
2024-05-13 8:18 ` [PATCH v3 07/11] reftable: use `uint16_t` to track restart interval Patrick Steinhardt
2024-05-13 8:18 ` [PATCH v3 08/11] refs/reftable: allow configuring " Patrick Steinhardt
2024-05-21 23:50 ` Justin Tobler
2024-05-22 7:19 ` Patrick Steinhardt
2024-05-13 8:18 ` [PATCH v3 09/11] refs/reftable: allow disabling writing the object index Patrick Steinhardt
2024-05-13 8:18 ` [PATCH v3 10/11] reftable: make the compaction factor configurable Patrick Steinhardt
2024-05-13 8:18 ` [PATCH v3 11/11] refs/reftable: allow configuring geometric factor Patrick Steinhardt
2024-05-17 8:14 ` [PATCH v3 00/11] reftable: expose write options as config Karthik Nayak
2024-05-17 8:26 ` Patrick Steinhardt
2024-05-21 23:54 ` Justin Tobler
2024-05-22 7:19 ` Patrick Steinhardt
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=be5bdc6dc13510890dbc9ed06e2de6dd866c29e5.1715336798.git.ps@pks.im \
--to=ps@pks.im \
--cc=git@vger.kernel.org \
--cc=jltobler@gmail.com \
--cc=karthik.188@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).