From: Gao Xiang <hsiangkao@linux.alibaba.com>
To: Yifan Zhao <zhaoyifan@sjtu.edu.cn>
Cc: linux-erofs@lists.ozlabs.org
Subject: Re: [PATCH v2 3/7] erofs-utils: mkfs: add --worker=# parameter
Date: Fri, 23 Feb 2024 00:40:30 +0800 [thread overview]
Message-ID: <63dc8e76-acde-4736-8df7-65670b17f8d9@linux.alibaba.com> (raw)
In-Reply-To: <20240220075525.684205-4-zhaoyifan@sjtu.edu.cn>
On 2024/2/20 15:55, Yifan Zhao wrote:
> This patch introduces a --worker=# parameter for the incoming
> multi-threaded compression support. It also introduces a segment size
> used in multi-threaded compression, which has the default value 16MB
> and cannot be modified.
It also introduces a concept called `segment size` to split large files
for multi-threading, which has the default value 16MB for now.
>
> Signed-off-by: Yifan Zhao <zhaoyifan@sjtu.edu.cn>
> ---
> include/erofs/config.h | 4 ++++
> lib/config.c | 4 ++++
> mkfs/main.c | 38 ++++++++++++++++++++++++++++++++++++++
> 3 files changed, 46 insertions(+)
>
> diff --git a/include/erofs/config.h b/include/erofs/config.h
> index 73e3ac2..d19094e 100644
> --- a/include/erofs/config.h
> +++ b/include/erofs/config.h
> @@ -75,6 +75,10 @@ struct erofs_configure {
> char c_force_chunkformat;
> /* < 0, xattr disabled and INT_MAX, always use inline xattrs */
> int c_inline_xattr_tolerance;
> +#ifdef EROFS_MT_ENABLED
> + u64 c_mt_segment_size;
I think mt_ prefix is not needed: c_segment_size;
> + u32 c_mt_worker_num;
c_mt_workers;
> +#endif
>
> u32 c_pclusterblks_max, c_pclusterblks_def, c_pclusterblks_packed;
> u32 c_max_decompressed_extent_bytes;
> diff --git a/lib/config.c b/lib/config.c
> index 947a183..8add06d 100644
> --- a/lib/config.c
> +++ b/lib/config.c
> @@ -38,6 +38,10 @@ void erofs_init_configure(void)
> cfg.c_pclusterblks_max = 1;
> cfg.c_pclusterblks_def = 1;
> cfg.c_max_decompressed_extent_bytes = -1;
> +#ifdef EROFS_MT_ENABLED
> + cfg.c_mt_segment_size = 16ULL * 1024 * 1024;
> + cfg.c_mt_worker_num = 1;
> +#endif
>
> erofs_stdout_tty = isatty(STDOUT_FILENO);
> }
> diff --git a/mkfs/main.c b/mkfs/main.c
> index 7aea64a..3882533 100644
> --- a/mkfs/main.c
> +++ b/mkfs/main.c
> @@ -73,6 +73,9 @@ static struct option long_options[] = {
> {"gzip", no_argument, NULL, 517},
> #endif
> {"offset", required_argument, NULL, 518},
> +#ifdef EROFS_MT_ENABLED
> + {"worker", required_argument, NULL, 519},
let's use `--workers=#` instead of `worker`.
> +#endif
> {0, 0, 0, 0},
> };
>
> @@ -175,6 +178,9 @@ static void usage(int argc, char **argv)
> " --product-out=X X=product_out directory\n"
> " --fs-config-file=X X=fs_config file\n"
> " --block-list-file=X X=block_list file\n"
> +#endif
> +#ifdef EROFS_MT_ENABLED
> + " --worker=# set the number of worker threads to # (default=1)\n"
--workers=#
Thanks,
Gao Xiang
next prev parent reply other threads:[~2024-02-22 16:40 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-02-20 7:55 [PATCH v2 0/7] erofs-utils: mkfs: introduce multi-threaded compression Yifan Zhao
2024-02-20 7:55 ` [PATCH v2 1/7] erofs-utils: introduce multi-threading framework Yifan Zhao
2024-02-20 7:55 ` [PATCH v2 2/7] erofs-utils: add a helper to get available processors Yifan Zhao
2024-02-20 7:55 ` [PATCH v2 3/7] erofs-utils: mkfs: add --worker=# parameter Yifan Zhao
2024-02-20 7:55 ` [PATCH v2 4/7] erofs-utils: mkfs: optionally print warning in erofs_compressor_init Yifan Zhao
2024-02-20 7:55 ` [PATCH v2 5/7] erofs-utils: mkfs: introduce inner-file multi-threaded compression Yifan Zhao
2024-02-20 7:55 ` [PATCH v2 6/7] erofs-utils: mkfs: introduce inter-file " Yifan Zhao
2024-02-20 7:55 ` [PATCH v2 7/7] erofs-utils: mkfs: use per-worker tmpfile for multi-threaded mkfs Yifan Zhao
2024-02-22 2:54 ` [PATCH v2 4/7] erofs-utils: mkfs: optionally print warning in erofs_compressor_init Gao Xiang
2024-02-22 16:40 ` Gao Xiang [this message]
2024-02-22 2:37 ` [PATCH v2 1/7] erofs-utils: introduce multi-threading framework Gao Xiang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=63dc8e76-acde-4736-8df7-65670b17f8d9@linux.alibaba.com \
--to=hsiangkao@linux.alibaba.com \
--cc=linux-erofs@lists.ozlabs.org \
--cc=zhaoyifan@sjtu.edu.cn \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).