* [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning
@ 2024-05-08 7:41 Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 01/19] zram: move from crypto API to custom comp backends API Sergey Senozhatsky
` (19 more replies)
0 siblings, 20 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:41 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
This patch set moves zram from crypto API to a custom compression
API which allows us to tune and configure compression algorithms,
something that crypto API, unfortunately, doesn't support. Basically,
this seroes brings back the bits of comp "backend" code that we had
many years ago. This means that if we want zram to support new
compression algorithms we need to implement corresponding backends.
Currently, zram supports a pretty decent number of comp backends:
lzo, lzorle, lz4, lz4hc, 842, deflate, zstd
At this point we handle 2 parameters: a compression level and
a pre-trained compression dictionary. Which seems like a good enough
start. The list will be extended in the future.
Examples:
- changes default compression level
echo "algo=zstd level=11" > /sys/block/zram0/comp_algorithm
- passes path to a pre-trained dictionary
echo "algo=zstd dict=/etc/dictionary" > /sys/block/zram0/comp_algorithm
Benchmarks
==========
*** zstd
/sys/block/zram0/mm_stat
1750302720 504600204 514416640 0 514416640 1 0 34204 34204
*** zstd level=5
/sys/block/zram0/mm_stat
1750331392 488449001 497905664 0 497905664 1 0 34204 34204
*** zstd dict=/etc/dictionary
/sys/block/zram0/mm_stat
1750335488 464838883 474210304 0 474210304 1 0 34204 34204
*** zstd level=5 dict=/etc/dictionary
/sys/block/zram0/mm_stat
1750319104 451907185 461299712 0 461299712 1 0 34204 34204
*** lz4
/sys/block/zram0/mm_stat
1750319104 664253635 676859904 0 676859904 1 0 34288 34288
*** lz4 dict=/etc/dictionary
/sys/block/zram0/mm_stat
1750319104 620602911 632705024 0 632705024 1 0 34288 34288
*** lz4hc
/sys/block/zram0/mm_stat
1750315008 609004936 621092864 0 621092864 1 0 34288 34288
*** lz4hc level=5 dict=/etc/dictionary
/sys/block/zram0/mm_stat
1750323200 501315128 511303680 0 511303680 1 0 34288 34288
v3:
-- support pre-trained dictionary in lz4
-- support pre-trained dictionary in lz4hc
v2:
-- forcibly select LZO when none backeneds were selected (Masahiro Yamada)
-- add 842 compression backend
-- added comp config preprocessing and caching
-- reworked zstd dict handling
Sergey Senozhatsky (19):
zram: move from crypto API to custom comp backends API
zram: add lzo and lzorle compression backends support
zram: add lz4 compression backend support
zram: add lz4hc compression backend support
zram: add zstd compression backend support
zram: pass estimated src size hint to zstd
zram: add zlib compression backend support
zram: add 842 compression backend support
zram: check that backends array has at least one backend
zram: introduce zcomp_config structure
zram: extend comp_algorithm attr write handling
zram: support compression level comp config
zram: add support for dict comp config
zram: add dictionary support to zstd backend
zram: add config init/release backend callbacks
zram: share dictionaries between per-CPU contexts
zram: add dictionary support to lz4
zram: add dictionary support to lz4hc
Documentation/zram: add documentation for algorithm parameters
Documentation/admin-guide/blockdev/zram.rst | 38 +++-
drivers/block/zram/Kconfig | 76 +++++--
drivers/block/zram/Makefile | 8 +
drivers/block/zram/backend_842.c | 79 +++++++
drivers/block/zram/backend_842.h | 10 +
drivers/block/zram/backend_deflate.c | 143 +++++++++++++
drivers/block/zram/backend_deflate.h | 10 +
drivers/block/zram/backend_lz4.c | 123 +++++++++++
drivers/block/zram/backend_lz4.h | 10 +
drivers/block/zram/backend_lz4hc.c | 128 ++++++++++++
drivers/block/zram/backend_lz4hc.h | 10 +
drivers/block/zram/backend_lzo.c | 55 +++++
drivers/block/zram/backend_lzo.h | 10 +
drivers/block/zram/backend_lzorle.c | 55 +++++
drivers/block/zram/backend_lzorle.h | 10 +
drivers/block/zram/backend_zstd.c | 221 ++++++++++++++++++++
drivers/block/zram/backend_zstd.h | 10 +
drivers/block/zram/zcomp.c | 155 +++++++-------
drivers/block/zram/zcomp.h | 42 +++-
drivers/block/zram/zram_drv.c | 120 ++++++++++-
drivers/block/zram/zram_drv.h | 1 +
21 files changed, 1203 insertions(+), 111 deletions(-)
create mode 100644 drivers/block/zram/backend_842.c
create mode 100644 drivers/block/zram/backend_842.h
create mode 100644 drivers/block/zram/backend_deflate.c
create mode 100644 drivers/block/zram/backend_deflate.h
create mode 100644 drivers/block/zram/backend_lz4.c
create mode 100644 drivers/block/zram/backend_lz4.h
create mode 100644 drivers/block/zram/backend_lz4hc.c
create mode 100644 drivers/block/zram/backend_lz4hc.h
create mode 100644 drivers/block/zram/backend_lzo.c
create mode 100644 drivers/block/zram/backend_lzo.h
create mode 100644 drivers/block/zram/backend_lzorle.c
create mode 100644 drivers/block/zram/backend_lzorle.h
create mode 100644 drivers/block/zram/backend_zstd.c
create mode 100644 drivers/block/zram/backend_zstd.h
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply [flat|nested] 33+ messages in thread
* [PATCHv3 01/19] zram: move from crypto API to custom comp backends API
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
@ 2024-05-08 7:41 ` Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 02/19] zram: add lzo and lzorle compression backends support Sergey Senozhatsky
` (18 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:41 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Crypto API is beautiful and powerful, however, being a
generic API, it lacks support for fine-grained per-algorithm
configuration. A number of compression algorithms provide
various knobs to tune characteristics for particular data
patterns. The simplest case is "compression level". A more
complicated and interesting case is user-space trained
dictionaries (e.g. lz4 and zstd).
Moving to custom backends implementation gives us ability to
have our own minimalistic and extendable API, and algorithms
tunings becomes possible.
The list of compression backends is empty at this point,
we will add backends in the followup patches.
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/Kconfig | 39 +----------
drivers/block/zram/zcomp.c | 127 +++++++++++++---------------------
drivers/block/zram/zcomp.h | 26 +++++--
drivers/block/zram/zram_drv.c | 9 ++-
4 files changed, 73 insertions(+), 128 deletions(-)
diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index 7b29cce60ab2..8ecb74f83a5e 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -2,7 +2,6 @@
config ZRAM
tristate "Compressed RAM block device support"
depends on BLOCK && SYSFS && MMU
- depends on CRYPTO_LZO || CRYPTO_ZSTD || CRYPTO_LZ4 || CRYPTO_LZ4HC || CRYPTO_842
select ZSMALLOC
help
Creates virtual block devices called /dev/zramX (X = 0, 1, ...).
@@ -15,45 +14,9 @@ config ZRAM
See Documentation/admin-guide/blockdev/zram.rst for more information.
-choice
- prompt "Default zram compressor"
- default ZRAM_DEF_COMP_LZORLE
- depends on ZRAM
-
-config ZRAM_DEF_COMP_LZORLE
- bool "lzo-rle"
- depends on CRYPTO_LZO
-
-config ZRAM_DEF_COMP_ZSTD
- bool "zstd"
- depends on CRYPTO_ZSTD
-
-config ZRAM_DEF_COMP_LZ4
- bool "lz4"
- depends on CRYPTO_LZ4
-
-config ZRAM_DEF_COMP_LZO
- bool "lzo"
- depends on CRYPTO_LZO
-
-config ZRAM_DEF_COMP_LZ4HC
- bool "lz4hc"
- depends on CRYPTO_LZ4HC
-
-config ZRAM_DEF_COMP_842
- bool "842"
- depends on CRYPTO_842
-
-endchoice
-
config ZRAM_DEF_COMP
string
- default "lzo-rle" if ZRAM_DEF_COMP_LZORLE
- default "zstd" if ZRAM_DEF_COMP_ZSTD
- default "lz4" if ZRAM_DEF_COMP_LZ4
- default "lzo" if ZRAM_DEF_COMP_LZO
- default "lz4hc" if ZRAM_DEF_COMP_LZ4HC
- default "842" if ZRAM_DEF_COMP_842
+ default "unset-value"
config ZRAM_WRITEBACK
bool "Write back incompressible or idle page to backing device"
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 8237b08c49d8..0d0d2e6dbaa9 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -15,31 +15,16 @@
#include "zcomp.h"
-static const char * const backends[] = {
-#if IS_ENABLED(CONFIG_CRYPTO_LZO)
- "lzo",
- "lzo-rle",
-#endif
-#if IS_ENABLED(CONFIG_CRYPTO_LZ4)
- "lz4",
-#endif
-#if IS_ENABLED(CONFIG_CRYPTO_LZ4HC)
- "lz4hc",
-#endif
-#if IS_ENABLED(CONFIG_CRYPTO_842)
- "842",
-#endif
-#if IS_ENABLED(CONFIG_CRYPTO_ZSTD)
- "zstd",
-#endif
+static struct zcomp_backend *backends[] = {
+ NULL
};
-static void zcomp_strm_free(struct zcomp_strm *zstrm)
+static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm)
{
- if (!IS_ERR_OR_NULL(zstrm->tfm))
- crypto_free_comp(zstrm->tfm);
+ if (zstrm->ctx)
+ comp->backend->destroy_ctx(zstrm->ctx);
vfree(zstrm->buffer);
- zstrm->tfm = NULL;
+ zstrm->ctx = NULL;
zstrm->buffer = NULL;
}
@@ -47,60 +32,55 @@ static void zcomp_strm_free(struct zcomp_strm *zstrm)
* Initialize zcomp_strm structure with ->tfm initialized by backend, and
* ->buffer. Return a negative value on error.
*/
-static int zcomp_strm_init(struct zcomp_strm *zstrm, struct zcomp *comp)
+static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm)
{
- zstrm->tfm = crypto_alloc_comp(comp->name, 0, 0);
+ zstrm->ctx = comp->backend->create_ctx();
+
/*
* allocate 2 pages. 1 for compressed data, plus 1 extra for the
* case when compressed size is larger than the original one
*/
zstrm->buffer = vzalloc(2 * PAGE_SIZE);
- if (IS_ERR_OR_NULL(zstrm->tfm) || !zstrm->buffer) {
- zcomp_strm_free(zstrm);
+ if (!zstrm->ctx || !zstrm->buffer) {
+ zcomp_strm_free(comp, zstrm);
return -ENOMEM;
}
return 0;
}
+static struct zcomp_backend *lookup_backend(const char *comp)
+{
+ int i = 0;
+
+ while (backends[i]) {
+ if (sysfs_streq(comp, backends[i]->name))
+ break;
+ i++;
+ }
+ return backends[i];
+}
+
bool zcomp_available_algorithm(const char *comp)
{
- /*
- * Crypto does not ignore a trailing new line symbol,
- * so make sure you don't supply a string containing
- * one.
- * This also means that we permit zcomp initialisation
- * with any compressing algorithm known to crypto api.
- */
- return crypto_has_comp(comp, 0, 0) == 1;
+ return lookup_backend(comp) != NULL;
}
/* show available compressors */
ssize_t zcomp_available_show(const char *comp, char *buf)
{
- bool known_algorithm = false;
ssize_t sz = 0;
int i;
- for (i = 0; i < ARRAY_SIZE(backends); i++) {
- if (!strcmp(comp, backends[i])) {
- known_algorithm = true;
+ for (i = 0; i < ARRAY_SIZE(backends) - 1; i++) {
+ if (!strcmp(comp, backends[i]->name)) {
sz += scnprintf(buf + sz, PAGE_SIZE - sz - 2,
- "[%s] ", backends[i]);
+ "[%s] ", backends[i]->name);
} else {
sz += scnprintf(buf + sz, PAGE_SIZE - sz - 2,
- "%s ", backends[i]);
+ "%s ", backends[i]->name);
}
}
- /*
- * Out-of-tree module known to crypto api or a missing
- * entry in `backends'.
- */
- if (!known_algorithm && crypto_has_comp(comp, 0, 0) == 1)
- sz += scnprintf(buf + sz, PAGE_SIZE - sz - 2,
- "[%s] ", comp);
-
- sz += scnprintf(buf + sz, PAGE_SIZE - sz, "\n");
return sz;
}
@@ -115,8 +95,8 @@ void zcomp_stream_put(struct zcomp *comp)
local_unlock(&comp->stream->lock);
}
-int zcomp_compress(struct zcomp_strm *zstrm,
- const void *src, unsigned int *dst_len)
+int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
+ const void *src, unsigned int *dst_len)
{
/*
* Our dst memory (zstrm->buffer) is always `2 * PAGE_SIZE' sized
@@ -132,21 +112,19 @@ int zcomp_compress(struct zcomp_strm *zstrm,
* the dst buffer, zram_drv will take care of the fact that
* compressed buffer is too big.
*/
- *dst_len = PAGE_SIZE * 2;
+ size_t dlen = PAGE_SIZE * 2;
+ int ret;
- return crypto_comp_compress(zstrm->tfm,
- src, PAGE_SIZE,
- zstrm->buffer, dst_len);
+ ret = comp->backend->compress(zstrm->ctx, src, zstrm->buffer, &dlen);
+ if (!ret)
+ *dst_len = dlen;
+ return ret;
}
-int zcomp_decompress(struct zcomp_strm *zstrm,
- const void *src, unsigned int src_len, void *dst)
+int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
+ const void *src, unsigned int src_len, void *dst)
{
- unsigned int dst_len = PAGE_SIZE;
-
- return crypto_comp_decompress(zstrm->tfm,
- src, src_len,
- dst, &dst_len);
+ return comp->backend->decompress(zstrm->ctx, src, src_len, dst);
}
int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node)
@@ -158,7 +136,7 @@ int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node)
zstrm = per_cpu_ptr(comp->stream, cpu);
local_lock_init(&zstrm->lock);
- ret = zcomp_strm_init(zstrm, comp);
+ ret = zcomp_strm_init(comp, zstrm);
if (ret)
pr_err("Can't allocate a compression stream\n");
return ret;
@@ -170,7 +148,7 @@ int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node)
struct zcomp_strm *zstrm;
zstrm = per_cpu_ptr(comp->stream, cpu);
- zcomp_strm_free(zstrm);
+ zcomp_strm_free(comp, zstrm);
return 0;
}
@@ -199,32 +177,21 @@ void zcomp_destroy(struct zcomp *comp)
kfree(comp);
}
-/*
- * search available compressors for requested algorithm.
- * allocate new zcomp and initialize it. return compressing
- * backend pointer or ERR_PTR if things went bad. ERR_PTR(-EINVAL)
- * if requested algorithm is not supported, ERR_PTR(-ENOMEM) in
- * case of allocation error, or any other error potentially
- * returned by zcomp_init().
- */
struct zcomp *zcomp_create(const char *alg)
{
struct zcomp *comp;
int error;
- /*
- * Crypto API will execute /sbin/modprobe if the compression module
- * is not loaded yet. We must do it here, otherwise we are about to
- * call /sbin/modprobe under CPU hot-plug lock.
- */
- if (!zcomp_available_algorithm(alg))
- return ERR_PTR(-EINVAL);
-
comp = kzalloc(sizeof(struct zcomp), GFP_KERNEL);
if (!comp)
return ERR_PTR(-ENOMEM);
- comp->name = alg;
+ comp->backend = lookup_backend(alg);
+ if (!comp->backend) {
+ kfree(comp);
+ return ERR_PTR(-EINVAL);
+ }
+
error = zcomp_init(comp);
if (error) {
kfree(comp);
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index e9fe63da0e9b..757b85017e23 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -12,13 +12,26 @@ struct zcomp_strm {
local_lock_t lock;
/* compression/decompression buffer */
void *buffer;
- struct crypto_comp *tfm;
+ void *ctx;
+};
+
+struct zcomp_backend {
+ int (*compress)(void *ctx, const unsigned char *src,
+ unsigned char *dst, size_t *dst_len);
+
+ int (*decompress)(void *ctx, const unsigned char *src, size_t src_len,
+ unsigned char *dst);
+
+ void *(*create_ctx)(void);
+ void (*destroy_ctx)(void *ctx);
+
+ const char *name;
};
/* dynamic per-device compression frontend */
struct zcomp {
struct zcomp_strm __percpu *stream;
- const char *name;
+ struct zcomp_backend *backend;
struct hlist_node node;
};
@@ -33,10 +46,9 @@ void zcomp_destroy(struct zcomp *comp);
struct zcomp_strm *zcomp_stream_get(struct zcomp *comp);
void zcomp_stream_put(struct zcomp *comp);
-int zcomp_compress(struct zcomp_strm *zstrm,
- const void *src, unsigned int *dst_len);
-
-int zcomp_decompress(struct zcomp_strm *zstrm,
- const void *src, unsigned int src_len, void *dst);
+int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
+ const void *src, unsigned int *dst_len);
+int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
+ const void *src, unsigned int src_len, void *dst);
#endif /* _ZCOMP_H_ */
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 4cf38f7d3e0a..298ef0e97e03 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1342,7 +1342,8 @@ static int zram_read_from_zspool(struct zram *zram, struct page *page,
ret = 0;
} else {
dst = kmap_local_page(page);
- ret = zcomp_decompress(zstrm, src, size, dst);
+ ret = zcomp_decompress(zram->comps[prio], zstrm,
+ src, size, dst);
kunmap_local(dst);
zcomp_stream_put(zram->comps[prio]);
}
@@ -1429,7 +1430,8 @@ static int zram_write_page(struct zram *zram, struct page *page, u32 index)
compress_again:
zstrm = zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]);
src = kmap_local_page(page);
- ret = zcomp_compress(zstrm, src, &comp_len);
+ ret = zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm,
+ src, &comp_len);
kunmap_local(src);
if (unlikely(ret)) {
@@ -1616,7 +1618,8 @@ static int zram_recompress(struct zram *zram, u32 index, struct page *page,
num_recomps++;
zstrm = zcomp_stream_get(zram->comps[prio]);
src = kmap_local_page(page);
- ret = zcomp_compress(zstrm, src, &comp_len_new);
+ ret = zcomp_compress(zram->comps[prio], zstrm,
+ src, &comp_len_new);
kunmap_local(src);
if (ret) {
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 02/19] zram: add lzo and lzorle compression backends support
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 01/19] zram: move from crypto API to custom comp backends API Sergey Senozhatsky
@ 2024-05-08 7:41 ` Sergey Senozhatsky
2024-05-09 11:23 ` kernel test robot
2024-05-08 7:41 ` [PATCHv3 03/19] zram: add lz4 compression backend support Sergey Senozhatsky
` (17 subsequent siblings)
19 siblings, 1 reply; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:41 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/Kconfig | 23 +++++++++++++++
drivers/block/zram/Makefile | 3 ++
drivers/block/zram/backend_lzo.c | 44 +++++++++++++++++++++++++++++
drivers/block/zram/backend_lzo.h | 10 +++++++
drivers/block/zram/backend_lzorle.c | 44 +++++++++++++++++++++++++++++
drivers/block/zram/backend_lzorle.h | 10 +++++++
drivers/block/zram/zcomp.c | 7 +++++
7 files changed, 141 insertions(+)
create mode 100644 drivers/block/zram/backend_lzo.c
create mode 100644 drivers/block/zram/backend_lzo.h
create mode 100644 drivers/block/zram/backend_lzorle.c
create mode 100644 drivers/block/zram/backend_lzorle.h
diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index 8ecb74f83a5e..5d329a887b12 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -14,8 +14,31 @@ config ZRAM
See Documentation/admin-guide/blockdev/zram.rst for more information.
+config ZRAM_BACKEND_LZO
+ bool "lzo and lzo-rle compression support"
+ depends on ZRAM
+ select LZO_COMPRESS
+ select LZO_DECOMPRESS
+
+choice
+ prompt "Default zram compressor"
+ default ZRAM_DEF_COMP_LZORLE
+ depends on ZRAM
+
+config ZRAM_DEF_COMP_LZORLE
+ bool "lzo-rle"
+ depends on ZRAM_BACKEND_LZO
+
+config ZRAM_DEF_COMP_LZO
+ bool "lzo"
+ depends on ZRAM_BACKEND_LZO
+
+endchoice
+
config ZRAM_DEF_COMP
string
+ default "lzo-rle" if ZRAM_DEF_COMP_LZORLE
+ default "lzo" if ZRAM_DEF_COMP_LZO
default "unset-value"
config ZRAM_WRITEBACK
diff --git a/drivers/block/zram/Makefile b/drivers/block/zram/Makefile
index de9e457907b1..2dcbc9b75d91 100644
--- a/drivers/block/zram/Makefile
+++ b/drivers/block/zram/Makefile
@@ -1,4 +1,7 @@
# SPDX-License-Identifier: GPL-2.0-only
+
+obj-$(CONFIG_ZRAM_BACKEND_LZO) += backend_lzorle.o backend_lzo.o
+
zram-y := zcomp.o zram_drv.o
obj-$(CONFIG_ZRAM) += zram.o
diff --git a/drivers/block/zram/backend_lzo.c b/drivers/block/zram/backend_lzo.c
new file mode 100644
index 000000000000..b88b408964cd
--- /dev/null
+++ b/drivers/block/zram/backend_lzo.c
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include <linux/lzo.h>
+
+#include "backend_lzo.h"
+
+static void *lzo_create(void)
+{
+ return kzalloc(LZO1X_MEM_COMPRESS, GFP_KERNEL);
+}
+
+static void lzo_destroy(void *ctx)
+{
+ kfree(ctx);
+}
+
+static int lzo_compress(void *ctx, const unsigned char *src,
+ unsigned char *dst, size_t *dst_len)
+{
+ int ret;
+
+ ret = lzo1x_1_compress(src, PAGE_SIZE, dst, dst_len, ctx);
+ return ret == LZO_E_OK ? 0 : ret;
+}
+
+static int lzo_decompress(void *ctx, const unsigned char *src, size_t src_len,
+ unsigned char *dst)
+{
+ size_t dst_len = PAGE_SIZE;
+ int ret;
+
+ ret = lzo1x_decompress_safe(src, src_len, dst, &dst_len);
+ return ret == LZO_E_OK ? 0 : ret;
+}
+
+struct zcomp_backend backend_lzo = {
+ .compress = lzo_compress,
+ .decompress = lzo_decompress,
+ .create_ctx = lzo_create,
+ .destroy_ctx = lzo_destroy,
+ .name = "lzo",
+};
diff --git a/drivers/block/zram/backend_lzo.h b/drivers/block/zram/backend_lzo.h
new file mode 100644
index 000000000000..377ccb7389e2
--- /dev/null
+++ b/drivers/block/zram/backend_lzo.h
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#ifndef __BACKEND_LZO_H__
+#define __BACKEND_LZO_H__
+
+#include "zcomp.h"
+
+extern struct zcomp_backend backend_lzo;
+
+#endif /* __BACKEND_LZO_H__ */
diff --git a/drivers/block/zram/backend_lzorle.c b/drivers/block/zram/backend_lzorle.c
new file mode 100644
index 000000000000..9bf1843021b0
--- /dev/null
+++ b/drivers/block/zram/backend_lzorle.c
@@ -0,0 +1,44 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include <linux/lzo.h>
+
+#include "backend_lzorle.h"
+
+static void *lzorle_create(void)
+{
+ return kzalloc(LZO1X_MEM_COMPRESS, GFP_KERNEL);
+}
+
+static void lzorle_destroy(void *ctx)
+{
+ kfree(ctx);
+}
+
+static int lzorle_compress(void *ctx, const unsigned char *src,
+ unsigned char *dst, size_t *dst_len)
+{
+ int ret;
+
+ ret = lzorle1x_1_compress(src, PAGE_SIZE, dst, dst_len, ctx);
+ return ret == LZO_E_OK ? 0 : ret;
+}
+
+static int lzorle_decompress(void *ctx, const unsigned char *src,
+ size_t src_len, unsigned char *dst)
+{
+ size_t dst_len = PAGE_SIZE;
+ int ret;
+
+ ret = lzo1x_decompress_safe(src, src_len, dst, &dst_len);
+ return ret == LZO_E_OK ? 0 : ret;
+}
+
+struct zcomp_backend backend_lzorle = {
+ .compress = lzorle_compress,
+ .decompress = lzorle_decompress,
+ .create_ctx = lzorle_create,
+ .destroy_ctx = lzorle_destroy,
+ .name = "lzo-rle",
+};
diff --git a/drivers/block/zram/backend_lzorle.h b/drivers/block/zram/backend_lzorle.h
new file mode 100644
index 000000000000..5c1db65a38a4
--- /dev/null
+++ b/drivers/block/zram/backend_lzorle.h
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#ifndef __BACKEND_LZORLE_H__
+#define __BACKEND_LZORLE_H__
+
+#include "zcomp.h"
+
+extern struct zcomp_backend backend_lzorle;
+
+#endif /* __BACKEND_LZORLE_H__ */
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 0d0d2e6dbaa9..58fb3ac91f4b 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -15,7 +15,14 @@
#include "zcomp.h"
+#include "backend_lzo.h"
+#include "backend_lzorle.h"
+
static struct zcomp_backend *backends[] = {
+#if IS_ENABLED(CONFIG_ZRAM_BACKEND_LZO)
+ &backend_lzorle,
+ &backend_lzo,
+#endif
NULL
};
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 03/19] zram: add lz4 compression backend support
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 01/19] zram: move from crypto API to custom comp backends API Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 02/19] zram: add lzo and lzorle compression backends support Sergey Senozhatsky
@ 2024-05-08 7:41 ` Sergey Senozhatsky
2024-05-10 7:00 ` Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 04/19] zram: add lz4hc " Sergey Senozhatsky
` (16 subsequent siblings)
19 siblings, 1 reply; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:41 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/Kconfig | 11 ++++++++
drivers/block/zram/Makefile | 1 +
drivers/block/zram/backend_lz4.c | 47 ++++++++++++++++++++++++++++++++
drivers/block/zram/backend_lz4.h | 10 +++++++
drivers/block/zram/zcomp.c | 4 +++
5 files changed, 73 insertions(+)
create mode 100644 drivers/block/zram/backend_lz4.c
create mode 100644 drivers/block/zram/backend_lz4.h
diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index 5d329a887b12..f1e76fc8431a 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -20,6 +20,12 @@ config ZRAM_BACKEND_LZO
select LZO_COMPRESS
select LZO_DECOMPRESS
+config ZRAM_BACKEND_LZ4
+ bool "lz4 compression support"
+ depends on ZRAM
+ select LZ4_COMPRESS
+ select LZ4_DECOMPRESS
+
choice
prompt "Default zram compressor"
default ZRAM_DEF_COMP_LZORLE
@@ -33,12 +39,17 @@ config ZRAM_DEF_COMP_LZO
bool "lzo"
depends on ZRAM_BACKEND_LZO
+config ZRAM_DEF_COMP_LZ4
+ bool "lz4"
+ depends on ZRAM_BACKEND_LZ4
+
endchoice
config ZRAM_DEF_COMP
string
default "lzo-rle" if ZRAM_DEF_COMP_LZORLE
default "lzo" if ZRAM_DEF_COMP_LZO
+ default "lz4" if ZRAM_DEF_COMP_LZ4
default "unset-value"
config ZRAM_WRITEBACK
diff --git a/drivers/block/zram/Makefile b/drivers/block/zram/Makefile
index 2dcbc9b75d91..1be5d2657960 100644
--- a/drivers/block/zram/Makefile
+++ b/drivers/block/zram/Makefile
@@ -1,6 +1,7 @@
# SPDX-License-Identifier: GPL-2.0-only
obj-$(CONFIG_ZRAM_BACKEND_LZO) += backend_lzorle.o backend_lzo.o
+obj-$(CONFIG_ZRAM_BACKEND_LZ4) += backend_lz4.o
zram-y := zcomp.o zram_drv.o
diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c
new file mode 100644
index 000000000000..697592dbabe2
--- /dev/null
+++ b/drivers/block/zram/backend_lz4.c
@@ -0,0 +1,47 @@
+#include <linux/kernel.h>
+#include <linux/lz4.h>
+#include <linux/vmalloc.h>
+
+#include "backend_lz4.h"
+
+static void *lz4_create(void)
+{
+ return vmalloc(LZ4_MEM_COMPRESS);
+}
+
+static void lz4_destroy(void *ctx)
+{
+ vfree(ctx);
+}
+
+static int lz4_compress(void *ctx, const unsigned char *src,
+ unsigned char *dst, size_t *dst_len)
+{
+ int ret;
+
+ ret = LZ4_compress_default(src, dst, PAGE_SIZE, *dst_len, ctx);
+ if (!ret)
+ return -EINVAL;
+ *dst_len = ret;
+ return 0;
+}
+
+static int lz4_decompress(void *ctx, const unsigned char *src,
+ size_t src_len, unsigned char *dst)
+{
+ int dst_len = PAGE_SIZE;
+ int ret;
+
+ ret = LZ4_decompress_safe(src, dst, src_len, dst_len);
+ if (ret < 0)
+ return -EINVAL;
+ return 0;
+}
+
+struct zcomp_backend backend_lz4 = {
+ .compress = lz4_compress,
+ .decompress = lz4_decompress,
+ .create_ctx = lz4_create,
+ .destroy_ctx = lz4_destroy,
+ .name = "lz4",
+};
diff --git a/drivers/block/zram/backend_lz4.h b/drivers/block/zram/backend_lz4.h
new file mode 100644
index 000000000000..a5fb5564835c
--- /dev/null
+++ b/drivers/block/zram/backend_lz4.h
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#ifndef __BACKEND_LZ4_H__
+#define __BACKEND_LZ4_H__
+
+#include "zcomp.h"
+
+extern struct zcomp_backend backend_lz4;
+
+#endif /* __BACKEND_LZ4_H__ */
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 58fb3ac91f4b..902bdaf7e299 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -17,11 +17,15 @@
#include "backend_lzo.h"
#include "backend_lzorle.h"
+#include "backend_lz4.h"
static struct zcomp_backend *backends[] = {
#if IS_ENABLED(CONFIG_ZRAM_BACKEND_LZO)
&backend_lzorle,
&backend_lzo,
+#endif
+#if IS_ENABLED(CONFIG_ZRAM_BACKEND_LZ4)
+ &backend_lz4,
#endif
NULL
};
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 04/19] zram: add lz4hc compression backend support
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (2 preceding siblings ...)
2024-05-08 7:41 ` [PATCHv3 03/19] zram: add lz4 compression backend support Sergey Senozhatsky
@ 2024-05-08 7:41 ` Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 05/19] zram: add zstd " Sergey Senozhatsky
` (15 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:41 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/Kconfig | 11 +++++
drivers/block/zram/Makefile | 5 ++-
drivers/block/zram/backend_lz4hc.c | 72 ++++++++++++++++++++++++++++++
drivers/block/zram/backend_lz4hc.h | 10 +++++
drivers/block/zram/zcomp.c | 4 ++
5 files changed, 100 insertions(+), 2 deletions(-)
create mode 100644 drivers/block/zram/backend_lz4hc.c
create mode 100644 drivers/block/zram/backend_lz4hc.h
diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index f1e76fc8431a..0d890a8d2dc4 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -26,6 +26,12 @@ config ZRAM_BACKEND_LZ4
select LZ4_COMPRESS
select LZ4_DECOMPRESS
+config ZRAM_BACKEND_LZ4HC
+ bool "lz4hc compression support"
+ depends on ZRAM
+ select LZ4HC_COMPRESS
+ select LZ4_DECOMPRESS
+
choice
prompt "Default zram compressor"
default ZRAM_DEF_COMP_LZORLE
@@ -43,6 +49,10 @@ config ZRAM_DEF_COMP_LZ4
bool "lz4"
depends on ZRAM_BACKEND_LZ4
+config ZRAM_DEF_COMP_LZ4HC
+ bool "lz4hc"
+ depends on ZRAM_BACKEND_LZ4HC
+
endchoice
config ZRAM_DEF_COMP
@@ -50,6 +60,7 @@ config ZRAM_DEF_COMP
default "lzo-rle" if ZRAM_DEF_COMP_LZORLE
default "lzo" if ZRAM_DEF_COMP_LZO
default "lz4" if ZRAM_DEF_COMP_LZ4
+ default "lz4hc" if ZRAM_DEF_COMP_LZ4HC
default "unset-value"
config ZRAM_WRITEBACK
diff --git a/drivers/block/zram/Makefile b/drivers/block/zram/Makefile
index 1be5d2657960..815b45471c7d 100644
--- a/drivers/block/zram/Makefile
+++ b/drivers/block/zram/Makefile
@@ -1,7 +1,8 @@
# SPDX-License-Identifier: GPL-2.0-only
-obj-$(CONFIG_ZRAM_BACKEND_LZO) += backend_lzorle.o backend_lzo.o
-obj-$(CONFIG_ZRAM_BACKEND_LZ4) += backend_lz4.o
+obj-$(CONFIG_ZRAM_BACKEND_LZO) += backend_lzorle.o backend_lzo.o
+obj-$(CONFIG_ZRAM_BACKEND_LZ4) += backend_lz4.o
+obj-$(CONFIG_ZRAM_BACKEND_LZ4HC) += backend_lz4hc.o
zram-y := zcomp.o zram_drv.o
diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
new file mode 100644
index 000000000000..1735a2598443
--- /dev/null
+++ b/drivers/block/zram/backend_lz4hc.c
@@ -0,0 +1,72 @@
+#include <linux/kernel.h>
+#include <linux/lz4.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+
+#include "backend_lz4hc.h"
+
+struct lz4hc_ctx {
+ void *mem;
+ s32 level;
+};
+
+static void lz4hc_destroy(void *ctx)
+{
+ struct lz4hc_ctx *zctx = ctx;
+
+ vfree(zctx->mem);
+ kfree(zctx);
+}
+
+static void *lz4hc_create(void)
+{
+ struct lz4hc_ctx *ctx;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return NULL;
+
+ ctx->mem = vmalloc(LZ4HC_MEM_COMPRESS);
+ if (!ctx->mem) {
+ lz4hc_destroy(ctx);
+ return NULL;
+ }
+
+ /* @FIXME: using a hardcoded LZ4HC_DEFAULT_CLEVEL for now */
+ ctx->level = LZ4HC_DEFAULT_CLEVEL;
+ return ctx;
+}
+
+static int lz4hc_compress(void *ctx, const unsigned char *src,
+ unsigned char *dst, size_t *dst_len)
+{
+ struct lz4hc_ctx *zctx = ctx;
+ int ret;
+
+ ret = LZ4_compress_HC(src, dst, PAGE_SIZE, *dst_len,
+ zctx->level, zctx->mem);
+ if (!ret)
+ return -EINVAL;
+ *dst_len = ret;
+ return 0;
+}
+
+static int lz4hc_decompress(void *ctx, const unsigned char *src,
+ size_t src_len, unsigned char *dst)
+{
+ int dst_len = PAGE_SIZE;
+ int ret;
+
+ ret = LZ4_decompress_safe(src, dst, src_len, dst_len);
+ if (ret < 0)
+ return -EINVAL;
+ return 0;
+}
+
+struct zcomp_backend backend_lz4hc = {
+ .compress = lz4hc_compress,
+ .decompress = lz4hc_decompress,
+ .create_ctx = lz4hc_create,
+ .destroy_ctx = lz4hc_destroy,
+ .name = "lz4hc",
+};
diff --git a/drivers/block/zram/backend_lz4hc.h b/drivers/block/zram/backend_lz4hc.h
new file mode 100644
index 000000000000..29c428a850e2
--- /dev/null
+++ b/drivers/block/zram/backend_lz4hc.h
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#ifndef __BACKEND_LZ4HC_H__
+#define __BACKEND_LZ4HC_H__
+
+#include "zcomp.h"
+
+extern struct zcomp_backend backend_lz4hc;
+
+#endif /* __BACKEND_LZ4HC_H__ */
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 902bdaf7e299..f04f5844a23c 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -18,6 +18,7 @@
#include "backend_lzo.h"
#include "backend_lzorle.h"
#include "backend_lz4.h"
+#include "backend_lz4hc.h"
static struct zcomp_backend *backends[] = {
#if IS_ENABLED(CONFIG_ZRAM_BACKEND_LZO)
@@ -26,6 +27,9 @@ static struct zcomp_backend *backends[] = {
#endif
#if IS_ENABLED(CONFIG_ZRAM_BACKEND_LZ4)
&backend_lz4,
+#endif
+#if IS_ENABLED(CONFIG_ZRAM_BACKEND_LZ4HC)
+ &backend_lz4hc,
#endif
NULL
};
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 05/19] zram: add zstd compression backend support
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (3 preceding siblings ...)
2024-05-08 7:41 ` [PATCHv3 04/19] zram: add lz4hc " Sergey Senozhatsky
@ 2024-05-08 7:41 ` Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 06/19] zram: pass estimated src size hint to zstd Sergey Senozhatsky
` (14 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:41 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/Kconfig | 11 ++++
drivers/block/zram/Makefile | 1 +
drivers/block/zram/backend_zstd.c | 97 +++++++++++++++++++++++++++++++
drivers/block/zram/backend_zstd.h | 10 ++++
drivers/block/zram/zcomp.c | 4 ++
5 files changed, 123 insertions(+)
create mode 100644 drivers/block/zram/backend_zstd.c
create mode 100644 drivers/block/zram/backend_zstd.h
diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index 0d890a8d2dc4..71cd0d5d8f35 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -32,6 +32,12 @@ config ZRAM_BACKEND_LZ4HC
select LZ4HC_COMPRESS
select LZ4_DECOMPRESS
+config ZRAM_BACKEND_ZSTD
+ bool "zstd compression support"
+ depends on ZRAM
+ select ZSTD_COMPRESS
+ select ZSTD_DECOMPRESS
+
choice
prompt "Default zram compressor"
default ZRAM_DEF_COMP_LZORLE
@@ -53,6 +59,10 @@ config ZRAM_DEF_COMP_LZ4HC
bool "lz4hc"
depends on ZRAM_BACKEND_LZ4HC
+config ZRAM_DEF_COMP_ZSTD
+ bool "zstd"
+ depends on ZRAM_BACKEND_ZSTD
+
endchoice
config ZRAM_DEF_COMP
@@ -61,6 +71,7 @@ config ZRAM_DEF_COMP
default "lzo" if ZRAM_DEF_COMP_LZO
default "lz4" if ZRAM_DEF_COMP_LZ4
default "lz4hc" if ZRAM_DEF_COMP_LZ4HC
+ default "zstd" if ZRAM_DEF_COMP_ZSTD
default "unset-value"
config ZRAM_WRITEBACK
diff --git a/drivers/block/zram/Makefile b/drivers/block/zram/Makefile
index 815b45471c7d..053fe35e346b 100644
--- a/drivers/block/zram/Makefile
+++ b/drivers/block/zram/Makefile
@@ -3,6 +3,7 @@
obj-$(CONFIG_ZRAM_BACKEND_LZO) += backend_lzorle.o backend_lzo.o
obj-$(CONFIG_ZRAM_BACKEND_LZ4) += backend_lz4.o
obj-$(CONFIG_ZRAM_BACKEND_LZ4HC) += backend_lz4hc.o
+obj-$(CONFIG_ZRAM_BACKEND_ZSTD) += backend_zstd.o
zram-y := zcomp.o zram_drv.o
diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
new file mode 100644
index 000000000000..4da49626f110
--- /dev/null
+++ b/drivers/block/zram/backend_zstd.c
@@ -0,0 +1,97 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include <linux/zstd.h>
+
+#include "backend_zstd.h"
+
+struct zstd_ctx {
+ zstd_cctx *cctx;
+ zstd_dctx *dctx;
+ void *cctx_mem;
+ void *dctx_mem;
+ s32 level;
+};
+
+static void zstd_destroy(void *ctx)
+{
+ struct zstd_ctx *zctx = ctx;
+
+ vfree(zctx->cctx_mem);
+ vfree(zctx->dctx_mem);
+ kfree(zctx);
+}
+
+static void *zstd_create(void)
+{
+ zstd_parameters params;
+ struct zstd_ctx *ctx;
+ size_t sz;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return NULL;
+
+ ctx->level = ZSTD_defaultCLevel();
+ params = zstd_get_params(ctx->level, 0);
+ sz = zstd_cctx_workspace_bound(¶ms.cParams);
+ ctx->cctx_mem = vzalloc(sz);
+ if (!ctx->cctx_mem)
+ goto error;
+
+ ctx->cctx = zstd_init_cctx(ctx->cctx_mem, sz);
+ if (!ctx->cctx)
+ goto error;
+
+ sz = zstd_dctx_workspace_bound();
+ ctx->dctx_mem = vzalloc(sz);
+ if (!ctx->dctx_mem)
+ goto error;
+
+ ctx->dctx = zstd_init_dctx(ctx->dctx_mem, sz);
+ if (!ctx->dctx)
+ goto error;
+
+ return ctx;
+
+error:
+ zstd_destroy(ctx);
+ return NULL;
+}
+
+static int zstd_compress(void *ctx, const unsigned char *src,
+ unsigned char *dst, size_t *dst_len)
+{
+ struct zstd_ctx *zctx = ctx;
+ const zstd_parameters params = zstd_get_params(zctx->level, 0);
+ size_t ret;
+
+ ret = zstd_compress_cctx(zctx->cctx, dst, *dst_len,
+ src, PAGE_SIZE, ¶ms);
+ if (zstd_is_error(ret))
+ return -EINVAL;
+ *dst_len = ret;
+ return 0;
+}
+
+static int zstd_decompress(void *ctx, const unsigned char *src, size_t src_len,
+ unsigned char *dst)
+{
+ struct zstd_ctx *zctx = ctx;
+ size_t ret;
+
+ ret = zstd_decompress_dctx(zctx->dctx, dst, PAGE_SIZE, src, src_len);
+ if (zstd_is_error(ret))
+ return -EINVAL;
+ return 0;
+}
+
+struct zcomp_backend backend_zstd = {
+ .compress = zstd_compress,
+ .decompress = zstd_decompress,
+ .create_ctx = zstd_create,
+ .destroy_ctx = zstd_destroy,
+ .name = "zstd",
+};
diff --git a/drivers/block/zram/backend_zstd.h b/drivers/block/zram/backend_zstd.h
new file mode 100644
index 000000000000..75d2d2c02768
--- /dev/null
+++ b/drivers/block/zram/backend_zstd.h
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#ifndef __BACKEND_ZSTD_H__
+#define __BACKEND_ZSTD_H__
+
+#include "zcomp.h"
+
+extern struct zcomp_backend backend_zstd;
+
+#endif /* __BACKEND_ZSTD_H__ */
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index f04f5844a23c..c16eb038f608 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -19,6 +19,7 @@
#include "backend_lzorle.h"
#include "backend_lz4.h"
#include "backend_lz4hc.h"
+#include "backend_zstd.h"
static struct zcomp_backend *backends[] = {
#if IS_ENABLED(CONFIG_ZRAM_BACKEND_LZO)
@@ -30,6 +31,9 @@ static struct zcomp_backend *backends[] = {
#endif
#if IS_ENABLED(CONFIG_ZRAM_BACKEND_LZ4HC)
&backend_lz4hc,
+#endif
+#if IS_ENABLED(CONFIG_ZRAM_BACKEND_ZSTD)
+ &backend_zstd,
#endif
NULL
};
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 06/19] zram: pass estimated src size hint to zstd
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (4 preceding siblings ...)
2024-05-08 7:41 ` [PATCHv3 05/19] zram: add zstd " Sergey Senozhatsky
@ 2024-05-08 7:41 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 07/19] zram: add zlib compression backend support Sergey Senozhatsky
` (13 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:41 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
zram works with PAGE_SIZE buffers, so we always know exact
size of the source buffer and hence can pass estimated_src_size
to zstd_get_params().
This hint on x86_64, for example, reduces the size of the work
memory buffer from 1303520 bytes down to 90080 bytes. Given that
compression streams are per-CPU that's quite some memory saving.
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/backend_zstd.c | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
index 4da49626f110..4a7734aa1a8a 100644
--- a/drivers/block/zram/backend_zstd.c
+++ b/drivers/block/zram/backend_zstd.c
@@ -35,7 +35,7 @@ static void *zstd_create(void)
return NULL;
ctx->level = ZSTD_defaultCLevel();
- params = zstd_get_params(ctx->level, 0);
+ params = zstd_get_params(ctx->level, PAGE_SIZE);
sz = zstd_cctx_workspace_bound(¶ms.cParams);
ctx->cctx_mem = vzalloc(sz);
if (!ctx->cctx_mem)
@@ -65,7 +65,7 @@ static int zstd_compress(void *ctx, const unsigned char *src,
unsigned char *dst, size_t *dst_len)
{
struct zstd_ctx *zctx = ctx;
- const zstd_parameters params = zstd_get_params(zctx->level, 0);
+ const zstd_parameters params = zstd_get_params(zctx->level, PAGE_SIZE);
size_t ret;
ret = zstd_compress_cctx(zctx->cctx, dst, *dst_len,
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 07/19] zram: add zlib compression backend support
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (5 preceding siblings ...)
2024-05-08 7:41 ` [PATCHv3 06/19] zram: pass estimated src size hint to zstd Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 08/19] zram: add 842 " Sergey Senozhatsky
` (12 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/Kconfig | 11 +++
drivers/block/zram/Makefile | 9 +-
drivers/block/zram/backend_deflate.c | 130 +++++++++++++++++++++++++++
drivers/block/zram/backend_deflate.h | 10 +++
drivers/block/zram/zcomp.c | 4 +
5 files changed, 160 insertions(+), 4 deletions(-)
create mode 100644 drivers/block/zram/backend_deflate.c
create mode 100644 drivers/block/zram/backend_deflate.h
diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index 71cd0d5d8f35..9dedd2edfb28 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -38,6 +38,12 @@ config ZRAM_BACKEND_ZSTD
select ZSTD_COMPRESS
select ZSTD_DECOMPRESS
+config ZRAM_BACKEND_DEFLATE
+ bool "deflate compression support"
+ depends on ZRAM
+ select ZLIB_DEFLATE
+ select ZLIB_INFLATE
+
choice
prompt "Default zram compressor"
default ZRAM_DEF_COMP_LZORLE
@@ -63,6 +69,10 @@ config ZRAM_DEF_COMP_ZSTD
bool "zstd"
depends on ZRAM_BACKEND_ZSTD
+config ZRAM_DEF_COMP_DEFLATE
+ bool "deflate"
+ depends on ZRAM_BACKEND_DEFLATE
+
endchoice
config ZRAM_DEF_COMP
@@ -72,6 +82,7 @@ config ZRAM_DEF_COMP
default "lz4" if ZRAM_DEF_COMP_LZ4
default "lz4hc" if ZRAM_DEF_COMP_LZ4HC
default "zstd" if ZRAM_DEF_COMP_ZSTD
+ default "deflate" if ZRAM_DEF_COMP_DEFLATE
default "unset-value"
config ZRAM_WRITEBACK
diff --git a/drivers/block/zram/Makefile b/drivers/block/zram/Makefile
index 053fe35e346b..91c07595d8b4 100644
--- a/drivers/block/zram/Makefile
+++ b/drivers/block/zram/Makefile
@@ -1,9 +1,10 @@
# SPDX-License-Identifier: GPL-2.0-only
-obj-$(CONFIG_ZRAM_BACKEND_LZO) += backend_lzorle.o backend_lzo.o
-obj-$(CONFIG_ZRAM_BACKEND_LZ4) += backend_lz4.o
-obj-$(CONFIG_ZRAM_BACKEND_LZ4HC) += backend_lz4hc.o
-obj-$(CONFIG_ZRAM_BACKEND_ZSTD) += backend_zstd.o
+obj-$(CONFIG_ZRAM_BACKEND_LZO) += backend_lzorle.o backend_lzo.o
+obj-$(CONFIG_ZRAM_BACKEND_LZ4) += backend_lz4.o
+obj-$(CONFIG_ZRAM_BACKEND_LZ4HC) += backend_lz4hc.o
+obj-$(CONFIG_ZRAM_BACKEND_ZSTD) += backend_zstd.o
+obj-$(CONFIG_ZRAM_BACKEND_DEFLATE) += backend_deflate.o
zram-y := zcomp.o zram_drv.o
diff --git a/drivers/block/zram/backend_deflate.c b/drivers/block/zram/backend_deflate.c
new file mode 100644
index 000000000000..949d402ea3dd
--- /dev/null
+++ b/drivers/block/zram/backend_deflate.c
@@ -0,0 +1,130 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include <linux/vmalloc.h>
+#include <linux/zlib.h>
+
+#include "backend_deflate.h"
+
+/* Use the same value as crypto API */
+#define DEFLATE_DEF_WINBITS 11
+#define DEFLATE_DEF_MEMLEVEL MAX_MEM_LEVEL
+
+struct deflate_ctx {
+ struct z_stream_s cctx;
+ struct z_stream_s dctx;
+ s32 level;
+};
+
+static void deflate_destroy(void *ctx)
+{
+ struct deflate_ctx *zctx = ctx;
+
+ if (zctx->cctx.workspace) {
+ zlib_deflateEnd(&zctx->cctx);
+ vfree(zctx->cctx.workspace);
+ }
+ if (zctx->dctx.workspace) {
+ zlib_inflateEnd(&zctx->dctx);
+ vfree(zctx->dctx.workspace);
+ }
+ kfree(zctx);
+}
+
+static void *deflate_create(void)
+{
+ struct deflate_ctx *ctx;
+ size_t sz;
+ int ret;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return NULL;
+
+ /* @FIXME: using a hardcoded Z_DEFAULT_COMPRESSION for now */
+ ctx->level = Z_DEFAULT_COMPRESSION;
+ sz = zlib_deflate_workspacesize(-DEFLATE_DEF_WINBITS, MAX_MEM_LEVEL);
+ ctx->cctx.workspace = vzalloc(sz);
+ if (!ctx->cctx.workspace)
+ goto error;
+
+ ret = zlib_deflateInit2(&ctx->cctx, ctx->level, Z_DEFLATED,
+ -DEFLATE_DEF_WINBITS, DEFLATE_DEF_MEMLEVEL,
+ Z_DEFAULT_STRATEGY);
+ if (ret != Z_OK)
+ goto error;
+
+ sz = zlib_inflate_workspacesize();
+ ctx->dctx.workspace = vzalloc(sz);
+ if (!ctx->dctx.workspace)
+ goto error;
+
+ ret = zlib_inflateInit2(&ctx->dctx, -DEFLATE_DEF_WINBITS);
+ if (ret != Z_OK)
+ goto error;
+
+ return ctx;
+
+error:
+ deflate_destroy(ctx);
+ return NULL;
+}
+
+static int deflate_compress(void *ctx, const unsigned char *src,
+ unsigned char *dst, size_t *dst_len)
+{
+ struct deflate_ctx *zctx = ctx;
+ struct z_stream_s *deflate;
+ int ret;
+
+ deflate = &zctx->cctx;
+ ret = zlib_deflateReset(deflate);
+ if (ret != Z_OK)
+ return -EINVAL;
+
+ deflate->next_in = (u8 *)src;
+ deflate->avail_in = PAGE_SIZE;
+ deflate->next_out = (u8 *)dst;
+ deflate->avail_out = *dst_len;
+
+ ret = zlib_deflate(deflate, Z_FINISH);
+ if (ret != Z_STREAM_END)
+ return -EINVAL;
+
+ *dst_len = deflate->total_out;
+ return 0;
+}
+
+static int deflate_decompress(void *ctx, const unsigned char *src,
+ size_t src_len, unsigned char *dst)
+{
+ struct deflate_ctx *zctx = ctx;
+ struct z_stream_s *inflate;
+ int ret;
+
+ inflate = &zctx->dctx;
+
+ ret = zlib_inflateReset(inflate);
+ if (ret != Z_OK)
+ return -EINVAL;
+
+ inflate->next_in = (u8 *)src;
+ inflate->avail_in = src_len;
+ inflate->next_out = (u8 *)dst;
+ inflate->avail_out = PAGE_SIZE;
+
+ ret = zlib_inflate(inflate, Z_SYNC_FLUSH);
+ if (ret != Z_STREAM_END)
+ return -EINVAL;
+
+ return 0;
+}
+
+struct zcomp_backend backend_deflate = {
+ .compress = deflate_compress,
+ .decompress = deflate_decompress,
+ .create_ctx = deflate_create,
+ .destroy_ctx = deflate_destroy,
+ .name = "deflate",
+};
diff --git a/drivers/block/zram/backend_deflate.h b/drivers/block/zram/backend_deflate.h
new file mode 100644
index 000000000000..49cef8fc1e77
--- /dev/null
+++ b/drivers/block/zram/backend_deflate.h
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#ifndef __BACKEND_DEFLATE_H__
+#define __BACKEND_DEFLATE_H__
+
+#include "zcomp.h"
+
+extern struct zcomp_backend backend_deflate;
+
+#endif /* __BACKEND_DEFLATE_H__ */
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index c16eb038f608..9fc5477a6259 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -20,6 +20,7 @@
#include "backend_lz4.h"
#include "backend_lz4hc.h"
#include "backend_zstd.h"
+#include "backend_deflate.h"
static struct zcomp_backend *backends[] = {
#if IS_ENABLED(CONFIG_ZRAM_BACKEND_LZO)
@@ -34,6 +35,9 @@ static struct zcomp_backend *backends[] = {
#endif
#if IS_ENABLED(CONFIG_ZRAM_BACKEND_ZSTD)
&backend_zstd,
+#endif
+#if IS_ENABLED(CONFIG_ZRAM_BACKEND_DEFLATE)
+ &backend_deflate,
#endif
NULL
};
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 08/19] zram: add 842 compression backend support
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (6 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 07/19] zram: add zlib compression backend support Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-09 12:42 ` Christoph Hellwig
2024-05-08 7:42 ` [PATCHv3 09/19] zram: check that backends array has at least one backend Sergey Senozhatsky
` (11 subsequent siblings)
19 siblings, 1 reply; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/Kconfig | 11 ++++++
drivers/block/zram/Makefile | 1 +
drivers/block/zram/backend_842.c | 68 ++++++++++++++++++++++++++++++++
drivers/block/zram/backend_842.h | 10 +++++
drivers/block/zram/zcomp.c | 4 ++
5 files changed, 94 insertions(+)
create mode 100644 drivers/block/zram/backend_842.c
create mode 100644 drivers/block/zram/backend_842.h
diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index 9dedd2edfb28..1e0e7e5910b8 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -44,6 +44,12 @@ config ZRAM_BACKEND_DEFLATE
select ZLIB_DEFLATE
select ZLIB_INFLATE
+config ZRAM_BACKEND_842
+ bool "842 compression support"
+ depends on ZRAM
+ select 842_COMPRESS
+ select 842_DECOMPRESS
+
choice
prompt "Default zram compressor"
default ZRAM_DEF_COMP_LZORLE
@@ -73,6 +79,10 @@ config ZRAM_DEF_COMP_DEFLATE
bool "deflate"
depends on ZRAM_BACKEND_DEFLATE
+config ZRAM_DEF_COMP_842
+ bool "842"
+ depends on ZRAM_BACKEND_842
+
endchoice
config ZRAM_DEF_COMP
@@ -83,6 +93,7 @@ config ZRAM_DEF_COMP
default "lz4hc" if ZRAM_DEF_COMP_LZ4HC
default "zstd" if ZRAM_DEF_COMP_ZSTD
default "deflate" if ZRAM_DEF_COMP_DEFLATE
+ default "842" if ZRAM_DEF_COMP_842
default "unset-value"
config ZRAM_WRITEBACK
diff --git a/drivers/block/zram/Makefile b/drivers/block/zram/Makefile
index 91c07595d8b4..029827a6ddac 100644
--- a/drivers/block/zram/Makefile
+++ b/drivers/block/zram/Makefile
@@ -5,6 +5,7 @@ obj-$(CONFIG_ZRAM_BACKEND_LZ4) += backend_lz4.o
obj-$(CONFIG_ZRAM_BACKEND_LZ4HC) += backend_lz4hc.o
obj-$(CONFIG_ZRAM_BACKEND_ZSTD) += backend_zstd.o
obj-$(CONFIG_ZRAM_BACKEND_DEFLATE) += backend_deflate.o
+obj-$(CONFIG_ZRAM_BACKEND_842) += backend_842.o
zram-y := zcomp.o zram_drv.o
diff --git a/drivers/block/zram/backend_842.c b/drivers/block/zram/backend_842.c
new file mode 100644
index 000000000000..8ea7a230b890
--- /dev/null
+++ b/drivers/block/zram/backend_842.c
@@ -0,0 +1,68 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#include <linux/kernel.h>
+#include <linux/slab.h>
+#include <linux/sw842.h>
+#include <linux/vmalloc.h>
+
+#include "backend_842.h"
+
+struct sw842_ctx {
+ void *mem;
+};
+
+static void destroy_842(void *ctx)
+{
+ struct sw842_ctx *zctx = ctx;
+
+ kfree(zctx->mem);
+ kfree(zctx);
+}
+
+static void *create_842(void)
+{
+ struct sw842_ctx *ctx;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return NULL;
+
+ ctx->mem = kmalloc(SW842_MEM_COMPRESS, GFP_KERNEL);
+ if (!ctx->mem)
+ goto error;
+
+ return ctx;
+
+error:
+ destroy_842(ctx);
+ return NULL;
+}
+
+static int compress_842(void *ctx, const unsigned char *src,
+ unsigned char *dst, size_t *dst_len)
+{
+ struct sw842_ctx *zctx = ctx;
+ unsigned int dlen = *dst_len;
+ int ret;
+
+ ret = sw842_compress(src, PAGE_SIZE, dst, &dlen, zctx->mem);
+ if (ret == 0)
+ *dst_len = dlen;
+ return ret;
+}
+
+static int decompress_842(void *ctx, const unsigned char *src, size_t src_len,
+ unsigned char *dst)
+{
+ unsigned int dlen = PAGE_SIZE;
+
+ return sw842_decompress(src, src_len, dst, &dlen);
+}
+
+struct zcomp_backend backend_842 = {
+ .compress = compress_842,
+ .decompress = decompress_842,
+ .create_ctx = create_842,
+ .destroy_ctx = destroy_842,
+ .name = "842",
+};
diff --git a/drivers/block/zram/backend_842.h b/drivers/block/zram/backend_842.h
new file mode 100644
index 000000000000..c03a2396d7b2
--- /dev/null
+++ b/drivers/block/zram/backend_842.h
@@ -0,0 +1,10 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+
+#ifndef __BACKEND_842_H__
+#define __BACKEND_842_H__
+
+#include "zcomp.h"
+
+extern struct zcomp_backend backend_842;
+
+#endif /* __BACKEND_842_H__ */
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 9fc5477a6259..2a38126f4da3 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -21,6 +21,7 @@
#include "backend_lz4hc.h"
#include "backend_zstd.h"
#include "backend_deflate.h"
+#include "backend_842.h"
static struct zcomp_backend *backends[] = {
#if IS_ENABLED(CONFIG_ZRAM_BACKEND_LZO)
@@ -38,6 +39,9 @@ static struct zcomp_backend *backends[] = {
#endif
#if IS_ENABLED(CONFIG_ZRAM_BACKEND_DEFLATE)
&backend_deflate,
+#endif
+#if IS_ENABLED(CONFIG_ZRAM_BACKEND_842)
+ &backend_842,
#endif
NULL
};
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 09/19] zram: check that backends array has at least one backend
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (7 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 08/19] zram: add 842 " Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 10/19] zram: introduce zcomp_config structure Sergey Senozhatsky
` (10 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Make sure that backends array has anything apart from the
sentinel NULL value.
We also select LZO_BACKEND if none backends were selected.
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/Kconfig | 19 +++++++++++++------
drivers/block/zram/zcomp.c | 8 ++++++++
2 files changed, 21 insertions(+), 6 deletions(-)
diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index 1e0e7e5910b8..6aea609b795c 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -14,12 +14,6 @@ config ZRAM
See Documentation/admin-guide/blockdev/zram.rst for more information.
-config ZRAM_BACKEND_LZO
- bool "lzo and lzo-rle compression support"
- depends on ZRAM
- select LZO_COMPRESS
- select LZO_DECOMPRESS
-
config ZRAM_BACKEND_LZ4
bool "lz4 compression support"
depends on ZRAM
@@ -50,6 +44,19 @@ config ZRAM_BACKEND_842
select 842_COMPRESS
select 842_DECOMPRESS
+config ZRAM_BACKEND_FORCE_LZO
+ depends on ZRAM
+ def_bool !ZRAM_BACKEND_LZ4 && !ZRAM_BACKEND_LZ4HC && \
+ !ZRAM_BACKEND_ZSTD && !ZRAM_BACKEND_DEFLATE && \
+ !ZRAM_BACKEND_842
+
+config ZRAM_BACKEND_LZO
+ bool "lzo and lzo-rle compression support" if !ZRAM_BACKEND_FORCE_LZO
+ depends on ZRAM
+ default ZRAM_BACKEND_FORCE_LZO
+ select LZO_COMPRESS
+ select LZO_DECOMPRESS
+
choice
prompt "Default zram compressor"
default ZRAM_DEF_COMP_LZORLE
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 2a38126f4da3..d49791f724e9 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -209,6 +209,14 @@ struct zcomp *zcomp_create(const char *alg)
struct zcomp *comp;
int error;
+ /*
+ * The backends array has a sentinel NULL value, so the minimum
+ * size is 1. In order to be valid the array, apart from the
+ * sentinel NULL element, should have at least one compression
+ * backend selected.
+ */
+ BUILD_BUG_ON(ARRAY_SIZE(backends) <= 1);
+
comp = kzalloc(sizeof(struct zcomp), GFP_KERNEL);
if (!comp)
return ERR_PTR(-ENOMEM);
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 10/19] zram: introduce zcomp_config structure
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (8 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 09/19] zram: check that backends array has at least one backend Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 11/19] zram: extend comp_algorithm attr write handling Sergey Senozhatsky
` (9 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
We will store a per-algorithm parameters there (compression level,
dictionary, dictionary size, etc.).
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/backend_842.c | 2 +-
drivers/block/zram/backend_deflate.c | 9 ++++++---
drivers/block/zram/backend_lz4.c | 2 +-
drivers/block/zram/backend_lz4hc.c | 10 ++++++----
drivers/block/zram/backend_lzo.c | 2 +-
drivers/block/zram/backend_lzorle.c | 2 +-
drivers/block/zram/backend_zstd.c | 8 ++++++--
drivers/block/zram/zcomp.c | 5 +++--
drivers/block/zram/zcomp.h | 14 ++++++++++++--
drivers/block/zram/zram_drv.c | 20 +++++++++++++++++++-
drivers/block/zram/zram_drv.h | 1 +
11 files changed, 57 insertions(+), 18 deletions(-)
diff --git a/drivers/block/zram/backend_842.c b/drivers/block/zram/backend_842.c
index 8ea7a230b890..12e716deb763 100644
--- a/drivers/block/zram/backend_842.c
+++ b/drivers/block/zram/backend_842.c
@@ -19,7 +19,7 @@ static void destroy_842(void *ctx)
kfree(zctx);
}
-static void *create_842(void)
+static void *create_842(struct zcomp_config *config)
{
struct sw842_ctx *ctx;
diff --git a/drivers/block/zram/backend_deflate.c b/drivers/block/zram/backend_deflate.c
index 949d402ea3dd..83c660adc722 100644
--- a/drivers/block/zram/backend_deflate.c
+++ b/drivers/block/zram/backend_deflate.c
@@ -32,7 +32,7 @@ static void deflate_destroy(void *ctx)
kfree(zctx);
}
-static void *deflate_create(void)
+static void *deflate_create(struct zcomp_config *config)
{
struct deflate_ctx *ctx;
size_t sz;
@@ -42,8 +42,11 @@ static void *deflate_create(void)
if (!ctx)
return NULL;
- /* @FIXME: using a hardcoded Z_DEFAULT_COMPRESSION for now */
- ctx->level = Z_DEFAULT_COMPRESSION;
+ if (config->level != ZCOMP_CONFIG_NO_LEVEL)
+ ctx->level = config->level;
+ else
+ ctx->level = Z_DEFAULT_COMPRESSION;
+
sz = zlib_deflate_workspacesize(-DEFLATE_DEF_WINBITS, MAX_MEM_LEVEL);
ctx->cctx.workspace = vzalloc(sz);
if (!ctx->cctx.workspace)
diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c
index 697592dbabe2..98d9c9274149 100644
--- a/drivers/block/zram/backend_lz4.c
+++ b/drivers/block/zram/backend_lz4.c
@@ -4,7 +4,7 @@
#include "backend_lz4.h"
-static void *lz4_create(void)
+static void *lz4_create(struct zcomp_config *config)
{
return vmalloc(LZ4_MEM_COMPRESS);
}
diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
index 1735a2598443..e1069f185437 100644
--- a/drivers/block/zram/backend_lz4hc.c
+++ b/drivers/block/zram/backend_lz4hc.c
@@ -18,7 +18,7 @@ static void lz4hc_destroy(void *ctx)
kfree(zctx);
}
-static void *lz4hc_create(void)
+static void *lz4hc_create(struct zcomp_config *config)
{
struct lz4hc_ctx *ctx;
@@ -26,14 +26,16 @@ static void *lz4hc_create(void)
if (!ctx)
return NULL;
+ if (config->level != ZCOMP_CONFIG_NO_LEVEL)
+ ctx->level = config->level;
+ else
+ ctx->level = LZ4HC_DEFAULT_CLEVEL;
+
ctx->mem = vmalloc(LZ4HC_MEM_COMPRESS);
if (!ctx->mem) {
lz4hc_destroy(ctx);
return NULL;
}
-
- /* @FIXME: using a hardcoded LZ4HC_DEFAULT_CLEVEL for now */
- ctx->level = LZ4HC_DEFAULT_CLEVEL;
return ctx;
}
diff --git a/drivers/block/zram/backend_lzo.c b/drivers/block/zram/backend_lzo.c
index b88b408964cd..79ecfae9c4c5 100644
--- a/drivers/block/zram/backend_lzo.c
+++ b/drivers/block/zram/backend_lzo.c
@@ -6,7 +6,7 @@
#include "backend_lzo.h"
-static void *lzo_create(void)
+static void *lzo_create(struct zcomp_config *config)
{
return kzalloc(LZO1X_MEM_COMPRESS, GFP_KERNEL);
}
diff --git a/drivers/block/zram/backend_lzorle.c b/drivers/block/zram/backend_lzorle.c
index 9bf1843021b0..b0937103b5fb 100644
--- a/drivers/block/zram/backend_lzorle.c
+++ b/drivers/block/zram/backend_lzorle.c
@@ -6,7 +6,7 @@
#include "backend_lzorle.h"
-static void *lzorle_create(void)
+static void *lzorle_create(struct zcomp_config *config)
{
return kzalloc(LZO1X_MEM_COMPRESS, GFP_KERNEL);
}
diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
index 4a7734aa1a8a..b2fb94902bef 100644
--- a/drivers/block/zram/backend_zstd.c
+++ b/drivers/block/zram/backend_zstd.c
@@ -24,7 +24,7 @@ static void zstd_destroy(void *ctx)
kfree(zctx);
}
-static void *zstd_create(void)
+static void *zstd_create(struct zcomp_config *config)
{
zstd_parameters params;
struct zstd_ctx *ctx;
@@ -34,7 +34,11 @@ static void *zstd_create(void)
if (!ctx)
return NULL;
- ctx->level = ZSTD_defaultCLevel();
+ if (config->level != ZCOMP_CONFIG_NO_LEVEL)
+ ctx->level = config->level;
+ else
+ ctx->level = ZSTD_defaultCLevel();
+
params = zstd_get_params(ctx->level, PAGE_SIZE);
sz = zstd_cctx_workspace_bound(¶ms.cParams);
ctx->cctx_mem = vzalloc(sz);
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index d49791f724e9..3f990a715487 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -61,7 +61,7 @@ static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm)
*/
static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm)
{
- zstrm->ctx = comp->backend->create_ctx();
+ zstrm->ctx = comp->backend->create_ctx(comp->config);
/*
* allocate 2 pages. 1 for compressed data, plus 1 extra for the
@@ -204,7 +204,7 @@ void zcomp_destroy(struct zcomp *comp)
kfree(comp);
}
-struct zcomp *zcomp_create(const char *alg)
+struct zcomp *zcomp_create(const char *alg, struct zcomp_config *config)
{
struct zcomp *comp;
int error;
@@ -221,6 +221,7 @@ struct zcomp *zcomp_create(const char *alg)
if (!comp)
return ERR_PTR(-ENOMEM);
+ comp->config = config;
comp->backend = lookup_backend(alg);
if (!comp->backend) {
kfree(comp);
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index 757b85017e23..345c78bc76db 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -5,6 +5,7 @@
#ifndef _ZCOMP_H_
#define _ZCOMP_H_
+
#include <linux/local_lock.h>
struct zcomp_strm {
@@ -15,6 +16,14 @@ struct zcomp_strm {
void *ctx;
};
+#define ZCOMP_CONFIG_NO_LEVEL INT_MIN
+
+struct zcomp_config {
+ s32 level;
+ size_t dict_sz;
+ void *dict;
+};
+
struct zcomp_backend {
int (*compress)(void *ctx, const unsigned char *src,
unsigned char *dst, size_t *dst_len);
@@ -22,7 +31,7 @@ struct zcomp_backend {
int (*decompress)(void *ctx, const unsigned char *src, size_t src_len,
unsigned char *dst);
- void *(*create_ctx)(void);
+ void *(*create_ctx)(struct zcomp_config *config);
void (*destroy_ctx)(void *ctx);
const char *name;
@@ -32,6 +41,7 @@ struct zcomp_backend {
struct zcomp {
struct zcomp_strm __percpu *stream;
struct zcomp_backend *backend;
+ struct zcomp_config *config;
struct hlist_node node;
};
@@ -40,7 +50,7 @@ int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node);
ssize_t zcomp_available_show(const char *comp, char *buf);
bool zcomp_available_algorithm(const char *comp);
-struct zcomp *zcomp_create(const char *alg);
+struct zcomp *zcomp_create(const char *alg, struct zcomp_config *config);
void zcomp_destroy(struct zcomp *comp);
struct zcomp_strm *zcomp_stream_get(struct zcomp *comp);
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 298ef0e97e03..6c36cd349431 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1994,6 +1994,20 @@ static void zram_slot_free_notify(struct block_device *bdev,
zram_slot_unlock(zram, index);
}
+static void zram_reset_comp_configs(struct zram *zram)
+{
+ u32 prio;
+
+ for (prio = 0; prio < ZRAM_MAX_COMPS; prio++) {
+ struct zcomp_config *config = &zram->configs[prio];
+
+ vfree(config->dict);
+ config->level = ZCOMP_CONFIG_NO_LEVEL;
+ config->dict_sz = 0;
+ config->dict = NULL;
+ }
+}
+
static void zram_destroy_comps(struct zram *zram)
{
u32 prio;
@@ -2007,6 +2021,8 @@ static void zram_destroy_comps(struct zram *zram)
zcomp_destroy(comp);
zram->num_active_comps--;
}
+
+ zram_reset_comp_configs(zram);
}
static void zram_reset_device(struct zram *zram)
@@ -2064,7 +2080,8 @@ static ssize_t disksize_store(struct device *dev,
if (!zram->comp_algs[prio])
continue;
- comp = zcomp_create(zram->comp_algs[prio]);
+ comp = zcomp_create(zram->comp_algs[prio],
+ &zram->configs[prio]);
if (IS_ERR(comp)) {
pr_err("Cannot initialise %s compressing backend\n",
zram->comp_algs[prio]);
@@ -2271,6 +2288,7 @@ static int zram_add(void)
if (ret)
goto out_cleanup_disk;
+ zram_reset_comp_configs(zram);
comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor);
zram_debugfs_register(zram);
diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h
index 37bf29f34d26..2c77f3bff166 100644
--- a/drivers/block/zram/zram_drv.h
+++ b/drivers/block/zram/zram_drv.h
@@ -107,6 +107,7 @@ struct zram {
struct zram_table_entry *table;
struct zs_pool *mem_pool;
struct zcomp *comps[ZRAM_MAX_COMPS];
+ struct zcomp_config configs[ZRAM_MAX_COMPS];
struct gendisk *disk;
/* Prevent concurrent execution of device init */
struct rw_semaphore init_lock;
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 11/19] zram: extend comp_algorithm attr write handling
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (9 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 10/19] zram: introduce zcomp_config structure Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 12/19] zram: support compression level comp config Sergey Senozhatsky
` (8 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Previously comp_algorithm device attr would accept only
algorithm name param, however in order to enabled comp
configuration we need to extend comp_algorithm_store()
with param=value support.
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/zram_drv.c | 26 +++++++++++++++++++++++++-
1 file changed, 25 insertions(+), 1 deletion(-)
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 6c36cd349431..bd8433363cbe 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1028,9 +1028,33 @@ static ssize_t comp_algorithm_store(struct device *dev,
size_t len)
{
struct zram *zram = dev_to_zram(dev);
+ char *args, *param, *val;
+ char *alg = NULL;
int ret;
- ret = __comp_algorithm_store(zram, ZRAM_PRIMARY_COMP, buf);
+ args = skip_spaces(buf);
+ while (*args) {
+ args = next_arg(args, ¶m, &val);
+
+ /*
+ * We need to support 'param' without value, which is an
+ * old format for this attr (algorithm name only).
+ */
+ if (!val || !*val) {
+ alg = param;
+ continue;
+ }
+
+ if (!strcmp(param, "algo")) {
+ alg = val;
+ continue;
+ }
+ }
+
+ if (!alg)
+ return -EINVAL;
+
+ ret = __comp_algorithm_store(zram, ZRAM_PRIMARY_COMP, alg);
return ret ? ret : len;
}
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 12/19] zram: support compression level comp config
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (10 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 11/19] zram: extend comp_algorithm attr write handling Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 13/19] zram: add support for dict " Sergey Senozhatsky
` (7 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Add support for compression level level=N comp configuration
to comp_algorithm and recomp_algorithm knobs.
Note that zram cannot verify ranges, it's a task of
corresponding backends to make sure that level makes
sense.
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/zram_drv.c | 30 ++++++++++++++++++++++++++++--
1 file changed, 28 insertions(+), 2 deletions(-)
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index bd8433363cbe..89a2eb37e26c 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1013,6 +1013,12 @@ static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf)
return 0;
}
+static int comp_config_store(struct zram *zram, u32 prio, s32 level)
+{
+ zram->configs[prio].level = level;
+ return 0;
+}
+
static ssize_t comp_algorithm_show(struct device *dev,
struct device_attribute *attr,
char *buf)
@@ -1030,6 +1036,7 @@ static ssize_t comp_algorithm_store(struct device *dev,
struct zram *zram = dev_to_zram(dev);
char *args, *param, *val;
char *alg = NULL;
+ s32 level = ZCOMP_CONFIG_NO_LEVEL;
int ret;
args = skip_spaces(buf);
@@ -1049,12 +1056,21 @@ static ssize_t comp_algorithm_store(struct device *dev,
alg = val;
continue;
}
+
+ if (!strcmp(param, "level")) {
+ ret = kstrtoint(val, 10, &level);
+ if (ret)
+ return ret;
+ continue;
+ }
}
if (!alg)
return -EINVAL;
- ret = __comp_algorithm_store(zram, ZRAM_PRIMARY_COMP, alg);
+ ret = comp_config_store(zram, ZRAM_PRIMARY_COMP, level);
+ if (!ret)
+ ret = __comp_algorithm_store(zram, ZRAM_PRIMARY_COMP, alg);
return ret ? ret : len;
}
@@ -1087,6 +1103,7 @@ static ssize_t recomp_algorithm_store(struct device *dev,
int prio = ZRAM_SECONDARY_COMP;
char *args, *param, *val;
char *alg = NULL;
+ s32 level = ZCOMP_CONFIG_NO_LEVEL;
int ret;
args = skip_spaces(buf);
@@ -1107,6 +1124,13 @@ static ssize_t recomp_algorithm_store(struct device *dev,
return ret;
continue;
}
+
+ if (!strcmp(param, "level")) {
+ ret = kstrtoint(val, 10, &level);
+ if (ret)
+ return ret;
+ continue;
+ }
}
if (!alg)
@@ -1115,7 +1139,9 @@ static ssize_t recomp_algorithm_store(struct device *dev,
if (prio < ZRAM_SECONDARY_COMP || prio >= ZRAM_MAX_COMPS)
return -EINVAL;
- ret = __comp_algorithm_store(zram, prio, alg);
+ ret = comp_config_store(zram, prio, level);
+ if (!ret)
+ ret = __comp_algorithm_store(zram, prio, alg);
return ret ? ret : len;
}
#endif
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 13/19] zram: add support for dict comp config
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (11 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 12/19] zram: support compression level comp config Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 14/19] zram: add dictionary support to zstd backend Sergey Senozhatsky
` (6 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Handle dict=path param so that we can read a pre-trained
compression algorithm dictionary which we then pass to the
backend configuration.
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/zram_drv.c | 54 ++++++++++++++++++++++++++++-------
1 file changed, 43 insertions(+), 11 deletions(-)
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 89a2eb37e26c..863e4e125eaa 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -33,6 +33,7 @@
#include <linux/debugfs.h>
#include <linux/cpuhotplug.h>
#include <linux/part_stat.h>
+#include <linux/kernel_read_file.h>
#include "zram_drv.h"
@@ -1013,9 +1014,35 @@ static int __comp_algorithm_store(struct zram *zram, u32 prio, const char *buf)
return 0;
}
-static int comp_config_store(struct zram *zram, u32 prio, s32 level)
+static void __reset_comp_config(struct zram *zram, u32 prio)
{
+ struct zcomp_config *config = &zram->configs[prio];
+
+ vfree(config->dict);
+ config->level = ZCOMP_CONFIG_NO_LEVEL;
+ config->dict_sz = 0;
+ config->dict = NULL;
+}
+
+static int comp_config_store(struct zram *zram, u32 prio, s32 level,
+ const char *dict_path)
+{
+ ssize_t sz = 0;
+
+ __reset_comp_config(zram, prio);
+
+ if (dict_path) {
+ sz = kernel_read_file_from_path(dict_path, 0,
+ &zram->configs[prio].dict,
+ INT_MAX,
+ NULL,
+ READING_POLICY);
+ if (sz < 0)
+ return -EINVAL;
+ }
+
zram->configs[prio].level = level;
+ zram->configs[prio].dict_sz = sz;
return 0;
}
@@ -1035,7 +1062,7 @@ static ssize_t comp_algorithm_store(struct device *dev,
{
struct zram *zram = dev_to_zram(dev);
char *args, *param, *val;
- char *alg = NULL;
+ char *alg = NULL, *dict_path = NULL;
s32 level = ZCOMP_CONFIG_NO_LEVEL;
int ret;
@@ -1063,12 +1090,17 @@ static ssize_t comp_algorithm_store(struct device *dev,
return ret;
continue;
}
+
+ if (!strcmp(param, "dict")) {
+ dict_path = val;
+ continue;
+ }
}
if (!alg)
return -EINVAL;
- ret = comp_config_store(zram, ZRAM_PRIMARY_COMP, level);
+ ret = comp_config_store(zram, ZRAM_PRIMARY_COMP, level, dict_path);
if (!ret)
ret = __comp_algorithm_store(zram, ZRAM_PRIMARY_COMP, alg);
return ret ? ret : len;
@@ -1102,7 +1134,7 @@ static ssize_t recomp_algorithm_store(struct device *dev,
struct zram *zram = dev_to_zram(dev);
int prio = ZRAM_SECONDARY_COMP;
char *args, *param, *val;
- char *alg = NULL;
+ char *alg = NULL, *dict_path = NULL;
s32 level = ZCOMP_CONFIG_NO_LEVEL;
int ret;
@@ -1131,6 +1163,11 @@ static ssize_t recomp_algorithm_store(struct device *dev,
return ret;
continue;
}
+
+ if (!strcmp(param, "dict")) {
+ dict_path = val;
+ continue;
+ }
}
if (!alg)
@@ -1139,7 +1176,7 @@ static ssize_t recomp_algorithm_store(struct device *dev,
if (prio < ZRAM_SECONDARY_COMP || prio >= ZRAM_MAX_COMPS)
return -EINVAL;
- ret = comp_config_store(zram, prio, level);
+ ret = comp_config_store(zram, prio, level, dict_path);
if (!ret)
ret = __comp_algorithm_store(zram, prio, alg);
return ret ? ret : len;
@@ -2049,12 +2086,7 @@ static void zram_reset_comp_configs(struct zram *zram)
u32 prio;
for (prio = 0; prio < ZRAM_MAX_COMPS; prio++) {
- struct zcomp_config *config = &zram->configs[prio];
-
- vfree(config->dict);
- config->level = ZCOMP_CONFIG_NO_LEVEL;
- config->dict_sz = 0;
- config->dict = NULL;
+ __reset_comp_config(zram, prio);
}
}
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 14/19] zram: add dictionary support to zstd backend
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (12 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 13/19] zram: add support for dict " Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 15/19] zram: add config init/release backend callbacks Sergey Senozhatsky
` (5 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
This adds support for pre-trained zstd dictionaries [1]
Dictionary is loaded once (per-config) and then loaded to Cctx
and Dctx by reference, so we don't allocate extra memory.
Regarding GFP_ATOMIC in Cctx customAlloc(), we probably would
want to do something about it. Either make sure that we always
(somehow) fully setup all Cctx contexts from non-atomic context
before we attempt to use them, come up with some sort of custom
allocator or stop calling zcomp_compress() from atomic context.
TEST
====
- default zstd
/sys/block/zram0/mm_stat
1750315008 504602831 514256896 0 514256896 1 0 34204 34204
- zstd level=7 dict=/etc/dictionary
/sys/block/zram0/mm_stat
1750310912 432540606 441712640 0 441712640 1 0 34187 34187
[1] https://github.com/facebook/zstd/blob/dev/programs/zstd.1.md#dictionary-builder
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/backend_zstd.c | 126 ++++++++++++++++++++++++------
1 file changed, 101 insertions(+), 25 deletions(-)
diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
index b2fb94902bef..006d0f40617d 100644
--- a/drivers/block/zram/backend_zstd.c
+++ b/drivers/block/zram/backend_zstd.c
@@ -12,23 +12,52 @@ struct zstd_ctx {
zstd_dctx *dctx;
void *cctx_mem;
void *dctx_mem;
+ ZSTD_customMem ctx_mem;
+ ZSTD_CDict *cdict;
+ ZSTD_DDict *ddict;
s32 level;
};
+/*
+ * Cctx allocator.customAlloc() can be called from zcomp_compress() under
+ * local-lock (per-CPU compression stream), in which case we must use
+ * GFP_ATOMIC.
+ */
+static void *zstd_ctx_alloc(void *opaque, size_t size)
+{
+ if (!preemptible())
+ return kvzalloc(size, GFP_ATOMIC);
+
+ return kvzalloc(size, GFP_KERNEL);
+}
+
+static void zstd_ctx_free(void *opaque, void *address)
+{
+ kvfree(address);
+}
+
static void zstd_destroy(void *ctx)
{
struct zstd_ctx *zctx = ctx;
- vfree(zctx->cctx_mem);
- vfree(zctx->dctx_mem);
+ if (zctx->cctx_mem)
+ vfree(zctx->cctx_mem);
+ else
+ ZSTD_freeCCtx(zctx->cctx);
+
+ if (zctx->dctx_mem)
+ vfree(zctx->dctx_mem);
+ else
+ ZSTD_freeDCtx(zctx->dctx);
+
+ ZSTD_freeCDict(zctx->cdict);
+ ZSTD_freeDDict(zctx->ddict);
kfree(zctx);
}
static void *zstd_create(struct zcomp_config *config)
{
- zstd_parameters params;
struct zstd_ctx *ctx;
- size_t sz;
ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
if (!ctx)
@@ -39,24 +68,62 @@ static void *zstd_create(struct zcomp_config *config)
else
ctx->level = ZSTD_defaultCLevel();
- params = zstd_get_params(ctx->level, PAGE_SIZE);
- sz = zstd_cctx_workspace_bound(¶ms.cParams);
- ctx->cctx_mem = vzalloc(sz);
- if (!ctx->cctx_mem)
- goto error;
-
- ctx->cctx = zstd_init_cctx(ctx->cctx_mem, sz);
- if (!ctx->cctx)
- goto error;
-
- sz = zstd_dctx_workspace_bound();
- ctx->dctx_mem = vzalloc(sz);
- if (!ctx->dctx_mem)
- goto error;
-
- ctx->dctx = zstd_init_dctx(ctx->dctx_mem, sz);
- if (!ctx->dctx)
- goto error;
+ ctx->ctx_mem.customAlloc = zstd_ctx_alloc;
+ ctx->ctx_mem.customFree = zstd_ctx_free;
+
+ if (config->dict_sz == 0) {
+ zstd_parameters params;
+ size_t sz;
+
+ params = zstd_get_params(ctx->level, PAGE_SIZE);
+ sz = zstd_cctx_workspace_bound(¶ms.cParams);
+ ctx->cctx_mem = vzalloc(sz);
+ if (!ctx->cctx_mem)
+ goto error;
+
+ ctx->cctx = zstd_init_cctx(ctx->cctx_mem, sz);
+ if (!ctx->cctx)
+ goto error;
+
+ sz = zstd_dctx_workspace_bound();
+ ctx->dctx_mem = vzalloc(sz);
+ if (!ctx->dctx_mem)
+ goto error;
+
+ ctx->dctx = zstd_init_dctx(ctx->dctx_mem, sz);
+ if (!ctx->dctx)
+ goto error;
+ } else {
+ ZSTD_compressionParameters params;
+
+ ctx->cctx = ZSTD_createCCtx_advanced(ctx->ctx_mem);
+ if (!ctx->cctx)
+ goto error;
+
+ ctx->dctx = ZSTD_createDCtx_advanced(ctx->ctx_mem);
+ if (!ctx->dctx)
+ goto error;
+
+ params = ZSTD_getCParams(ctx->level, PAGE_SIZE,
+ config->dict_sz);
+
+ ctx->cdict = ZSTD_createCDict_advanced(config->dict,
+ config->dict_sz,
+ ZSTD_dlm_byRef,
+ ZSTD_dct_auto,
+ params,
+ ctx->ctx_mem);
+ if (!ctx->cdict)
+ goto error;
+
+ ctx->ddict = ZSTD_createDDict_advanced(config->dict,
+ config->dict_sz,
+ ZSTD_dlm_byRef,
+ ZSTD_dct_auto,
+ ctx->ctx_mem);
+ if (!ctx->ddict)
+ goto error;
+ }
return ctx;
@@ -72,8 +139,12 @@ static int zstd_compress(void *ctx, const unsigned char *src,
const zstd_parameters params = zstd_get_params(zctx->level, PAGE_SIZE);
size_t ret;
- ret = zstd_compress_cctx(zctx->cctx, dst, *dst_len,
- src, PAGE_SIZE, ¶ms);
+ if (!zctx->cdict)
+ ret = zstd_compress_cctx(zctx->cctx, dst, *dst_len,
+ src, PAGE_SIZE, ¶ms);
+ else
+ ret = ZSTD_compress_usingCDict(zctx->cctx, dst, *dst_len,
+ src, PAGE_SIZE, zctx->cdict);
if (zstd_is_error(ret))
return -EINVAL;
*dst_len = ret;
@@ -86,7 +157,12 @@ static int zstd_decompress(void *ctx, const unsigned char *src, size_t src_len,
struct zstd_ctx *zctx = ctx;
size_t ret;
- ret = zstd_decompress_dctx(zctx->dctx, dst, PAGE_SIZE, src, src_len);
+ if (!zctx->ddict)
+ ret = zstd_decompress_dctx(zctx->dctx, dst, PAGE_SIZE,
+ src, src_len);
+ else
+ ret = ZSTD_decompress_usingDDict(zctx->dctx, dst, PAGE_SIZE,
+ src, src_len, zctx->ddict);
if (zstd_is_error(ret))
return -EINVAL;
return 0;
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 15/19] zram: add config init/release backend callbacks
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (13 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 14/19] zram: add dictionary support to zstd backend Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 16/19] zram: share dictionaries between per-CPU contexts Sergey Senozhatsky
` (4 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Some backends can create a backend-specific private data
for comp config, e.g. parse a dictionary and use it to
init all of the ctx-s later on. Introduce two zcomp_config
callbacks to create and destroy per-config backend private
data. This is also the place where config can be validated.
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/backend_842.c | 11 +++++++++++
drivers/block/zram/backend_deflate.c | 20 +++++++++++++++-----
drivers/block/zram/backend_lz4.c | 11 +++++++++++
drivers/block/zram/backend_lz4hc.c | 20 +++++++++++++++-----
drivers/block/zram/backend_lzo.c | 11 +++++++++++
drivers/block/zram/backend_lzorle.c | 11 +++++++++++
drivers/block/zram/backend_zstd.c | 20 +++++++++++++++-----
drivers/block/zram/zcomp.c | 6 ++++++
drivers/block/zram/zcomp.h | 4 ++++
drivers/block/zram/zram_drv.c | 3 +++
10 files changed, 102 insertions(+), 15 deletions(-)
diff --git a/drivers/block/zram/backend_842.c b/drivers/block/zram/backend_842.c
index 12e716deb763..1522aa88ac35 100644
--- a/drivers/block/zram/backend_842.c
+++ b/drivers/block/zram/backend_842.c
@@ -11,6 +11,15 @@ struct sw842_ctx {
void *mem;
};
+static int init_config_842(struct zcomp_config *config)
+{
+ return 0;
+}
+
+static void release_config_842(struct zcomp_config *config)
+{
+}
+
static void destroy_842(void *ctx)
{
struct sw842_ctx *zctx = ctx;
@@ -64,5 +73,7 @@ struct zcomp_backend backend_842 = {
.decompress = decompress_842,
.create_ctx = create_842,
.destroy_ctx = destroy_842,
+ .init_config = init_config_842,
+ .release_config = release_config_842,
.name = "842",
};
diff --git a/drivers/block/zram/backend_deflate.c b/drivers/block/zram/backend_deflate.c
index 83c660adc722..2e166b507f5a 100644
--- a/drivers/block/zram/backend_deflate.c
+++ b/drivers/block/zram/backend_deflate.c
@@ -17,6 +17,18 @@ struct deflate_ctx {
s32 level;
};
+static int deflate_init_config(struct zcomp_config *config)
+{
+ if (config->level == ZCOMP_CONFIG_NO_LEVEL)
+ config->level = Z_DEFAULT_COMPRESSION;
+
+ return 0;
+}
+
+static void deflate_release_config(struct zcomp_config *config)
+{
+}
+
static void deflate_destroy(void *ctx)
{
struct deflate_ctx *zctx = ctx;
@@ -42,11 +54,7 @@ static void *deflate_create(struct zcomp_config *config)
if (!ctx)
return NULL;
- if (config->level != ZCOMP_CONFIG_NO_LEVEL)
- ctx->level = config->level;
- else
- ctx->level = Z_DEFAULT_COMPRESSION;
-
+ ctx->level = config->level;
sz = zlib_deflate_workspacesize(-DEFLATE_DEF_WINBITS, MAX_MEM_LEVEL);
ctx->cctx.workspace = vzalloc(sz);
if (!ctx->cctx.workspace)
@@ -129,5 +137,7 @@ struct zcomp_backend backend_deflate = {
.decompress = deflate_decompress,
.create_ctx = deflate_create,
.destroy_ctx = deflate_destroy,
+ .init_config = deflate_init_config,
+ .release_config = deflate_release_config,
.name = "deflate",
};
diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c
index 98d9c9274149..560fcf139301 100644
--- a/drivers/block/zram/backend_lz4.c
+++ b/drivers/block/zram/backend_lz4.c
@@ -4,6 +4,15 @@
#include "backend_lz4.h"
+static int lz4_init_config(struct zcomp_config *config)
+{
+ return 0;
+}
+
+static void lz4_release_config(struct zcomp_config *config)
+{
+}
+
static void *lz4_create(struct zcomp_config *config)
{
return vmalloc(LZ4_MEM_COMPRESS);
@@ -43,5 +52,7 @@ struct zcomp_backend backend_lz4 = {
.decompress = lz4_decompress,
.create_ctx = lz4_create,
.destroy_ctx = lz4_destroy,
+ .init_config = lz4_init_config,
+ .release_config = lz4_release_config,
.name = "lz4",
};
diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
index e1069f185437..431a44f0fcfd 100644
--- a/drivers/block/zram/backend_lz4hc.c
+++ b/drivers/block/zram/backend_lz4hc.c
@@ -10,6 +10,18 @@ struct lz4hc_ctx {
s32 level;
};
+static int lz4hc_init_config(struct zcomp_config *config)
+{
+ if (config->level == ZCOMP_CONFIG_NO_LEVEL)
+ config->level = LZ4HC_DEFAULT_CLEVEL;
+
+ return 0;
+}
+
+static void lz4hc_release_config(struct zcomp_config *config)
+{
+}
+
static void lz4hc_destroy(void *ctx)
{
struct lz4hc_ctx *zctx = ctx;
@@ -26,11 +38,7 @@ static void *lz4hc_create(struct zcomp_config *config)
if (!ctx)
return NULL;
- if (config->level != ZCOMP_CONFIG_NO_LEVEL)
- ctx->level = config->level;
- else
- ctx->level = LZ4HC_DEFAULT_CLEVEL;
-
+ ctx->level = config->level;
ctx->mem = vmalloc(LZ4HC_MEM_COMPRESS);
if (!ctx->mem) {
lz4hc_destroy(ctx);
@@ -70,5 +78,7 @@ struct zcomp_backend backend_lz4hc = {
.decompress = lz4hc_decompress,
.create_ctx = lz4hc_create,
.destroy_ctx = lz4hc_destroy,
+ .init_config = lz4hc_init_config,
+ .release_config = lz4hc_release_config,
.name = "lz4hc",
};
diff --git a/drivers/block/zram/backend_lzo.c b/drivers/block/zram/backend_lzo.c
index 79ecfae9c4c5..75e0e3d297aa 100644
--- a/drivers/block/zram/backend_lzo.c
+++ b/drivers/block/zram/backend_lzo.c
@@ -6,6 +6,15 @@
#include "backend_lzo.h"
+static int lzo_init_config(struct zcomp_config *config)
+{
+ return 0;
+}
+
+static void lzo_release_config(struct zcomp_config *config)
+{
+}
+
static void *lzo_create(struct zcomp_config *config)
{
return kzalloc(LZO1X_MEM_COMPRESS, GFP_KERNEL);
@@ -40,5 +49,7 @@ struct zcomp_backend backend_lzo = {
.decompress = lzo_decompress,
.create_ctx = lzo_create,
.destroy_ctx = lzo_destroy,
+ .init_config = lzo_init_config,
+ .release_config = lzo_release_config,
.name = "lzo",
};
diff --git a/drivers/block/zram/backend_lzorle.c b/drivers/block/zram/backend_lzorle.c
index b0937103b5fb..d1dc2e35061b 100644
--- a/drivers/block/zram/backend_lzorle.c
+++ b/drivers/block/zram/backend_lzorle.c
@@ -6,6 +6,15 @@
#include "backend_lzorle.h"
+static int lzorle_init_config(struct zcomp_config *config)
+{
+ return 0;
+}
+
+static void lzorle_release_config(struct zcomp_config *config)
+{
+}
+
static void *lzorle_create(struct zcomp_config *config)
{
return kzalloc(LZO1X_MEM_COMPRESS, GFP_KERNEL);
@@ -40,5 +49,7 @@ struct zcomp_backend backend_lzorle = {
.decompress = lzorle_decompress,
.create_ctx = lzorle_create,
.destroy_ctx = lzorle_destroy,
+ .init_config = lzorle_init_config,
+ .release_config = lzorle_release_config,
.name = "lzo-rle",
};
diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
index 006d0f40617d..df59584b0337 100644
--- a/drivers/block/zram/backend_zstd.c
+++ b/drivers/block/zram/backend_zstd.c
@@ -36,6 +36,18 @@ static void zstd_ctx_free(void *opaque, void *address)
kvfree(address);
}
+static int zstd_init_config(struct zcomp_config *config)
+{
+ if (config->level == ZCOMP_CONFIG_NO_LEVEL)
+ config->level = ZSTD_defaultCLevel();
+
+ return 0;
+}
+
+static void zstd_release_config(struct zcomp_config *config)
+{
+}
+
static void zstd_destroy(void *ctx)
{
struct zstd_ctx *zctx = ctx;
@@ -63,11 +75,7 @@ static void *zstd_create(struct zcomp_config *config)
if (!ctx)
return NULL;
- if (config->level != ZCOMP_CONFIG_NO_LEVEL)
- ctx->level = config->level;
- else
- ctx->level = ZSTD_defaultCLevel();
-
+ ctx->level = config->level;
ctx->ctx_mem.customAlloc = zstd_ctx_alloc;
ctx->ctx_mem.customFree = zstd_ctx_free;
@@ -173,5 +181,7 @@ struct zcomp_backend backend_zstd = {
.decompress = zstd_decompress,
.create_ctx = zstd_create,
.destroy_ctx = zstd_destroy,
+ .init_config = zstd_init_config,
+ .release_config = zstd_release_config,
.name = "zstd",
};
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 3f990a715487..a7013a4b6575 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -187,18 +187,24 @@ static int zcomp_init(struct zcomp *comp)
if (!comp->stream)
return -ENOMEM;
+ ret = comp->backend->init_config(comp->config);
+ if (ret)
+ goto cleanup;
+
ret = cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node);
if (ret < 0)
goto cleanup;
return 0;
cleanup:
+ comp->backend->release_config(comp->config);
free_percpu(comp->stream);
return ret;
}
void zcomp_destroy(struct zcomp *comp)
{
+ comp->backend->release_config(comp->config);
cpuhp_state_remove_instance(CPUHP_ZCOMP_PREPARE, &comp->node);
free_percpu(comp->stream);
kfree(comp);
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index 345c78bc76db..aa604e5db7ad 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -22,6 +22,7 @@ struct zcomp_config {
s32 level;
size_t dict_sz;
void *dict;
+ void *private;
};
struct zcomp_backend {
@@ -34,6 +35,9 @@ struct zcomp_backend {
void *(*create_ctx)(struct zcomp_config *config);
void (*destroy_ctx)(void *ctx);
+ int (*init_config)(struct zcomp_config *config);
+ void (*release_config)(struct zcomp_config *config);
+
const char *name;
};
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 863e4e125eaa..d667fe2d0cd9 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1018,6 +1018,9 @@ static void __reset_comp_config(struct zram *zram, u32 prio)
{
struct zcomp_config *config = &zram->configs[prio];
+ /* config->private should be freed by the backend */
+ WARN_ON_ONCE(config->private);
+
vfree(config->dict);
config->level = ZCOMP_CONFIG_NO_LEVEL;
config->dict_sz = 0;
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 16/19] zram: share dictionaries between per-CPU contexts
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (14 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 15/19] zram: add config init/release backend callbacks Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 17/19] zram: add dictionary support to lz4 Sergey Senozhatsky
` (3 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
zstd's CDict and DDict are accessed read-only during compression
and decompression, so instead of allocation per-context dictionaries
we can create just one CDict and DDict in init_config() and make
per-CPU contexts use them. This saves quite a lot of memory: on my
system CDict requires 408128 bytes and DDict requires 27352 bytes.
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/backend_zstd.c | 106 ++++++++++++++++++++----------
1 file changed, 70 insertions(+), 36 deletions(-)
diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
index df59584b0337..99115b21c264 100644
--- a/drivers/block/zram/backend_zstd.c
+++ b/drivers/block/zram/backend_zstd.c
@@ -7,14 +7,18 @@
#include "backend_zstd.h"
+struct zstd_ctx_data {
+ ZSTD_customMem ctx_mem;
+ ZSTD_CDict *cdict;
+ ZSTD_DDict *ddict;
+};
+
struct zstd_ctx {
zstd_cctx *cctx;
zstd_dctx *dctx;
void *cctx_mem;
void *dctx_mem;
- ZSTD_customMem ctx_mem;
- ZSTD_CDict *cdict;
- ZSTD_DDict *ddict;
+ struct zstd_ctx_data *ctx_data;
s32 level;
};
@@ -38,32 +42,81 @@ static void zstd_ctx_free(void *opaque, void *address)
static int zstd_init_config(struct zcomp_config *config)
{
+ struct zstd_ctx_data *ctx_data = config->private;
+ ZSTD_compressionParameters params;
+
+ /* Already initialized */
+ if (ctx_data)
+ return 0;
+
if (config->level == ZCOMP_CONFIG_NO_LEVEL)
config->level = ZSTD_defaultCLevel();
+ if (config->dict_sz == 0)
+ return 0;
+
+ ctx_data = kzalloc(sizeof(*ctx_data), GFP_KERNEL);
+ if (!ctx_data)
+ return -ENOMEM;
+
+ ctx_data->ctx_mem.customAlloc = zstd_ctx_alloc;
+ ctx_data->ctx_mem.customFree = zstd_ctx_free;
+
+ params = ZSTD_getCParams(config->level, PAGE_SIZE, config->dict_sz);
+
+ ctx_data->cdict = ZSTD_createCDict_advanced(config->dict,
+ config->dict_sz,
+ ZSTD_dlm_byRef,
+ ZSTD_dct_auto,
+ params,
+ ctx_data->ctx_mem);
+ if (!ctx_data->cdict)
+ goto error;
+
+ ctx_data->ddict = ZSTD_createDDict_advanced(config->dict,
+ config->dict_sz,
+ ZSTD_dlm_byRef,
+ ZSTD_dct_auto,
+ ctx_data->ctx_mem);
+ if (!ctx_data->ddict)
+ goto error;
+
+ config->private = ctx_data;
return 0;
+
+error:
+ ZSTD_freeCDict(ctx_data->cdict);
+ ZSTD_freeDDict(ctx_data->ddict);
+ kfree(ctx_data);
+ return -EINVAL;
}
static void zstd_release_config(struct zcomp_config *config)
{
+ struct zstd_ctx_data *ctx_data = config->private;
+
+ if (!ctx_data)
+ return;
+
+ config->private = NULL;
+ ZSTD_freeCDict(ctx_data->cdict);
+ ZSTD_freeDDict(ctx_data->ddict);
+ kfree(ctx_data);
}
static void zstd_destroy(void *ctx)
{
struct zstd_ctx *zctx = ctx;
+ /* Don't free zctx->ctx_data, it's done in release_config() */
if (zctx->cctx_mem)
vfree(zctx->cctx_mem);
else
ZSTD_freeCCtx(zctx->cctx);
-
if (zctx->dctx_mem)
vfree(zctx->dctx_mem);
else
ZSTD_freeDCtx(zctx->dctx);
-
- ZSTD_freeCDict(zctx->cdict);
- ZSTD_freeDDict(zctx->ddict);
kfree(zctx);
}
@@ -75,9 +128,8 @@ static void *zstd_create(struct zcomp_config *config)
if (!ctx)
return NULL;
+ ctx->ctx_data = config->private;
ctx->level = config->level;
- ctx->ctx_mem.customAlloc = zstd_ctx_alloc;
- ctx->ctx_mem.customFree = zstd_ctx_free;
if (config->dict_sz == 0) {
zstd_parameters params;
@@ -102,35 +154,15 @@ static void *zstd_create(struct zcomp_config *config)
if (!ctx->dctx)
goto error;
} else {
- ZSTD_compressionParameters params;
+ struct zstd_ctx_data *ctx_data = ctx->ctx_data;
- ctx->cctx = ZSTD_createCCtx_advanced(ctx->ctx_mem);
+ ctx->cctx = ZSTD_createCCtx_advanced(ctx_data->ctx_mem);
if (!ctx->cctx)
goto error;
- ctx->dctx = ZSTD_createDCtx_advanced(ctx->ctx_mem);
+ ctx->dctx = ZSTD_createDCtx_advanced(ctx_data->ctx_mem);
if (!ctx->dctx)
goto error;
-
- params = ZSTD_getCParams(ctx->level, PAGE_SIZE,
- config->dict_sz);
-
- ctx->cdict = ZSTD_createCDict_advanced(config->dict,
- config->dict_sz,
- ZSTD_dlm_byRef,
- ZSTD_dct_auto,
- params,
- ctx->ctx_mem);
- if (!ctx->cdict)
- goto error;
-
- ctx->ddict = ZSTD_createDDict_advanced(config->dict,
- config->dict_sz,
- ZSTD_dlm_byRef,
- ZSTD_dct_auto,
- ctx->ctx_mem);
- if (!ctx->ddict)
- goto error;
}
return ctx;
@@ -144,15 +176,16 @@ static int zstd_compress(void *ctx, const unsigned char *src,
unsigned char *dst, size_t *dst_len)
{
struct zstd_ctx *zctx = ctx;
+ struct zstd_ctx_data *ctx_data = zctx->ctx_data;
const zstd_parameters params = zstd_get_params(zctx->level, PAGE_SIZE);
size_t ret;
- if (!zctx->cdict)
+ if (!ctx_data)
ret = zstd_compress_cctx(zctx->cctx, dst, *dst_len,
src, PAGE_SIZE, ¶ms);
else
ret = ZSTD_compress_usingCDict(zctx->cctx, dst, *dst_len,
- src, PAGE_SIZE, zctx->cdict);
+ src, PAGE_SIZE, ctx_data->cdict);
if (zstd_is_error(ret))
return -EINVAL;
*dst_len = ret;
@@ -163,14 +196,15 @@ static int zstd_decompress(void *ctx, const unsigned char *src, size_t src_len,
unsigned char *dst)
{
struct zstd_ctx *zctx = ctx;
+ struct zstd_ctx_data *ctx_data = zctx->ctx_data;
size_t ret;
- if (!zctx->ddict)
+ if (!ctx_data)
ret = zstd_decompress_dctx(zctx->dctx, dst, PAGE_SIZE,
src, src_len);
else
ret = ZSTD_decompress_usingDDict(zctx->dctx, dst, PAGE_SIZE,
- src, src_len, zctx->ddict);
+ src, src_len, ctx_data->ddict);
if (zstd_is_error(ret))
return -EINVAL;
return 0;
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 17/19] zram: add dictionary support to lz4
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (15 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 16/19] zram: share dictionaries between per-CPU contexts Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 18/19] zram: add dictionary support to lz4hc Sergey Senozhatsky
` (2 subsequent siblings)
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Support pre-trained dictionary param. lz4 doesn't
mandate specific format of the dictionary and even
zstd --train can be used to train a dictionary for
lz4, according to [1].
TEST
====
- default lz4
/sys/block/zram0/mm_stat
1750323200 664258735 676990976 0 676990976 2 0 34288 34288
- lz4 dict=/etc/dictionary
/sys/block/zram0/mm_stat
1750310912 620608254 632852480 0 632852480 1 0 34288 34288
[1] https://github.com/lz4/lz4/issues/557
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/backend_lz4.c | 77 +++++++++++++++++++++++++++++---
1 file changed, 71 insertions(+), 6 deletions(-)
diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c
index 560fcf139301..2c3b64058adf 100644
--- a/drivers/block/zram/backend_lz4.c
+++ b/drivers/block/zram/backend_lz4.c
@@ -1,9 +1,20 @@
#include <linux/kernel.h>
#include <linux/lz4.h>
+#include <linux/slab.h>
#include <linux/vmalloc.h>
#include "backend_lz4.h"
+struct lz4_ctx {
+ void *mem;
+ LZ4_streamDecode_t *dstrm;
+ LZ4_stream_t *cstrm;
+
+ /* Shared between C/D streams */
+ void *dict;
+ size_t dict_sz;
+};
+
static int lz4_init_config(struct zcomp_config *config)
{
return 0;
@@ -13,22 +24,65 @@ static void lz4_release_config(struct zcomp_config *config)
{
}
-static void *lz4_create(struct zcomp_config *config)
+static void lz4_destroy(void *ctx)
{
- return vmalloc(LZ4_MEM_COMPRESS);
+ struct lz4_ctx *zctx = ctx;
+
+ kfree(zctx->dstrm);
+ kfree(zctx->cstrm);
+ vfree(zctx->mem);
+ kfree(zctx);
}
-static void lz4_destroy(void *ctx)
+static void *lz4_create(struct zcomp_config *config)
{
- vfree(ctx);
+ struct lz4_ctx *ctx;
+
+ ctx = kzalloc(sizeof(*ctx), GFP_KERNEL);
+ if (!ctx)
+ return NULL;
+
+ if (!config->dict) {
+ ctx->mem = vmalloc(LZ4_MEM_COMPRESS);
+ if (!ctx->mem)
+ goto error;
+ } else {
+ ctx->dstrm = kzalloc(sizeof(*ctx->dstrm), GFP_KERNEL);
+ if (!ctx->dstrm)
+ goto error;
+
+ ctx->cstrm = kzalloc(sizeof(*ctx->cstrm), GFP_KERNEL);
+ if (!ctx->cstrm)
+ goto error;
+
+ ctx->dict = config->dict;
+ ctx->dict_sz = config->dict_sz;
+ }
+
+ return ctx;
+error:
+ lz4_destroy(ctx);
+ return NULL;
}
static int lz4_compress(void *ctx, const unsigned char *src,
unsigned char *dst, size_t *dst_len)
{
+ struct lz4_ctx *zctx = ctx;
int ret;
- ret = LZ4_compress_default(src, dst, PAGE_SIZE, *dst_len, ctx);
+ if (!zctx->cstrm) {
+ ret = LZ4_compress_default(src, dst, PAGE_SIZE, *dst_len,
+ zctx->mem);
+ } else {
+ /* Cstrm needs to be reset */
+ ret = LZ4_loadDict(zctx->cstrm, zctx->dict, zctx->dict_sz);
+ if (ret != zctx->dict_sz)
+ return -EINVAL;
+ ret = LZ4_compress_fast_continue(zctx->cstrm, src, dst,
+ PAGE_SIZE, *dst_len,
+ LZ4_ACCELERATION_DEFAULT);
+ }
if (!ret)
return -EINVAL;
*dst_len = ret;
@@ -38,10 +92,21 @@ static int lz4_compress(void *ctx, const unsigned char *src,
static int lz4_decompress(void *ctx, const unsigned char *src,
size_t src_len, unsigned char *dst)
{
+ struct lz4_ctx *zctx = ctx;
int dst_len = PAGE_SIZE;
int ret;
- ret = LZ4_decompress_safe(src, dst, src_len, dst_len);
+ if (!zctx->dstrm) {
+ ret = LZ4_decompress_safe(src, dst, src_len, dst_len);
+ } else {
+ /* Dstrm needs to be reset */
+ ret = LZ4_setStreamDecode(zctx->dstrm, zctx->dict,
+ zctx->dict_sz);
+ if (!ret)
+ return -EINVAL;
+ ret = LZ4_decompress_safe_continue(zctx->dstrm, src, dst,
+ src_len, dst_len);
+ }
if (ret < 0)
return -EINVAL;
return 0;
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 18/19] zram: add dictionary support to lz4hc
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (16 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 17/19] zram: add dictionary support to lz4 Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 19/19] Documentation/zram: add documentation for algorithm parameters Sergey Senozhatsky
2024-05-09 12:43 ` [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Christoph Hellwig
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Support pre-trained dictionary param. Just like lz4,
lz4hc doesn't mandate specific format of the dictionary
and zstd --train can be used to train a dictionary for
lz4, according to [1].
TEST
====
- default lz4hc
/sys/block/zram0/mm_stat
1750315008 609010918 621006848 0 621006848 2 0 34288 34288
- lz4hc dict=/etc/dictionary
/sys/block/zram0/mm_stat
1750319104 499629401 509485056 0 509485056 1 0 34288 34288
[1] https://github.com/lz4/lz4/issues/557
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
drivers/block/zram/backend_lz4hc.c | 58 ++++++++++++++++++++++++++----
1 file changed, 51 insertions(+), 7 deletions(-)
diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
index 431a44f0fcfd..c928f94f30df 100644
--- a/drivers/block/zram/backend_lz4hc.c
+++ b/drivers/block/zram/backend_lz4hc.c
@@ -8,6 +8,12 @@
struct lz4hc_ctx {
void *mem;
s32 level;
+ LZ4_streamDecode_t *dstrm;
+ LZ4_streamHC_t *cstrm;
+
+ /* Shared between C/D streams */
+ void *dict;
+ size_t dict_sz;
};
static int lz4hc_init_config(struct zcomp_config *config)
@@ -26,6 +32,8 @@ static void lz4hc_destroy(void *ctx)
{
struct lz4hc_ctx *zctx = ctx;
+ kfree(zctx->dstrm);
+ kfree(zctx->cstrm);
vfree(zctx->mem);
kfree(zctx);
}
@@ -39,12 +47,27 @@ static void *lz4hc_create(struct zcomp_config *config)
return NULL;
ctx->level = config->level;
- ctx->mem = vmalloc(LZ4HC_MEM_COMPRESS);
- if (!ctx->mem) {
- lz4hc_destroy(ctx);
- return NULL;
+ if (!config->dict) {
+ ctx->mem = vmalloc(LZ4HC_MEM_COMPRESS);
+ if (!ctx->mem)
+ goto error;
+ } else {
+ ctx->dstrm = kzalloc(sizeof(*ctx->dstrm), GFP_KERNEL);
+ if (!ctx->dstrm)
+ goto error;
+
+ ctx->cstrm = kzalloc(sizeof(*ctx->cstrm), GFP_KERNEL);
+ if (!ctx->cstrm)
+ goto error;
+
+ ctx->dict = config->dict;
+ ctx->dict_sz = config->dict_sz;
}
+
return ctx;
+error:
+ lz4hc_destroy(ctx);
+ return NULL;
}
static int lz4hc_compress(void *ctx, const unsigned char *src,
@@ -53,8 +76,18 @@ static int lz4hc_compress(void *ctx, const unsigned char *src,
struct lz4hc_ctx *zctx = ctx;
int ret;
- ret = LZ4_compress_HC(src, dst, PAGE_SIZE, *dst_len,
- zctx->level, zctx->mem);
+ if (!zctx->cstrm) {
+ ret = LZ4_compress_HC(src, dst, PAGE_SIZE, *dst_len,
+ zctx->level, zctx->mem);
+ } else {
+ /* Cstrm needs to be reset */
+ LZ4_resetStreamHC(zctx->cstrm, zctx->level);
+ ret = LZ4_loadDictHC(zctx->cstrm, zctx->dict, zctx->dict_sz);
+ if (ret != zctx->dict_sz)
+ return -EINVAL;
+ ret = LZ4_compress_HC_continue(zctx->cstrm, src, dst,
+ PAGE_SIZE, *dst_len);
+ }
if (!ret)
return -EINVAL;
*dst_len = ret;
@@ -64,10 +97,21 @@ static int lz4hc_compress(void *ctx, const unsigned char *src,
static int lz4hc_decompress(void *ctx, const unsigned char *src,
size_t src_len, unsigned char *dst)
{
+ struct lz4hc_ctx *zctx = ctx;
int dst_len = PAGE_SIZE;
int ret;
- ret = LZ4_decompress_safe(src, dst, src_len, dst_len);
+ if (!zctx->dstrm) {
+ ret = LZ4_decompress_safe(src, dst, src_len, dst_len);
+ } else {
+ /* Dstrm needs to be reset */
+ ret = LZ4_setStreamDecode(zctx->dstrm, zctx->dict,
+ zctx->dict_sz);
+ if (!ret)
+ return -EINVAL;
+ ret = LZ4_decompress_safe_continue(zctx->dstrm, src, dst,
+ src_len, dst_len);
+ }
if (ret < 0)
return -EINVAL;
return 0;
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [PATCHv3 19/19] Documentation/zram: add documentation for algorithm parameters
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (17 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 18/19] zram: add dictionary support to lz4hc Sergey Senozhatsky
@ 2024-05-08 7:42 ` Sergey Senozhatsky
2024-05-09 12:43 ` [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Christoph Hellwig
19 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-08 7:42 UTC (permalink / raw
To: Andrew Morton, Minchan Kim; +Cc: linux-kernel, linux-block, Sergey Senozhatsky
Document brief description of compression algorithms' parameters:
compression level and pre-trained dictionary.
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
Documentation/admin-guide/blockdev/zram.rst | 38 ++++++++++++++++-----
1 file changed, 29 insertions(+), 9 deletions(-)
diff --git a/Documentation/admin-guide/blockdev/zram.rst b/Documentation/admin-guide/blockdev/zram.rst
index 091e8bb38887..58d79f9099e3 100644
--- a/Documentation/admin-guide/blockdev/zram.rst
+++ b/Documentation/admin-guide/blockdev/zram.rst
@@ -102,15 +102,26 @@ Examples::
#select lzo compression algorithm
echo lzo > /sys/block/zram0/comp_algorithm
-For the time being, the `comp_algorithm` content does not necessarily
-show every compression algorithm supported by the kernel. We keep this
-list primarily to simplify device configuration and one can configure
-a new device with a compression algorithm that is not listed in
-`comp_algorithm`. The thing is that, internally, ZRAM uses Crypto API
-and, if some of the algorithms were built as modules, it's impossible
-to list all of them using, for instance, /proc/crypto or any other
-method. This, however, has an advantage of permitting the usage of
-custom crypto compression modules (implementing S/W or H/W compression).
+For the time being, the `comp_algorithm` content shows only compression
+algorithms that are supported by zram.
+
+It is also possible to pass algorithm specific configuration parameters::
+
+ #set compression level to 8
+ echo "zstd level=8" > /sys/block/zram0/comp_algorithm
+
+Note that `comp_algorithm` also supports `algo=name` format::
+
+ #set compression level to 8
+ echo "algo=zstd level=8" > /sys/block/zram0/comp_algorithm
+
+Certain compression algorithms support pre-trained dictionaries, which
+significantly change algorithms' characteristics. In order to configure
+compression algorithm to use external pre-trained dictionary, pass full
+path to the dictionary along with other parameters::
+
+ #pass path to pre-trained dictionary
+ echo "algo=zstd dict=/etc/dictioary" > /sys/block/zram0/comp_algorithm
4) Set Disksize
===============
@@ -442,6 +453,15 @@ configuration:::
#select deflate recompression algorithm, priority 2
echo "algo=deflate priority=2" > /sys/block/zramX/recomp_algorithm
+The `recomp_algorithm` also supports algorithm configuration parameters, e.g.
+compression level and pre-trained dircionary::
+
+ #pass compression level
+ echo "algo=zstd level=8" > /sys/block/zramX/recomp_algorithm
+
+ #pass path to pre-trained dictionary
+ echo "algo=zstd dict=/etc/dictioary" > /sys/block/zramX/recomp_algorithm
+
Another device attribute that CONFIG_ZRAM_MULTI_COMP enables is recompress,
which controls recompression.
--
2.45.0.rc1.225.g2a3ae87e7f-goog
^ permalink raw reply related [flat|nested] 33+ messages in thread
* Re: [PATCHv3 02/19] zram: add lzo and lzorle compression backends support
2024-05-08 7:41 ` [PATCHv3 02/19] zram: add lzo and lzorle compression backends support Sergey Senozhatsky
@ 2024-05-09 11:23 ` kernel test robot
2024-05-10 5:33 ` Sergey Senozhatsky
0 siblings, 1 reply; 33+ messages in thread
From: kernel test robot @ 2024-05-09 11:23 UTC (permalink / raw
To: Sergey Senozhatsky, Andrew Morton, Minchan Kim
Cc: llvm, oe-kbuild-all, Linux Memory Management List, linux-kernel,
linux-block, Sergey Senozhatsky
Hi Sergey,
kernel test robot noticed the following build errors:
[auto build test ERROR on axboe-block/for-next]
[also build test ERROR on akpm-mm/mm-everything linus/master v6.9-rc7 next-20240509]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Sergey-Senozhatsky/zram-move-from-crypto-API-to-custom-comp-backends-API/20240508-154917
base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux-block.git for-next
patch link: https://lore.kernel.org/r/20240508074223.652784-3-senozhatsky%40chromium.org
patch subject: [PATCHv3 02/19] zram: add lzo and lzorle compression backends support
config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20240509/202405091921.320BxOyE-lkp@intel.com/config)
compiler: clang version 18.1.4 (https://github.com/llvm/llvm-project e6c3289804a67ea0bb6a86fadbe454dd93b8d855)
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240509/202405091921.320BxOyE-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202405091921.320BxOyE-lkp@intel.com/
All errors (new ones prefixed by >>, old ones prefixed by <<):
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/dma/dmatest.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/dma/ioat/ioatdma.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/virtio/virtio_dma_buf.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/xen/xen-evtchn.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/xen/xen-privcmd.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/tty/n_hdlc.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/tty/n_gsm.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/char/agp/intel-gtt.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/char/lp.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/char/ppdev.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/char/tlclk.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/gpu/drm/tiny/bochs.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/gpu/drm/tiny/cirrus.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/gpu/drm/i915/kvmgt.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/base/regmap/regmap-i2c.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/base/regmap/regmap-spi.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/block/brd.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/block/loop.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/dax/hmem/dax_hmem.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/dax/device_dax.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/dax/kmem.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/dax/dax_pmem.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/scsi/isci/isci.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/cdrom/cdrom.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/usb/serial/usb_debug.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/media/tuners/tda9887.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/media/rc/rc-core.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/media/dvb-frontends/au8522_decoder.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/media/dvb-frontends/mb86a16.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/media/v4l2-core/v4l2-async.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/media/v4l2-core/v4l2-fwnode.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hwmon/asus_atk0110.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/thermal/intel/intel_soc_dts_iosf.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/mmc/core/mmc_core.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/mmc/core/sdio_uart.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-a4tech.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-apple.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-aureal.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-belkin.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-cherry.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-chicony.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-cypress.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-dr.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-elecom.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-ezkey.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-gyration.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-ite.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-kensington.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-keytouch.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-kye.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-lcpower.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-lenovo.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-logitech.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-lg-g15.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-logitech-dj.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-logitech-hidpp.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-microsoft.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-monterey.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-ortek.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-pl.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-petalynx.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-primax.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-saitek.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-samsung.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-sjoy.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-speedlink.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-steelseries.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-sunplus.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-gaff.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-tmff.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-tivo.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-topseed.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-twinhan.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-xinmo.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-zpff.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-zydacron.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/hid-waltop.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hid/intel-ish-hid/intel-ishtp.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/platform/x86/intel/intel-hid.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/platform/x86/intel/intel-vbtn.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/platform/x86/intel/intel-rst.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/platform/x86/amilo-rfkill.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/platform/x86/classmate-laptop.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/ras/amd/atl/amd_atl.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/hwtracing/intel_th/intel_th_msu_sink.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/parport/parport.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/nvdimm/libnvdimm.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/nvdimm/nd_pmem.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/nvdimm/nd_btt.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/nvdimm/nd_e820.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/uio/uio.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/uio/uio_cif.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/uio/uio_aec.o
WARNING: modpost: missing MODULE_DESCRIPTION() in drivers/dca/dca.o
>> ERROR: modpost: "backend_lzorle" [drivers/block/zram/zram.ko] undefined!
>> ERROR: modpost: "backend_lzo" [drivers/block/zram/zram.ko] undefined!
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 08/19] zram: add 842 compression backend support
2024-05-08 7:42 ` [PATCHv3 08/19] zram: add 842 " Sergey Senozhatsky
@ 2024-05-09 12:42 ` Christoph Hellwig
2024-05-10 4:43 ` Sergey Senozhatsky
0 siblings, 1 reply; 33+ messages in thread
From: Christoph Hellwig @ 2024-05-09 12:42 UTC (permalink / raw
To: Sergey Senozhatsky; +Cc: Andrew Morton, Minchan Kim, linux-kernel, linux-block
On Wed, May 08, 2024 at 04:42:01PM +0900, Sergey Senozhatsky wrote:
> Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
Not a very useful commit message, made worse by the magic '842'
name. How is anyone stumbling over git-blame or looking at git
logs supposed to make sense of this?
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
` (18 preceding siblings ...)
2024-05-08 7:42 ` [PATCHv3 19/19] Documentation/zram: add documentation for algorithm parameters Sergey Senozhatsky
@ 2024-05-09 12:43 ` Christoph Hellwig
2024-05-10 5:15 ` Sergey Senozhatsky
19 siblings, 1 reply; 33+ messages in thread
From: Christoph Hellwig @ 2024-05-09 12:43 UTC (permalink / raw
To: Sergey Senozhatsky
Cc: Andrew Morton, Minchan Kim, linux-kernel, linux-block, Herbert Xu,
David S. Miller, linux-crypto
On Wed, May 08, 2024 at 04:41:53PM +0900, Sergey Senozhatsky wrote:
> This patch set moves zram from crypto API to a custom compression
> API which allows us to tune and configure compression algorithms,
> something that crypto API, unfortunately, doesn't support.
[...]
> 21 files changed, 1203 insertions(+), 111 deletions(-)
Why can't it? This is an awful lot of crazy code duplication just
to pass a few parameters.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 08/19] zram: add 842 compression backend support
2024-05-09 12:42 ` Christoph Hellwig
@ 2024-05-10 4:43 ` Sergey Senozhatsky
0 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-10 4:43 UTC (permalink / raw
To: Christoph Hellwig
Cc: Sergey Senozhatsky, Andrew Morton, Minchan Kim, linux-kernel,
linux-block
On (24/05/09 05:42), Christoph Hellwig wrote:
> On Wed, May 08, 2024 at 04:42:01PM +0900, Sergey Senozhatsky wrote:
> > Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
>
> Not a very useful commit message, made worse by the magic '842'
> name. How is anyone stumbling over git-blame or looking at git
> logs supposed to make sense of this?
I can probably add something to the commit message:
This adds s/w 842 compression support
Not sure if that's significantly more informative, but maybe.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning
2024-05-09 12:43 ` [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Christoph Hellwig
@ 2024-05-10 5:15 ` Sergey Senozhatsky
2024-05-10 7:40 ` Herbert Xu
0 siblings, 1 reply; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-10 5:15 UTC (permalink / raw
To: Christoph Hellwig
Cc: Sergey Senozhatsky, Andrew Morton, Minchan Kim, linux-kernel,
linux-block, Herbert Xu, David S. Miller, linux-crypto
On (24/05/09 05:43), Christoph Hellwig wrote:
> On Wed, May 08, 2024 at 04:41:53PM +0900, Sergey Senozhatsky wrote:
> > This patch set moves zram from crypto API to a custom compression
> > API which allows us to tune and configure compression algorithms,
> > something that crypto API, unfortunately, doesn't support.
>
> [...]
>
> > 21 files changed, 1203 insertions(+), 111 deletions(-)
>
> Why can't it?
Well, I asked crypto folks if that's doable and the only reply was
"did you try using compression libs directly". And that's not a
bad response, I take it.
The handling of parameters becomes quite intrusive very quickly.
It's not as simple as just passing a new "struct crypto_tfm" to all
sort of API abstractions that crypto has, it's a little more than that.
Just as an example. For zstd we can work in two modes
1) load the dictionary by_copy
2) load the dictionary by_ref
In (2) we need to guarantee that the dictionary memory outlives any
comp contexts, so cyrpto_tfm-s now begin to have "external" dependency.
But if we load the dictionary by_ref then what we can do is a
pre-processing of the dictionary buffer - we get CDict and DDict
pointers (specific only to zstd backend) which all contexts now can
share (contexts access C/D Dict in read-only mode). For this we need
to have a pre-processing stage somewhere in the API and keep the
"compression's backend private data" somewhere, then somehow pass it to
context cra_init and release that memory when all context were destroyed.
In zram I just went with "we do only by_ref" and handle all the
dependencies/guarantees, it's very simple because all of this stays
in zram.
But in general case, a typical crypto API usage
tfm = crypto_alloc_comp(comp->name, 0, 0);
should become much more complex. I'd say that, probably, developing
an entirely new sub-set of API would be simpler.
So I implemented a simple zram comp API. I can't tell how much effort
it'll be to handle all of this in crypto, I'm not really familiar with
crypto, and I'm not sure if crypto API folks are even interested.
> This is an awful lot of crazy code duplication just
> to pass a few parameters.
I see what you mean, but the majority of the code is unique, there
isn't too much code duplication in fact. Params handling is unique,
dictionary handling is unique, zstd implementation is entirely
different and pretty much specific to zram (we don't handle all sort
of cases that zstd API support, we focus on things that we need),
lz4/lz4hc implementations are also different, etc. etc. Things like
lzo/lzorle may count as code duplication, but those are like 20 lines
of code or maybe even less (which isn't that crazy).
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 02/19] zram: add lzo and lzorle compression backends support
2024-05-09 11:23 ` kernel test robot
@ 2024-05-10 5:33 ` Sergey Senozhatsky
0 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-10 5:33 UTC (permalink / raw
To: kernel test robot
Cc: Sergey Senozhatsky, Andrew Morton, Minchan Kim, llvm,
oe-kbuild-all, Linux Memory Management List, linux-kernel,
linux-block
On (24/05/09 19:23), kernel test robot wrote:
> config: x86_64-rhel-8.3-rust (https://download.01.org/0day-ci/archive/20240509/202405091921.320BxOyE-lkp@intel.com/config)
[..]
> >> ERROR: modpost: "backend_lzorle" [drivers/block/zram/zram.ko] undefined!
> >> ERROR: modpost: "backend_lzo" [drivers/block/zram/zram.ko] undefined!
... and that's how learned that my .config has ZRAM=y
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 03/19] zram: add lz4 compression backend support
2024-05-08 7:41 ` [PATCHv3 03/19] zram: add lz4 compression backend support Sergey Senozhatsky
@ 2024-05-10 7:00 ` Sergey Senozhatsky
0 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-10 7:00 UTC (permalink / raw
To: Sergey Senozhatsky; +Cc: Andrew Morton, Minchan Kim, linux-kernel, linux-block
On (24/05/08 16:41), Sergey Senozhatsky wrote:
> +static int lz4_compress(void *ctx, const unsigned char *src,
> + unsigned char *dst, size_t *dst_len)
> +{
> + int ret;
> +
> + ret = LZ4_compress_default(src, dst, PAGE_SIZE, *dst_len, ctx);
> + if (!ret)
> + return -EINVAL;
> + *dst_len = ret;
> + return 0;
> +}
Apparently lz4 supports compression level tunable, which is
"acceleration". I'll slightly rewrite lz4/lz4hc backends in v4.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning
2024-05-10 5:15 ` Sergey Senozhatsky
@ 2024-05-10 7:40 ` Herbert Xu
2024-05-10 8:08 ` Sergey Senozhatsky
0 siblings, 1 reply; 33+ messages in thread
From: Herbert Xu @ 2024-05-10 7:40 UTC (permalink / raw
To: Sergey Senozhatsky
Cc: Christoph Hellwig, Andrew Morton, Minchan Kim, linux-kernel,
linux-block, David S. Miller, linux-crypto
On Fri, May 10, 2024 at 02:15:09PM +0900, Sergey Senozhatsky wrote:
>
> Well, I asked crypto folks if that's doable and the only reply was
> "did you try using compression libs directly". And that's not a
> bad response, I take it.
Sorry, I've been busy but I was going to get back to you on this.
> But in general case, a typical crypto API usage
>
> tfm = crypto_alloc_comp(comp->name, 0, 0);
>
> should become much more complex. I'd say that, probably, developing
> an entirely new sub-set of API would be simpler.
We could easily add a setparams interface for acomp to support
this. The form of parameters would be specific to each individual
algorithm (but obviously all drivers for the same algorithm must
use the same format).
Let me hack something up for you.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning
2024-05-10 7:40 ` Herbert Xu
@ 2024-05-10 8:08 ` Sergey Senozhatsky
2024-05-10 8:12 ` Herbert Xu
0 siblings, 1 reply; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-10 8:08 UTC (permalink / raw
To: Herbert Xu
Cc: Sergey Senozhatsky, Christoph Hellwig, Andrew Morton, Minchan Kim,
linux-kernel, linux-block, David S. Miller, linux-crypto
On (24/05/10 15:40), Herbert Xu wrote:
> > But in general case, a typical crypto API usage
> >
> > tfm = crypto_alloc_comp(comp->name, 0, 0);
> >
> > should become much more complex. I'd say that, probably, developing
> > an entirely new sub-set of API would be simpler.
>
> We could easily add a setparams interface for acomp to support
> this. The form of parameters would be specific to each individual
> algorithm (but obviously all drivers for the same algorithm must
> use the same format).
For some algorithms params needs to be set before ctx is created.
For example zstd, crypto/zstd calls zstd_get_params(ZSTD_DEF_LEVEL, 0)
to estimate workspace size, which misses the opportunity to configure
it an way zram/zswap can benefit from, because those work with PAGE_SIZE
source buffer. So for zram zstd_get_params(ZSTD_DEF_LEVEL, PAGE_SIZE)
is much better (it saves 1.2MB per ctx, which is per-CPU in zram). Not
to mention that zstd_get_params(param->level, 0) is what we need at the
end.
And then drivers need to be re-implemented to support params. For
example, crypto/lz4 should call LZ4_compress_fast() instead of
LZ4_compress_default(), because fact() accepts compression level,
which is a tunable value.
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning
2024-05-10 8:08 ` Sergey Senozhatsky
@ 2024-05-10 8:12 ` Herbert Xu
2024-05-10 8:28 ` Sergey Senozhatsky
0 siblings, 1 reply; 33+ messages in thread
From: Herbert Xu @ 2024-05-10 8:12 UTC (permalink / raw
To: Sergey Senozhatsky
Cc: Christoph Hellwig, Andrew Morton, Minchan Kim, linux-kernel,
linux-block, David S. Miller, linux-crypto
On Fri, May 10, 2024 at 05:08:27PM +0900, Sergey Senozhatsky wrote:
>
> For some algorithms params needs to be set before ctx is created.
> For example zstd, crypto/zstd calls zstd_get_params(ZSTD_DEF_LEVEL, 0)
> to estimate workspace size, which misses the opportunity to configure
> it an way zram/zswap can benefit from, because those work with PAGE_SIZE
> source buffer. So for zram zstd_get_params(ZSTD_DEF_LEVEL, PAGE_SIZE)
> is much better (it saves 1.2MB per ctx, which is per-CPU in zram). Not
> to mention that zstd_get_params(param->level, 0) is what we need at the
> end.
For these algorithms where the overhead of allocating a default
set of parameters and then changing them on a setparam call is
too high, we could stipulate that the tfm can only be used after
a setparam call (just as we require a setkey before cipher ops).
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning
2024-05-10 8:12 ` Herbert Xu
@ 2024-05-10 8:28 ` Sergey Senozhatsky
2024-05-10 8:30 ` Herbert Xu
0 siblings, 1 reply; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-10 8:28 UTC (permalink / raw
To: Herbert Xu
Cc: Sergey Senozhatsky, Christoph Hellwig, Andrew Morton, Minchan Kim,
linux-kernel, linux-block, David S. Miller, linux-crypto
On (24/05/10 16:12), Herbert Xu wrote:
> On Fri, May 10, 2024 at 05:08:27PM +0900, Sergey Senozhatsky wrote:
> >
> > For some algorithms params needs to be set before ctx is created.
> > For example zstd, crypto/zstd calls zstd_get_params(ZSTD_DEF_LEVEL, 0)
> > to estimate workspace size, which misses the opportunity to configure
> > it an way zram/zswap can benefit from, because those work with PAGE_SIZE
> > source buffer. So for zram zstd_get_params(ZSTD_DEF_LEVEL, PAGE_SIZE)
> > is much better (it saves 1.2MB per ctx, which is per-CPU in zram). Not
> > to mention that zstd_get_params(param->level, 0) is what we need at the
> > end.
>
> For these algorithms where the overhead of allocating a default
> set of parameters and then changing them on a setparam call is
> too high, we could stipulate that the tfm can only be used after
> a setparam call (just as we require a setkey before cipher ops).
OK. I guess for drivers' params support (dictionaries handling etc.)
we take take some code from this series. You mentioned acomp, does this
mean setparam is for async compression only?
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning
2024-05-10 8:28 ` Sergey Senozhatsky
@ 2024-05-10 8:30 ` Herbert Xu
2024-05-10 8:40 ` Sergey Senozhatsky
0 siblings, 1 reply; 33+ messages in thread
From: Herbert Xu @ 2024-05-10 8:30 UTC (permalink / raw
To: Sergey Senozhatsky
Cc: Christoph Hellwig, Andrew Morton, Minchan Kim, linux-kernel,
linux-block, David S. Miller, linux-crypto
On Fri, May 10, 2024 at 05:28:50PM +0900, Sergey Senozhatsky wrote:
>
> OK. I guess for drivers' params support (dictionaries handling etc.)
> we take take some code from this series. You mentioned acomp, does this
> mean setparam is for async compression only?
It would be for both acomp and scomp. I have no intention to
add it to the legacy comp interface.
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning
2024-05-10 8:30 ` Herbert Xu
@ 2024-05-10 8:40 ` Sergey Senozhatsky
0 siblings, 0 replies; 33+ messages in thread
From: Sergey Senozhatsky @ 2024-05-10 8:40 UTC (permalink / raw
To: Herbert Xu
Cc: Sergey Senozhatsky, Christoph Hellwig, Andrew Morton, Minchan Kim,
linux-kernel, linux-block, David S. Miller, linux-crypto
On (24/05/10 16:30), Herbert Xu wrote:
> On Fri, May 10, 2024 at 05:28:50PM +0900, Sergey Senozhatsky wrote:
> >
> > OK. I guess for drivers' params support (dictionaries handling etc.)
> > we take take some code from this series. You mentioned acomp, does this
> > mean setparam is for async compression only?
>
> It would be for both acomp and scomp. I have no intention to
> add it to the legacy comp interface.
Alright, I'll wait for the patches and then will take a look
at how to use them in zram and how I can help with the drivers
(if needed).
^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2024-05-10 8:40 UTC | newest]
Thread overview: 33+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2024-05-08 7:41 [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 01/19] zram: move from crypto API to custom comp backends API Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 02/19] zram: add lzo and lzorle compression backends support Sergey Senozhatsky
2024-05-09 11:23 ` kernel test robot
2024-05-10 5:33 ` Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 03/19] zram: add lz4 compression backend support Sergey Senozhatsky
2024-05-10 7:00 ` Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 04/19] zram: add lz4hc " Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 05/19] zram: add zstd " Sergey Senozhatsky
2024-05-08 7:41 ` [PATCHv3 06/19] zram: pass estimated src size hint to zstd Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 07/19] zram: add zlib compression backend support Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 08/19] zram: add 842 " Sergey Senozhatsky
2024-05-09 12:42 ` Christoph Hellwig
2024-05-10 4:43 ` Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 09/19] zram: check that backends array has at least one backend Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 10/19] zram: introduce zcomp_config structure Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 11/19] zram: extend comp_algorithm attr write handling Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 12/19] zram: support compression level comp config Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 13/19] zram: add support for dict " Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 14/19] zram: add dictionary support to zstd backend Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 15/19] zram: add config init/release backend callbacks Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 16/19] zram: share dictionaries between per-CPU contexts Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 17/19] zram: add dictionary support to lz4 Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 18/19] zram: add dictionary support to lz4hc Sergey Senozhatsky
2024-05-08 7:42 ` [PATCHv3 19/19] Documentation/zram: add documentation for algorithm parameters Sergey Senozhatsky
2024-05-09 12:43 ` [PATCHv3 00/19] zram: convert to custom compression API and allow algorithms tuning Christoph Hellwig
2024-05-10 5:15 ` Sergey Senozhatsky
2024-05-10 7:40 ` Herbert Xu
2024-05-10 8:08 ` Sergey Senozhatsky
2024-05-10 8:12 ` Herbert Xu
2024-05-10 8:28 ` Sergey Senozhatsky
2024-05-10 8:30 ` Herbert Xu
2024-05-10 8:40 ` Sergey Senozhatsky
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for read-only IMAP folder(s) and NNTP newsgroup(s).