multi-threading option #185
Description
We may add a support for multi-threading to lzbench in the future but it will be a new feature. Looking for volunteers :)
Had a look how is it organised and I thought - Is there a way to simply pass an argument to the de/compressor?
Maybe, instead of multiplying arguments of function, like param1, param2... paramN
inline int64_t lzbench_compress(lzbench_params_t *params, std::vector<size_t>& chunk_sizes, compress_func compress, std::vector<size_t> &compr_sizes, uint8_t *inbuf, uint8_t *outbuf, size_t outsize, size_t param1, size_t param2, char* workmem);
aggregate all those additional, optional parameters in a struct and pass a struct to the function.
inline int64_t lzbench_compress(lzbench_params_t *params, std::vector<size_t>& chunk_sizes, compress_func compress, std::vector<size_t> &compr_sizes, uint8_t *inbuf, uint8_t *outbuf, size_t outsize, struct options *codec_params);
struct codec_options { param1; param2; (...) paramN; };
It would allow to seamless extension in the future as only thing passed would be a pointer (to struct).
Also that could require to make new column called 'threads' that would be turn on in case of threads option to the program. Just to give justice to that codec and the others that don't necessarily use threads.
Well, at least to show it after header.
it's much easier to reach memory or storage limits
It the user's choice.
what skews results
Like how?
only modern compressors support multithreading
And modern hardware, which is likely to ba able to handle load and requirements of the task. Again - user's choice.
should we compare e.g. a single threaded lzf with multithreaded lz4?
If it's clearly stated it multi-threaded result it's user's choice.
even if a compressor support multithreading we will compare not just compressors but also how effective is its multi-threading implementation vs other implementations
It would only use 'internal', 'codec's implementation'. Again - user's choice.
Most, if not all, multi-threaded compressors I can think of operate on client not codec level. And I think it's a good thing.
I wouldn't worry about that here. If codec offers multi-threading, it's their job to do it efficiently.
to make testing fair we can implement multi-threading in lzbench for all compressors and disable internal multi-threading implementation (but this is a new feature)
Again, if it was clearly marked as X-threaded, then it's user's decision to make conclusions.
All in all, allowing 'codec's internal' threading is ok as long as it's clearly stated it's multi-threaded measure.