replicated_deduplication_window_* MergeTree table settings

These settings are available in system.merge_tree_settings and are autogenerated from ClickHouse source.

replicated_deduplication_window

The number of most recently inserted blocks for which ClickHouse Keeper stores hash sums to check for duplicates.

Possible values:

• Any positive integer.
• 0 (disable deduplication)

For insert deduplication, when writing into replicated tables, ClickHouse writes deduplication hash sums into ClickHouse Keeper. Hash sums are stored only for the most recent replicated_deduplication_window blocks. The oldest hash sums are removed from ClickHouse Keeper.

The hash sum covers the whole inserted block, so an insert is deduplicated only when its entire data matches a previous insert (a retry), not per individual part.

A large number for replicated_deduplication_window slows down Inserts because more entries need to be compared.

replicated_deduplication_window_for_async_inserts

Legacy setting retained for mixed-version rolling upgrades. New inserts deduplicate with the unified hash governed by replicated_deduplication_window; this setting now only bounds how many legacy async-insert hashes are kept in the async_blocks directory in ClickHouse Keeper, which is still written by older replicas during a rolling upgrade and cleaned up by the current leader. A value of 0 retains none of them — the next cleanup pass removes every entry — so keep the default (or a larger value) to preserve the legacy async deduplication window during the upgrade.

replicated_deduplication_window_seconds

The number of seconds after which the hash sums of the inserted blocks are removed from ClickHouse Keeper.

Possible values:

• Any positive integer.

Similar to replicated_deduplication_window, replicated_deduplication_window_seconds specifies how long to store hash sums of blocks for insert deduplication. Hash sums older than replicated_deduplication_window_seconds are removed from ClickHouse Keeper, even if they are less than replicated_deduplication_window.

The time is relative to the time of the most recent record, not to the wall time. If it’s the only record it will be stored forever.

replicated_deduplication_window_seconds_for_async_inserts

Legacy setting retained for mixed-version rolling upgrades. Together with replicated_deduplication_window_for_async_inserts it bounds how long legacy async-insert hashes are kept in the async_blocks directory in ClickHouse Keeper (written by older replicas during a rolling upgrade and cleaned up by the current leader). New inserts use replicated_deduplication_window_seconds.