Link Search Menu Expand Document

Configuration

Clonos adds a number of configuration parameters on top of base Flink, which are described below.

Deployment Parameters

Standby

parameter Description Recommended
jobmanager.execution.failover-strategy Sets the failover strategy for Clonos. Either “full” or “standbytask”. standbytask
jobmanager.execution.num-standby-tasks Sets the number of standby tasks Clonos should maintain per task. 1

In-Flight log

parameter Description Recommended
taskmanager.inflight.type Sets the type of inflight log. Either “inmemory” or “spillable” spillable
taskmanager.inflight.spill.policy Sets the spill strategy of the spillable inflight log. Either “eager”, “availability”. availability
taskmanager.infight.spill.availability-trigger Sets the spill threshold of the availability strategy. A value of 0.4 triggers a spill when only 40% of network buffers are available. 0.4
taskmanager.inflight.spill.sleep Interval in ms between checks of buffer availability 100
taskmanager.inflight.num-prefetch-buffers Number of buffers used for pre-fetching during in-fligh replay. 10
taskmanager.network.memory.sender-extra-buffers-per-channel Sets the size of the in-flight log buffer pool. For each channel add this many buffers to it. 10
taskmanager.network.memory.sender-extra-buffers-per-gate Sets the base size of the in-flight log buffer pool. 200

Causal Log Manager

parameter Description Recommended
taskmanager.network.netty.determinantMemorySteal Percentage of network (off-heap) memory stolen from the network manager for the Causal Log Manager 0.2
taskmanager.network.netty.determinantBufferSize Size of buffers used to implement causal logs. If each TaskManager must track a large number of tasks, reducing this can help in the face of OOM errors. 32768
taskmanager.network.netty.determinantBuffersPerJob How many buffers each job is allocated for causal logging. 100
taskmanager.network.netty.enableDeltaSharingOptimizations Only disabled for debugging. Avoids sharing causal logs along non-causal paths. true
taskmanager.network.netty.determinantDeltaEncodingStrategy Chooses the encoding strategy of causal logs on buffers. Either “hierarchical” or “flat”. Hierarchical strategy is more compact. hierarchical

Runtime Parameters

These parameters can be set at runtime by each stream processing job.


// The granularity with which time is measured in Clonos.
// Increasing this value results in less nondeterminism at the cost of less granular computations.
// Between 1 and 10, this is mostly unnoticeable.
env.getConfig.setAutoTimeSetterInterval(5)

// The determinant sharing depth. Use -1 for full sharing.
env.getJavaEnv.setDeterminantSharingDepth(1)

Back to top

Copyright © 2021-2021 TU Delft - Web Information Systems.