Restructure Configuration Docs to separate "Getting Started", "Common Operations" and "Tuning/Debugging"

In order to make the configuration documentation easier to understand for users, I would propose to he these major sections.

• Basic Setup / Getting Started
• Common Setup Options
• Deployment Targets (K8s, Yarn, Mesos)
• State Backends
• Expert/Internal
• Deprecated Options

 

The details of what to put into these sections is as follows:

Basic Setups

This section can be manual, does not need to be auto-generated. It should also be a bit more verbose and not just list options and descriptions.

• "Master and TaskManager Processes"
  • taskmanager.memory.process.size, taskmanager.memory.flink.size, jobmanager.heap.size
  • rest.address, rest.port
  • jobmanager.rpc.address, jobmanager.rpc.port

• "Parallelism"
  • taskmanager.numberOfTaskSlots

• "Checkpointing"
  • state.backend
  • state.checkpoints.dir
  • state.savepoints.dir

• Directories
  • io.tmp.dirs
  • web.upload.dir

• Miscellaneous
  • web.submit.enable 
  • cluster.evenly-spread-out-slots

 

Setup and Common Operations

(These are options everyone may need at some point. They are not to alter behavior/debug/tune, but to configure the behavior and interaction with the environment)

Hosts and Ports

• rest.address
• rest.port
• rest.bind-port
• jobmanager.rpc.address
• jobmanager.rpc.port
• taskmanager.host
• taskmanager.rpc.port
• taskmanager.data.port

Memory Configuration

• all the FLIP-49 memory options

Fault Tolerance

• restart-strategy
• restart strategy specific options

State Backends

High Availability

• High Availability (HA)
• ZooKeeper-based HA Mode

Security

• SSL Settings
• ZooKeeper Security
• Kerberos-based Security

Metrics

File Systems

History Server

Python Runtime

Queryable State

Deployment Resource Management

• Kubernetes
• Yarn
• Mesos

 

State Backends

(these options are mainly for specific setups and performance tuning of the respective backends)

• RocksDB State Backend Options
• RocksDB Configurable Options (rename to RocksDB Native Options)
• RocksDB Native Metrics

 

Debugging & Expert Setups

(Normally, users should not need to touch any of these options)

Debugging ClassLoading

• Preamble links to https://ci.apache.org/projects/flink/flink-docs-master/monitoring/debugging_classloading.html
• classloader.resolve-order
• classloader.parent-first-patterns.default
• classloader.parent-first-patterns.additional

Debugging Fault Tolerance and Scheduling

• jobmanager.execution.failover-strategy
• heartbeat.interval
• heartbeat.timeout
• cluster.registration.*
• slot.idle.timeout
• slot.request.timeout

Full Rest Options

Full JobManager Options

Full TaskManager Options

• taskmanager.debug.memory.log
• taskmanager.debug.memory.log-interval

Resource Manager Options

Akka Options

Network Options

• Netty Shuffle Environment
• Network Communication (via Netty)

 

Deprecated

• DataSet Optimizer
• DataSet Algorithms