Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5131 Fine-grained Resource Configuration
  3. FLINK-5138

Add the method in state backend to estimate memory by hint of state size in ResourceSpec

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Workaround
    • None
    • None
    • None

    Description

      Users may specify the state size in setResources API, then the state backend would roughly estimate the different kinds of memory usages based on the state size. Currently the specific state backend (FsStateBackend, MemoryStateBackend, etc) can implement the default unknown resource for it to work around.

      There are two advantages to do this:

      • This estimate memory will be considered together when requesting resource for container, so the total memory usage will not exceed the container limit.
      • Especially for RocksDB backend, the proper memory setting for RocksDB may get better performance in read and write operation.

      Attachments

        Activity

          People

            zjwang Zhijiang
            zjwang Zhijiang
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: