Uploaded image for project: 'Apache Drill'
  1. Apache Drill
  2. DRILL-8283

Add a configurable recursive file listing size limit

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 1.20.2
    • 1.20.3
    • Storage - Other
    • None

    Description

      Currently a malicious or merely unwitting user can crash their Drill foreman by sending

      select * from dfs.huge_workspace limit 10
      

      causing the query planner to recurse over every file in huge_workspace and culminating in

      2022-08-09 15:13:22,251 [1d0da29f-e50c-fd51-43d9-8a5086d52c4e:foreman] ERROR o.a.drill.common.CatastrophicFailure - Catastrophic Failure Occurred, exiting. Information message: Unable to handle out of memory condition in Foreman.java.lang.OutOfMemoryError: null 

      if there are enough files in huge_workspace. A SHOW FILES command can produce the same effect. This issue proposes a new BOOT option named drill.exec.storage.file.recursive_listing_max_size with a default value of, say 10 000. If a file listing task exceeds this limit then the initiating operation is terminated with a UserException preventing runaway resource usage.

      Attachments

        Activity

          People

            dzamo James Turton
            dzamo James Turton
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: