S3A FileSystem does not correctly delete directories with fake entries

# First step: Create an new and empty bucket in S3
$ aws s3 mb s3://dimajix-tmp                                                                                                                                                            make_bucket: dimajix-tmp

$ aws s3 ls
2020-08-21 11:19:50 dimajix-tmp

# Upload some data
$ aws s3 cp some_file.txt s3://dimajix-tmp/tmp/
upload: ./some_file.txt to s3://dimajix-tmp/tmp/some_file.txt

$ aws s3 ls s3://dimajix-tmp/tmp/                                                                                                                                                       2020-08-21 11:23:35          0 some_file.txt

# Check that Hadoop can list the file
$ /opt/hadoop/bin/hdfs dfs -ls s3a://dimajix-tmp/
Found 1 items
drwxrwxrwx   - kaya kaya          0 2020-08-21 11:24 s3a://dimajix-tmp/tmp

$ /opt/hadoop/bin/hdfs dfs -ls s3a://dimajix-tmp/tmp/
-rw-rw-rw-   1 kaya kaya          0 2020-08-21 11:23 s3a://dimajix-tmp/tmp/some_file.txt


# Evil step: Create fake directory entry in S3
$ aws s3api put-object --bucket dimajix-tmp --key tmp/
{
    "ETag": "\"d41d8cd98f00b204e9800998ecf8427e\""
}

# Look into S3, ensure that fake directory entry was created
$ aws s3 ls s3://dimajix-tmp/tmp/
2020-08-21 11:25:40          0
2020-08-21 11:23:35          0 some_file.txt

# Look into S3 using Hadoop CLI, ensure that everything looks okay (which is the case)
$ /opt/hadoop/bin/hdfs dfs -ls s3a://dimajix-tmp/tmp/
Found 1 items
-rw-rw-rw-   1 kaya kaya          0 2020-08-21 11:23 s3a://dimajix-tmp/tmp/some_file.txt

# Bug: Now try to delete the directory with Hadoop CLI
$ /opt/hadoop/bin/hdfs dfs -rm s3a://dimajix-tmp/tmp/
rm: `s3a://dimajix-tmp/tmp': Is a directory

# Okay, that didn't work out, Hadoop interprets the prefix as a directory (which is fine). It also did not delete anything, as we can see in S3:
$ aws s3 ls s3://dimajix-tmp/tmp/
2020-08-21 11:25:40          0
2020-08-21 11:23:35          0 some_file.txt

# Now let's try a little bit harder by trying to recursively delete the directory
$ /opt/hadoop/bin/hdfs dfs -rm -r s3a://dimajix-tmp/tmp/
Deleted s3a://dimajix-tmp/tmp

# Everything looked fine so far. But let's inspect S3 directly. We'll find that only the prefix (fake directory entry) has been removed. The file in the directory is still there.
$ aws s3 ls s3://dimajix-tmp/tmp/
2020-08-21 11:23:35          0 some_file.txt

# We can also use Hadoop CLI to check that the directory containing the file is still present, although we wanted to delete it above.
$ /opt/hadoop/bin/hdfs dfs -ls s3a://dimajix-tmp/tmp/
Found 1 items
-rw-rw-rw-   1 kaya kaya          0 2020-08-21 11:23 s3a://dimajix-tmp/tmp/some_file.txt

# Now let's perform the same action again to remove the directory and all its contents
$ /opt/hadoop/bin/hdfs dfs -rm -r s3a://dimajix-tmp/tmp/
Deleted s3a://dimajix-tmp/tmp

# Finally everything was cleaned up.
$ aws s3 ls s3://dimajix-tmp/tmp/

$ /opt/hadoop/bin/hdfs dfs -ls s3a://dimajix-tmp/tmp/
ls: `s3a://dimajix-tmp/tmp/': No such file or directory

$ /opt/hadoop/bin/hdfs dfs -ls s3a://dimajix-tmp/