Facet offset not returning enough results with solr cloud

A collection has four shards.
A field has >5K terms according to the web ui for one core of the collection.
A json facet query with limit 20 and offset 0 returns 20 buckets as expected.

The same query with an offset of 20 returns only seven more buckets.

With debugQuery= true I see that the first query with offset = 0 the four cores return 27 different vals which are merged. The top 20 are collected for the result.

With offset=20 I see the same 27 vals for the cores. But now the first 20 are skipped and seven are returned.
And with offset=40 the same is returned as for offset=20.

Example:
http://mysolrserver/solr/my_collection/select?wt=json&indent=false&q=*&rows=0&json.facet={"my_facet"%3A+

{"field"%3A+"some_field"%2C+"limit"%3A+20%2C+"offset"%3A+20%2C+"sort"%3A+"count+desc"%2C+"type"%3A+"field"}

}