[Python] Allow custom reader/writer implementation for arrow dataset read/write path

We're implementing a "versionable" data format where the read/write path has some metadata handling which we currently can't plug into the native pyarrow write_dataset and pa.dataset.dataset mechanism.

What we've done currently is have our own `lance.write_dataset` and `lance.dataset` interfaces which knows about the versioning. And if you use the native arrow ones, it reads/writes an unversioned dataset.

It would be great if:

1. the arrow interfaces provided a way for custom data formats to provide their own Arrow compliant reader/writer implementations, so we can delete our custom interface and stick with native pyarrow interface.

2. the pyarrow interface can support custom kwargs like "version=5" or "as_of=<timestamp>" or "version='latest'"

for reference, this is what our custom C++ dataset implementation looks like: https://github.com/eto-ai/lance/blob/main/cpp/include/lance/arrow/dataset.h