Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
None
Description
It occurred to me that we can likely improve the performance and scalability of Table.to_pandas or other to_pandas methods by using the active MemoryPool to allocate memory for the array rather than letting NumPy use the system allocator. We would need to use the PyCapsule approach to setting a shared_ptr<Buffer> as the base of the created NumPy arrays
This has the additional benefit of tracking NumPy-related allocations in the MemoryPool so we will have a more precise accounting of allocated memory.
Attachments
Issue Links
- links to