Make PyDataFrame.inner_df() public

[ryansun96]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Current python bindings allow users to implement rust extension while majority of their logic stays in python. In my project, I'm trying to do the opposite: having majority of the logic in rust with some entry point in Python. The user would make a DataFrame from some raw data in Python and pass it to the rust logic for processing.

Current implementation of PyDataFrame does not allow the "real / inner" rust data frame to be accessed, because inner_df is pub(crate).

Describe the solution you'd like

Make PyDataFrame.inner_df() pub.

Describe alternatives you've considered

Open to advice if the above workflow is not recommended, as I'm quite new to datafusion and rust.

Additional context

This should not cause any breaking change since we are expanding the visibility.

[timsaucer]

I recommend reading this section of the online manual, specifically the section about the unstable ABI: https://datafusion.apache.org/python/contributor-guide/ffi.html

This is the reason we do not wish to makeinner_df pub. It is not recommended to use datafusion-python as a direct dependency. Rather, we wish users to use datafusion dependencies and work through FFI wherever possible.

This is an area of active development! We'd love to have additional people get involved.

Can you give us a little more information about what specifically you're trying to do? We may already have work underway that can help you.

[ryansun96]

Hi @timsaucer thanks for your reply! Happy to share more context:

We are rewriting a Python system in rust, in a phased approach. In simplified terms, our current system looks like this:

class BaseCls:
    def method1(df: pd.DataFrame):
        pass #Some pandas logic
    def method2():
        pass

To make the change as transparent as possible, in Phase 1, we plan to implement certain compute and memory intensive operations in rust, using datafusion, exposed via PyO3 bindings. In other words, what we are trying to do is:

class BaseCls:
    def method1(df: pd.DataFrame): #Now a native method implemented in Rust, using datafusion for various data frame operations
        
    def method2():
        pass # Same as before, implemented in Py

Thus, we would like to access & manipulate the dataframe from rust, rather than using the exposed py bindings. We will implement rust UDFs, object store registries, etc., without using those items from py at all.

What is your recommended approach for this?

[timsaucer]

It sounds like your final use case will not have datafusion-python in the loop as a Python dependency at all, correct?

[timsaucer]

Approach 1: Using df-python

• Write all of your rust code for UDFs, etc and expose via PyO3 using the FFI approach.
• Register these with df-python
• Write python style dataframe operations
• Everything still runs at full native speed during execution with zero copy between UDFs and core DF code

Approach 2: Using df-python as a rust dependency only

• We update the method to pub as requested
• You expose all of the specific endpoints you want to use to python, so probably something that is taking in that pandas dataframe in your method signature
• Everything is running in Rust and the results are sent back probably as arrow record batch or record batch stream
• You will not get any of the datafusion-python python API (but maybe not an issue)

Approach 3: Mixture of the two

• Write a table provider that takes in as an input that pandas dataframe, does all the operations you want under the hood using its own session context (not the datafusion-python session context)
• Follow on operations from the output of the table provider could use datafusion-python interfaces

[ryansun96]

final use case will not have datafusion-python in the loop as a Python dependency at all, correct?

That is correct. One detail that I did not mention in my previous post but seems to be significant based on the proposed approaches, is that in our project, we do not need to send the rust processing results back to Python, which makes Approach 2 much simpler.

With that I do think Approach 3 has value outside of our use case right now. To check my understanding, approach 3 is essentially using a table provider as a "pipe" operator that embeds custom logic, rather than simply reading & returning rows?

does all the operations you want under the hood using its own session context (not the datafusion-python session context)

Does it imply the rule of thumb that "If I want to use functions from different libraries in the same session context, those libraries need to be compiled together"? I.e. even if use datafusion-python as a py dependency, as long as we depend on source, not the artifact published to PyPI, we will be fine even if we do data transfer "naively"?

[ryansun96]

Would it be cool for me to open a PR to implement Approach 2, update the method to pub?

[timsaucer]

I'm not opposed to the PR, but have you verified locally it actually solves your problem? Also if you aren't sending anything back to Python and you're not starting with a datafusion dataframe (or are you?) then maybe all you really need is a method to get arrow record batches (streaming or otherwise) into datafusion core. You could potentially lift the appropriate snippet and reduce your compile times significantly.

[timsaucer]

https://github.com/apache/arrow-rs/blob/caf1c6022c71af00ef712e9e865acfee74169f0d/arrow-pyarrow/src/lib.rs#L461 shows how you can get arrow stream from Python, which can then be fed into a table provider in DataFusion.