Fix pandas bug

[QuanMPhm]

While not entirely clear, it seems the recent Pandas relase (3.0.0)
changed read_csv() cast to pyarrow datatypes, causing an error.
Specifying the pyarrow engine seems to fix the issue

[knikolla]

I believe the right way to go about this is to explicitly specify the engine.

It seems that the error points to an issue with the default engine c and switching to pyarrow fixes it.

df = pandas.read_csv(
    location,
    engine="pyarrow",
    dtype={INVOICE_COLUMN_COST: pandas.ArrowDtype(pyarrow.decimal128(12, 2))},
)

[QuanMPhm]

@knikolla I see. That makes sense. I'll make the change too on the invoicing code later today. Out of curiosity, how did you arrive at this solution? I didn't realize engine was an option, or solution to this, at least from the googling I did. The error stack trace referred to deep pandas internals that I didn't look closely into.

[knikolla]

@knikolla I see. That makes sense. I'll make the change too on the invoicing code later today. Out of curiosity, how did you arrive at this solution? I didn't realize engine was an option, or solution to this, at least from the googling I did. The error stack trace referred to deep pandas internals that I didn't look closely into.

@QuanMPhm For obscure internal errors that require digging into documentation, I find something like Gemini to be pretty helpful (50% of the time). I pasted the stack trace and it gave me the following information.

Then I verified that by reading the pandas docs with regards to the engine option and tested the code myself.

---

The error pyarrow.lib.ArrowInvalid: Got bytestring of length 8 (expected 16) is a bit of a "low-level" protest from Arrow.

In short: the pandas C engine (the default) is trying to pass data to the Arrow dtype, but they aren't speaking the same language. The C engine processes the CSV as strings/bytes, and when it hands those bytes to the Arrow decimal handler, the internal byte-length doesn't match what a decimal128 expects.

To fix this, you need to tell pandas to use the Arrow engine for the entire reading process, not just for the final data type.

The Fix: Switch the Engine
Add engine="pyarrow" to your read_csv call. This ensures that PyArrow handles the parsing from the very first byte.

[knikolla]

Going to trial having Copilot help with code reviews.

[Copilot]

Pull request overview

Adjusts invoice CSV ingestion in the daily billable usage management command to avoid a pandas 3.0 dtype-casting regression when reading cost values.

Changes:

• Forces pandas.read_csv to use the pyarrow engine for invoice CSV parsing.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[QuanMPhm]

Interesting

[QuanMPhm]

@knikolla I'll let you resolve all the comments and merge.