Add fuzz test, property-based test, and table test for PV function by paulzuradzki · Pull Request #100 · numpy/numpy-financial

paulzuradzki · 2024-01-07T21:11:43Z

Description

This change request adds testing to the pv / present value function.

Tests
- Adds fuzz tests and property-based tests (PBT) using Hypothesis
- Adds more example-based tests (with pytest.mark.parametrize)
- The hope is that these tests can be extended to other functions as well.
Add exception handling for InvalidOperation, TypeError, DivisionByZero, Overflow. ~~I opted to mirror numpy and return -0.0. Arguably, we would want a runtime error and can ignore it in the fuzz tests.~~

Context:

We use fuzz tests for extreme value testing (e.g., empty values, very low, very high, or zero) using a randomized input generator.
We use property-based tests to verify that certain conditions hold true across a range of inputs.
I used type hints and Hypothesis Ghostwriter CLI feature to help write the strategy.
- We can generate random np.ndarray of small dimensions, but this was a bit slow. I opted to test array-input directly (tests/test_financial.py::TestPV::test_pv_examples[rates_as_array).

Testing

If you want to select only the related tests, you can run them like so.

$ pytest tests/test_financial.py::TestPV -vv

tests/test_financial.py::TestPV::test_pv PASSED                                                                                                                                          [  9%]
tests/test_financial.py::TestPV::test_pv_decimal PASSED                                                                                                                                  [ 18%]
tests/test_financial.py::TestPV::test_pv_examples[default_fv_and_when] PASSED                                                                                                            [ 27%]
tests/test_financial.py::TestPV::test_pv_examples[specify_fv_and_when] PASSED                                                                                                            [ 36%]
tests/test_financial.py::TestPV::test_pv_examples[when_1] PASSED                                                                                                                         [ 45%]
tests/test_financial.py::TestPV::test_pv_examples[when_1_and_fv_1000] PASSED                                                                                                             [ 54%]
tests/test_financial.py::TestPV::test_pv_examples[fv>0] PASSED                                                                                                                           [ 63%]
tests/test_financial.py::TestPV::test_pv_examples[negative_rate] PASSED                                                                                                                  [ 72%]
tests/test_financial.py::TestPV::test_pv_examples[rates_as_array] PASSED                                                                                                                 [ 81%]
tests/test_financial.py::TestPV::test_pv_fuzz PASSED                                                                                                                                     [ 90%]
tests/test_financial.py::TestPV::test_pv_interest_rate_sensitivity PASSED                                                                                                                [100%]

====================================================================================== 11 passed in 0.82s ======================================================================================

paulzuradzki · 2024-01-07T21:12:47Z

Here are some of the example errors caught in fuzz testing. We can handle or ignore these cases.

InvalidOperation

rate=fraction_to_decimal(Fraction(0, 1)),
nper=0,
pmt=Decimal('NaN'),
fv=Decimal('NaN'),
when='end',


rate=fraction_to_decimal(Fraction(0, 1)),
nper=array([0], dtype=int8),
pmt=Decimal('NaN'),
fv=Decimal('NaN'),
when='end',

ValueError


Trying example: test_pv_fuzz(
    self=<tests.test_financial.TestPV object at 0x11179c550>,
    rate=0.0,
    nper=array([False, False]),
    pmt=array([False, False, False]),
    fv=1.9565376279601164e+16,
    when='begin',
)
>           return -(fv + pmt * fact) / temp
E           ValueError: operands could not be broadcast together with shapes (3,) (2,)

DivisionByZero

E           Falsifying example: test_pv_fuzz(
E               self=<tests.test_financial.TestPV object at 0x109ba8e50>,
E               rate=1,
E               nper=Decimal('-Infinity'),
E               pmt=0,
E               fv=1,
E               when='end',  # or any other generated value
E           )

numpy_financial/_financial.py:607: DivisionByZero

Overflow

    | decimal.Overflow: [<class 'decimal.Overflow'>]
    | Falsifying example: test_pv_fuzz(
    |     self=<tests.test_financial.TestPV object at 0x12059c790>,
    |     rate=fraction_to_decimal(Fraction(15150, 123698847)),
    |     nper=fraction_to_decimal(Fraction(141745421979347, 7539)),
    |     pmt=Decimal('NaN'),
    |     fv=0,
    |     when='end',
    | )
    +---------------- 2 ----------------
    | Traceback (most recent call last):
    |   File "/Users/pzuradzki/repos/numpy-financial/tests/test_financial.py", line 143, in test_pv_fuzz
    |     npf.pv(rate, nper, pmt, fv, when)
    |   File "/Users/pzuradzki/repos/numpy-financial/numpy_financial/_financial.py", line 605, in pv
    |     temp = (1 + rate) ** nper
    |            ~~~~~~~~~^~~~~~~~~
    | decimal.Overflow: [<class 'decimal.Overflow'>]
    | Falsifying example: test_pv_fuzz(
    |     self=<tests.test_financial.TestPV object at 0x12059c790>,
    |     rate=fraction_to_decimal(Fraction(3, 1)),
    |     nper=fraction_to_decimal(Fraction(1660965, 1)),
    |     pmt=Decimal('NaN'),
    |     fv=Decimal('NaN'),
    |     when='end',
    | )
    +------------------------------------

paulzuradzki · 2024-01-07T21:17:32Z

numpy_financial/_financial.py


-def pv(rate, nper, pmt, fv=0, when='end'):
+def pv(
+    rate: Union[int, float, Decimal, np.ndarray],


I added type hints here, but I see that they're not conventional in the repo. NP to remove the type hints if preferred for consistency; else, can keep it as optional.

Using Union since the package supports Python 3.9. Starting in Python 3.10, we can do int|float|Decimal|np.ndarray.

Type hints are great. Adding type hints is actually part of what I'll be working on once I have the broadcasting done.

numpy_financial/_financial.py

paulzuradzki · 2024-01-07T21:21:04Z

tests/test_financial.py

@@ -90,13 +93,167 @@ def test_decimal_with_when(self):


 class TestPV:


fwiw, I would remove the test classes and rely on function naming convention (ex: test_pv*). Out of scope.

I agree on principle, however having them in classes makes it easier to run groups of tests for each function.

paulzuradzki · 2024-01-07T21:22:25Z

tests/test_financial.py

+        },
+    }
+
+    # Randomized input strategies for fuzz tests & property-based tests


Providing a "strategy" limits the values that the fuzzer (Hypothesis) will supply. The strategies that we re-use are defined here.

Kai-Striega · 2024-01-07T22:51:45Z

Thanks for taking the time to contribute. This looks excellent!

I'm a bit short of time today/tomorrow. I'll try to get a full review done ASAP as there are a couple of other people touching these tests.

FYI I'm currently working on a broadcasting + numba rewrite so many of these errors should be fixed after that.

numpy_financial/_financial.py

Kai-Striega · 2024-01-07T23:01:00Z

pyproject.toml



 [tool.poetry.group.test.dependencies]
+hypothesis = "^6.92.2"


I think adding hypothesis is the right idea here, however I don't have much experience with it. It might take me a while to thoroughly understand this

No problem. Here are some links that I found helpful depending how far down the rabbit hole you want to go with it.

What is Property Based Testing? - Hypothesis

Quick start guide — Hypothesis 6.92.2 documentation

Beyond Units by Hillel Wayne

Choosing properties for property-based testing | F# for fun and profit (fsharpforfunandprofit.com)

Thanks! I've read the Hypothesis documentation and done some basic property based tests, but not much beyond that.

tests/test_financial.py

paulzuradzki · 2024-01-07T23:14:24Z

tests/test_financial.py

+        expected = float(npf.pv(rate=rate + 0.1, nper=nper, pmt=pmt, when=when))
+
+        # As interest rate increases, present value decreases
+        assert round(result, 4) >= round(expected, 4)


I tried to use something like result >= pytest.approx(expected), but pytest.approx doesn't work with floats and greater-than-or-equal-to operator.

remove logging

paulzuradzki · 2024-01-07T23:32:47Z

tests/test_financial.py

+            Decimal("-127128.1709461939327295222005"),
+        )
+
+    @pytest.mark.parametrize("test_case", test_cases.values(), ids=test_cases.keys())


ids= paramater gets us test case labels.

With ids

$ pytest tests/test_financial.py::TestPV::test_pv_examples -vv tests/test_financial.py::TestPV::test_pv PASSED [ 9%] tests/test_financial.py::TestPV::test_pv_decimal PASSED [ 18%] tests/test_financial.py::TestPV::test_pv_examples[default_fv_and_when] PASSED [ 27%] tests/test_financial.py::TestPV::test_pv_examples[specify_fv_and_when] PASSED [ 36%] tests/test_financial.py::TestPV::test_pv_examples[when_1] PASSED [ 45%] tests/test_financial.py::TestPV::test_pv_examples[when_1_and_fv_1000] PASSED [ 54%] tests/test_financial.py::TestPV::test_pv_examples[fv>0] PASSED [ 63%] tests/test_financial.py::TestPV::test_pv_examples[negative_rate] PASSED [ 72%] tests/test_financial.py::TestPV::test_pv_examples[rates_as_array] PASSED [ 81%] tests/test_financial.py::TestPV::test_pv_fuzz PASSED [ 90%] tests/test_financial.py::TestPV::test_pv_interest_rate_sensitivity PASSED [100%]

Without ids (no labels)

$ pytest tests/test_financial.py::TestPV::test_pv_examples -vv tests/test_financial.py::TestPV::test_pv_examples[test_case0] PASSED [ 27%] tests/test_financial.py::TestPV::test_pv_examples[test_case1] PASSED [ 36%] tests/test_financial.py::TestPV::test_pv_examples[test_case2] PASSED [ 45%] tests/test_financial.py::TestPV::test_pv_examples[test_case3] PASSED [ 54%] tests/test_financial.py::TestPV::test_pv_examples[test_case4] PASSED [ 63%] tests/test_financial.py::TestPV::test_pv_examples[test_case5] PASSED [ 72%] tests/test_financial.py::TestPV::test_pv_examples[test_case6] PASSED [ 81%]

Kai-Striega · 2024-03-11T23:46:00Z

@paulzuradzki sorry for not being in touch. I moved states and then had some health issues. I'm back now.

This work looks great and is definitely a candidate for merging. I'll get a formal review done this week if you are still interested in contributing.

add PBT and table tests for PV func

34eebab

paulzuradzki force-pushed the tests/add-hypothesis-pbt-and-fuzz-tests branch from aaccf23 to 34eebab Compare January 7, 2024 21:14