Problem Description
Currently, SDGym computes a "win" by determining whether the synthesizer's quality score (for a particular dataset) beats the GaussianCopulaSynthesizer's quality score. This is done intentionally, because GaussianCopula is a reliable, longstanding statistical method that provides a robust framework and baseline for synthetic data generation.
However, there are a few problems with computing wins in this way:
- The GaussianCopulaSynthesizer itself will never "win" by this definition, since it must beat its own quality score
- Any synthesizer that produces higher quality synthetic data will be considered a winner, even if it show extremely poor performance (as compared to other synthesizers)
Expected behavior
To solve the problems above, we should (a) update the "wins" computation to also include the Pareto front, accounting for the quality/performance tradeoff, and (b) enforce that the quality score is >= Gaussian Copula's quality score.
So overall a synthesizer is a "winner" for a dataset if BOTH of the following are true:
- It is on the Pareto frontier when you consider all the synthesizers run on that dataset. This means that there is no other synthesizer in the benchmarking run for this dataset that has beat this one in terms of both quality and performance* AND
- Its quality score is equal to or better than the quality score from GaussianCopulaSynthesizer**
With this change, there are still multiple winners for a given dataset.
Additional context
*A synthesizer "beats" another one if it's quality score is higher and if its overall runtime is lower
**With this definition, the GaussianCopulaSynthesizer is also able to be a winner. It will always pass the quality criteria, but it may or may not pass the Pareto frontier criteria.
Problem Description
Currently, SDGym computes a "win" by determining whether the synthesizer's quality score (for a particular dataset) beats the GaussianCopulaSynthesizer's quality score. This is done intentionally, because GaussianCopula is a reliable, longstanding statistical method that provides a robust framework and baseline for synthetic data generation.
However, there are a few problems with computing wins in this way:
Expected behavior
To solve the problems above, we should (a) update the "wins" computation to also include the Pareto front, accounting for the quality/performance tradeoff, and (b) enforce that the quality score is
>=Gaussian Copula's quality score.So overall a synthesizer is a "winner" for a dataset if BOTH of the following are true:
With this change, there are still multiple winners for a given dataset.
Additional context
*A synthesizer "beats" another one if it's quality score is higher and if its overall runtime is lower
**With this definition, the GaussianCopulaSynthesizer is also able to be a winner. It will always pass the quality criteria, but it may or may not pass the Pareto frontier criteria.