Description
Add DPG (Descriptive Prompt Generation) benchmarks for image understanding evaluation.
Details
- Source: GitHub CSV from
TencentQQGYLab/ELLA repo
- Subcategories: entity, attribute, relation, global, other
- Collate:
prompt_with_auxiliaries_collate
Implementation
- Add
setup_dpg_dataset in src/pruna/data/datasets/prompt.py
- Support
category param for filtering subcategories
- Register in
base_datasets
- Add
BenchmarkInfo entry with metrics: ["qa_accuracy"], subsets list
- Auxiliaries should include
questions list and category_broad for evaluation
- Add test
Acceptance Criteria
Description
Add DPG (Descriptive Prompt Generation) benchmarks for image understanding evaluation.
Details
TencentQQGYLab/ELLArepoprompt_with_auxiliaries_collateImplementation
setup_dpg_datasetinsrc/pruna/data/datasets/prompt.pycategoryparam for filtering subcategoriesbase_datasetsBenchmarkInfoentry with metrics:["qa_accuracy"], subsets listquestionslist andcategory_broadfor evaluationAcceptance Criteria
PrunaDataModule.from_string("DPG")works (all subcategories)PrunaDataModule.from_string("DPG", category="entity")worksquestionsandcategory_broadfields