Add processing tools for exclusion sets (as methods)

[egrace479]

This is for test-set filtering, removal of human faces, removal of images of labels (vs. specimens), removal of empty camera trap frames (from GBIF), general cleaning for citizen science data (remove cluttered occurrences and set camera traps to proper processing), etc. in more detail...

• All lookup table exclude creators from TreeOfLife-dev (many of these are here) should be in src/ for this repo:
  • PDQ hash process
  • Image type separator (museum, camera trap, and citizen science categories)
  • Museum Specimen processing: label vs specimen filter
  • camera trap processing: megadetector & reducer
  • citizen science image processing
  • face detection
• Add slurm scripts that run these processes from TreeOfLife-dev to root-level scripts/ directory (these will be more template (your/path/here) style.

This whole process then also needs clear documentation for how to use each of these tools and the pipeline.

[egrace479]

Adding these to a processing/ folder at the repo root in #27 to be worked into the package. This does include documentation explaining the code and the repo format at this stage.

[egrace479]

Animal and Face detection pipelines (consistent with the TreeOfLife data format(s)) are being added to hpc-inference (name-change pending). @NetZissou, I believe the repo also contains CLIP distributed embedding and other processing steps more generalized from this one. Should we be refocusing this issue to providing instructions and slurm scripts for using that tool in this context, thus, leaving what's here more as a record of our original process?

With that idea in mind, I would assume we still need the following implemented as methods in this repo:

• PDQ hash process
• Image type separator (museum, camera trap, and citizen science categories)
• Museum Specimen processing: label vs specimen filter
• citizen science image processing

Additionally, with the shift to HDF5 format for images, these may need updates since they're currently designed for working solely with parquet.

@NetZissou, does that seem right to you?