Ingestion filters. Fixes #1192

[eilmiv]

Summary of changes

• Added filtering that happens after reading from a remote for ingestion and is configurable by the user
• Support for allow list and block list
• Support for may different kinds of filters, including e.g. a filter that matches a substring in the description
• All string filters are case-insensitive

Motivation and context

It is useful to not ingest everything from a source. Solves #1192. Important part of the exchange feature in the mTeSS-x project.

Screenshots

Checklist

• I have read and followed the CONTRIBUTING guide.
• I confirm that I have the authority necessary to make this contribution on behalf of its copyright owner and agree
  to license it to the TeSS codebase under the
  BSD license.

[Copilot]

Pull request overview

This PR adds a comprehensive filtering system for ingestion sources, allowing users to configure allow lists and block lists to selectively import materials and events. This addresses issue #1192 and supports the mTeSS-x project's exchange feature.

Key Changes

• Introduced a new SourceFilter model with support for multiple filter types (keyword, title, description, URL, etc.) and modes (allow/block)
• Added filtering logic to the ingestion pipeline that executes after reading from remote sources
• Created a dynamic UI for managing filters with JavaScript-based form controls

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
db/migrate/20251209112056_create_source_filters.rb	Creates the source_filters table with mode, type, and value fields
db/migrate/20251218100418_remove_keyword_filter_from_sources.rb	Removes deprecated keyword_filter column from sources
app/models/source_filter.rb	New model implementing filter matching logic for various field types
app/models/source.rb	Adds association to source_filters and implements passes_filter? method
app/controllers/sources_controller.rb	Updates controller to handle nested source_filters attributes
app/views/sources/_form.html.erb	Adds UI sections for allow list and block list filter configuration
app/views/sources/_source_filter_form.html.erb	Partial for individual filter form fields
app/assets/javascripts/source_filters.js	JavaScript for dynamically adding/removing filter forms
app/assets/stylesheets/sources.scss	Styling for filter form layout
lib/ingestors/ingestor.rb	Integrates filter method into ingestion write process
app/workers/source_test_worker.rb	Applies filters during source testing
test/unit/ingestors/ingestor_test.rb	Comprehensive tests for filter behavior
test/fixtures/source_filters.yml	Test fixtures for various filter configurations
test/fixtures/materials.yml	Test materials for filter testing
test/fixtures/events.yml	Test events for filter testing
test/models/source_filter_test.rb	Empty test file placeholder

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[fbacall]

Looks good, just a few suggestions

[eilmiv]

A major change since the last review is that the allow list now has a conjunctive semantic (e4bc5d7). This is strictly more expressive. All boolean expressions of the provided predicates are now supported (if multiple "sources" for a given training metadata collection are considered).