Add invariant tests #4326
Open
Add invariant tests #4326
+334
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
denik
requested review from
andrewnester,
anton-107,
pietern and
shreyas-goenka
as code owners
Collaborator
|
Commit: 63ee262
41 interesting tests: 22 KNOWN, 13 FAIL, 4 BUG, 1 SKIP, 1 RECOVERED
Top 48 slowest tests (at least 2 minutes):
|
Add 9 minimal resource configuration templates to test the no-drift invariant across different resource types (jobs, pipelines, experiments, models, volumes, schemas, clusters, model serving endpoints). Changes: - Add configs/ directory with minimal .yml.tmpl files for each resource type - Add data/ directory for supporting files (pipeline.py) - Update no_drift/script to copy data files at test start - Update test.toml to include new configs and ignore *.py files - Update out.test.toml with expanded config matrix All configs tested and verified on both testserver and AWS cloud with direct and terraform deployment engines. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add 4 more resource configurations to the no_drift invariant test, bringing the total to 13 working configs. New configs: - alerts (with .dbalert.json file) - dashboards (with .lvdash.json file) - apps (with source_code_path directory) - secret_scopes Also created sql_warehouse.yml.tmpl but excluded from test due to drift issues. Changes: - Add 5 new config templates in configs/ - Add 3 new data files in data/ (alert.dbalert.json, dashboard.lvdash.json, app/) - Update test.toml to include new configs and ignore *.json and app directory - Update out.test.toml with expanded config matrix All 13 configs tested on testserver with both direct and terraform engines. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Complete the no_drift invariant test coverage by adding the remaining resource types, bringing the total to 17 working configs. New configs: - quality_monitor (with dependent schema) - database_instance - database_catalog (with dependent database_instance) - synced_database_table (with dependent instance and catalog) Changes: - Add 3 new config templates in configs/ - Fix quality_monitor to use $CURRENT_USER_NAME - Update test.toml with 4 additional configs - Update out.test.toml with full config matrix All 17 configs tested successfully on testserver with both direct and terraform deployment engines, showing zero drift after deploy. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
App names must be between 2-30 characters. Changed from 'test-app-' to 'app-' to fit within the limit. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
denik
force-pushed
the
denik/invariant-tests
branch
from
63ee262 to
a195594
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
Add a new kind of test - invariant test for DABs (a subtype of acceptance test).
Invariant tests are acceptance tests that can be run against many configs to check for certain properties.
Unlike regular acceptance tests full output is not recorded, unless the condition is not met.
In this PR there is one invariant test added - no_drift, that checks that there are no actions planned after successful deploy. If that's not the case, the test will dump full JSON plan to the output.
Why
Simplify resource implementation testing - implementors only need to provide a config to get all invariants check.
With no_drift in particular, we've had several cases where this condition is not met and it's still the case for some configurations of permissions and grants. It's also going to become more important as we replace "server_side_default" with something more precise.
The other use of invariant tests is fuzzing - since they work on any config, they can be used on randomly generated configs to find bugs.