Plugin discovery fails with circular import when plugin config inherits from SingleColumnConfig

[webmaxru]

Summary

When a third-party plugin's config module imports from data_designer.config, a circular import race condition causes plugin discovery to fail with a misleading error message, even though the plugin is correctly implemented.

Problem

During plugin discovery, the Plugin._load() method uses importlib.import_module() to load the config class. However, if the plugin's config module is currently being imported (i.e., partially initialized), getattr(module, class_name) fails because the class definition hasn't been executed yet.

Error Message

🛑 Failed to load plugin from entry point 'my-plugin': Could not find class 'MyPluginConfig' in module 'my_plugin.config'

This is confusing because the class does exist in the file — it's just not available yet due to import timing.

Reproduction

1. Create a plugin with this structure:

my-plugin/
├── pyproject.toml
└── src/
    └── my_plugin/
        ├── __init__.py
        ├── config.py
        ├── generator.py
        └── plugin.py

2. Plugin config (config.py):

from data_designer.config.column_configs import SingleColumnConfig
from typing import Literal

class MyPluginConfig(SingleColumnConfig):
    column_type: Literal["my-plugin"] = "my-plugin"
    # ... other fields

3. Plugin entry point (plugin.py):

from data_designer.plugins import Plugin, PluginType

plugin = Plugin(
    impl_qualified_name="my_plugin.generator.MyPluginGenerator",
    config_qualified_name="my_plugin.config.MyPluginConfig",
    plugin_type=PluginType.COLUMN_GENERATOR,
)

4. User script:

from my_plugin.config import MyPluginConfig  # This triggers the bug!

from data_designer.interface import DataDesigner
# ...

Import chain that causes the issue:

1. User imports my_plugin.config
2. config.py imports from data_designer.config.column_configs import SingleColumnConfig
3. This triggers data_designer.config initialization
4. Which triggers plugin discovery via PluginManager
5. Plugin discovery tries to validate my_plugin.config.MyPluginConfig
6. _load() calls importlib.import_module("my_plugin.config")
7. But that module is already being imported (step 1), so sys.modules["my_plugin.config"] exists but is incomplete
8. getattr(module, "MyPluginConfig") fails → PluginLoadError

Proposed Solution

Modify Plugin._load() to handle partially-initialized modules by attempting a reload:

@staticmethod
def _load(fully_qualified_object: str) -> type:
    module_name, object_name = _get_module_and_object_names(fully_qualified_object)
    module = importlib.import_module(module_name)

    # Handle case where module is partially initialized (circular import during plugin discovery).
    # If the class isn't available yet, try reloading the module.
    if not hasattr(module, object_name):
        # Module may be partially loaded due to circular imports during plugin discovery.
        # Try to reload it to complete initialization.
        try:
            module = importlib.reload(module)
        except Exception:
            pass  # If reload fails, fall through to the error below

    try:
        return getattr(module, object_name)
    except AttributeError:
        raise PluginLoadError(f"Could not find class {object_name!r} in module {module_name!r}")

Why This Fix Is Safe

Scenario	Before	After
Normal load (class exists)	✅ Works	✅ Works (no change)
Class doesn't exist	❌ PluginLoadError	❌ Same error
Circular import (class not yet defined)	❌ PluginLoadError	✅ Reload succeeds

• The reload only triggers when hasattr() returns False — meaning it would have failed anyway
• Same exception type and message for actual missing classes
• No API changes

Workarounds (current)

Users can avoid this by:

1. Using uv run python instead of system Python (ensures correct venv)
2. Importing DataDesigner before importing plugin config classes
3. Using lazy imports in the plugin's __init__.py

However, these are unintuitive and the error message doesn't guide users toward them.

Environment

• Data Designer version: latest (main branch)
• Python: 3.11+
• OS: Windows/Linux/macOS

[webmaxru]

Context: I work on Declarative Colums plugin - to specify columns via YAML

[johnnygreco]

Hey @webmaxru – thank you very much for raising this issue!

If possible, we'd like to avoid reloading modules – hoping to make improve the foundations to be less brittle for plugin developers wrt to circular imports.

To address this issue, we just merged two PRs: #291 and #287.

Note there is a small breaking change with this update. SingleColumnConfig now lives in data_designer.config.base:

from data_designer.config.base import SingleColumnConfig

If you have a chance to try this out with your plugin, please let us know if this resolves the issue for you!

Thanks again 🙌

[webmaxru]

Thank you for your quick response!

Actually, I refactored my plugin - it now only accepts full column set (before that I provided possibility to use YAML for column-by-column build) and the need for reloading modules became unnecessary. Please, feel free to close this issue.

Btw here is my plugin: https://pypi.org/project/data-designer-declarative-columns/
I created it as a helper to build Data Designer Copilot but hopefully DD dev community will find it useful too.

I see that Available Plugin List is empty in DD documentation website so I'll be happy to get Declarative Columns listed there - do you accept 3rd party contributions?

[johnnygreco]

Hey @webmaxru – glad to hear things are working for you!

Thank for creating and sharing the declarative columns package and Data Designer Copilot – sooooooo cool! Please reach out if you have any questions / feedback.

Indeed, plugins are still a bit experimental, though I think we are converging on the design (most up-to-date pattern is in our e2e tests). And yes, we most definitely want to include third-party plugins in our "Available Plugin List"!

I haven't had a chance to dig too deep into the implementation, but one question I had about your plugin is whether DD config builder's from_config method could be used to deserialize your yaml configurations? Are your declarative columns using a different config?

[webmaxru]

Hello @johnnygreco !
You are absolutely right: current version is compatible with YAML format that can be consumed by from_config but I overengineered it (I admit :) ) by adding own validation as preparation for the next versions where I plan to add higher-level abstractions for the ways to define columns. On my roadmap:

• coulmn templates with parameters
• schema-first DSL
• composable fragments
• variants / feature flags
• column bundles for common domains
• maybe, natural language definitions :)