Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
dcab3d1
feat: Add database adapter interface for multi-backend support (Phase 2)
dimitri-yatsenko Jan 17, 2026
1cec906
feat: Add backend configuration to DatabaseSettings
dimitri-yatsenko Jan 17, 2026
2ece79c
feat: Add get_cursor() method to database adapters
dimitri-yatsenko Jan 17, 2026
b76a099
feat: Integrate database adapters into Connection class
dimitri-yatsenko Jan 17, 2026
8692c99
feat: Use database adapters for SQL generation in table.py (Phase 5)
dimitri-yatsenko Jan 17, 2026
1365bf9
feat: Add json_path_expr() method to database adapters (Phase 6 Part 1)
dimitri-yatsenko Jan 17, 2026
77e2d4c
feat: Use adapter for WHERE clause generation (Phase 6 Part 2)
dimitri-yatsenko Jan 17, 2026
5ddd3b7
feat: Use adapter for query expression SQL (Phase 6 Part 3)
dimitri-yatsenko Jan 17, 2026
a1c5cef
feat: Add DDL generation adapter methods (Phase 7 Part 1)
dimitri-yatsenko Jan 17, 2026
ca5ea6c
feat: Thread adapter through declare.py for backend-agnostic DDL (Pha…
dimitri-yatsenko Jan 17, 2026
53cfbc8
feat: Add multi-backend testing infrastructure (Phase 1)
dimitri-yatsenko Jan 17, 2026
6ef7b2c
docs: Add comprehensive multi-backend testing design
dimitri-yatsenko Jan 17, 2026
99b9396
fix: Set autocommit=True by default in database adapters
dimitri-yatsenko Jan 17, 2026
5e1dc6f
fix: Replace hardcoded MySQL syntax with adapter methods
dimitri-yatsenko Jan 17, 2026
7eb7846
fix: Make heading.py backend-agnostic for column and index queries
dimitri-yatsenko Jan 17, 2026
5547ea4
feat: Add backend-agnostic upsert and complete heading.py fixes
dimitri-yatsenko Jan 17, 2026
f865143
fix: Complete foreign key and primary key support for PostgreSQL
dimitri-yatsenko Jan 17, 2026
691704c
fix: Use table instances instead of classes in len() calls
dimitri-yatsenko Jan 17, 2026
b96c52d
fix: Use backend-agnostic COUNT DISTINCT for multi-column primary keys
dimitri-yatsenko Jan 17, 2026
9800381
feat: Add backend-agnostic cascade delete support
dimitri-yatsenko Jan 17, 2026
5fa0f56
fix: Backend-agnostic fixes for cascade delete and FreeTable
dimitri-yatsenko Jan 18, 2026
6d6460f
fix: Complete cascade delete support for PostgreSQL
dimitri-yatsenko Jan 18, 2026
566c5b5
fix: Resolve mypy and ruff linting errors
dimitri-yatsenko Jan 18, 2026
338e7ea
feat: Add PostgreSQL support to CI test dependencies
dimitri-yatsenko Jan 18, 2026
57f376d
fix: Fix cascade delete for multi-column FKs and renamed attributes
dimitri-yatsenko Jan 18, 2026
5b7f6d7
style: Apply pre-commit formatting fixes
dimitri-yatsenko Jan 18, 2026
664ff34
fix: Add column name aliases for MySQL information_schema queries
dimitri-yatsenko Jan 18, 2026
075d96d
fix: Add column name aliases for all MySQL information_schema queries
dimitri-yatsenko Jan 18, 2026
b6a4f6f
fix: Update test_foreign_keys to pass adapter parameter
dimitri-yatsenko Jan 18, 2026
d88c308
fix: Mark describe() bugs as xfail and fix PostgreSQL SSL/multiproces…
dimitri-yatsenko Jan 18, 2026
450d2b9
fix: Add missing pytest import in test_foreign_keys.py
dimitri-yatsenko Jan 18, 2026
cb0d544
fix: Restore database.backend config after multi-backend tests
dimitri-yatsenko Jan 18, 2026
ddca0ed
fix: Change connection_by_backend to function scope
dimitri-yatsenko Jan 18, 2026
9ff1eb5
fix: Track connection closed state internally
dimitri-yatsenko Jan 18, 2026
12ae73b
fix: Correct SSL configuration format for MySQL
dimitri-yatsenko Jan 18, 2026
efb8482
fix: Make MySQL adapter accept use_tls parameter
dimitri-yatsenko Jan 18, 2026
e1ad919
fix: Enable SSL by default when use_tls not specified
dimitri-yatsenko Jan 18, 2026
7cdcf3d
fix: Explicitly enable SSL with ssl_disabled=False
dimitri-yatsenko Jan 18, 2026
ba702d3
test: Mark test_secure_connection as xfail pending investigation
dimitri-yatsenko Jan 18, 2026
ad127be
fix: Preserve nullable and unique modifiers in describe() for FK attr…
dimitri-yatsenko Jan 19, 2026
b6cf20f
test: Remove xfail markers for describe() tests now that nullable is …
dimitri-yatsenko Jan 19, 2026
618097d
chore: Bump version to 2.1.0a1 for PostgreSQL multi-backend support
dimitri-yatsenko Jan 19, 2026
c416807
fix: Add warning on SSL fallback and improve TLS tests
dimitri-yatsenko Jan 19, 2026
f1563db
fix: Use datajoint/mysql image for testcontainers (has SSL configured)
dimitri-yatsenko Jan 19, 2026
33abfeb
test: Skip SSL tests when not using external containers
dimitri-yatsenko Jan 19, 2026
688855f
Merge pre/v2.0: Fix nested parentheses in index parsing
dimitri-yatsenko Jan 19, 2026
da0ac48
test: Remove xfail from test_describe now that JSON index reconstruct…
dimitri-yatsenko Jan 19, 2026
bd35a7e
fix: Include EXPRESSION column in MySQL get_indexes_sql for functiona…
dimitri-yatsenko Jan 19, 2026
1d25762
fix: Unescape single quotes in MySQL expression indexes
dimitri-yatsenko Jan 19, 2026
aeb7535
Merge pull request #1338 from datajoint/feat/database-adapters
dimitri-yatsenko Jan 19, 2026
4797151
docs: Remove multi-backend-testing.md (moved to datajoint-docs)
dimitri-yatsenko Jan 20, 2026
05d2512
fix: PostgreSQL adapter bugs for multi-backend support
dimitri-yatsenko Jan 20, 2026
ab99193
fix: Clean up PostgreSQL enum types when dropping tables
dimitri-yatsenko Jan 20, 2026
d8b15c5
fix: PostgreSQL adapter bugs for multi-backend support
dimitri-yatsenko Jan 20, 2026
e54e4a7
fix: Clean up PostgreSQL enum types when dropping tables
dimitri-yatsenko Jan 20, 2026
be7d079
style: Fix linting issues
dimitri-yatsenko Jan 20, 2026
8a8423b
fix: Escape % in LIKE patterns for MySQL
dimitri-yatsenko Jan 20, 2026
6b2b7e4
fix: use single quotes for SQL literals (PostgreSQL compatibility)
dimitri-yatsenko Jan 20, 2026
0469a72
fix: use PostgreSQL-specific queries for dependencies loading
dimitri-yatsenko Jan 20, 2026
a4ed877
fix: convert double-quoted defaults to single quotes
dimitri-yatsenko Jan 20, 2026
e56a5a6
fix: generate COMMENT ON COLUMN for PostgreSQL blob codecs
dimitri-yatsenko Jan 20, 2026
97db517
fix: escape single quotes in PostgreSQL COMMENT statements
dimitri-yatsenko Jan 20, 2026
3c34d31
fix: PostgreSQL compatibility in jobs.py
dimitri-yatsenko Jan 20, 2026
aa78497
fix: add current_user_expr() for backend-agnostic user retrieval
dimitri-yatsenko Jan 20, 2026
c795c3a
fix: use adapter for identifier quoting in SQL generation
dimitri-yatsenko Jan 20, 2026
f113f92
fix: convert memoryview to bytes for PostgreSQL blob unpacking
dimitri-yatsenko Jan 20, 2026
b1ef634
fix: make table name quoting backend-agnostic
dimitri-yatsenko Jan 20, 2026
e49a2ef
refactor: move get_master regex to adapter methods
dimitri-yatsenko Jan 20, 2026
1ffb157
fix: use adapter.quote_identifier in metaclass full_table_name
dimitri-yatsenko Jan 20, 2026
6506bad
fix: strip both backticks and double quotes from lineage table names
dimitri-yatsenko Jan 20, 2026
56a8df4
fix: handle PostgreSQL enum types and USER-DEFINED columns
dimitri-yatsenko Jan 20, 2026
6576b43
fix: address CI lint and test failures
dimitri-yatsenko Jan 20, 2026
b7e800b
style: apply ruff-format formatting fixes
dimitri-yatsenko Jan 20, 2026
86bd95e
Merge origin/pre/v2.1 into fix/postgresql-adapter-bugs
dimitri-yatsenko Jan 20, 2026
fd4e011
fix: use adapter quoting in autopopulate progress()
dimitri-yatsenko Jan 20, 2026
d2e89ba
fix: handle psycopg2 auto-deserialized JSON in codecs
dimitri-yatsenko Jan 20, 2026
bc245d3
fix: PostgreSQL compatibility improvements for DataJoint 2.1
dimitri-yatsenko Jan 20, 2026
ae2dc57
fix: include table_comment in PostgreSQL get_table_info_sql
dimitri-yatsenko Jan 20, 2026
fd31b22
feat: add DJ_USE_TLS environment variable support
dimitri-yatsenko Jan 20, 2026
2f61cbd
fix: PostgreSQL compatibility for diagrams and date functions
dimitri-yatsenko Jan 20, 2026
79e712b
fix: improve PostgreSQL SQL function translations
dimitri-yatsenko Jan 20, 2026
30b7130
fix: handle PostgreSQL double-quote format in _get_tier
dimitri-yatsenko Jan 20, 2026
9775d0a
fix: Remove deprecated ___ separator support
dimitri-yatsenko Jan 22, 2026
864b258
feat: Add singleton tables (empty primary keys)
dimitri-yatsenko Jan 22, 2026
996e820
fix: PostgreSQL compatibility for singleton tables
dimitri-yatsenko Jan 22, 2026
25354ad
style: Format test file with ruff
dimitri-yatsenko Jan 22, 2026
94fa4a8
chore: Bump version to 2.1.0a4
dimitri-yatsenko Jan 22, 2026
d3c7afb
fix: Backend-agnostic JSON path expressions
dimitri-yatsenko Jan 23, 2026
215bc9c
chore: Bump version to 2.1.0a5
dimitri-yatsenko Jan 23, 2026
5010b85
refactor: Remove boolean_true_literal, use TRUE for both backends
dimitri-yatsenko Jan 23, 2026
e0a7c6d
Merge pull request #1341 from datajoint/feat/singleton-tables
MilagrosMarin Jan 23, 2026
648bd1a
docs: fix string quoting in docstring example
dimitri-yatsenko Jan 23, 2026
1698acf
Merge pull request #1344 from datajoint/fix/docstring-postgresql-compat
MilagrosMarin Jan 23, 2026
f98cdbf
feat(diagram): add direction, Mermaid output, and schema grouping
dimitri-yatsenko Jan 23, 2026
0dd5a69
feat: always group diagram nodes by schema with module labels
dimitri-yatsenko Jan 23, 2026
903e6b2
chore: bump version to 2.1.0a6
dimitri-yatsenko Jan 23, 2026
d41b75f
feat: improve schema grouping labels with fallback logic
dimitri-yatsenko Jan 23, 2026
80489fc
feat: add collapse() method for high-level pipeline views
dimitri-yatsenko Jan 23, 2026
3292a06
fix: properly merge diagrams from different schemas
dimitri-yatsenko Jan 23, 2026
c3c4c0f
fix: diagram improvements for collapse and display
dimitri-yatsenko Jan 23, 2026
ba1a237
fix: collapse chaining for multiple collapsed diagrams
dimitri-yatsenko Jan 23, 2026
26264d4
fix: reset alias node counter on dependencies clear
dimitri-yatsenko Jan 23, 2026
09cf50d
fix: don't collapse fresh diagrams that were never combined
dimitri-yatsenko Jan 23, 2026
77ebfb5
fix: use database schema name for collapsed nodes when module is ambi…
dimitri-yatsenko Jan 23, 2026
de00340
fix: place collapsed nodes inside schema clusters for proper layout
dimitri-yatsenko Jan 23, 2026
4d6b7ac
feat: change default diagram direction from TB to LR
dimitri-yatsenko Jan 23, 2026
9e87106
fix: collapsed nodes show only table count, not redundant name
dimitri-yatsenko Jan 23, 2026
d5bdf51
refactor: simplify collapse logic to use single _expanded_nodes set
dimitri-yatsenko Jan 23, 2026
e59eeb3
fix: break long line in diagram.py to pass lint
dimitri-yatsenko Jan 24, 2026
810ceee
style: apply ruff format to diagram.py
dimitri-yatsenko Jan 24, 2026
a8b0734
Merge pull request #1345 from datajoint/feat/diagram-improvements
MilagrosMarin Jan 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -195,3 +195,8 @@ datajoint.json

# Test outputs
*_test_summary.txt

# Swap files
*.swp
*.swo
*~
19 changes: 19 additions & 0 deletions docker-compose.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,19 @@ services:
timeout: 30s
retries: 5
interval: 15s
postgres:
image: postgres:${POSTGRES_VER:-15}
environment:
- POSTGRES_PASSWORD=${PG_PASS:-password}
- POSTGRES_USER=${PG_USER:-postgres}
- POSTGRES_DB=${PG_DB:-test}
ports:
- "5432:5432"
healthcheck:
test: [ "CMD-SHELL", "pg_isready -U postgres" ]
timeout: 30s
retries: 5
interval: 15s
minio:
image: minio/minio:${MINIO_VER:-RELEASE.2025-02-28T09-55-16Z}
environment:
Expand Down Expand Up @@ -52,6 +65,8 @@ services:
depends_on:
db:
condition: service_healthy
postgres:
condition: service_healthy
minio:
condition: service_healthy
environment:
Expand All @@ -61,6 +76,10 @@ services:
- DJ_TEST_HOST=db
- DJ_TEST_USER=datajoint
- DJ_TEST_PASSWORD=datajoint
- DJ_PG_HOST=postgres
- DJ_PG_USER=postgres
- DJ_PG_PASS=password
- DJ_PG_PORT=5432
- S3_ENDPOINT=minio:9000
- S3_ACCESS_KEY=datajoint
- S3_SECRET_KEY=datajoint
Expand Down
9 changes: 7 additions & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ test = [
"pytest-cov",
"requests",
"graphviz",
"testcontainers[mysql,minio]>=4.0",
"testcontainers[mysql,minio,postgres]>=4.0",
"polars>=0.20.0",
"pyarrow>=14.0.0",
]
Expand All @@ -98,14 +98,16 @@ test = [
s3 = ["s3fs>=2023.1.0"]
gcs = ["gcsfs>=2023.1.0"]
azure = ["adlfs>=2023.1.0"]
postgres = ["psycopg2-binary>=2.9.0"]
polars = ["polars>=0.20.0"]
arrow = ["pyarrow>=14.0.0"]
test = [
"pytest",
"pytest-cov",
"requests",
"s3fs>=2023.1.0",
"testcontainers[mysql,minio]>=4.0",
"testcontainers[mysql,minio,postgres]>=4.0",
"psycopg2-binary>=2.9.0",
"polars>=0.20.0",
"pyarrow>=14.0.0",
]
Expand Down Expand Up @@ -227,6 +229,9 @@ ignore-words-list = "rever,numer,astroid"
markers = [
"requires_mysql: marks tests as requiring MySQL database (deselect with '-m \"not requires_mysql\"')",
"requires_minio: marks tests as requiring MinIO object storage (deselect with '-m \"not requires_minio\"')",
"mysql: marks tests that run on MySQL backend (select with '-m mysql')",
"postgresql: marks tests that run on PostgreSQL backend (select with '-m postgresql')",
"backend_agnostic: marks tests that should pass on all backends (auto-marked for parameterized tests)",
]


Expand Down
54 changes: 54 additions & 0 deletions src/datajoint/adapters/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
"""
Database adapter registry for DataJoint.

This module provides the adapter factory function and exports all adapters.
"""

from __future__ import annotations

from .base import DatabaseAdapter
from .mysql import MySQLAdapter
from .postgres import PostgreSQLAdapter

__all__ = ["DatabaseAdapter", "MySQLAdapter", "PostgreSQLAdapter", "get_adapter"]

# Adapter registry mapping backend names to adapter classes
ADAPTERS: dict[str, type[DatabaseAdapter]] = {
"mysql": MySQLAdapter,
"postgresql": PostgreSQLAdapter,
"postgres": PostgreSQLAdapter, # Alias for postgresql
}


def get_adapter(backend: str) -> DatabaseAdapter:
"""
Get adapter instance for the specified database backend.

Parameters
----------
backend : str
Backend name: 'mysql', 'postgresql', or 'postgres'.

Returns
-------
DatabaseAdapter
Adapter instance for the specified backend.

Raises
------
ValueError
If the backend is not supported.

Examples
--------
>>> from datajoint.adapters import get_adapter
>>> mysql_adapter = get_adapter('mysql')
>>> postgres_adapter = get_adapter('postgresql')
"""
backend_lower = backend.lower()

if backend_lower not in ADAPTERS:
supported = sorted(set(ADAPTERS.keys()))
raise ValueError(f"Unknown database backend: {backend}. " f"Supported backends: {', '.join(supported)}")

return ADAPTERS[backend_lower]()
Loading
Loading