Architecture Overview

The SDK is organized around a core HTTP client layer, endpoint wrappers that model the iMednet API, workflow helpers that combine multiple endpoint calls, and a CLI built on top of those pieces.

Components

graph TD CLI[CLI] --> |invokes| Workflows Workflows --> |coordinate| Endpoints Endpoints --> |use| Client["(HTTP Client)"] Client --> |httpx| API

Core Client

The synchronous Client implements authentication, retry handling, and JSON serialization for each API request. It inherits from HTTPClientBase and is shared by all endpoint classes. HTTP transport is handled by httpx.

HTTP Transport Contract

The transport layer in imednet.core.http provides the following guarantees:

Retry policy

• Retries are evaluated by imednet.core.retry.DefaultRetryPolicy.

• Idempotent methods (GET, PUT, DELETE, HEAD, OPTIONS) retry on:

  • httpx.RequestError network failures

  • HTTP 5xx responses

• All methods (including POST/PATCH) retry on HTTP 429 rate limits.

• Non-idempotent methods do not retry on network errors or 5xx responses.

• Maximum attempts are controlled by the retries argument on Client, AsyncClient, ImednetSDK, and AsyncImednetSDK.

• Backoff uses Tenacity’s randomized exponential wait strategy with backoff_factor as the multiplier.

• If a retriable response includes Retry-After, that delay overrides the backoff schedule.

Timeouts

• timeout is forwarded directly into httpx.Client / httpx.AsyncClient.

• You can pass either a float (single timeout applied to all phases) or httpx.Timeout for per-phase timeouts.

Status-code error mapping

Response errors are mapped via imednet.errors.get_error_class():

Status code	Raised exception
400	imednet.errors.BadRequestError
401	imednet.errors.UnauthorizedError
403	imednet.errors.ForbiddenError
404	imednet.errors.NotFoundError
409	imednet.errors.ConflictError
429	imednet.errors.RateLimitError
500-599	imednet.errors.ServerError
other	imednet.errors.ApiError

Credential redaction

• Transport logs never include raw credential headers.

• URLs logged by request monitoring are sanitized through imednet.utils.url.redact_url_query(), redacting sensitive query params such as api_key, security_key, token, secret, and password.

• Retry exhaustion raises imednet.errors.RequestError with a generic message that does not expose credentials.

Async Client

AsyncClient provides the same features as the sync client but leverages async/await for concurrency. The public SDK surface is split into two client classes:

• ImednetSDK for synchronous usage only.

• AsyncImednetSDK for asynchronous usage only.

Migration note: if you previously used async behavior from ImednetSDK, move to AsyncImednetSDK and manage lifecycle with async with or await sdk.aclose().

Endpoints

Each endpoint, such as StudiesEndpoint, wraps a related set of API operations. Endpoints now use composition: GenericListGetEndpoint composes operation classes (for example ListOperation and FilterGetOperation) rather than inheriting deep operation mixin chains. This keeps method resolution straightforward, improves IDE autocomplete, and avoids MRO-related typing ambiguity. Endpoints can cache responses when called without filters and expose list/get methods that return typed models.

Workflows

Workflows orchestrate several endpoints to perform higher level tasks. For example, RecordUpdateWorkflow validates record payloads, submits them, and polls resulting jobs. Workflows have sync and async variants and are available under sdk.workflows.

Caching

Caching Behavior describes how endpoint and schema data are cached. Cached values can be refreshed by passing refresh=True to endpoint methods or calling schema.refresh() on a validator.

CLI

The Command Line Interface (CLI) uses Typer to expose common workflows on the command line. Each command creates an ImednetSDK instance, invokes a workflow, and closes the SDK when finished.

Data Flow

graph LR User --> |runs| CLI User --> |imports| SDK CLI --> |uses| Workflows SDK --> |exposes| Endpoints Workflows --> |call| Endpoints Endpoints --> |delegate| Client Client --> |talks to| API Client --> |uses| Cache["(Caches)"]

Extension Points

graph TD BaseEndpoint --> NewEndpoint[Custom Endpoint] Workflows --> NewWorkflow[Custom Workflow] NewEndpoint --> |register| SDK NewWorkflow --> |expose via| CLI

Adding New Endpoints

• Subclass GenericEndpoint or GenericListGetEndpoint.

• Register the class in _ENDPOINT_REGISTRY within imednet.sdk so ImednetSDK exposes it.

• Document the endpoint in docs/endpoints/ and add tests.

Adding New Workflows

• Create a workflow under imednet/workflows and provide sync and async methods where appropriate.

• Instantiate the workflow in Workflows inside imednet.sdk.

• Add CLI commands or examples that demonstrate the workflow.

• Update documentation and tests.

Export Sink Architecture

The SDK supports four distinct export paths. Choose the right path based on the shape of data you want to land at the destination.

Export Path Decision Matrix

Path	When to use	Key components
Tabular	Destination is a flat/columnar store (CSV, Excel, SQL, DuckDB, Parquet). All variables for a form can be represented as columns.	RecordMapper → pandas.DataFrame → sink function in imednet.integrations.export
Document	Destination is a document store (for example MongoDB) and should keep nested record_data payloads with metadata envelope fields.	DataExtractionWorkflow → typed Record list → MongoDbExportSink
Graph	Destination models relationships natively (for example Neo4j) and the hierarchy Study → Subject → Visit → Record should be preserved.	DataExtractionWorkflow → typed Record list → Neo4jExportSink
Warehouse	Destination is a cloud data warehouse with a native bulk loader. Records are staged as Parquet files and then COPY’d in a single command for best throughput.	RecordMapper → Parquet staging → SnowflakeExportSink

Data flow diagram

graph TD SDK["ImednetSDK"] --> RM["RecordMapper (tabular)"] SDK --> DEW["DataExtractionWorkflow (structured)"] RM --> DF["pandas.DataFrame"] DF --> T1["CSV / Excel / JSON"] DF --> T2["SQL / DuckDB"] DF --> T3["Parquet (local)"] DEW --> REC["Typed Record list"] REC --> D["MongoDbExportSink"] REC --> G["Neo4jExportSink"] T3 --> W["SnowflakeExportSink (COPY INTO)"]

Shared ExportSink contract

All non-tabular sinks subclass ExportSink and must implement three methods:

write_batch(records, *, batch_id) -> int

Write one batch to the destination. The batch_id is a caller-supplied idempotency key in the format "<study_key>/<form_key>/<batch_n>". Returns the number of records written.

flush() -> None

Flush any internal buffers.

close() -> None

Release all held resources. Must be idempotent.

Sinks are used as context managers; flush is called automatically on clean exit and close is always called via __exit__.

Shared SinkConfig fields:

Field	Default	Description
batch_size	500	Records per write_batch() call.
max_retries	3	Retry attempts on transient errors.
retry_backoff	1.0	Base delay (seconds); grows as backoff * 2^attempt.
idempotent	True	Use upsert / MERGE / FORCE=FALSE semantics.

Error propagation

• Transient errors are retried up to max_retries times with exponential back-off.

• After all retries are exhausted, sinks raise ExportBatchError (carries batch_id) so that callers can log and resume partial exports.

• Misconfiguration (missing credentials, invalid URIs) raises ExportConfigurationError immediately before any data is written.

• Credentials and connection URIs are never written to logs. Pass URIs through _redact_uri() before logging them.

Optional dependency conventions

Each sink module calls _require_optional_dep() at connection time (not at import time). This means:

• import imednet.integrations.graph never raises even when neo4j is not installed.

• The error is raised when the first sink instance is created.

• The error message tells the user which imednet[<extra>] to install.

Extras naming:

pip install 'imednet[neo4j]'       # Neo4j driver
pip install 'imednet[mongodb]'     # PyMongo client
pip install 'imednet[snowflake]'   # Snowflake connector + pyarrow
pip install 'imednet[export]'      # Tabular path (pandas, SQL, Parquet, DuckDB)

Public API exposure

• imednet.integrations re-exports all tabular helpers and all three new sink classes (backward-compatible).

• Graph/document/warehouse sinks can also be imported directly from their submodules:

  • imednet.integrations.graph

  • imednet.integrations.document

  • imednet.integrations.warehouse

• apache_airflow_providers_imednet.export wraps only the tabular helpers. Airflow DAGs that need graph or warehouse sinks should import them directly from imednet.integrations.

Adding New Export Sinks

1. Create a module under packages/core/src/imednet/integrations/.

2. Subclass ExportSink and implement write_batch, flush, and close.

3. Call _require_optional_dep() inside the constructor (not at module level).

4. Add the optional dependency to packages/core/pyproject.toml and define a new [tool.poetry.extras] key.

5. Re-export the new class from imednet.integrations.

6. Add unit tests; run pytest --cov=imednet.integrations.