Design — pinky-connect
Update date : 2026-05-31 19:26
Unified external connector layer for Snowflake pipelines. Handles authentication, pagination, and error recovery so Snowpark SPs don't reimplement HTTP plumbing. Everything runs inside Snowflake — no external scheduler, no intermediary service.
See ADR-0001 for the rationale behind this constraint.
Module inventory
| Module | Description | Status |
|---|---|---|
http.py |
Generic HTTP client — OAuth2, Basic, Bearer, API key; paginate(); concurrent_map() for vectorized UDFs |
planned |
notify.py |
Unified notify() — Email (SYSTEM$SEND_EMAIL), Teams Adaptive Cards, Slack block messages |
planned |
workday.py |
RAAS REST + SOAP (single / paged / parallel), WQL pagination, OAuth2 token refresh | planned |
insee.py |
Melodi file download / query / catalog, INSEE Metadata geo lookups, SIRENE Parquet refs + S3 constants | planned |
Future directions
- Connector registry — declarative YAML listing connectors + auth scheme, so snowflake-provider can provision required secrets automatically
- Retry strategies — configurable backoff per connector (transient HTTP 5xx, rate limits) beyond the current Workday-only retry
- Stream-to-stage — chunked upload for files >100 MB via multipart, bypasses the in-memory limit in
stage_write() - Additional connectors — Salesforce REST, SAP OData, SharePoint Graph API, S3/GCS/Azure Blob (zero-compute COPY via external stage)
- Openflow bridge — once Snowflake exposes
CREATE OPENFLOWDDL, map existing connectors to Openflow configs