Skip to content

Features — pinky-connect

Update date : 2026-05-31 23:07

Feature list ordered by priority. stable = in production. planned = not yet built, order = priority.

Planned

Ordered by priority — closest to implementation first.

MVP — migration from backup

All four modules are production-ready in _BACKUP/snowflake-kit-connect-bak. MVP = clean migration into the pinky-connect repo structure, not new development.

Module Description
http.py Generic HTTP client: OAuth2 (client_credentials + password), Basic, Bearer, API key; paginate() with Link header / next_key / next_param; concurrent_map() for vectorized UDFs
notify.py Unified notify() entry point: Email via SYSTEM$SEND_EMAIL, Teams Adaptive Cards, Slack block messages; NotifyResult pydantic model
workday.py RAAS REST, SOAP (single + paged + parallel), WQL with LIMIT/OFFSET pagination and OAuth2 token refresh
insee.py Melodi file download / query / catalog / ranges, INSEE Metadata geo lookups, data.gouv.fr SIRENE Parquet file references + S3 constants

Also in scope: repo scaffold — pyproject.toml, conda env, pre-commit, CI.

http.py — retry decorator

Configurable backoff for transient HTTP 5xx and rate limits, applicable to any connector beyond Workday.

insee.py — stream-to-stage

Chunked multipart upload for files >100 MB. Primary use case: INSEE full SIRENE export.

Salesforce REST connector

SOQL query + bulk API v2 for high-volume extracts.

SAP OData connector

Generic OData v4 connector with entity set pagination.

ADP connector

File parser for ADP proprietary payroll formats. French ADP formats are notoriously non-standard — scope and format to be confirmed before implementation.

Kyriba connector

Generic config-driven payment file pipeline: decrypt → SHA256 integrity check → re-encrypt with Kyriba public key → Kyriba naming convention → stage write. Partner config (KyribaPartnerConfig) drives filename templates and accepted extensions. Depends on pgpy.

Backlog

  • SharePoint Graph API — document library read/write
  • S3/GCS/Azure Blob — zero-compute COPY via external stage config
  • Openflow bridge — map connectors to CREATE OPENFLOW DDL once SDK supports it
  • Connector registry — YAML-driven secret provisioning via snowflake-provider