Case study
flow-watcher
Multi-engine data platform monitoring with SLOs, anomaly detection, and lineage.
Public code pending: the repository will be published once the ownership and sanitisation review is complete.
The problem
Data platforms can fail silently: a missing run does not always produce an error capable of triggering an alert.
Decisions
- Plugin-based connectors, so a new engine can be added without changing the core
- Explicit SLOs per source instead of global thresholds
- A Prometheus exporter instead of a custom dashboard
Architecture
A Python core with connectors for data engines and infrastructure services, anomaly detection, SLO, lineage, and schema drift modules, a Prometheus exporter, and an MCP server.
Results
- Unifies monitoring across different engines through a shared connector interface
- Detects schema drift and missing executions, not only explicit errors