Every data project in your organisation starts the same way. A data engineer spends two weeks setting up infrastructure: provisioning compute, configuring storage, wiring up orchestration, writing boilerplate ingestion code. The project is unique. The setup is not.
Software engineering solved this years ago with platform teams and golden paths. Data teams are still hand-rolling bespoke infrastructure per project, burning engineering time on undifferentiated work.
An internal data platform changes this. Not a vendor product. Not a data catalogue with a rebrand. A self-serve layer that lets data engineers and scientists ship work without filing infrastructure tickets.
Strip away the marketing. A useful internal data platform provides five things:
Self-serve compute. Data engineers can spin up processing environments (Spark clusters, dbt runners, notebook servers) without waiting for DevOps. Guardrails prevent runaway costs.
Standardised storage. One pattern for landing raw data, one for transformed data, one for serving. Not three different S3 bucket conventions invented by three different teams.
Pipeline orchestration as a service. A shared Airflow or Dagster instance with templates. Teams define DAGs; the platform handles scheduling, monitoring, alerting, and retries.
Data cataloguing. Every dataset is discoverable with schema, ownership, freshness, and lineage. If an analyst has to Slack someone to find a table, the platform has failed.
Environment provisioning. Development, staging, and production environments that are consistent and reproducible. Infrastructure as code, not infrastructure as tribal knowledge.
A data platform is not a product you buy. It's the internal tooling that eliminates the undifferentiated setup work from every data project.
A golden path is an opinionated, well-supported way to accomplish a common task. It's not a mandate. It's the path of least resistance that happens to follow best practices.
Ingestion golden path. A templated connector framework. To ingest a new source: fill in a config file (source type, credentials, schedule, destination), run a CLI command, get a pipeline. No custom code for the 90% case.
Transformation golden path. A dbt project structure with standard naming conventions, pre-configured testing, and CI/CD integration. New models follow the template. Code review catches deviations.
Serving golden path. A standard pattern for exposing data to consumers: materialised views for dashboards, API endpoints for applications, exports for external partners. Each pattern is documented with examples.
Teams can deviate from golden paths when they have a genuine reason. But the default should be fast, safe, and boring.
Golden paths aren't mandates. They're the path of least resistance that happens to follow best practices. Make the right thing the easy thing.
A platform exists to make data teams faster. Measure that, not vanity metrics.
| Metric | What it tells you | Target |
|---|---|---|
| Time to first pipeline | How long from "new project" to "data flowing" | Under 1 day |
| Ticket deflection rate | How often teams self-serve vs filing requests | Over 80% |
| Onboarding time | How long for a new data engineer to ship their first change | Under 1 week |
| Pipeline incident rate | How often platform-managed pipelines break | Trending down quarter over quarter |
| Developer satisfaction (survey) | Whether the platform is actually helping or just adding process | Above 7/10 |
If your platform adds process without removing friction, it's governance theatre, not a platform.
| Layer | Build when | Buy when |
|---|---|---|
| Orchestration | You need deep customisation or have existing Airflow expertise | Managed Airflow (MWAA, Astronomer) covers your needs |
| Catalogue | Your metadata model is unique to your domain | DataHub, Atlan, or a cloud-native catalogue fits |
| Compute provisioning | You're multi-cloud or have complex isolation requirements | Single-cloud with native services (EMR, Dataproc) |
| Data quality | Quality rules are domain-specific and tightly coupled to transformations | Great Expectations or Soda covers standard checks |
| CI/CD for data | You have specific testing and deployment requirements | GitHub Actions or GitLab CI with dbt Cloud |
The general principle: buy commodity layers, build differentiated ones. Orchestration and compute are commodity. Your specific ingestion patterns and data product interfaces are differentiated.
The platform that nobody asked for. A platform team builds elaborate tooling without consulting the data engineers who will use it. Adoption is near zero. Build with your users, not for them.
Mandating the platform before it's ready. Forcing all teams onto a half-built platform breeds resentment. Ship a minimum viable platform, get one team using it successfully, then expand.
Platform as gatekeeper. If the platform team becomes a bottleneck that teams must route through for every change, you've replaced one problem with another. Self-serve is the goal.
Ignoring the feedback loop. Platforms without usage metrics and user feedback stagnate. Instrument everything. Run quarterly surveys. Deprecate features nobody uses.
If your data platform adds process without removing friction, it's governance theatre. Measure adoption and developer satisfaction, not feature count.
You don't need a dedicated platform team of five to begin. Start with one engineer and one golden path:
The platform grows from proven patterns, not architecture diagrams. Ship the template before you ship the vision.