Product & Engineering department
Data engineering, run as an ongoing function
Most reporting problems start upstream: a pipeline that fails silently, a spreadsheet export someone forgot, two tables that both claim to be the customer list. Nexoryl runs data engineering as a managed department: ingestion, pipelines, warehouse, data models and quality monitoring, built and operated remotely in your cloud and data stack.
The first button opens an email to info@nexoryl.com.
You bring
- Your source systems and APIs
- Your cloud account and warehouse, or a choice of one
- The questions reports need to answer
Nexoryl runs
- Ingestion
- ELT pipelines
- Data models
- Quality tests
- Monitoring
You get
- One trusted copy of the data
- Pipelines that alert before reports break
- Models analysts can build on
- Operating model
- A managed data engineering function with an architecture, code in your repositories and pipeline monitoring with named owners.
- Delivery model
- Remote. Pipelines run in your cloud accounts; code lives in your repositories with CI.
- Usual starting scope
- A platform assessment, then the most important sources moved into a modelled warehouse.
- Typical roles
- A data engineering lead, data engineers and an analytics engineer.
- Runs inside
- Snowflake, BigQuery, Redshift or Databricks, with dbt and an orchestrator such as Airflow or Dagster.
- Engagement
- An assessment and platform build, then a monthly retainer for operations and changes.
Overview
What is data engineering?
Data engineering is the work of moving data out of the systems that create it, cleaning and modelling it, and keeping it available and correct for the people and tools that use it.
In practice that means ingestion from applications and APIs, pipelines into a warehouse, data models analysts can query, and the tests and monitoring that stop bad data reaching a dashboard.
Nexoryl model
What does Nexoryl's Data Engineering department do?
It builds and runs your data platform: connecting sources, building and scheduling pipelines, designing the warehouse, modelling data in code, and monitoring freshness and quality so problems are caught before anyone reports on them.
Reporting, dashboards and analysis sit with Data & Analytics. Cloud infrastructure and cost controls are shared with DevOps & Cloud.
- Responsible for
- Ingestion, ELT and ETL pipelines, orchestration, warehouse design, data modelling, data quality tests, monitoring and documentation.
- Not included
- Building dashboards and reports, machine learning model development, and physical server or data-centre work.
- Connects to
- Data & Analytics, DevOps & Cloud, AI & Automation and Software Engineering.
The data flow
How data gets from your systems to a decision.
Six layers, each with an owner and a test. A problem in one shows up in that layer's monitoring, not in a board report.
- 01
Sources
Applications, databases, SaaS tools and files that create the data.
- 02
Ingest
Data extracted on a schedule or in near real time.
- 03
Store
Raw data landed unchanged in your warehouse or lake.
- 04
Transform
Cleaned, joined and modelled into tables people can trust.
- 05
Test
Freshness, volume and validity checked on every run.
- 06
Serve
Models exposed to BI tools, analysts and applications.
Where data platforms fail
Four problems that start upstream of every report.
When a dashboard is wrong, the cause is usually here, not in the dashboard.
Silent pipeline failures
- Shows up as
- Dashboards show yesterday's numbers and nobody notices.
- We look at
- Orchestration, alerting and freshness checks.
- What changes
- Freshness and volume tests, with alerts routed to a named owner.
Several versions of the truth
- Shows up as
- Two customer tables with different counts.
- We look at
- The source of record and the modelling layers.
- What changes
- One modelled layer with documented definitions.
Manual exports
- Shows up as
- Someone downloads a CSV every Monday and pastes it somewhere.
- We look at
- Which data still moves by hand.
- What changes
- Automated ingestion, so the export becomes a pipeline.
Unknown warehouse cost
- Shows up as
- The warehouse bill rises every month and nobody knows why.
- We look at
- Query patterns, schedules and storage.
- What changes
- Usage monitoring and cost-aware scheduling with DevOps & Cloud.
What the department covers
Data engineering services.
Built in code, tested on every run and documented, so the platform doesn't depend on one person's memory.
Ingestion and integration
Data out of the systems that create it.
- SaaS and API connectors
- Database replication
- File and event ingestion
- Change data capture
Pipelines and orchestration
Data moved on time, in the right order.
- ELT and ETL pipelines
- Airflow, Dagster or Prefect orchestration
- Scheduling and dependencies
- Backfills and reprocessing
Warehouse and lakehouse
A platform designed for how you use data.
- Platform selection support
- Warehouse setup
- Schema and layer design
- Role-based access
Data modelling
Tables analysts can trust and build on.
- dbt models
- Analytics-ready dimensional tables
- Metric definitions in code
- Documentation and lineage
Data quality and observability
Problems caught before they reach a report.
- Freshness, volume and validity tests
- Anomaly alerts
- Incident runbooks
- Data contracts with source owners
Operations and cost
The platform kept healthy and affordable.
- Pipeline monitoring
- Incident response within agreed hours
- Warehouse cost monitoring
- Platform upgrades
How remote delivery works
How a data engineering engagement runs.
Old and new are reconciled before anything switches over, so reports don't change without explanation.
Step 1: Assess
Sources, current pipelines, warehouse and pain points reviewed.
Step 2: Design
Target architecture and modelling approach agreed with your team.
Step 3: Build
Pipelines, models and tests written in your repositories, with CI.
Step 4: Migrate and validate
Existing reports reconciled against the new models before switching.
Step 5: Operate
Monitoring, incidents and change requests handled, with a monthly review.
Code lives in your repositories and runs in your cloud accounts. Credentials are scoped to what each pipeline needs.
Scope and boundaries
What's included, and what stays with you.
Data engineering stops where reporting starts. Each side has its own department.
Included
- Ingestion and pipelines
- Warehouse or lakehouse setup
- Data models and documentation
- Data quality tests and monitoring
- Orchestration and CI for data code
- Ongoing pipeline operations
Not included
- Dashboards and reports (Data & Analytics)
- Machine learning model development
- Physical servers or data-centre work
- Cloud and tool licence costs, which are billed to you
What you provide
- Access to source systems or API credentials
- Cloud account and warehouse access
- Repository access
- Data owners for definitions and approvals
Tools and platforms
Where the work happens.
The stack fits your scale. A small team doesn't need the same platform as an enterprise.
- Warehouses
- SnowflakeBigQueryAmazon RedshiftDatabricksPostgreSQL
- Ingestion
- FivetranAirbyteCustom Python connectors
- Transformation and orchestration
- dbtAirflowDagsterPrefect
- Quality and observability
- dbt testsGreat ExpectationsYour observability tool
Named as examples of tools we work in. They don't imply a partnership or certification.
Measurement
How a data engineering department is measured.
Reliability first: data that arrives on time and is correct.
Reliability
- Pipeline success rate
- Data freshness against targets
- Incidents caught before users report them
Quality
- Test coverage on key models
- Open data issues
- Reconciliation differences
Delivery
- Sources onboarded
- Time to add a new source
- Change requests completed
Cost
- Warehouse spend trend
- Cost per pipeline or domain
- Idle resources removed
These are measurement categories, not promised figures.
Who does the work
The people building your data platform.
Sized to the number of sources and the pace of change.
Data Engineering Lead
Owns the architecture, standards and the monthly review.
Data Engineers
Build and run ingestion, pipelines and orchestration.
Analytics Engineer
Models data in dbt and keeps definitions documented.
A good fit when
- Reports break or disagree and nobody can say why
- Data still moves by manual export
- You're setting up a warehouse for the first time
- Your data platform depends on one person
Probably not the right fit when
- You only need dashboards on clean data (see Data & Analytics)
- You need machine learning models built
- You need on-premises hardware installed
Ways to start
Three ways to begin.
The assessment shows which sources and pipelines carry the most risk.
Data platform assessment
Sources, pipelines, warehouse and costs reviewed, with a target architecture.
Platform build
Priority sources ingested, modelled and tested in a warehouse as a scoped project.
Managed data engineering
Ongoing operations, new sources and changes on a monthly retainer.
Fees depend on the number of sources, data volumes and platform choice, and are agreed after the assessment. Cloud and tool costs are billed to you.
Questions
Questions about data engineering.
If your question isn't here, email us. Most answers depend on your sources and volumes.
What's the difference between data engineering and data analytics?
Data engineering builds and runs the pipelines, warehouse and models that make data available and correct. Data & Analytics turns that data into reports, dashboards and analysis.
Which data warehouses do you work with?
Snowflake, BigQuery, Amazon Redshift, Databricks and PostgreSQL, among others. If you don't have one yet, we help you choose based on your scale and existing cloud.
Do you use dbt?
Usually, for modelling and testing in code. Where another approach fits your stack better, we explain why before using it.
Can data engineering be done remotely?
Yes. Pipelines run in your cloud accounts and code lives in your repositories, so all of the work is done over secure remote access.
How do you stop pipelines failing silently?
Every key model has freshness, volume and validity tests, with alerts routed to a named owner and a runbook for common failures.
Will our existing reports change?
Not without explanation. Existing reports are reconciled against the new models before switching, and any differences are documented and agreed.
Who pays for cloud and tool costs?
You do, directly to the providers. The department monitors usage and recommends changes that reduce cost.
What do you need from us to start?
Access to source systems or API credentials, your cloud and warehouse accounts, repository access, and data owners who can agree definitions.
Next step
Fix the data before you fix the dashboard.
Tell us where your data lives and what keeps breaking. We'll assess the platform and show you where the risk sits.