Data engineering
Reliable ELT pipelines, clear data models and quality tests so your reports and models rest on data you can trust and that refreshes without manual work.
Updated:
Metrics are calculated in different places with slightly different rules. Nobody knows which one is right.
A pipeline that fails silently, takes too long or is fully rebuilt every day makes everything else slower and more expensive.
When a source column changes, nobody finds out until a report goes wrong.
Queries scan more data than necessary and the warehouse bill rises without any gain for the business.
The pattern I usually use separates responsibilities into layers. In base (or staging) each source is loaded almost as is, with consistent names, corrected types and little else: it is the boundary between the outside world and yours. In core, business entities — customers, transactions, products — are modelled with rules, deduplication and well-defined keys. In marts, tables are built ready for each use: a dashboard, a predictive model, an export. Every layer has its tests and documentation, and dbt manages dependencies between models.
Tests are not a final add-on. From the first model I define uniqueness and not-null tests on keys, relationships between tables and business rules (for example, that an amount is not negative except for refunds). I add freshness checks to detect sources that have stopped arriving and, where possible, reconcile totals with the source system. Failures produce concrete alerts instead of silently wrong reports.
To run pipelines on a schedule I use Airflow when the project has enough dependencies to justify it; for simpler needs, simple scheduling is enough. In BigQuery, most of the saving comes from three decisions: querying only the columns you need, partitioning and clustering large tables according to how they are filtered, and avoiding full rebuilds when an incremental load is enough.
I have worked with SQL, dbt and BigQuery in a fintech analytics team (described at a high level for confidentiality) and I maintain a personal lab with more than 250,000 synthetic transactions that illustrates layer organisation, tests and orchestration. You will find it under projects.
Ingestion from your sources into the warehouse (BigQuery or another), with incremental loads where appropriate and error handling.
Base, core and marts layers with naming conventions, documentation and lineage, ready for analytics and models.
Tests for uniqueness, nulls, relationships and business rules; freshness checks and reconciliation against the source.
Scheduled runs and dependencies with Airflow or another tool suited to the size of the project, with failure alerts.
Review of queries and models, partitioning, clustering and materialisations to reduce processing time and cost.
What data exists, who uses it, which decisions depend on it and which figures must reconcile.
I define entities, granularity and metrics with your team before writing transformations, to avoid rework.
One source and one use case end to end before expanding. Every model with its tests from day one.
I compare results with figures you already know and document differences and their cause.
Orchestration, alerts, documentation and code review so your team can maintain it.
Layer structure, conventions and tests from the beginning, with a first dashboard on reliable data.
Turn scattered logic into versioned, tested and documented models.
Analyse why a daily load takes so long and move from rebuilding everything to incremental loads where the data allows.
Normalise different structures into a common model with standardised records.
In ETL, data is transformed before being loaded into the warehouse; in ELT it is loaded first and transformed inside the warehouse with SQL. ELT with tools such as dbt is now the norm for cloud warehouses because it uses their compute and keeps transformations versioned.
My professional experience is with BigQuery and dbt. Modelling, testing and orchestration principles carry over to other SQL warehouses; if your environment is different I will say so clearly before starting.
dbt lets you write transformations as versioned SQL queries with tests and automatically generated documentation. It turns data logic into reviewable code instead of loose queries.
Yes. A scoped review identifies fragile points, costly queries and missing tests, and ends with a prioritised list of improvements.
The experience page describes, at a high level, my work with SQL, dbt and BigQuery at a fintech, and the projects page includes a personal lab with synthetic data showing how I organise layers.
Data engineeringPersonal project
A personal lab that recreates an analytics engineering environment on synthetic financial-crime data, with base, core and marts layers, quality tests and orchestration.
Find where AI adds value in your operations and take it to production with human oversight.
View serviceTurn manual, repetitive tasks into scheduled, traceable and maintainable workflows.
View serviceAn assistant that answers from your documents, cites its sources and respects permissions.
View serviceForecasts and classifications honestly evaluated against simple alternatives.
View serviceA first conversation is enough to know whether it makes sense and where to start.