Deniz

Hi, I'm Deniz 👋

Notes on data engineering and distributed systems. From data warehouses to modern lakehouse architectures; writing about end-to-end data pipelines that carry source data into analytics and machine learning.

Read the posts About

Latest posts

All →

How Many Piano Tuners Are in Chicago? Fermi Problems and the Art of Estimation

How do you give a reasonable answer to an enormous question when you have no data, no measuring instrument, and no sources? Enrico Fermi was famous for doing exactly that. This post covers what Fermi problems are, walks step by step through the famous piano tuner question, explains why the method produces surprisingly accurate results, and why companies from Google to McKinsey still ask these questions in interviews — then applies the same method to a data engineering capacity estimate.

Fermi ProblemsEstimationProblem SolvingInterviewsCapacity PlanningData Engineering
Read more →

Which Data Goes Where: Data Warehouse, Lakehouse, and the Real-Time Analytics Store

Analytical data has three classic homes: the clean, modeled Data Warehouse; the Lakehouse, which keeps raw files and open tables under one roof; and the real-time analytics store that answers queries over streaming events in under a second (ClickHouse, Druid, Pinot, Azure Data Explorer). This post answers "which data goes where?" through each store's identity, the decision signals, the traps that exams and architects love, and how all three flow together in the real world.

Data WarehouseLakehouseReal-Time AnalyticsOLAPData ArchitectureBig Data
Read more →

Language > Framework > Library: A Correct but Incomplete Hierarchy

The familiar ordering is right on the scope axis, but it hides the line that actually separates a framework from a library: who holds control. Independent versus framework-bound libraries, when a library is enough and when a framework is worth it, how to diagnose a tool without asking anyone — and the unexpected place Spark, Airflow, Kafka, and dbt occupy in this picture.

FrameworkLibraryPythonSparkData EngineeringBackend
Read more →

Why SQL Isn't Considered "Flexible" — and Why "SQL or NoSQL" Is the Wrong Question

If ALTER TABLE adds a column in seconds, why does SQL get called "inflexible"? Because flexibility means two different things: the micro-flexibility of adding a column, and the architectural flexibility of changing billions of rows without downtime, storing schemaless data, and scaling horizontally. This post argues that SQL's rigidity is not a flaw but a deliberate safety trade-off — and that the real-world answer is combining both in one project (polyglot persistence).

SQLNoSQLDatabasePolyglot PersistenceScalingBackend
Read more →

Same SELECT, Entirely Different Engine: What Is a DataFrame, and Why PySpark?

The SELECT inside spark.sql("SELECT ...") is the very one you've written in Oracle or PostgreSQL for years — so what does the PySpark around it actually do? From the DataFrame concept and the pandas–Spark divide, to why transformations aren't written in plain SQL, to the industry's SQL/PySpark balance; and most importantly, the distinction that keeps the syntax identical while completely changing the engine, where the data lives, and how it scales — built from classic-SQL reflexes.

DataFramePySparkSpark SQLSQLData EngineeringBackend
Read more →