Deniz

Posts

12 posts in total

How Many Piano Tuners Are in Chicago? Fermi Problems and the Art of Estimation

How do you give a reasonable answer to an enormous question when you have no data, no measuring instrument, and no sources? Enrico Fermi was famous for doing exactly that. This post covers what Fermi problems are, walks step by step through the famous piano tuner question, explains why the method produces surprisingly accurate results, and why companies from Google to McKinsey still ask these questions in interviews — then applies the same method to a data engineering capacity estimate.

Fermi ProblemsEstimationProblem SolvingInterviewsCapacity PlanningData Engineering
Read more →

Which Data Goes Where: Data Warehouse, Lakehouse, and the Real-Time Analytics Store

Analytical data has three classic homes: the clean, modeled Data Warehouse; the Lakehouse, which keeps raw files and open tables under one roof; and the real-time analytics store that answers queries over streaming events in under a second (ClickHouse, Druid, Pinot, Azure Data Explorer). This post answers "which data goes where?" through each store's identity, the decision signals, the traps that exams and architects love, and how all three flow together in the real world.

Data WarehouseLakehouseReal-Time AnalyticsOLAPData ArchitectureBig Data
Read more →

Language > Framework > Library: A Correct but Incomplete Hierarchy

The familiar ordering is right on the scope axis, but it hides the line that actually separates a framework from a library: who holds control. Independent versus framework-bound libraries, when a library is enough and when a framework is worth it, how to diagnose a tool without asking anyone — and the unexpected place Spark, Airflow, Kafka, and dbt occupy in this picture.

FrameworkLibraryPythonSparkData EngineeringBackend
Read more →

Why SQL Isn't Considered "Flexible" — and Why "SQL or NoSQL" Is the Wrong Question

If ALTER TABLE adds a column in seconds, why does SQL get called "inflexible"? Because flexibility means two different things: the micro-flexibility of adding a column, and the architectural flexibility of changing billions of rows without downtime, storing schemaless data, and scaling horizontally. This post argues that SQL's rigidity is not a flaw but a deliberate safety trade-off — and that the real-world answer is combining both in one project (polyglot persistence).

SQLNoSQLDatabasePolyglot PersistenceScalingBackend
Read more →

Same SELECT, Entirely Different Engine: What Is a DataFrame, and Why PySpark?

The SELECT inside spark.sql("SELECT ...") is the very one you've written in Oracle or PostgreSQL for years — so what does the PySpark around it actually do? From the DataFrame concept and the pandas–Spark divide, to why transformations aren't written in plain SQL, to the industry's SQL/PySpark balance; and most importantly, the distinction that keeps the syntax identical while completely changing the engine, where the data lives, and how it scales — built from classic-SQL reflexes.

DataFramePySparkSpark SQLSQLData EngineeringBackend
Read more →

Why No Distributed System Can Be "CA": From CAP to PACELC via the Stock Market, Instagram, and LoL

How do thousands of servers agree on the price of the same stock? Putting the stock market, Instagram, and online games side by side, we rebuild the CAP theorem, its blind spot, and its successor PACELC: why a partition forces you to sacrifice C or A, why the objection "isn't the system CA when nothing is wrong?" doesn't hold, why the speed of light makes a pure CA system physically impossible — and why an exchange won't even settle for a quorum.

Distributed SystemsCAP TheoremPACELCConsistencyConsensusBackend
Read more →

Where Does Big Data Begin? Three Misconceptions, One Real Criterion

Big data does not mean "unstructured data," "a large company's data," or "a real-time system." So can structured data alone be big data? What separates the data an organization shows off from the data that actually flows behind the scenes? Does real-time require big data? This post takes apart three common misconceptions and rebuilds the one criterion that actually draws the line — the architectural difference between traditional tools and big data tools.

Big DataDistributed SystemsScalingReal-TimeData EngineeringBackend
Read more →

No One at the Center: How Open-Source Tools Recognize Each Other

Coming from the Oracle world, where everything lives under one roof, it's startling how open-source tools click together like lego bricks. How does PowerShell recognize git, what does pip install actually do, and which contract lets Airflow talk to Spark? And the real question: who governs any of this? A piece that builds up, step by step, how an order with no one at the center actually works.

Open SourcePATHCLIPythonEcosystemStandardsBackend
Read more →