CLICKHOUSE DATA EVENT
AI Loops on Data Streams
Pipelines that keep themselves healthy, around the clock
Fatih Yildiz · Co-founder, Edge Delta · Seattle
ABOUT ME
Hi, I'm Fatih
Living in Leschi, loving Seattle
Father of two boys No dogs yet, still in negotiations 🐶
Obsessed with writing code
Ex-Microsoft
Wrote code, dodged meetings
Ex-Twitter
Transcoded cat videos
Founder @ Edge Delta
Building better observability
Edge Delta · AI Loops on Data Streams
Everyone builds their pipeline once. Nobody maintains it 24/7.
Like my Nest at home: hot day, it cools. Cold day, it heats. I set the temperature once. It keeps the house stable.
PIPELINES 101
Three ways telemetry gets in
01
Push
Data is sent to you. CDNs and Lambda functions, where no agent can run.
02
Pull
You scrape an API on a schedule and fetch what's new.
03
Collect
An agent at the edge reads data where it's produced.
Edge Delta · AI Loops on Data Streams
PIPELINES 101
The work happens in between
Sources
Push Pull Collect
→
Transform · OTTL, Lua, JS
Sample · Parse · Mask PII Aggregate · Roll up metrics Drop dimensions · Tail-sample traces
→
Destinations
ClickHouse Splunk, Datadog Elastic
Common tools: Fluentd, Fluent Bit, OpenTelemetry Collector, Edge Delta
Edge Delta · AI Loops on Data Streams
PIPELINES 101
What every pipeline must be
Performant
Agents run next to your apps. Every CPU cycle is a cost.
Flexible
A transformation language that can reshape any record.
Broad coverage
Many inputs and outputs, so data gets where it's needed.
Pipeline throughput benchmark, mask scenario, logs per second over 18 runs from May to September 2026. Edge Delta rises from about 28 thousand to about 52 thousand logs per second. OpenTelemetry Collector stays near 7 thousand, Fluentd near 4.5 thousand, and Cribl near 2 thousand.
~7× the OpenTelemetry Collector while masking PII
Edge Delta pipeline benchmark · mask scenario, logs/sec · 18 runs, May–Sep 2026
THE PROBLEM
Your goals stay put. Your data drifts.
Volume
Debug logs left on. DNS errors spike.
WHY IT MATTERS
Higher cost for network, ingest, indexing and storage. Slower storage.
Security
PII slips past masking.
WHY IT MATTERS
Breaks compliance: SOC 2, HIPAA, FedRAMP.
Schema
host_name · host.name · hostname
WHY IT MATTERS
Slow queries, broken correlation, poor indexing.
Edge Delta · AI Loops on Data Streams
A pipeline is never done.
Someone has to watch volume, security and schema, every hour of every day. That job is a perfect fit for AI.
WHY AI WORKS HERE
Why verifiability matters
Chart titled Asymmetry of verification can be improved with privileged information. Tasks are plotted by how hard they are to generate and to verify. Best diet and writing a factual essay are hard to verify. Sudoku and building Twitter are easy to verify. Data processing code moves from hard to easy to verify when test cases are written beforehand, and competition math moves down with an answer key.
Below the line, checking is easier than doing. That's where AI thrives.
Privileged information moves a task down: test cases, an answer key.
Capture and test are the test cases for data processing.
Chart: Jason Wei, "Asymmetry of verification and verifier's law" (2025)
THE BUILDING BLOCKS
The verifiable loop
A loop of four steps: capture, test, deploy, verify. An arrow returns from test to capture, labeled iterate five to six times until the test passes. A larger arrow returns from verify to capture, labeled drift detected, run the loop again.
Edge Delta · AI Loops on Data Streams
THE BUILDING BLOCKS
Every step leaves evidence
Capture
Before and after samples at every node. Cheap enough for production, so benchmark it.
Test
Run a change on captured data. See the after-state through the whole chain. No waiting on indexing.
Deploy & verify
Check locally, then on the live agent. Canary and roll out as the customer needs.
Edge Delta · AI Loops on Data Streams
THE BUILDING BLOCKS
Why CLIs beat MCP for telemetry
Small context
Telemetry output is huge. The agent trims it with jq or head before the model reads it.
Composable
Pipes, jq, diff, kubectl, git. One shell line instead of many tool calls.
Familiar to models
Trained on years of shell usage. Fewer wrong calls, better recovery.
MCP still fits where there's no shell, or where you need scoped, governed access.
Edge Delta · AI Loops on Data Streams
LIVE DEMO
One prompt, three goals
Security
Every record passes PII checks
Volume
Noisy sources get cut down
Schema
One unified structure
FROM DEMO TO 24/7
The same loop, around the clock
01
Watch
Volume in and out of every node
02
Detect
A trend shifts or a goal fails
03
Investigate
Captures and indexed data find the source
04
Propose
A fix, tested in a sandbox, with before and after
05
Apply
After approval, then verify again
Edge Delta · AI Loops on Data Streams
FROM DEMO TO 24/7
Autonomy is earned step by step
Recommend
"If I apply this, here's before and after. Approve?"
Fast approve
Approvals become quick as trust builds.
Autonomous
Acts on its own inside guardrails.
Sandbox gives it a safe place to act. Guardrails limit what it may change.
Edge Delta · AI Loops on Data Streams
BUILD THE THERMOSTAT
Three things to take home
01
Make every pipeline goal verifiable.
02
Give AI capture and test tools, not just dashboards.
03
Prefer CLIs, and run the loop in a sandbox with guardrails.
Thank you. Questions?
Fatih Yildiz · Co-founder, Edge Delta fatih@edgedelta.com · edgedelta.com