Writing a PySpark Custom Data Source
Adding a JSONPlaceholder connector to the pyspark-data-sources repo: how the Python Data Source API works, and where a fake-data source is useful for testing in Databricks.
Data Engineer building scalable systems
Welcome to my digital garden where I share learnings and ideas on data and software engineering.
Recent Posts
Building muse — a private multi-model AI tool
Why I built muse: a BYOK AI tool that queries multiple LLMs, synthesizes one answer with a trust score, and runs on-device via MLX. Now with an iOS app.
Building a Modern Data Pipeline: Scala 3 + Kafka + Flink + Spark
How I built a streaming data pipeline using Scala 3 and Scala 2.13 together — with Kafka, Flink, Spark, and ZIO orchestration.
Building a Second Brain with LLMs
How I built a working knowledge graph of my own work in Obsidian using Andrej Karpathy's LLM Wiki approach.
slidesmd — Make Your PowerPoint Presentations Queryable by AI
Auto-index your .pptx files into a single agents.md that any AI can read. Drop a file in the folder, agents.md updates automatically. Ask Claude or a local Ollama model anything across all your decks.
ClipAI CLI — Local AI Text Assistant for the Terminal
A terminal-native AI text assistant powered by Ollama. No API key, no cloud — runs entirely on your machine with clipboard, pipe, and interactive TUI modes.