James Kinyua Njuguna

Crafting High-Performance Distributed Systems, Real-Time Data Lakehouses & Agentic AI

Senior Software & AI Engineer specializing in resilient event-driven architectures, production Retrieval-Augmented Generation (RAG), and cloud data pipelines. Focused on driving business outcomes through clean monorepo patterns, automated compliance constraints, zero-downtime microservices, and end-to-end observability.

Explore My Work View Resume

About Me

I am a Senior Software & AI Engineer dedicated to crafting robust software architectures that drive measurable business velocity. Over the last decade, my work has focused on leading distributed system design, real-time data engineering, and agentic AI integrations that solve mission-critical challenges.

Whether orchestrating e-commerce Lambda Architecture streaming platforms with PySpark and Delta Lake, engineering safety-critical AI assistants using Google Vertex AI and local HNSW vector indexes, or building deterministic workforce management engines in Nx monorepos, I prioritize engineering excellence. I enforce strict automated test suites (E2E/Unit), zero-trust security (WIF/ADC), and real-time observability to ensure flawless production reliability.

When I'm not architecting systems, you can find me cycling outdoors on mountain bike trails or taking peaceful evening walks with my partner. I believe keeping an active balance fuels sharper problem-solving at work.

James Kinyua Njuguna

Lead Engineering Experience

Core impact outcomes and engineering achievements during my Lead Software Engineer Consultancy period.

Lead Software Engineer Consultant

Independent Consulting & Architecture

Delivered safety-critical AI capabilities by architecting Astra Aviation RAG and Astra Document Summary using Vertex AI, Vercel AI SDK, and HNSW vector indexing to automate NTSB accident analysis and complex document synthesis.
Architected an e-commerce Lambda Architecture platform using Apache Kafka, PySpark Structured Streaming, and Delta Lake, engineering 10-second sliding windows and 15-minute session gaps for sub-second tracking of user conversion funnels.
Orchestrated automated ELT pipelines across 7 analytical marts using dbt, DuckDB, and Apache Airflow, exposing live metrics via Streamlit dashboards on a 15-minute refresh cycle within a hermetic Pantsbuild monorepo.
Engineered ShiftSync, a NestJS/Next.js workforce management platform that automated labor law compliance (10-hour rest bounds, daily limits) and multi-location scheduling across disparate time zones via WebSockets.
Prevented I/O blocking during concurrent streaming by architecting a cloud-native video platform that decoupled upload and playback pipelines using Docker, Kubernetes microservices, and event-driven communication.

Projects Portfolio

Explore key engineering projects spanning Agentic AI systems, Real-Time Data Streaming, and Distributed Microservices.

Data & Streaming

Real-Time Data Lakehouse Platform

Lambda Architecture platform engineering 10-second sliding windows and 15-minute session gaps for sub-second tracking of user conversion funnels using Kafka, PySpark Structured Streaming, Delta Lake, dbt, DuckDB, Airflow, and Streamlit.

PySpark 3.5 Kafka Avro Delta Lake dbt / DuckDB Apache Airflow

Source Code

AI & Agentic

Astra Aviation RAG

Safety-critical conversational assistant parsing NTSB aviation accident/incident reports using Retrieval-Augmented Generation (RAG) and local Hierarchical Navigable Small World (HNSW) vector indexing.

Next.js 15 RAG Search HNSWLib Vertex AI

Live Demo

AI & Agentic

Astra Document Summary

Conversational AI web assistant powered by Google Gemini, Vertex AI, and Vercel AI SDK to automate complex document parsing, multi-file chunking, and streaming synthesis.

Next.js 15 Vertex AI Vercel AI SDK Upstash Redis

Live Demo

AI & Agentic

Astra Interview Assistant

Conversational AI application simulating technical job interviews, asking tailored questions based on candidate resumes and job descriptions with structured LLM evaluation loops.

Next.js 15 Vertex AI TypeScript

Live Demo

Distributed & Cloud

ShiftSync Workforce Platform

Multi-location workforce management platform automating labor law compliance (10-hour rest bounds, 12-hour daily limits) and cross-timezone shift scheduling via WebSockets and Drizzle ORM.

NestJS Next.js PostgreSQL Drizzle ORM WebSockets Playwright E2E

Source Code

Distributed & Cloud

FlixTube Microservices

Cloud-native video streaming platform preventing I/O blocking during concurrent streaming by decoupling upload and playback pipelines using Docker, Kubernetes microservices, and RabbitMQ.

Kubernetes Docker RabbitMQ Node.js

Source Code

Technical Expertise

Core technology stacks, developer platforms, and architectural domains mastered over a decade of engineering.

AI & Agentic Systems

LangChain Vertex AI Vercel AI SDK Model Context Protocol (MCP) Retrieval-Augmented Gen (RAG) Vector Search (HNSWLib) Google Antigravity

Data Engineering & Storage

PySpark 3.5 Spark Structured Streaming Delta Lake dbt & DuckDB Apache Kafka GCP (BigQuery / GCS) PostgreSQL & MySQL MongoDB & Redis Apache Airflow Avro & Schema Registry

Backend & APIs

Node.js NestJS Express.js Microservices Architecture REST & GraphQL WebSockets & gRPC

DevOps & Developer Platforms

Docker Kubernetes Terraform Pantsbuild Monorepo GitHub Actions CI/CD Prometheus & Grafana

Languages & Core

Python TypeScript JavaScript SQL Bash / Linux Shell

Testing & Governance

Playwright E2E Jest & Integration Tests PyDeequ & Great Expectations Structured Pino Logging Workload Identity (ADC/OIDC)

Education

Bachelor of Science in Information Technology

Mount Kenya University • Thika, Kenya

2012 - 2016 Research & Innovation Students Association

Let's Keep in Touch!

I'm always open to discussing new engineering leadership opportunities, architecture consultations, or technical collaborations. Drop me a line!

LinkedIn Profile

Or explore my work on GitHub