Skip to content

AI/ML Systems Architect · Infrastructure & MLOps @ eUp Group

Taking AI modelsinto real production.

Speech (ASR, pronunciation and tone scoring) and embedding services on GCP and DigitalOcean for users in VN, JP, KR and CN. Cost-conscious by default: GPU right-sizing, CPU-first inference, self-hosted alternatives to paid APIs.

Hire MeDownload CV
  • Kubernetes & GitOps (GKE)
  • Production ML Serving
  • Agentic RAG & LLM Systems
  • IaC (Terraform · Ansible)
  • Cloud Cost Engineering
  • Multi-cloud (GCP · AWS · OCI)

01Công Anh DũngHanoi, VietnamVI / EN

Years building and operating AI systems in production
3+
Speech scoring services live across JP, KR, CN and EN markets
4
Cloud platforms in production (GCP, AWS, DigitalOcean, OCI, Cloudflare)
5
Japanese ASR models published on Hugging Face
3

Core Expertise

How I build and operate AI infrastructure.

Four areas I work in depth: architecture and cost control, Kubernetes operations, model serving, and LLM application design.

  • 01

    Solution Architecture & Cost Engineering

    System design for ML serving, benchmark-driven GPU sizing, and root-cause analysis — a measured p95 of 1.86s under concurrent load at ~8% GPU utilization proved a commodity CUDA GPU was enough where an H100 plan would have cost ~$2,475/mo.

  • 02

    Cloud & Kubernetes Infrastructure

    End-to-end ownership: Terraform/Ansible provisioning, Docker, and Kubernetes on GKE, DigitalOcean, and bare-metal kubeadm — plus AWS EC2 deployment automation, ArgoCD GitOps, and Cloudflare Tunnel.

  • 03

    ML Serving & MLOps

    Shipping ASR (faster-whisper/CTranslate2), TTS, pronunciation scoring (wav2vec2), and embedding (Qwen3) services with full CI/CT/CD pipelines, quality gates, and Prometheus/CloudWatch observability.

  • 04

    Agentic RAG & LLM Application Architecture

    Multi-agent orchestration (Google ADK) with hybrid keyword/semantic retrieval, rule-based slot-filling, and grounding verification — so code owns every figure and the LLM only phrases it. Provider-agnostic serving across Gemini, Claude, and self-hosted vLLM/Ollama.

  • Migrating remaining production ML services from Docker Swarm to GKE with Helm and ArgoCD App-of-Apps
  • Hardening the internal Qwen3 embedding service and its RAG code-review pipeline
  • Treating the home LAN like production — Ansible IaC for the Raspberry Pi, Slack ChatOps alerts, and GPG-encrypted config backups
  • Designing agentic RAG systems where code owns orchestration and retrieval, and the LLM only phrases the answer

Selected Work

Infrastructure, ML serving, and platform engineering work.

Nine systems that shipped, plus one case study taken from problem to measured result.

AI Gateway: a resilient reactive layer for multi-provider LLM traffic

How I turned fragmented model integrations into a unified, fault-tolerant gateway.

  • Reactive Spring WebFlux gateway routing OpenAI / Gemini / Anthropic / DashScope traffic
  • Resilience4j circuit breaker, bulkhead, and retry for graceful degradation
  • Per-request token-usage tracking under one OpenAI-compatible API contract
Repository
Problem
Teams needed one stable API across multiple LLM providers, but each had different request/response semantics, rate limits, and failure patterns.
Architecture
A Java Spring WebFlux gateway handles routing, resilience, and policy control; a Python FastAPI worker serves local models and embedding tasks behind it.
Trade-offs
Chose reliability and observability over minimal complexity — more moving parts, but consistent behavior when a provider degrades or rate-limits.
Result
A reusable gateway baseline for future AI products that reduces integration overhead and standardizes production controls.
What I'd improve next
Per-provider latency dashboards, token-cost analytics, and automated fallback tuning driven by live traffic signals.

  • eUp · 2025-2026

    Multi-Market Speech Scoring Platform

    Four FastAPI/Gunicorn services scoring pronunciation for JLPT (Japanese), TOPIK (Korean), HSKK (Mandarin) and English, each with a multi-engine STT layer — Kotoba-Whisper, faster-whisper/CTranslate2, SenseVoice ONNX, ReazonSpeech — behind automatic fallback.

    Scoring is linguistic, not generic: Needleman-Wunsch alignment between reference and hypothesis, wav2vec2 CTC goodness-of-pronunciation with espeak-ng G2P, Japanese pitch accent via SudachiPy/MeCab/pykakasi, Mandarin tone classification, and Praat/parselmouth prosody behind a concurrency semaphore. Under benchmarked concurrent load the serving path holds p95 below two seconds (1.86s) at roughly 8% utilisation of one commodity CUDA GPU — capacity here is bounded by infrastructure budget, not by the models — and in production a 30-minute window on the Japanese market alone ran 740 consecutive scoring requests without a single non-2XX response.

    • FastAPI
    • Gunicorn
    • CTranslate2
    • wav2vec2 CTC
    • ONNX Runtime
    • parselmouth

  • eUp · 2025-2026

    Hey Translate

    Translation service that runs Claude, Gemini, GPT and Qwen in parallel on the same input rather than trusting a single provider.

    • Python
    • Claude
    • Gemini
    • GPT
    • Qwen
    • LLM-as-judge

  • eUp · 2025-2026

    Ultimate Lesson

    Pipeline that turns a YouTube URL into a structured language lesson: yt-dlp to Whisper transcription to GPT-4o-mini subtitle cleanup, then Gemini 2.5 Flash for generation with a fine-tuned Flash Lite extractor on Vertex AI.

    • Gemini 2.5 Flash
    • Vertex AI
    • Whisper
    • GPT-4o-mini
    • Redis
    • yt-dlp

  • eUp · 2025

    Internal Embedding Service

    Self-hosted Qwen3-Embedding-4B exposing an OpenAI-compatible /v1/embeddings endpoint on an RTX 4080, replacing the OpenAI Embedding API for internal workloads.

    • Qwen3-Embedding-4B
    • FastAPI
    • LanceDB
    • Docker Compose
    • GitLab CI
    • RTX 4080

  • 2025-2026

    AI Gateway

    Java 21 Spring WebFlux gateway plus a Python FastAPI worker that unify multi-provider LLM access (OpenAI, Gemini, Anthropic, DashScope) behind one API, with Bucket4j rate limiting and Redis-backed state.

    • Java 21
    • Spring WebFlux
    • Resilience4j
    • Bucket4j
    • Terraform
    • GKE Autopilot
    Repository

  • 2025-2026

    Homelab Kubernetes & GitOps Platform

    A 3-node Kubernetes v1.31 cluster bootstrapped by hand with kubeadm instead of a managed distro: Flannel CNI, MetalLB (L2), ingress-nginx, local-path storage and a self-hosted registry.

    • kubeadm
    • ArgoCD
    • MetalLB
    • Prometheus
    • Grafana
    • Loki
    Repository

  • 2026

    Whisper Finetune JA

    Reproducible LoRA fine-tuning pipeline for Japanese Whisper ASR on ReazonSpeech data, with training, INT8 export and inference scripts.

    • PyTorch
    • LoRA/PEFT
    • CTranslate2 INT8
    • Kaggle
    • GitHub Actions
    • Hugging Face
    RepositoryModels on Hugging Face

  • 2026

    Raspberry Pi Homelab — Ansible IaC & Slack ChatOps

    The Raspberry Pi serving my whole LAN turned into infrastructure as code: one idempotent Ansible playbook rebuilds it from a bare OS (fstab mounts, Docker CE, journald caps, cloudflared, Tailscale, AdGuard Home, cleanup cron).

    • Ansible
    • Docker
    • AdGuard Home / DoH
    • Cloudflare Worker
    • Tailscale
    • Slack API

  • 2025

    NewsPulse Reco Engine

    Event-driven Vietnamese news platform: crawling, Spark ETL, Kafka streaming, embeddings, trend detection and FCM push via an n8n orchestration flow.

    • Scala 3
    • Kafka
    • Spark
    • Elasticsearch
    • n8n
    • Firebase FCM
    Repository

Experience

Three years shipping and operating AI systems in production.

Three years across three places: conversation analytics at FPT Smart Cloud, AI engineering at AMELA, now owning AI/ML infrastructure at eUp Group.

  1. Jan 2025 — Present

    AI/ML Systems Architect — Infrastructure & MLOps @ eUp Group

    Own AI infrastructure end-to-end for the HeyJapan product line — provisioning, delivery, model serving and observability — and architect the speech, translation and lesson-generation systems on top of it.

    • Architected a multi-market speech scoring platform (JLPT, TOPIK, HSKK, English): 4 FastAPI/Gunicorn services with multi-engine STT — Kotoba-Whisper, faster-whisper/CTranslate2, SenseVoice ONNX, ReazonSpeech — behind automatic fallback.
    • Ruled out a ~$2,475/mo H100 plan with a benchmark instead of an opinion: p95 1.86s under concurrent load on ~8% of a commodity CUDA GPU. Also ran the RCA that closed a 3x ASR latency gap between two environments.
    • Migrated production ML services from Docker Swarm to Kubernetes (GKE and bare-metal kubeadm) with Terraform, Ansible, Helm and ArgoCD App-of-Apps; Prometheus, Grafana and Loki for observability.
    • Replaced the OpenAI Embedding API with a self-hosted Qwen3-Embedding-4B service and built the RAG code-review pipeline it feeds, gated by an exit-code check in GitLab CI.
    • Build and release engineering: Jenkins signed tags, GitLab CI with pytest and Docker-in-Docker, Harbor registry, gitleaks and Qodana quality gates.
    • Terraform
    • Ansible
    • Kubernetes
    • ArgoCD
    • FastAPI
    • CTranslate2
    • Prometheus

  2. Nov 2024 — Jan 2025

    AI Engineer @ AMELA Technology

    Delivered computer vision and retrieval-augmented generation features for client products.

    • Built a Japanese handwriting OCR pipeline, from data preparation through model training to a serving endpoint.
    • Shipped RAG chatbots over client document sets, and an AI-guided cosmetic eyeliner feature driven by facial landmark detection.
    • PyTorch
    • OCR
    • RAG
    • OpenCV

  3. Mar 2023 — Nov 2024

    AI Engineer @ FPT Smart Cloud

    Worked on FPT AI Enhance, the conversation analytics platform serving Home Credit, FE Credit, MB Bank, FPT Long Chau and FPT Shop.

    • Developed conversation analytics features on production call-centre data for enterprise banking and retail customers.
    • Implemented LLM guardrails, and built virtual assistant chatbots for the State Securities Commission of Vietnam.
    • Python
    • NLP
    • LLM Guardrails
    • Chatbot

VNU University of Science (HUS)

Computer and Information Science · Mar 2019 — Jun 2025

AWS Certified Solutions Architect — Associate (SAA-C03)

Engineering Playbook

Operating principles for production AI infrastructure.

The tools I use daily, and the five principles that decide how I use them.

  • Terraform
  • Ansible
  • Kubernetes (GKE · kubeadm · Autopilot)
  • GCP (GKE · Vertex AI · Artifact Registry)
  • AWS (EC2 · CloudWatch)
  • DigitalOcean
  • faster-whisper / CTranslate2
  • Kotoba-Whisper
  • SenseVoice ONNX
  • LangChain / LangGraph
  • Google ADK
  • MCP
  • Python · FastAPI
  • Java 17/21 · Spring WebFlux
  • Scala 3
  • GitLab CI (pytest · DinD)
  • GitHub Actions
  • Jenkins

  • Infrastructure & Platform

    • Terraform
    • Ansible
    • Kubernetes (GKE · kubeadm · Autopilot)
    • ArgoCD
    • Helm
    • Kustomize
    • Docker
    • Harbor
    • Cloudflare Tunnel
    • Tailscale

  • Cloud

    • GCP (GKE · Vertex AI · Artifact Registry)
    • AWS (EC2 · CloudWatch)
    • DigitalOcean
    • OCI
    • Workload Identity
    • Preemptible pools
    • vast.ai

  • ML & Speech

    • faster-whisper / CTranslate2
    • Kotoba-Whisper
    • SenseVoice ONNX
    • ReazonSpeech
    • wav2vec2 CTC (GOP)
    • LoRA / PEFT
    • espeak-ng G2P
    • Praat / parselmouth
    • SudachiPy · MeCab · pykakasi

  • LLM & Agents

    • LangChain / LangGraph
    • Google ADK
    • MCP
    • A2A Protocol
    • Gemini · Claude · GPT · Qwen
    • vLLM · Ollama
    • Qwen3 Embedding
    • LanceDB · Milvus

  • Backend & Data

    • Python · FastAPI
    • Java 17/21 · Spring WebFlux
    • Scala 3
    • Next.js
    • Kafka
    • Spark Streaming
    • Delta Lake
    • Elasticsearch
    • Redis
    • PostgreSQL

  • Delivery & Observability

    • GitLab CI (pytest · DinD)
    • GitHub Actions
    • Jenkins
    • Prometheus
    • Grafana
    • Loki
    • gitleaks
    • Qodana
    • n8n

  1. 1

    Decide with frameworks, not gut

    Compare GPU and cloud options on price, data-center location, SLA, and VN-JP-KR latency together — never on raw price alone.

  2. 2

    Cost-conscious by default

    Size hardware to the measured workload (wav2vec2 ~315M in FP16 needs ~2GB VRAM) and settle CPU-vs-GPU trade-offs with a benchmark before provisioning anything.

  3. 3

    CLI-first, reproducible operations

    Automate provisioning and deploys with Terraform, Ansible, and GitOps so any environment is rebuildable from code.

  4. 4

    Ship through GitOps and quality gates

    Deliver via ArgoCD App-of-Apps and CI exit-code gates that fail fast on corruption or regressions before they reach users.

  5. 5

    Instrument and degrade gracefully

    Observe every service with kube-prometheus-stack and treat provider or GPU failure as normal, with predefined fallback paths.

Data Snapshot

GitHub activity and repository signals.

Every number below is read straight from the GitHub API and frozen at the moment of the snapshot, so the page always shows exactly the data the build saw — nothing on this band is drawn by hand.

GitHub snapshot

Public repositories
64
Own repositories, not forks
49
In scope since 2025
15
Languages used in scope
7

Language mix

  • Python7 · 47%
  • Unknown3 · 20%
  • HTML1 · 7%
  • Java1 · 7%
  • Makefile1 · 7%
  • Scala1 · 7%

Update rhythm by month

OctNovDecJan '26FebMarAprMayJunJulAugSep

8 repos

Repository freshness

  • Updated <= 30 days1
  • Updated 31-90 days1
  • Updated 91-180 days2
  • Updated > 180 days11

Most recently active repositories

Scope: public repositories with commits since 2025. University coursework is excluded, and most production work at eUp, AMELA and FPT lives in private repositories, so it does not appear here.

Technical notes

Infrastructure, MLOps and what cost engineering taught me.

Things I tried, broke and fixed — mostly around GPU cost, LLM serving and operations.

  • Why I stopped letting the agent orchestrate itself

    A hybrid pattern for agentic RAG: the framework owns the runtime, code owns orchestration and every number, and the LLM is restricted to phrasing — with grounding verification as the enforcement layer.

    Repositorydungca1512/research-agent

  • Provider-agnostic LLM serving: cloud API or local vLLM behind one env var

    Designing the LLM layer as an interface so a CPU-only VM ships on a cloud API today and swaps to on-premise vLLM later without touching application code.

  • GPU Cost Engineering: let the benchmark pick the hardware

    Measuring before buying: p95 1.86s under concurrent load on ~8% of a commodity GPU, why the H100 line item (~$2,475/mo) was never justified, and how CPU-only serving died on a ~1 req/s throughput wall.

  • Home infrastructure deserves production discipline

    Rewriting a hand-configured Raspberry Pi as one idempotent Ansible playbook: DNS latency 199ms -> 27ms, Slack ChatOps for status and outage alerts, lint/secret-scan CI gates, and GPG-encrypted backups with a recovery runbook.

  • Docker Swarm to GKE: migrating production ML services

    Notes on moving live ML workloads to Kubernetes with Helm, ArgoCD App-of-Apps, and a bare-metal kubeadm homelab for testing.

    Repositorydungca1512/homelab

  • AI Gateway: reactive resilience for LLM traffic

    A practical breakdown of provider routing, Resilience4j controls, and a unified API contract for multi-provider LLM products.

    Repositorydungca1512/ai-gateway
Read the blog

Contact

Building AI infrastructure or an ML platform?

I am available for AI/ML infrastructure, DevOps, and MLOps roles focused on production model serving, Kubernetes platforms, and cost-efficient cloud architecture.

Hanoi, Vietnam · AI/ML Systems Architect · Infrastructure & MLOps @ eUp Group