What is an AI tech stack?
An AI tech stack is the full set of technologies used to build and run an AI-powered application. Like any tech stack, it spans layers — but instead of just a front end, back end, and database, an AI stack adds the pieces unique to working with models: the model itself, a way to give it your data, and the tools to orchestrate, deploy, and monitor it.
The right stack depends on what you're building. A simple "chat with your docs" app needs far less than a production system running fine-tuned models at scale, but they draw from the same menu of layers below.
The layers of an AI tech stack
A modern AI stack typically covers these layers:
- Models / LLMs: the brain — hosted APIs like OpenAI or Anthropic, or open models (Llama, Mistral) via Hugging Face
- Data & embeddings: your content turned into vectors so a model can search it — embedding models plus a pipeline to keep it fresh
- Vector database: stores and searches embeddings for retrieval (RAG) — Pinecone, Weaviate, Qdrant, or pgvector
- Orchestration / frameworks: chain prompts, tools, and steps together — LangChain, LlamaIndex, or a lightweight custom layer
- Compute & deployment: run models and jobs — cloud (AWS, GCP), or model-serving platforms like Modal, Replicate, or Baseten
- Monitoring & evaluation: track quality, cost, and drift — LangSmith, Langfuse, or Weights & Biases
AI tech stack examples
Three common shapes, from simplest to most involved:
- RAG app ("chat with your data"): OpenAI or Anthropic + an embedding model + pgvector/Pinecone + LlamaIndex, deployed as a simple API
- AI agent / assistant: a strong LLM + LangChain (tools and memory) + a vector DB + LangSmith for tracing, on serverless or Modal
- Production ML product: fine-tuned or open models via Hugging Face + a serving platform (Baseten/Modal) + a feature/vector store + W&B for evaluation
How to choose your AI stack
Start with the model and the framework, because they shape everything else. For most apps, a hosted LLM (OpenAI/Anthropic) plus a light framework gets you to a working prototype fastest; reach for open models and serving platforms when cost, privacy, or customization demand it.
Add a vector database the moment you need the model to answer from your own content (RAG), and add monitoring before you go to production — AI systems fail quietly, and you can't improve what you don't measure.