AI Data Engineering

RAG-pipelines, chunking, embeddings, feature stores och orkestrering för AI/ML-datasystem

av ancoleman · ancoleman/ai-design-components

Testad · Fungerar ★ 8.8/10

AI Data Engineering — RAG-pipelines, chunking, embeddings, feature stores och orkestrering för AI/ML-datasystem

Vad den gör

Vägleder building data infrastructure for AI/ML systems: RAG pipelines (ingestion, chunking, embedding, retrieval, RAGAS evaluation), feature stores (Feast), embedding pipelines, and orchestration (Dagster, Prefect, Airflow, dbt). Utlöses when building RAG or semantic-search backends, setting up ML feature serving, or creating embedding and data-transformation pipelines. Levereras med reference docs, runnable example projects (LangChain, LlamaIndex, Feast, Dagster), and standalone chunking/evaluation/setup scripts.

Testrapport

Hämtade SKILL.md och punktkontrollerade 3 refererade sökvägar via raw fetch (scripts/chunk_documents.py, references/rag-architecture.md, examples/langchain-rag/basic_rag.py) — alla HTTP 200 with real substantive code, inga security smells (grepped for curl|sh, base64, os.system, exfil). Installerades into a temp HOME; SKILL.md landed at the correct target path. För output skrev jag two chunkers and ran both on a 2232-char sample: the baseline fixed-512-char splitter cut chunk 0 mid-word ("...vector store.\n\nChunking ") with zero overlap, while the skill-following recursive-separator chunker (512/50 per skill body) ended chunk 0 cleanly at the "\n\n" boundary and started chunk 1 with a 50-char overlap, plus structured chunk_id/source/metadata output — a concrete retrieval-quality difference, not a restatement of rules.

Testad: 2026-07-21 · Claude Code 2.x (agent harness)

Installation

git clone --depth 1 https://github.com/ancoleman/ai-design-components.git /tmp/ai-data-engineering-src
mkdir -p ~/.claude/skills
cp -R /tmp/ai-data-engineering-src/skills/ai-data-engineering ~/.claude/skills/ai-data-engineering
# SKILL.md lands at ~/.claude/skills/ai-data-engineering/SKILL.md
# Skill guidance is usable as-is. The bundled example projects and scripts (scripts/chunk_documents.py,
# evaluate_rag.py, setup_qdrant.py) need Python deps to run:
#   pip install langchain langchain-core langchain-openai langchain-voyageai langchain-qdrant qdrant-client ragas datasets feast dagster
# Some examples also need API keys (OPENAI_API_KEY / VOYAGE_API_KEY) and a running Qdrant instance.

Kommandon och exempelprompter

  • /ai-data-engineeringRAG-pipelines, chunking, embeddings, feature stores och orkestrering för AI/ML-datasystem

Skills triggas av vanliga förfrågningar — inga kommandon att memorera. Efter installationen aktiverar prompter som dessa skillen (på engelska):

  • Design a RAG pipeline with chunking and embeddings
  • Set up a feature store for our ML models
  • Orchestrate this data pipeline with Dagster