System Diagrams — Architecture, Pipeline & Data Flow
Key Insights
- Comprehensive Mermaid architecture diagrams including the new Source Framework (registry + 5 fetcher types + health/DLQ), Admin Panel (Flask routes, API, templates), and RSS Ingestion Pipeline — 13 diagrams total.
System Diagrams
This page provides comprehensive architectural, pipeline, and flow diagrams for the AcaciaFund DataOps platform. Each diagram is rendered as an SVG image with a simple, clear visual style.
New for June 2026: diagrams for the Source Framework (registry, 5 fetcher types, health tracking), Admin Panel (Flask routes, API, templates), and RSS Ingestion Pipeline (8 feed sources → classification → build).
1. Admin Panel - Routes, API & Templates
View source admin_panel.mmd on GitHub ↗
2. Build Process - Sequence Diagram
View source build_sequence.mmd on GitHub ↗
3. Content Model - UML Class Diagram
View source content_model.mmd on GitHub ↗
4. DataOps Pipeline - 8 Stages
View source dataops_pipeline.mmd on GitHub ↗
5. Module Interconnections & Data Flow
View source module_interconnections.mmd on GitHub ↗
6. Pillar Taxonomy - Content Classification
View source pillar_taxonomy.mmd on GitHub ↗
7. Pipeline Quality Gates & Observability
View source pipeline_quality.mmd on GitHub ↗
8. RSS Ingestion Pipeline
View source rss_ingestion.mmd on GitHub ↗
9. Search Index Architecture
View source search_index.mmd on GitHub ↗
10. Source Framework - Registry, Fetchers & Health
View source source_framework.mmd on GitHub ↗
11. Source Ingestion & Content Flow
View source source_ingestion.mmd on GitHub ↗
12. AcaciaFund System Architecture
View source system_architecture.mmd on GitHub ↗
13. User Journey - Site Navigation
View source user_journey.mmd on GitHub ↗
Diagram Reference
| # | File | Description | Type |
|---|---|---|---|
| 1 | admin_panel.mmd | Flask admin panel: 11 page routes + 7 API endpoints + 12 Jinja2 templates + 4 data sources | Flowchart |
| 2 | build_sequence.mmd | Build sequence: git push → CF Pages → generator → dist → deploy | Sequence Diagram |
| 3 | content_model.mmd | Content data model: RegistryData → ContentItem → BloomQuestion + Flashcard | Class Diagram |
| 4 | dataops_pipeline.mmd | 8-stage pipeline: ingest → validate → transform → catalog → visualize → render → serve → observe | Flowchart |
| 5 | module_interconnections.mmd | UML class diagram: config → validates → reads → invokes → renders | Class Diagram |
| 6 | pillar_taxonomy.mmd | Mindmap of 3 pillars × 6-7 topics + cross-cutting: AML/Markets/Science/DataOps | Mindmap |
| 7 | pipeline_quality.mmd | Quality gates: Pydantic validation → SQI → domain % → flags + observability | Flowchart |
| 8 | rss_ingestion.mmd | RSS ingestion flow: 8 feed sources → RSSFetcher → health/DLQ → ingest.py → pillar classification → build → pages | Flowchart |
| 9 | search_index.mmd | Search: build-time JSON generation → client-side fetch/filter | Flowchart |
| 10 | source_framework.mmd | Source registry architecture: etc/sources.toml → BaseFetcher ABC → 5 fetcher types → health + DLQ → admin dashboard | Flowchart |
| 11 | source_ingestion.mmd | Source ingestion: APIs → seed_articles.py → NLP enrichment → registry.json | Flowchart |
| 12 | system_architecture.mmd | End-to-end system: user → CDN → build pipeline → 236 pages output | Flowchart |
| 13 | user_journey.mmd | User navigation: Home → Research/Learn/Knowledge → Article/Lesson/Page | Flowchart |
Article Metadata
Further Reading
Databricks Blog
Lakehouse, Spark, Delta Lake, Unity Catalog — engineering blog
Apache Kafka
Kafka documentation, KIPs, and ecosystem updates
Apache Flink
Flink documentation and release notes
Apache Iceberg
Iceberg table format — specs, REST catalog, performance
dbt Blog
dbt Labs engineering blog — analytics engineering, Semantic Layer
Dagster Blog
Dagster orchestration — software-defined assets, IO managers
Feynman Concept Cards
Master each concept: read the ELI5, explore analogies, work examples, and teach it back.
DataOps is a concept in best practices. In simple terms, DataOps covers best practices in Data Engineering. This data engineering concept addresses key topics in the best practices in data engineering domain. Also known as: DataOps practices, data operation
Analogy
Example
Find Gaps
Explain DataOps as if teaching a colleague who is new to best practices. Cover: what it is, how it works, and why it matters.
Create
Create a checklist that demonstrates DataOps in a real-world best practices scenario. Walk through your design decisions.
Show solution
A checklist for DataOps should include: 1. The core components of dataops 2. How they interact 3. Expected outcomes or outputs
Lakehouse Architecture is a concept in architecture. In simple terms, The Lakehouse architecture, formalized by Armbrust et al. (2021), combines the flexibility of data lakes (cheap object storage, diverse data types) with the reliability of data warehouses (ACID transa
Analogy
Example
Find Gaps
Explain Lakehouse Architecture as if teaching a colleague who is new to architecture. Cover: what it is, how it works, and why it matters.
Create
Create a diagram that demonstrates Lakehouse Architecture in a real-world architecture scenario. Walk through your design decisions.
Show solution
A diagram for Lakehouse Architecture should include: 1. The core components of lakehouse architecture 2. How they interact 3. Expected outcomes or outputs