BigDataFlowing / Data × AI Knowledge Portal

BigDataFlowing: AI and Data Navigator

Built for data leaders, technical teams, and people evaluating enterprise AI adoption. It curates tools and writing across AI engineering, big data, data governance, enterprise agents, data platforms, and knowledge graphs. This is not a generic tool directory. It is the public knowledge entry point for BigDataFlowing.

View ProfileRead Articles
Search Tools

107 AI and data tools curated across AI engineering, governance, analytics, processing, and storage.

Portal Map

One Entry Point for the Knowledge System

The main site builds public trust through a connected chain of profile, tool map, selected writing, case context, and lab work.

Profile

Understand Dugufeng’s career transition, graduate exam journey, data governance practice, and AI engineering direction.

Enter

Tool Map

Browse AI and data tools by AI engineering, governance, analytics, processing, and storage.

Enter

Selected Writing

Read structured views on data governance, AI for Data, Data for AI, and enterprise AI agent engineering.

Enter

Case Context

Place data middle platforms, metadata, lineage, quality, knowledge bases, and agents back into enterprise scenarios.

Enter

Lab

Track explorations around CourseMotion AI, governance agents, enterprise semantic layers, and AI engineering evaluation.

Enter
Categories

Tool Map

A curated map connecting open-source projects, commercial platforms, and core enterprise data capabilities through AI engineering and data governance.

AI Engineering

LLM apps, agents, RAG, local models, MCP, evaluation, and enterprise AI orchestration.

Open Category

Data Governance

Metadata, data catalogs, lineage, quality, standards, and data asset management.

Open Category

Analytics and Visualization

BI, data exploration, reporting, visualization, and analytics platforms.

Open Category

Data Processing

Batch, streaming, scheduling, integration, and compute frameworks.

Open Category

Data Storage

Data lakes, object storage, distributed storage, and lakehouse foundations.

Open Category
Governance Focus

Selected Data Governance Tools

The original data navigation foundation remains visible, with DataHub, OpenMetadata, Atlas, Great Expectations, and other metadata, lineage, quality, and data asset tools.

Alation

AI-powered data catalog

Data GovernanceView Details

Amundsen

Open-source data catalog for discovery and metadata search

Data GovernanceView Details

Atlan

Data catalog and governance collaboration platform for modern data teams

Data GovernanceView Details

Apache Atlas

Metadata management and governance framework

Data GovernanceView Details

Bigeye

Enterprise data quality monitoring and observability platform

Data GovernanceView Details

Collibra

Enterprise data governance platform

Data GovernanceView Details
Big Data Foundation

Big Data and Data Platform Tools

Hadoop, Spark, Flink, Kafka, Iceberg, MinIO, and related projects remain part of BigDataFlowing core. AI engineering extends from these data foundations.

Airbyte

Open-source ELT data integration and connector platform

Data ProcessingView Details

Apache Airflow

Workflow orchestration system

Data ProcessingView Details

Azkaban

Distributed workflow manager

Data ProcessingView Details

Apache Beam

Unified batch and stream processing model

Data ProcessingView Details

Dagster

Software-defined orchestration platform centered on data assets

Data ProcessingView Details

dbt Core

SQL transformation and modeling framework for analytics engineering

Data ProcessingView Details
AI Engineering Extension

AI Engineering Additions

Dify, Ollama, OpenClaw, Hermes, and related tools are framed through enterprise AI applications, agents, RAG, and data governance context.

AnythingLLM

Desktop and server tool for team knowledge bases, RAG, and AI agents

AI EngineeringView Details

Arize Phoenix

Open-source observability platform for LLM, RAG, and ML systems

AI EngineeringView Details

AutoGen

Microsoft open-source framework for building collaborative multi-agent systems

AI EngineeringView Details

BentoML

Platform for AI model serving and inference deployment

AI EngineeringView Details

Chroma

Embedding database for AI-native applications

AI EngineeringView Details

CrewAI

Multi-agent orchestration framework for role-based task collaboration

AI EngineeringView Details