Data Engineering and Analytics Research Index

A starting point for the data-platform research already collected in Cyberbase: self-hosted ingestion and transformation, ongoing data pipelines, and analytics for structured, nested, or tabular data. Much of the earlier exploration grew out of security monitoring, but the underlying tools and questions extend beyond security and GRC.

Historical research, not a current product comparison

This index connects existing notes without replacing them. Tool mentions range from bookmarks to workflow ideas; they do not establish that a product is lightweight, currently self-hostable, maintained, or suitable for a particular workload. Licensing, deployment requirements, and old pricing claims need fresh verification.

Start here

1. Modern Data Engineering Stack for Cyber

Open the original research hub

The broadest entry point: ETL/ELT, APIs, Pandas, data integration, single-source-of-truth approaches, orchestration, ingestion, and streaming. Useful for recovering the original vocabulary and related research branches.

2. ETL, Data Integration, and Data Replication

Open the ETL and integration notes

The closest match to the self-hosted pipeline question. Covers Airbyte and Jitsu, alongside n8n, Pipedream, Make, and Zapier. Includes observations about transformation limitations and sketches a source-to-target workflow using containerized ingestion tools. The original deployment labels are historical notes, not independently verified hosting guidance.

3. Pipeline and analytics research

Additional archived research is not selected for this public site, so this index does not link to or publish those source notes. The useful research threads to revisit are:

  • Security data pipelines: Tenzir, Tarsal, Mezmo, and related pipeline research.
  • General analytics and exploration: Metabase, Pentaho, Kibana, Jupyter, and broader data-platform search terms.
  • ETL, transformation, stream processing, and data pipelines: Airbyte, Jitsu, dbt, Cribl Stream alternatives, Kafka, Airflow, Flink, and NiFi.
  • Data formats: JSON, CSV, Parquet, Arrow, and other formats.
  • Additional transformation tools: Singer, StreamSets, and Fluent Bit.

Finding alternatives to Cribl Stream is a useful thread to resume separately from general-purpose analytics.

4. Awesome SecOps Data Engineering

Open the SecOps data-engineering collection

A short navigation note with a link to the 2024 MAD data landscape. Some of its older internal references may point to material that is not published here. Its recorded dates are June 20, 2024 (created) and January 31, 2025 (modified); these are note metadata, not dates for a particular research session.

5. Self-hosting leads

Meltano was saved as a self-hosting research lead alongside Matano. This is a discovery lead rather than a completed evaluation. The original bookmark note is not selected for this public site and remains unchanged.

Find the right research thread

QuestionBest starting point
How can APIs or databases feed a self-hosted pipeline?ETL and integration notes
What alternatives exist for collecting, shaping, and routing events?Cribl alternatives and security data pipelines
Which tools handle scheduling versus event streaming?Airflow, Kafka, Flink, and NiFi
What could provide exploration, reports, or dashboards?Metabase, Pentaho, Kibana, and Jupyter
Which self-hosting bookmark is worth revisiting?Meltano
Where are the broader terminology references?Modern Data Engineering hub

Questions for the next research pass

  • Continuous versus streaming: is a reliable scheduled incremental refresh enough, or must each event be processed immediately?
  • Data shape: how are nested objects, arrays, schema changes, joins, and tabular outputs handled?
  • Lightweight operation: what are the minimum services, memory/storage needs, backup requirements, and upgrade burden?
  • Reliability and security: how are checkpoints, retries, duplicate prevention, credential storage, access control, and private datasets handled?

These questions organize future evaluation; the existing notes do not yet resolve them into a tested shortlist. Linking a note here does not change whether that note is selected for public publication.