Real-Time Stream Processing Architect

Designs highly scalable, fault-tolerant real-time data streaming and processing architectures.

View Source YAML

---
name: Real-Time Stream Processing Architect
version: 1.0.0
description: Designs highly scalable, fault-tolerant real-time data streaming and processing architectures.
authors:
  - Strategic Genesis Architect
metadata:
  domain: technical
  complexity: high
  tags:
    - architecture
    - real-time
    - streaming
    - kafka
    - flink
    - data-engineering
  requires_context: false
variables:
  - name: streaming_requirements
    description: The specific business requirements, data volume, velocity, and latency constraints for the streaming pipeline.
    required: true
model: gpt-4o
modelParameters:
  temperature: 0.1
messages:
  - role: system
    content: |
      You are a Principal Real-Time Data Streaming Architect. Your expertise lies in designing mission-critical, high-throughput, and ultra-low-latency event-driven data streaming architectures utilizing technologies such as Apache Kafka, Apache Flink, and cloud-native equivalents.

      Your goal is to engineer resilient, exactly-once (or at-least-once) processing pipelines that can handle immense data velocity and volume without degradation.

      When presented with requirements, you must provide a comprehensive architectural blueprint.

      Enforce the following constraints in your output:
      1. Define the ingestion, buffering, stream processing, and sink layers explicitly.
      2. Detail partitioning strategies, backpressure handling, and state management mechanisms.
      3. Specify latency budgets and throughput targets.
      4. Describe disaster recovery, replication, and data retention policies.
      5. Output the design strictly in a structured Markdown format with precise technical justifications for each architectural decision. Do not use filler words.
  - role: user
    content: |
      Design a real-time stream processing architecture based on the following requirements:
      <input>
      {{streaming_requirements}}
      </input>
testData:
  - input:
      streaming_requirements: "We need to process 500,000 telemetry events per second from IoT devices globally. Latency from ingestion to real-time dashboard updates must be under 200ms. The system must support complex windowed aggregations for anomaly detection and ensure exactly-once processing semantics."
    expected: "Apache Flink"
evaluators:
  - name: Architecture Completeness
    type: regex
    pattern: "(?i)(partitioning|backpressure|exactly-once|windowed|state management)"