NextGen APIs - Event Streaming

Introduction

Think of event streaming as the central nervous system of modern software architecture. Instead of applications directly asking each other for information (a request-response model), they produce and consume a continuous, real-time flow of events. An event is simply a record of something that happened, like an order being scheduled. This stream of events is published to a central log or broker. Other applications, known as consumers, can then subscribe to this stream and react to events as they occur, without ever needing to know about the original application (the producer).

This approach fundamentally changes how we build systems, leading to several key advantages:

• Decoupling: Services are independent. A producer sends an event and doesn't care who consumes it. This makes systems more resilient, flexible, and easier to maintain.

• Scalability: We can easily add more consumers to process the same event stream in parallel, allowing us to handle massive volumes of data and traffic.

• Real-time data: It enables powerful, real-time capabilities like fraud detection, live dashboards, and instant user notifications, as data is processed the moment it's created.

Supported providers

• AWS SNS (Simple Notification Service): A fully managed publish/subscribe (pub/sub) messaging service. It allows a single message from a producer to be "fanned out" and delivered to a large number of subscribers simultaneously (e.g., sending a notification to mobile apps, email, and other microservices all at once).

• AWS SQS (Simple Queue Service): A fully managed message queue. Its primary job is to reliably hold messages sent between different software components. It's excellent for decoupling background tasks and ensuring that no work is lost, even if a service is temporarily down.

• Apache Kafka: A high-throughput, distributed streaming platform. It's the powerhouse for handling massive, ordered streams of events and is often used as the core data backbone for real-time analytics and event-driven architectures.

Design

Events

Payload

• No structured data (e.g.: LOBs)

• No data from entities different than the one considered

• Suggested maximum payload size is 64 KB

Default payload

• Physical key(s): ID or IDs for relational entities

• Logical key(s): CODE or set of fields for composed logical keys

• Parent: ID of the parent entity in parent-child relationship entities

Targeted payload

Set of fields that carry relevant information of the entity. This part of the payload is tailored to the specific entity.

Suggested extended payload

Set of fields that the product suggests developers to add in the extension section of the payload, leveraging the Extension mechanism.

• History ID: Correlated entity carrying the historic data of updates per record of the source entity

• External System ID: External system that is typically the owner of the entity

• Origin ID: Indicates a generic entity/process that generated the entity

Attributes

Attributes are the metadata of an Event.

These are set by our Core Services - Foundation:

• dto

• scope

• user

• tenant

• hostname

• timestamp

• sequence

The dto attribute is set by NextGen FSM whenever publishing a new Event. It identifies the Event that happened in the system (e.g.: WorkOrderHeaderCreated).

These are set directly by AWS:

• uuid

Commands

Related Docs

OverIT Internal Documentation: https://overit-spa.atlassian.net/wiki/spaces/NEXTGEN/pages/104050393241/Event+Streaming