Engineering

Before designing a real-time market-data system, it helps to understand the information it handles. Financial markets are places where people and institutions buy and sell assets: stocks such as Apple shares, options contracts, currencies, cryptocurrencies, and more. A market-data system does not execute those trades. Its job is to observe what is happening in the market and quickly send that information to applications, traders, charts, and algorithms. The small events that move a market The basic events are quotes and trades. ...

When a platform sends a stock-purchase request to a broker API, it cannot know what happened merely because the connection ends badly. The broker may have received and executed the order before the response timed out. This design makes one synchronous attempt, records its outcome, and uses PostgreSQL to reconcile uncertain transactions in the background. The important boundary is between a response received from the broker and no response received. When the broker returns a response, the platform preserves the normal behavior: it stores the outcome and returns the broker’s 2xx or 4xx result to its client. A timeout, a server-closed socket, or a 5xx response is not treated as a failed purchase. It is treated as an UNKNOWN purchase that must be reconciled. ...

Integrations often need to reach systems that customers deliberately keep off the public internet. That can mean connecting an integration pipeline to a customer’s CMMS, ERP, or industrial IoT systems while those systems remain inside a private network protected by a VPN. The approach presented in this talk is to give the customer environment an Agent and let that Agent establish a long-lived, bidirectional gRPC connection to a Proxy. The Agent connects outward, registers its identity, and keeps the stream open. When the Integration Pipeline needs to perform work, it sends a request containing an Agent ID. The Proxy uses that identity to select the corresponding live session and sends the request back through the same connection. ...