6point6 worked with a large-scale customer to design and develop a scalable and extensible Shared Data Platform. The immediate goals were to persist and access essential data across multiple client programmes, including migrating from a legacy estate as well as multiple modern systems. Key requirements for this platform were availability, zero data loss and the avoidance of transactions. This talk will cover how Kafka and KStreams were leveraged to deliver a platform built around event sourcing. The solution boasts both high throughput and semi-strong consistency, whilst providing flexibility and the ability for multiple customers to interact with the same record. It will also cover the before and after architectures, the issues experienced, how development teams were impacted, and how platform performance and reliability were improved.
Learn how we:
- Scaled the platform on Amazon Web Services to handle billions of messages.
- Achieved low-latency eventual consistency at 50,000+ messages per second.
- Leveraged a blockchain-like ledger of changes to guarantee data integrity with point-in-time snapshots.
- Provided an atomic and idempotent platform without using distributed transactions.