As Expedia, the world’s largest online travel agency, moved to a multi Virtual Private Cloud (VPC) strategy in AWS, we faced the challenge of making our systems accessible, or using other systems, across many VPCs. In most cases, a secure internet-facing endpoint or doing VPC Peering should do the work, right?
But what if the system isn’t a typical HTTP-based microservice? What if it’s a distributed, partitioned and binary protocol-based system, where anyone talks to everyone all the time? That’s exactly what we encountered when we tried to make our Kafka accessible to our clients. We solved this problem by leveraging Apache Kafka’s distributive nature, using AWS’s new VPC Endpoint technology and their recent Network Load Balancer, some Route53 records and a bit of creativity!
In this session, I’ll dive into:
- Our use case: Kafka accessible to other VPCs
- Why we didn’t go with internet-facing endpoint or use VPC Peering
- A brief description on how VPC endpoints work
- Our solution to the problem: That’s where the fun starts.