Kafka Summit Logo
Organized by

Kafka Summit San Francisco 2018

October 16-17, 2018 | San Francisco

Connecting Kafka Across Multiple AWS VPCs

Session Level: Intermediate
Video & Slides

As Expedia, the world’s largest online travel agency, moved to a multi Virtual Private Cloud (VPC) strategy in AWS, we faced the challenge of making our systems accessible, or using other systems, across many VPCs. In most cases, a secure internet-facing endpoint or doing VPC Peering should do the work, right?

But what if the system isn’t a typical HTTP-based microservice? What if it’s a distributed, partitioned and binary protocol-based system, where anyone talks to everyone all the time? That’s exactly what we encountered when we tried to make our Kafka accessible to our clients. We solved this problem by leveraging Apache Kafka’s distributive nature, using AWS’s new VPC Endpoint technology and their recent Network Load Balancer, some Route53 records and a bit of creativity!

In this session, I’ll dive into:

  • Our use case: Kafka accessible to other VPCs
  • Why we didn’t go with internet-facing endpoint or use VPC Peering
  • A brief description on how VPC endpoints work
  • Our solution to the problem: That’s where the fun starts.

We use cookies to understand how you use our site and to improve your experience. Click here to learn more or change your cookie settings. By continuing to browse, you agree to our use of cookies.