At Flipkart, the leading e-commerce company in India, our team processes the financial data for millions of transactions every day. Financial data processing requires exactly-once semantics because of customer sensitivity and financial regulatory requirements. The financial data entities have a long lifecycle for updates and we maintain the idempotency for many months and sometimes years. In addition, we provide near real-time view to our stakeholders on the ‘state’ of the processing. Let’s consider a specific example of a data processing request where we aggregate the Invoices into Payments for our sellers. We have to keep the number of payments as Constant to cope with the varying and increased number of invoices. In addition, the system has to maintain an audit trail of paid invoices versus the corresponding payment indents, to comply with Auditing procedures. The need for scalability and real-time Service License Agreement (SLA) steered us towards adopting a stream processing solution such as Kafka.
In this talk, we explain our efforts towards a generic framework using Kafka for all use cases of Financial Data Processing in Flipkart and how we are faring on that challenge. We also explain: – The techniques used to achieve the exactly-once semantics within constraints specific to Financial data processing at Flipkart – Various challenges faced in the process and the use of technologies such as Storm Trident and MongoDB to overcome them. In addition, we also describe how this success has emboldened us to rethink our entire architecture on top of streaming infrastructure.