Gojek: Scaling up consumer experience with real-time data and analytics

Aveekshith Bushan:

Thanks Deboshree. I want to welcome everyone to the fireside chat with Gojek on scaling up consumer experience with real-time data analytics. It's my pleasure to welcome Ali Elgamal from Gojek. Ali is the head of technology at Gojek and he's been leading seven teams to not only solve current problems but future class of problems at scale within Gojek. So Ali, welcome to the session here.

Ali Elgamal:

Thank you so much, Aveek. Excited to be with you here.

Aveekshith Bushan:

Likewise. So I mean, the session is very interesting, Ali. I mean, obviously it's about consumer experience. We're talking about that scale with real-time data and analytics, which is really relevant across a whole bunch of industries. But what you guys have been solving in Gojek is pretty unique given the kind of areas that you focus on. To start with, I'd like to understand, and obviously Gojek has evolved over a period of time, so just to understand from you, from Gojek's perspective, how do you see it having evolved over the years from additional adoption perspective?

Ali Elgamal:

Sure. No, I mean, like any startups very early on with very limited resources, Gojek had a very humble beginning, started as a call center to allow people to book bikes, motorbikes as taxis, which we call Ojek in Indonesia. Several years after that we moved into launching the first app that we started with and to allow people to use their mobile phone to book these Ojeks. And from there, we started to see huge potential and people really were excited about the service, which allows us to scale into offering other on-demand services, starting with food delivery. And then we expanded to many other services, 20 plus services actually including payments and financial services and groceries and so on.

Last year in particular was also a very exciting year for us where we had our merger with Tokopedia, and we started to even offer e-commerce as part of a GoTo group. And we have a very elaborate ecosystem that really focuses on consumers and solving their daily needs. So it has been a progressive improvement of the consumer experience journey that led to adoption of a lot more services at GoTo group.

Aveekshith Bushan:

Got it. Well, that's a lot of things that you've gotten into the recent past. Kudos for that, but also from... and obviously given the topic that we are talking about, customer experience, consumer experience, and understanding the customer and so on, and providing the services which are relevant to the user. In that context, obviously Gojek has been doing a lot in this space. It'd be good to understand from you what kind of recent improvements you've seen from a consumer experience perspective, especially from the projects that you launched.

Ali Elgamal:

Yeah. Recently, I mean, here very recently I'm talking about 2021 and 2022, we launched a new revamped loyalty program for Gojek, we call it GoClub. It's very different from your traditional loyalty programs. It has elements of surprise. It's a tiered loyalty program. It's not your typical earn and burn type of points system. And the program itself, it's available for all Gojek users, and we have seen quite good adoption from users and it also allowed them to explore some of other services that may not have explored before. So that has been very exciting for us recently. Another thing that was very recent for us as well was, as part of after the migration with Tokopedia, the merger with Tokopedia, we actually started to have joint events. So that is in Indonesia, we have a TV show that Tokopedia used to run on a monthly basis called the WID Show.

And that show usually we have offers on TV, there are some music, some very renowned artists come to that show, and then we actually offer discounts for the users as they are enjoying the shows. And Gojek was... the first time for Gojek to participate in such live events was only in 2021. And that requires us to really ensure the scalability of our systems, to ensure that we can handle all the requests that come as the event starts. It was really a good experience for us and we really demonstrate how we can scale our platform services very easily to accommodate massive spikes. And one of the key elements or services that powered such events was what we call segmentation service, which basically allows all the different services on our platform to target segments of users with the different discounts. So both of these services really, if you look at the loyalty use case or the segmentation use case, we're talking about millions of users, the segmentation use case, I think at peak during these events, it can go to millions of requests per minute.

Aveekshith Bushan:

Wow, that's impressive. Not just from a perspective of the data that you're crunching and analyzing and storing and so on, but also in terms of agility, as you mentioned how quickly you have to scale up and ramp up your operations and so on. That's very impressive. So I think in all this, we've seen with a whole bunch of leaders in various verticals, you guys have really taken data to the next level. So how do you see, and you briefly alluded to it, but how do you see data powering these improvements, especially segmentation and so on that you're alluded to?

Ali Elgamal:

Well, it's everywhere. Data is everywhere, from the usage patterns of our services to how the users interact with the different screens or components in terms of views and interactions. This all data is very valuable to us because it allows us to understand what are the interests of the user and how we can offer them the right discounts or promotions that will make them... that are relevant to them. There is no point, we are not a typical advertising company that we target the masses, but we actually offer the user something relevant that will help them solve the next daily friction. So data plays a huge part of the personalization of Gojek services to our end consumers.

Aveekshith Bushan:

Got it. And from our perspective, AeroSpike perspective, very well, that's one of the key areas that we also focused on with the customers globally across industry verticals, understanding the customer, personalizing it based on the likes and preferences of the customer. And like you said, based on what he or she has done recently, offering them something that's a big focus area for us. So you are using AeroSpike in the data part that you talked about and how you're building these platforms. Obviously a whole bunch of other technologies involved, but from an AeroSpike perspective, that's a data platform that you're using in these use cases, both the segmentation part as well as the loyalty area, right Ali?

Ali Elgamal:

Yes, correct. AeroSpike is really the primary data store for us in both use cases, the loyalty and the segmentation use case.

Aveekshith Bushan:

Got it. And since you mentioned the evolution of Gojek over the years and how, obviously you had a loyalty platform and you're beginning to understand user more and more impacting the digital world, it's all about getting people online and getting them the best service, right? So what were you using, prior to using AeroSpike? Was there a data platform that you are using? And also related to that, why did you choose AeroSpike? What are the things that you considered when you actually chose AeroSpike for these use cases?

Ali Elgamal:

Sure. No, certainly. So the loyalty program is launched. So we used to have some sort of other loyalty program multiple years ago, but that was decommissioned. So the new loyalty program was something that we built from scratch, and we started from scratch building it using AeroSpike as one of the primary data store technologies for us. The segmentation was an existing use case, an existing service that has been with us for many years already. And that service has evolved from a stack of Postgres and Redis to AeroSpike. Of course, whether it is the loyalty use case or the segmentation use case, we rely heavily on Kafka for events processing. So in some cases, it could be a stream of commands and events that come out or events and some actions that are being reactions that we take on top of these events. So those type of patterns happened for us.

Segmentation use case, in particular, because it's a very core service for our platform, like almost all of our services rely on segmentation one way or another. It has a very tight SLO in terms of latencies. So this is one of the services that we expect to operate in less than five milliseconds at the 99th percentile. Considering the volume of the data, we are talking about hundreds of gigabytes of data there. So it's a large data store, very tight latency SLO. That's also why the original architecture was using a hybrid of Redis and Postgres to achieve that level of latency. But as your data started to grow to hundreds of gigabytes, that setup is starting to fall apart, and we needed to really think about the next generation solution for segmentation in particular. And that's when AeroSpike came into the picture. There are a couple of things that made us really choose AeroSpike for that use case.

Of course, one is the hybrid memory model that AeroSpike offers. So we actually don't need a caching layer in front of AeroSpike. We already get the latency that we need right out of the box. It simplifies our architecture. So we don't really need to think about the consistency between a caching layer and a database layer because it's all nicely handled and encapsulated by AeroSpike. The other thing, of course, is the availability, the resiliency that the digital tool provides. Managing Redis clusters and Postgres, you can use Patroni clusters or you can use managed services on the cloud. These are all possible options, but from a reliability standpoint, having active-active nodes and replication is something that really made a big deal for us. And when you look at that total cost of ownership, it's actually similar, but gives you a much higher scale that you can operate at. So if you are going for very high scale, the TCO with AeroSpike would be much lower to compared to additional stacks.

Aveekshith Bushan:

Awesome. So yeah, I mean, like you said, the TCO part, the scale part, which is very critical for you and the HMA, the hybrid memory architecture part that you talked about, where data is typically on SSD drives and your index information is in memory means that you get persistence and the power of a cache that you otherwise would've had two layers to do it, right? So that was the other aspect of what you said. And you also talked interestingly about reliability and active-active, right? Those are things which become very critical in these kinds of use cases, which are very, very mission-critical and real time and has to be running all the time.

So those are some great points that you alluded to. You also talked about data flowing in from Kafka and moving out based on certain events or triggers that you want to do something based on that thing, which is very interesting because in a lot of these real-time use cases that becomes critical to, one is the data part and how you store it and so on, but also the overall architecture, which I'm sure you've designed it in a way that makes it a real-time. So is there something you can share with that? I mean, how does that data flow in? What do you do from an application standpoint and how does it actually flow out? Especially when it comes to a lot of these use cases, which are from a real-time positioning perspective, also has have access to data in real time.

Ali Elgamal:

Sure. So we do have a very elaborate data platform with tooling that allows us to work with events and transform events that come out of the different services. So typically mos of our services publish... most of our publish events to Kafka. So Kafka is actually, we manage it ourselves. It's not using any cloud platform at the moment. It's something that we self-manage, but it actually operates a skill of a team who really takes care of it and they're doing a very great job running it for all the company. And we have tools built on top that allows you to actually process events and translate events from one format to the other. A service like the segmentation use case, what typically happens is either a service would actually publish events directly to manage a segment for a user, or you'll create what we call a dagger, which is a configuration for an event processing that would read events off a certain topic and create and write events to another topic.

And the topic that is going to be writing to is basically adding user to a segment or removing user from a segment. The segmentation use case, the segmentation service itself will be listening to that event, that topic, and then it'll apply these commands to the database, which is AeroSpike in this case. And once that event happens, we will then publish events that the user segments have changed. So other services, who basically depend on these events, can also do other actions. So that's a very simple simplistic view of how things happen. Of course, the detail is a bit more complicated than that, but I'm trying to give a simple explanation.

Aveekshith Bushan:

Awesome. Awesome. That's great actually. So it gives a sense of how you build your architecture. Like you said, there's obviously much more complexity in the details, but let's take that for another day. One question that I had, you've obviously mentioned about your Redis and Postgres architecture before you move to AeroSpike. This is relevant to a lot of new prospects who are probably looking at moving not necessarily from Redis to AeroSpike, but from really from one platform to another. How easy or difficult was it for you to move from your existing platform to AeroSpike and were you able to do it quickly or were there challenges you faced there and how did you supplement those?

Ali Elgamal:

Well, I mean, the important thing to understand is you need to design your data. You model your data in a way that is going to be suitable for data store that you're using. So segmentation use case was one of the use cases that was very easily mapped to a simple NoSQL model. And it fit very nicely with AeroSpike where it was just, you can think of it as one set with multiple bin and the user ID as the primary key. So it was very straightforward for us to model it. Movement of data and the migration of data for us was a little bit tricky because we have a policy that we try to do any data migration with zero downtime and all the migrations need to be reversible.

So the migration of data took us some time to iron out and get it right, but now we are running 100% on AeroSpike. We actually completely decommissioned Redis and Postgres from the segmentation use case specifically. So it's 100% on AeroSpike. We're even now thinking considering to move the segmentation use case to AeroSpike managed service because we really want to focus on building functionality and features that power consumer experiences and not do the heavy lifting of managing the data store ourselves. We prefer to keep the data store management to the experts of the data store. And that's where AeroSpike managed service makes a lot of sense for us. And that will be the next step for us in terms of AeroSpike adoption. So moving more towards managed service and expanding the number of use cases and services that would benefit from AeroSpike in terms of latency or cost or reliability or something.

Aveekshith Bushan:

Awesome. I think you've talked about migration in a lot of detail, so thanks for it. And really, again, I think you mentioned it briefly, but just to understand from you, given what you have implemented with AeroSpike, obviously AeroSpike is trying to get into a lot of, I wouldn't say different areas, but in line with the areas that we've been working on, which is high performance use cases primarily is a key value database, but now we are getting into areas like graph time series and document use cases with JSON support and so on. And obviously the connector ecosystem where data flows into AeroSpike and beyond, like you said with the Kafka connector and so on. Those areas that we are focusing on with active-active and so on. But we to get a sense from you, given your experience using AeroSpike and in these areas where all do you see AeroSpike potentially helping out in the days to come in the areas that you're focusing on?

Ali Elgamal:

The stack that I mentioned of Redis and Postgres behind is actually a very common stack in our system. We have tens or maybe hundreds of services that are using a stack like that. And the idea here is this type of stack is really high on maintainability, multiple points of failure, not really active-active. And I do see a world where most of the services that are using that stack would transition to an AeroSpike use case or AeroSpike setup, especially as we continue to grow and scale that stack is not going to help us on the long term. So there's a lot more services. Hopefully one day, I don't need to have any caching layer in front of my database. That's really would be one of the things that I would be hoping to get to. But it'll take us multiple years to get there.

Aveekshith Bushan:

Hopefully we can help shorten that time that you're talking about. But thank you so much, Ali. I think you've shared so many insights with us with the audience as well, which would be very element to everybody really looking at customer experience and making it more real time.

So thank you for all those insights. But just as a summary to what Ali just spoke about, you look at any consumer experience platform, it's about not just offering what the user is looking for, but to be able to do that, you need to have the right insights about the user. You got to have insights based on current data, but also historical data, like Ali was mentioning. And that's when it becomes real time and it becomes relevant to the user. We've been working with a number of industry leaders like Gojek in their segment with Telcos, with banks and so on, where the key thing is being able to analyze data across a whole bunch of areas, bring it together and show the user what is relevant to them. So in that sense, I think what Ali shared is really something path-breaking and something which could be relevant to a whole bunch of industry verticals. So thank you again, Ali and back to Deboshree now for any questions that the people might have. Thank you very much.