Haven Studios is incredibly excited to welcome Jalal El Mansouri to the team as our Principal Architect and Head of R&D for Haven Labs. Along with his team of visionary cloud developers, they will drive our studios' use of the cloud forward in innovative ways.
Haven has been a cloud native studio from day 1, founded with the idea that cloud would be a game changer for game development, particularly for a brand new studio kicking off remotely in the middle of a pandemic.
We sat down for a conversation with Jalal and Leon O’Reilly (our Head of Technology) to discuss their vision for cloud and how it will tangibly improve game development at Haven (and if we are successful and across the entire industry).
“Cloudy, with a chance of ML” -Image by craiyon.com
Haven: It’s great to see you both together in the same room again!
Leon: Yes, it’s a very exciting time! We are thrilled that Jalal and his team are joining Haven and getting everyone back together. We worked together at Google, and even before that at Ubisoft as well, so I’m pleased that we can continue this collaboration. Jalal has a great vision and passion to push game development into the future using the potential of the cloud.
Jalal: I’m also excited to join Haven. It’s an amazing opportunity to join an exceptionally talented team that shares a vision for cloud native development and creating the future of games as a service. Modern development pipelines are essentially 20 years old and require a major shift forward in order to meet future needs. Haven is the best place to focus on that.
Leon: We’ve both been working on games for a long time and have seen plenty of things that work, and plenty that don’t.
Jalal: The most eye opening experience for me was working on Rainbow Six: Siege. The game started with a mindset of “shipping the disc”, and then had to pivot to a live operations mentality. It was like changing the tires on a moving car! We learned so much about the challenges of scaling to a live game, and now we have the opportunity to address it all, by building a new pipeline from the ground up, rather than continuing to patch up the old ones.
Leon: So many of those things are from a different era, and they no longer hold up for live titles, but you rarely ever have the time to re-architect from the ground up. Luckily for us, working in a startup with a blank canvas, we have the opportunity to build things in a more modern and effective way.
Jalal: The cloud is an incredible accelerator, and delivers opportunities to move faster and co-create better. You already see this in other industries, like how Figma and Google Docs fundamentally changed how work is done. Games need to start moving in that direction if we want to stay cutting edge.
Haven: Speaking of Google - you were both there together for a time, what did you learn?
Leon: Being at Google was eye opening in a lot of ways, seeing how things were built there. Having a planet scale computer opens up a world of possibilities and changes the way you think about technology. Game developers can sometimes be a bit “cowboy style”, which has its own advantages, but it’s always great to learn from other industries. Now we want to bring the best of the game world and the best of the tech world together.
Jalal: That was what drew me to Google - understanding how they think about scale, and build their products with faster deliveries and seamless functionality across platforms. It’s a workflow that we need to adapt to games.
Haven: You both have strong visions on how you think the cloud will change game development, can you tell us more about them?
Leon: When we started Haven, we didn’t really have a choice: putting the foundations of the studio in the cloud was a necessity for us to get started. Now it’s become an enormous advantage for us, making us incredibly flexible in how we work. It’s given us a chance to re-evaluate how we want to make games. Thankfully, we’ve been able to move many familiar tools to the cloud which has been useful for the short and medium term to keep us all productive, but longer term we want to push further and further into cloud native technologies. This will help us tackle problems like connectivity and bandwidth and the enormously vast data sets we’re going to need to throw around. Jalal and his team have an exciting vision for how we can do this all better in the cloud.
Jalal: We want to make a truly native cloud engine. If you were to remove the cloud part, it would no longer exist or cease to make any sense. Think of Google Docs, it needs the cloud to function.
To enable that for game development, we need to rethink engine design from the ground up, and look to find ways to leverage horizontal scalability, bandwidth availability and connectivity for every single component. We are trying to build workflows that previously took several hours so that they now only take a few seconds.
Let’s look at asset management for example. The transition to 4k was an immense challenge, as textures and meshes that were already one of the most difficult resources to manage nearly quadrupled in size! All of that data needed to be moved around as big “chunks” because that’s how the old pipelines were designed to access it. Scaling productions to that growth was quite difficult as optimizations were built around data locality and having everything on a user machine became a major bottleneck. Significant efforts were undertaken to make the pipelines that touch textures less local, such as just in time source asset downloads, but they never solved the root problem.
Asset sizes are continuing to grow by multiple orders of magnitude, and the commonly used engines were architected when there was only 512MB of memory, whereas now we need to fill more than 20 times that amount. This is where cloud native thinking can truly shine, given that you design the engine and its components accordingly, in order to stay future proof and keep iteration time low.
Leon: Iteration time is the key. Game development is all about iteration: nothing is ever good enough in V1! Being able to quickly access all of your data, without syncing massive files, or fighting a slow internet connection will have enormous benefits to productivity.
Jalal: Yes, and iteration time only becomes more important in GaaS game creation: with the need to have multiple simultaneous development contexts, each with their own set of data. You’ve got multiple seasons, events, specialty content, and balance patches all overlapping. The cloud makes having all of these in flight much more manageable, and quick iteration can be achieved.
Haven: You’ve mentioned the challenge of asset size and transfers, how exactly does the cloud solve that?
Leon: Games have gotten so big that packages are regularly larger than 100GB, and naively distributing these to the team can be slow and cumbersome. There are technologies that we can build in the cloud that do this in a smarter way, and significantly reduce what each individual has to download from version to version.
Jalal: One of our goals is to provide a virtual file system that will allow seamless real-time access to data. Backed by a content addressable store and multiple levels of caching, existing systems can access all content just-in-time, but it just looks like a normal access from their perspective. Data is even split and cached according to the access pattern, which makes it so much faster to access it in an incredibly efficient manner. Processes that may have taken hours in the past now would only take seconds. Essentially, a level designer no longer needs to download an entirely new build of the game to test a level, just sync a tiny portion and get back to work!
Haven: Isn’t rethinking game tech pipelines within the context of a new studio a bit scary?
Leon: Making games is always scary but we have such an experienced team here at Haven, and now integrating Jalal’s crew of incredibly talented people, and adding the support of Playstation, I honestly believe that we have what it takes to do something genuinely new and innovative.
Jalal: Since we all left Google, I was watching the Haven team start on their amazing adventure as I assembled all of the best engineers to work on our vision for the future of GaaS technology, but… we didn’t have a game team - we didn't have collaborators on the content side to partner with. Now we have bridged that gap, and can make the best technology for the best development team. We are already helping to solve current problems and improve the game team’s workflows with cloud based solutions.
Also it’s just great to be back together with our shared passion and vision of the future, all in an environment where we understand that you make players happy by making developers happy.
Haven Studios: We are always looking for more engineers and developers who want to push for innovation. Reach out to us if you’d like to join our team!