On January 13, 2018, 1.4 million people across Hawai’i received an SMS from the state’s Emergency Management Agency: “BALLISTIC MISSILE THREAT INBOUND TO HAWAII. SEEK IMMEDIATE SHELTER. THIS IS NOT A DRILL.” For 38 minutes, people sought shelter, wrestled with existential questions, and came face-to-face with the threat of nuclear weaponry. And now, On the Morning You Wake (To the End of the World)—an immersive VR documentary—captures their experience to share with others on Quest 2.
On the Morning You Wake recently won the Jury Award for XR Experiences at SXSW. In the words of the jury statement, “On The Morning You Wake (To the End of the World) is an emotionally impactful and beautifully told story, delivered with stunning technical craftsmanship. This project explores the potential of immersive experiences, refining the grammar of spatial narrative. This particular story deals with the urgency of nuclear disarmament that has very unfortunately come into sharp focus due to current events. It effectively presents a massive geopolitical issue and grounds it in emotional and personal stories, translating what are usually abstract concepts into an embodied context.”
We sat down with one of the film’s co-creators, Steve Jamison, to learn more.
Tell us about your experience as a VR creator. What is it that draws you to VR as a medium?
Steve Jamison: First, I’d like to make clear that I was just one of the lead creators on this project, along with Dr. Jamaica Heolimeleikalani Osorio, Arnaud Colinart and Pierre Zandrowicz of Atlas V, and Mike Brett, who is both my creative partner and fellow co-founder of Archer’s Mark.
Whilst Mike and I produced VR experiences as a studio before, including the award-winning Notes on Blindness, this was our first experience as lead creators.
Having come from a more traditional film background, we absolutely love the fact that VR is a fundamentally collaborative medium. As a storyteller, you can often feel like you’re expected to have all the answers or a 360° perspective on every single detail relating to your project—but the very nature of the medium requires a more collegiate approach, and that’s something we really feed off.
It’s also incredibly exciting to be creating work for a platform at the same time as the platform itself is evolving, at great speed. It’s an experience we liken to driving a bullet train whilst laying the track directly in front of you. Even during the time that we’ve been working on this project there have been significant technical innovations that have changed the scope of what we were able to create or the scale of the audience that we’re now able to reach. And as a creator, that’s an incredibly daunting but exciting prospect.
If people take one thing away from On the Morning You Wake, what do you hope that would be and why?
SJ: The objectives of the project were:
What made you want to focus on this topic? Has working on the project made you feel differently?
SJ: Personally, before working on this project, I felt quite overwhelmed by the scale of the nuclear issue and powerless as an individual to have any impact on the big decisions that will define the kind of world we’re leaving for our children.
But four years on, I feel more empowered to take action and more connected to a community of people who are committed to abolishing nuclear weapons. What I’ve learned from Jamaica and the other collaborators on this project is that we do have the tools to lift the nuclear shadow. Any system that was built by us can be dismantled by us. And the first step to doing so is having the courage to imagine that we could live a different way. Meaningful change is possible, but meaningful change always comes from ordinary people getting organized from the ground up.
I guess, in a nutshell, that’s how we want people to feel after completing the experience.
Tell us about your first experience using VR. How has that influenced your past and present projects?
SJ: My first meaningful experience in VR was when our company was co-producing Notes on Blindness and I tried the very first prototype. The experience brought me to tears—a powerful emotional response that I hadn’t anticipated—and I understood in that moment the potential for VR to create a strong connection between people and a given subject matter.
For me, it’s about total immersion. If I’m watching a film, I always try to get as close to this as possible: total darkness, total silence, zero distractions, huge screen, volume up, etc.
And as a storyteller, you’re always hoping to create the strongest link possible between the viewer and the subject. Whilst I dearly love the collective experience of cinema, I also love the total immersion that’s possible with VR and how that can influence a viewer’s response to the material.
As a creative team, we share a belief that story is the most powerful agent for change that we have as a society—and that the intersection between art and technology has never been more important in shaping the opinions of the public and our political leadership.
And we also believe that the total immersion of VR can provoke a stronger emotional response than is possible through other forms of storytelling.
How do you feel about VR and its ability to expose us to the lived realities of other people?
SJ: Let’s face it: The reality of all-out nuclear war is incredibly abstract and something that the majority of people have a very hard time imagining. And it’s certainly something that they struggle to imagine happening to them—often it’s seen as something that’s a problem in another part of the globe or within another community.
What we hoped to do with this experience and what VR enables us to do is to place users right there at the center, to help them understand what it feels like to receive such a text message and to know your world is about to end.
This simply wouldn’t have been possible, certainly not to the same extent, in any other artform. But by providing access to the realities of others, we hope to create empathy and a greater connectivity with the core questions: Why do we live in a world where this is even possible? And: How can I take ownership of this problem to help effect meaningful change?
SJ: Full credit goes to Games for Change and Princeton University’s Department for Science and Global Security as the true originators of the project. The brief evolved out of a conversation between them in 2017 and the objectives mentioned above, specifically to raise awareness about nuclear threat and reach a new XR audience. They subsequently brought Archer’s Mark and Atlas V on board to help develop the idea and bring it to fruition.
As a team of creators, it was incredible to be able to call upon the expertise of Princeton’s nuclear experts (Alex Glaser and Tamara Lilinoe Patton) who helped condense decades of research in this space and inform our key choices about the direction, tone, and detail of the narrative. In fact, Tamara was born and raised in Hawai’i, and it was her personal experience in January 2018—fielding deeply traumatic text messages and calls from friends and relatives on the island—that led us to center the experience around the Hawaiian false alarm. As Tamara shared more of those memories and messages with us, we realized that her personal connection to the story provided an incredibly powerful point of entry to the subject matter.
Now that production is complete, we’re blessed to have the full force of the Games for Change team behind us. Led by President Susanna Pollock, they’re driving our global impact campaign, engaging fellows from the nuclear space to create an education syllabus, and helping to bring the experience to policymakers, public institutions, and centers of education.
Tell us about your production pipeline.
SJ: In truth, our production pipeline has evolved several times throughout the four-year production process, partly because of the technological advances that occurred in that time and partly due to the pandemic and changes that travel restrictions imposed on our plan.
From the outset, we wanted our characters to be as organic and authentic as possible to maximize the emotional connection with the viewer. This meant mo-cap and CG were off the table. We started by creating a vertical slice or prototype back in 2018, using Microsoft Kinect to capture performance and a mix of photogrammetry and lidar to bring environments to life in rich, authentic detail. From there, the plan was to shoot real people and real locations in Hawai’i—but then the pandemic changed everything.
But whilst the production hiatus and travel restrictions were a huge setback, the delay also meant that some technical advances that hadn’t been available to us for the prototype were now becoming more of a possibility. And so we pivoted the entire pipeline towards full volumetric capture of the characters, with environments built entirely within the game engine.
To my knowledge, no previous Quest experience has featured this much vol-cap or this many characters. And so our technical teams had to solve plenty of challenges relating to the optimization of those vol-cap assets, in order to make sure we had enough performance budget remaining to render all of the other environments and real-time particle shaders.
Our final production pipeline looked something a little like:
Your production team introduced bespoke technical innovations throughout the production pipeline. What can you tell us about that?
SJ: Straight out of the gate, we set a couple of big challenges for our technical team.
The first related to the art direction, which we wanted to be photorealistic in the opening scenes but then degrade to a particle-based point cloud from the moment the first text message arrives—that was representative of the invisible nuclear shadow we all live under constantly. We also wanted the ability to switch between these two worlds at any given time, so you can imagine the joy when we asked our team to create two separate looks and build in such a way that this would be possible—all whilst wrangling the enormous files generated by volumetric capture.
The second key challenge was finding a way to lock audio to the vol-cap assets so that some of the characters could appear to lip-synch key lines of dialogue, something that is incredibly challenging in the game engine.
The experience was created in Unity as it provided the most flexible environment for developing innovative rendering tools required to meet the challenges above—and to seamlessly mix different types of content, from volumetric captures to particle VFX and point clouds.
For the advanced particle effects, our technical team at Novelab were required to write custom shaders to give us full control of the particle motion while keeping the performance cost as low as possible. They also created a lot of specific shaders for the transitions between scenes, from vol-cap meshes dissolving into particles to environments transforming into raining polygons.
All volumetric captures were authored in 3D software to clean any remaining artifacts from the physical shoot and relight the characters in accordance with the lighting set-ups we had created inside the Unity build.
We then encoded the edited frames using Microsoft’s Mixed Reality Capture tools as .mp4 files containing vertex information. The tool allowed us to edit real time in the engine whilst compressing the assets as much as possible, meaning we could maximize the number of vol-cap characters / scenes whilst minimizing the GB size of the project.
For the environments, we developed a number of internal tools for creating a point cloud look that was reflective of the atomic themes, could be rendered using a custom shader, and to which we could apply various degrees of motion and transition. Even so, some scenes were difficult to understand just by using points alone, and so some elements required additional layers to give depth to the 3D space. For these we created bespoke destructed proxy meshes from the point clouds using surface reconstruction algorithms in MeshLab.
Whilst there were a great many different technical solutions involved, we hope the overall effect is one that feels consistent and organic across the various scenes and always in service of the driving narrative.
You worked with remote teams in the US, UK, France, and Germany. Did that present any unique challenges?
SJ: Anyone who has produced an XR experience in the past two years will recognize the very real challenges of working remotely throughout the pandemic. The creative process is massively iterative, and that’s obviously problematic when each iteration needs to be tested, optimized, and shared with several collaborators in different locations. We had the additional challenge of team members in the UK, France, Germany, and both coasts of the US and Hawai’i.
Had we all been in the same room, this process would have been infinitely easier, but I think it’s a huge credit to everyone involved that we managed to bring the project to market without ever sitting around the same desk. In fact, we didn’t actually meet face-to-face with our co-writer, Jamaica, until our SXSW premiere and still haven’t met the majority of our technical team in France.
We have so much respect and gratitude for all of the late nights or early mornings or for those who have been working in second languages throughout. Not to sound trite, but in a way we kinda hope this spirit of international collaboration speaks directly to the themes of the project—this is not a national issue or a political issue, but a human issue and one that needs collective international solutions.
How did the team come together? What role did Jamaica’s poetry play?
SJ: Ultimately, there were a lot of key issues that related to nuclear threat—for example the historical impact of colonialism and militarism on indigenous communities, or the decades of testing in the Pacific—that it wouldn’t be right for a team of British or French creators to express, since our nations have often been the most culpable.
Having spoken to Jamaica about her experiences on January 13, 2018, we knew that, as a Professor of Indigenous and Native Hawaiian Politics, she would be much better placed to contextualize some of those issues that we could not. And so we invited her to co-write the project with us and asked if she’d be willing to write a poem, bringing those issues into the foreground in a lyrical way.
We had no idea how that poem would evolve or how it would come to inform the structure of the narrative—and indeed the title of the experience—but as soon as Jamaica shared her first draft (which moved us all to tears), those ideas started to take shape.
If the challenge of our original creative brief from Games for Change and Princeton was to humanize and personalize the sometimes abstract issue of nuclear threat, then Jamaica’s poetry demonstrates beyond all doubt the power of her words and perspective to do just that.
Anything else you’d like to share with our readers?
SJ: Simply this—if you are moved by this experience, need aftercare support, or would like to know what you can do to help abolish nuclear weapons, then please join our community.
Visit our website onthemorningyouwake.com where we’ve collated a bunch of resources and created a toolkit to help you understand what steps you can take as groups or as an individual.
On the Morning You Wake (To the End of the World) is available now for free on Quest 2.