Improving space missions using software-based procedures and communications – with Epsilon3

Episode 28 of the Space Industry podcast is a discussion with Laura Crabtree, co-founder and CEO of satsearch member company Epsilon3, about how to improve space missions using software-based communication tools, practices, and operations tracking procedures.

Contents

• Episode show notes
• Epsilon3’s OS for Space Operations
• Transcript

---

Episode show notes

Epsilon3 is a California, USA, based space software developer that creates tools to better communicate and manage operational procedures.

The team includes engineering and design professionals from Northrop, Google, and SpaceX, with experience that includes first-hand operational management of sending American astronauts to the ISS.

In the podcast we discuss how improved team communication and project lifecycle management can result in better engineering outcomes and help avoid mission failure. We cover:

• The gaps that can exist between designers and operators
• The impacts of a lack of procedure tracking on space missions
• How to improve team communication, and why this is important
• Examples of missions and services that benefit from enhanced operations management

---

Epsilon3’s OS for Space Operations

---

Episode transcript

Hywel: Hello everybody. I’m your host Hywel Curtis. And I’d like to welcome you to the space industry by satsearch, where we share stories about the companies taking us into orbit. In this podcast, we delve into the opinions and expertise of the people behind the commercial space organizations of today who could become the household names of tomorrow.

Before we get started with the episode. Remember, you can find out more information about the suppliers, products, and innovations that are mentioned in this discussion on the global marketplace for space at satsearch.com.

Hello, and welcome to today’s episode. Today I’m joined by Laura Crabtree from Epsilon3. Epsilon3 develops operational tools, for space missions, and they’ve begun with web-based procedures.

The company supports project life cycles all the way from integration, right through to operations. And today we’re going to discuss a little bit about saving space missions from failure, using software-based procedures and communications systems. So Laura, welcome to the space industry podcast today. Is there anything you’d like to add to that introduction?

Laura: No, that was a pretty fantastic introduction. Thanks, Hywel.

Hywel: Okay. No problem. You’ve had pretty extensive experience working with teams within companies like SpaceX, running operations for space missions. In your view, based on that experience, what sort of gaps do you think exists today in production or testing, you know, between engineers who design and develop a procedure and then the operators who will need to routinely use them?

Laura: Yeah, it’s a really a detailed question. I think there are a lot of different pieces that can be looked at. The first thing I would probably look at is what tools are people using. And, you know, I’ve spent a lot of time in the industry and I’ve talked to a lot of people, the tools that people use are tools that I use very early on in my career.

So, you know, you can ask anybody, you know, how do you collect data? How do you look at different datasets from different tests? And most people will say, you know, I have CSV files or I have Excel spreadsheets, or, you know, I take the data and I then create a report from it. So I think it would stem from that is kind of the start.

But then, the second part of your question, which was between the engineers who are designing and developing either a system or a part, and then the people who are operating or testing the system, um, there is a handoff, so I’ve seen a couple of different models as to how that’s worked, but very commonly you’ll have a responsible engineer that is designing a piece of a system.

And then that responsible engineer will write down a test procedure and hand it over to a technician. Now the test procedure, maybe it is on Word document. Maybe it is on a confluence page. Maybe it’s on a PDF. Maybe it’s on a different system. That’s an internal tool, but then there is something that happens when you hand it off and the technician maybe doesn’t have all of the supporting documentation.

Maybe there’s a breakdown in what they understand or don’t understand. So we’re creating this system where you can link other documentation. You can have supporting documentation sort of in line, and then you can also suggest different edits to your test procedure in the moment. Um, because as you are probably fully aware, you are always learning something when you were testing or operating a system and you’re going to learn something new.

And if you forget to write it down, if you forget to tell the person that actually wrote the documentation, now you have to learn that again and again, and you’re going to save time if you can just communicate that change the first. There’s also the operator group. So, you know, we talked a little bit just now about the technicians, but the same is true for operators.

You’ll have responsible engineers and many times, you know, at SpaceX, a lot of our responsible engineers turned into operators, but you have to take them from that responsible engineering mindset and teach them the operations mindset because operating your system in space is not the same as testing it on the ground.

There’s a lot more complication that comes into that. And so, you know, being able to translate from your test all the way to how you operate your vehicle, you’re going to learn things in your test. And so if you’re using the same system to do your tests and your operations, I think that’s where the value comes in and getting kind of everyone to come together and bring that information to the same place.

The best way I could see how the communication could be enhanced between the different groups of people.

Hywel: Absolutely. Okay. So it’s, um, very much about context, but also you need a technical solution to be able to capture and use that context. And then as you said, as well, the, the mindset is important as well.

Laura: You also want to take the different tools and make sure that they’re talking together. You know, you don’t want to gather information in one place, have to email it, rely on someone actually saving the attachment from an email to then create a report. You want something that is more localized, something that is automatic. I run a test and everything from that test is automatically saved.

You have timestamps, you have supporting documentation. You have any closeout photos, any pictures, any plots, all in the same. I’ve run into a lot of problems where, you know, you have a notepad where people are taking notes on things. You have an email, you have an instant messenger conversation and whatever tool they’re using.

And if you want it to play back and or look at what happened at any given time, it’s really hard to piece all those things together. And it takes a lot of time to do that. Especially if you have a team that is more than a few.

Hywel: And on that. How often do you think such teams end up, not just being slowed down by the process, but where it hits in a brick wall due to the lack of the communication that exists? Or the ability to track the procedures as they are in use?

Laura: I think a lot of people don’t like to talk about the problems that they’ve had. So you might not find a lot of them in the news, but in every operation, there’s this communication breakdown. If you have, you know, you always have some.

Some type of voice communication if you have more than one person. And if that voice communication is taken up by nominal tasking, now you don’t have open lines of communication if something was to go wrong. So the way that I look at sort of the, the types of communication that are needed and the timing at which you need them, there’s a couple of different kind of tiers of communication.

The first one is nominal actions. So things that you can take your time with things that are well planned out in advance, that don’t require you to react on a five minute basis or one minute or two second basis. And these things don’t necessarily need to be communicated via voice communications.

And so you should plan those out. You should do them. You should record that you did them, but you don’t need to be talking about them because everybody knows that they’re being done. Then there’s contingency things where they maybe don’t need immediately. But you have a couple of minutes to react and maybe you have time to call people in to evaluate the system.

And that’s the time when you want people to be communicating via whatever your procedure or whatever your operational system is. You don’t want people to be using instant messenger. You probably don’t want people to be sending random emails. You want all information kind of coming into one place. And so you still kind of have time on your side.

And that is when. Sometimes use the voice loops, and sometimes you can just communicate via electronic procedures system, lastly, but most importantly, there’s this hair on fire situation where you have to take action imminently. And these are the types of things that we do a lot of training for. It’s maybe less than a minute, maybe, you know, 10 seconds there.

Aren’t and there shouldn’t be many actions that you have in this, in the course of space operations that require that type of, you know, time period. But if the voice loops or some type of voice communication is taken up with nominal or like contingency actions that you have time for now, you can think about times when maybe you don’t have time to call your friend. And you just have to say, I’m doing this right now, but if the voice clips are taken up with one of the other tiers of, of actions, now you have a problem.

Hywel: Do you think you could give us any example or an example of when you know, this lack of communication and tracking ability is cause a real problem for a team?

Laura: There are a couple of situations that I’ve run into there’s one that was in the news a lot. Actually, I wouldn’t say it was a breakdown in communication, but I would say that it was one of those like needs of voice loops and all communication channels open for contingency, such that if the team hadn’t been trained and hadn’t had a good way of communicating and coordinating, it could have gone a completely different way.

And this is the mission known as CRS2, and in this mission, we had multiple check valves that failed and did not open. And the vehicle did not have control of its attitude. And we were unable to perform maneuvers in the first couple of orbits of the mission. So this was many, many years ago, but it’s one of those things that’s like burned into my memory.

So we got on orbit and we realized that we had a problem. And at that moment, we called in specialists and those specialists, you know, started looking into what to do, but the operations team, they were focused solely on sending commands to the vehicle to try to open the valves. And they had to send probably, I think on the order of hundreds of commands to try to open the valves because it was tumbling.

So we had to establish a connection, send a command. And at that point, The connection that acquisition of signal was coming in and out. So we had to send commands rapidly to make sure that they would be accepted by the vehicle. And you can imagine if some of the nominal actions were being taken up on the voice loops that may not have happened in the time period that was needed.

So there’s a good example of, you know, we had taken at this point, you know, multiple other missions to kind of like hammer out our operation. And a lot of those somewhat nominal or more automated tasks, we didn’t have to talk about them. We just did them. And it was communicated via, you know, electronic documentation to the rest of the team.

And so that’s a really good example of where things could have taken a different turn, had we not been really prepared?

Hywel: You know, you mentioned the, sort of the three tiers of communication that how you consider organizing this stuff in, when we talk about procedures, but does the size and all the scale of, of the team or the company as a whole affect their ability to communicate in these ways, um, based on your experience, you know, can smaller teams kind of get away with the product a bit more easily, or is this more rigorous approach, something that even they should?

Laura: And that’s something that’s definitely come up a couple of times, we’re supporting teams that are, you know, 10 all the way up to multiple hundreds. And it depends on what they’re doing. But I would say that teams that are, you know, less than five, they benefit a lot from having traceability in the system.

So if you think about, you know, an Excel spreadsheet or a word document, you have to remember save as record timestamp, record this, send an email and they have benefited a lot from taking out a lot of those manual tasks. You can think about it as the larger teams benefit a lot from the communication and coordination.

And so depending on the size of the team, the value is slightly different. And kind of increases as your team size increases, but it does hold a lot of value for small teams. I’ve talked to 5 person teams that are just tired of writing things in word documents, and then saving as, and then forgetting where they saved something.

With our system, you know, it’s always available. You have basically on a website, you go to the website and everything is there. You can search the historical reference of everything that you’ve done and see who did it rather than always having to remember the folder structure and where that attachment was and look through the pictures of the closeout photos. That’s the difference in value is grows exponentially, but it doesn’t start out near zero. It starts out at a very high value.

Hywel: I mentioned in the introduction that the initial kind of product suite you’re focusing on is, um, uh, a web-based procedure system. What are the immediate problems that Epsilon3, that you are looking to address and, and what types of teams are you hoping, you know, would benefit from the solution that you’re developing?

Laura: The first thing that I wanted to do was, take a look at what tools companies were using for their operation. And I heard a couple of things when I first started looking into creating Epsilon3, and the first thing was we’re developing tools in-house and those tools are developed by aerospace and mechanical engineers that aren’t traditional software engineers and the tools work, but they’re not amazing. We want something that, you know, scales easily that is tailor made for this rather than building a one-off in house. So that’s one thing I heard.

The second thing I heard was that you have software engineers and then you have your operations and integration and test engineers. And you either have to teach your software engineers how to do operations integration tests, or vice versa in order to get the right product for the application. And so, you know, you can think about teaching software engineers, operations, or teaching operations engineers, how to code both of those things are not the best use of either of those people’s time.

And so when I started coming up with the idea for, you know, really serving this portion of the industry, it became really clear that there aren’t a lot of tools that are strictly geared towards operations. And so that’s where we wanted to make a huge impact. The immediate problems that we’re looking to address is mostly in how we perform operations. And what kinds of tasks we have to do manually. So I want to take a lot of that manual action out of operations.

So you can think about, you know, creating a report, sending something to someone for review, and let’s say, I want to attach something in an email. Well, why are you attaching it in an email? Well, there’s no other way to do it. So I want to take a lot of those tasks out so that we can really look at operations as a holistic picture and really standardize and make everyone more efficient in that way.

And then lastly, you know, that the communication and the coordination piece. So how do you coordinate with other people? I mean, we’re all working from home half the time. I’m at home now and I work from home and how do I coordinate with someone that’s halfway across the world? How do I communicate right now?

We set up a zoom and we zoom each other. If I was to do something and without sending you a notice, you would actually get an update that says, Hey Laura did this thing on the satellite. I think that would be really powerful and probably easier than setting up a zoom or sending you a slack message.

Hywel: Yeah, yeah, yeah. Set it up. Zoom only to say let’s take this offline and do it separately anyway.

Laura: Yeah, exactly.

Hywel: Could you share a couple of examples or an example of the use cases where Epsilon3 is you know, made an impact on a team or a mission.

Laura: Yeah, definitely so I mentioned earlier that, you know, we’re supporting teams that are really small and also, you know, multiple hundreds of people that, um, a couple of really good examples.

We started supporting a couple of teams when they were really small and we started scaling with them. One of the first companies that we started supporting is Stoke space and they’re developing fully reusable rockets. And they’ve started using us for a lot of their engine testing and a lot of their integration.

And they’ve found that a lot of what they did previously, we’ve taken out a lot of the manual tasking for what they’ve done. We’ve also helped ensure that their vehicles and their people are safe because if you have a well-oiled machine in an operation, you can ensure that, you know, X is done before Y but if you have something that isn’t automatically updating, I have to call you up and say, Hey, did you close that valve? Did you make sure that this was safe? And you can imagine that this is going to make everyone a lot safer.

Similarly, a small team that is growing is Inversion Space. They’re doing a lot of their engine testing using our software and are just so happy that they don’t have to call each other up and they don’t have to edit on word documents anymore because a lot of the times they were complaining to me about formatting and editing and copy and paste and things that, we just accept are the norm, but it doesn’t have to be that way. We’re just trying to help them take a lot of those things out. And so we’ve been really happy to support them and help them grow.

Hywel: Finally, I guess I’m looking ahead. So where you see this aspect of the industry, you change it. How do you see the role of software? You know, when it comes to the management of procedures and communications sort of maturing over the next five years, if we look beyond the horizon of, uh, of missions already planned and in progress, And looking at both satellite missions and even human space flight or deep space, robotic missions, what sort of routines, do you think could be built into such missions by then or what sort of tools or paradigms might be in use?

Laura: We’re looking really hard at integrations with other software packages. We know that there are a lot of other software packages that people are using out there. And we want to encourage all of the software packages that are out there to be integrated together so that we’re building a whole suite of software that supports operational landscape.

I’ve heard so many times that people, again, don’t want to always build tools in house. So we want to build the entire suite of software to support these operational procedures that people are doing. In addition, you know, as we grow, we’re looking towards automation. Obviously when you build a spacecraft, you build in as much automation, you can think of, but as I’m sure you also know when you’re in space, there are things that happen that you may not be able to always build automation on the spacecraft. So we want to help on the ground either in automation for testing and automating procedures from space. So I can think of many times when I learned things on the ground.

That I couldn’t always upload to tell the vehicle to do. If I could automate the task from the ground to just send a couple of commands, check the telemetry, and then send more commands to reconfigure the spacecraft. Now I can get myself off console and not have to sit and stare or be called in to perform actions.

I can be freed up to do other things. So we’re looking at that. Obviously looking towards the human exploration and human space flight business, as it’s growing, we would love to support that industry and make sure that our humans, as they reach further will be safe and efficient, when they’re operating vehicles, you know, running tests, doing maintenance on their spacecraft or space station. Those are the things that I want to see us expand into.

Hywel: Fantastic. Well, I think there’s some, a really good place to wrap up. Thank you very much, Laura, for sharing all those insights and that information about how teams companies can communicate better and the track the procedures better, generally develop better missions, better technology.

On behalf of satsearch and Space Industry podcast. Thank you very much for spending time with us today.

Laura: Thanks so much for having me. It was great chatting with you.

Hywel: And to all the listeners at the end, remember you couldn’t find out more about Epsilon3 on the satsearch platform, and we’ll put all the links to the company in the show notes and on the platform, you can use our free request tool to make requests for documentation, uh, introductions to the company, further information or whatever else you might need for your, uh, procurement purposes or trade studies.

Thank you for listening to this episode of the Space industry by satsearch. I hope you enjoy today’s story about one of the companies taking us into orbit. We’ll be back soon with more in-depth behind the scenes insights from private space businesses.

In the meantime, you can go to satsearch.com for more information on the space industry today, or find us on social media, if you have any questions or comments. Stay up to date, please subscribe to our weekly newsletter and you can also get each podcast on demand on iTunes, Spotify, the Google play store, or whichever podcast service you typically use.