the ConversAItion: Season 4 Episode 23

Zoox Pops the Hood on the AI Behind Self-Driving Cars

We hear a lot about autonomous vehicles these days, but they remain a black box for most. This week on the ConversAItion, Jim sits down with Janek Hudecek, Director of Planning and Control at California-based autonomous vehicle company Zoox to unpack the tech that makes self-driving cars possible. They discuss Zoox’s unique approach to autonomy and how the company is rethinking personal transportation with vehicles custom-built for passengers.
Listen on Apple Podcasts badge
“We have built a car specifically for riders where every seat offers the same experience, including full control over infotainment and temperature. The idea is that you don't have to buy your own vehicle, or worry about where to park it, when to bring it to a dealer for maintenance or fueling it up. You just call it and enjoy your ride.”
Janek Hudecek

About Janek Hudecek

Janek is the Director of Planning and Control at Zoox, a role he has held since July 2020. He has been at the self-driving car company since 2016, when it was still in stealth mode. Before joining Zoox, he studied informatics and mechanical engineering at RWTH Aachen University and worked as an engineer for eight years in Germany. You can find him on LinkedIn here.

Short on time? Here are 5 quick takeaways:

  1. Zoox sees a future built for riders, not drivers.

    Today’s cars are built to meet the needs of the driver, providing them with maximum comfort and control. Front passengers have less access to the infotainment system and seat adjustments, and backseat passengers have even less space, fewer safety measures, no infotainment access and limited AC control.

    Zoox has created a vehicle purpose-built for passengers, with the goal of reinventing personal transportation to make it safer, cleaner and more enjoyable. In December, Zoox unveiled its first robotaxi, the Level 5, bringing their unique vision of autonomy to life. By rethinking the interior of a vehicle, and focusing on first-class safety and comfort for passengers, Zoox’s purpose-built vehicle offers every rider the same experience with full control over amenities, like infotainment and climate control. 

  2. The AI behind autonomous vehicles controls steering, brakes and navigation, with the goal of maximum passenger comfort.

    Built for riders from the ground up, Zoox’s new vehicles don’t have steering wheels. Instead, the company’s AI technology controls a steering rack with the help of an electric motor, enhancing its ability to electronically control the components typically guided by a physical turn of the wheel. Ultimately, this four wheel steering mechanism provides even more control than Zoox’s driver-supported vehicles. Zoox’s AI provides brake commands in a similar fashion. 

    Steering and brake commands are used to control the vehicle’s trajectory—how it steers or changes lanes, for example—to navigate safely and comfortably through an environment. To generate trajectories, the AI perceives the environment around it through a sensor stack shared by all of Zoox’s vehicles, which allows the company to collect data and train the perception system with both test and production vehicles.

    Ultimately, Zoox’s goal is to achieve maximum confidence and smoothness, as if a well-trained driver was at the wheel—and it’s working. While acting as a safety driver, Janek led trips where passengers couldn’t even tell AI was driving.

  3. Zoox’s AI can locate itself in even the most challenging landscapes.

    Another crucial piece of Zoox’s technology is localization—the process of mapping where the vehicle is in the world. The company developed algorithms that localize with high precision, even in the most challenging conditions. Highways, for example, typically have very few distinct features like building or specific signs and remain fairly unvaried, but have other challenges, like underpasses or tunnels where you might lose a GPS signal. The localization system is able to detect the vehicle’s environment, locate it on the map and provide this information to the AI system so that it can continuously plan its trajectory. Ultimately, this enables the vehicle to get from point A to point B seamlessly.

  4. The technology already navigates dense environments with confidence, and will scale to handle all possible driving scenarios in the future.

    To plan a trajectory, Zoox’s AI maps out a route from the vehicle’s current position to the destination. The most challenging aspect is accurately predicting interactions with other traffic participants, like a low speed lane change in dense traffic where the car has to squeeze in between two adjacent lanes. 

    Zoox’s sophisticated perception algorithms enable its vehicles to navigate these challenging scenarios. The company’s perception algorithms can detect dynamic agents, like other vehicles, bikes, pedestrians and animals; static obstacles like trash cans, trees and parked vehicles; and non-rigid matter like steam and exhaust. The technology is even able to detect specific characteristics about agents, down to whether a pedestrian is looking at their phone and may be distracted, or what direction they are facing. The company has made great progress in demonstrating how AI can handle complex situations in large, busy cities like San Francisco and Las Vegas, and continues to iterate for all environments. 

  5. Safety and quality control in autonomous vehicles are the keys to widespread adoption.

    Autonomous vehicles can be polarizing, particularly to people who aren’t familiar with the ins and outs of the technology. Zoox believes that a vehicle should not be on the road unless it’s safe for both occupants and surrounding vehicles and pedestrians, and has prioritized safety to ensure consumers feel fully at ease during a ride. In fact,the company actually sees potential for autonomous vehicles to be safer than today’s cars. In addition to built-in safety software and extensive testing, Zoox’s vehicles include more than 100 safety innovations that do not exist in conventional cars today, with even more in the works. 

    Zoox also closely monitors quality control—meaning how often the vehicle reaches its target destination without getting stuck. The company’s goal is to offer a smooth and pleasant ride without any surprises, which is attractive to the passenger and surrounding vehicles and pedestrians. 

Read the transcript


EPISODE 23: Janek Hudecek

Jim Freeze Hi! And welcome. I’m Jim Freeze, and this is The ConversAItion, a podcast airing viewpoints on the impact of artificial intelligence on business and society. 


Today, we’re joined by Janek Hudecek, Director of Planning and Control at Zoox, a California-based autonomous vehicle company.  Zoox is rethinking personal transportation, and designing a future built around riders, rather than drivers, that is safer, cleaner and more enjoyable. In December, the company unveiled its highly-anticipated robotaxi for the first time.

Janek has been with Zoox for several years, working on autonomous software systems, algorithms and motion planning. Today, he joins us to talk about Zoox’s unique approach to autonomy, and the tech that makes self-driving cars possible.

Janek, welcome to the ConversAItion, we’re so excited to have you here. 

Janek Hudecek Hi, Jim. Thanks for having me.

Jim Freeze So let’s start out with a little bit about your background. You’ve been with Zoox for about five years. What drew you to the company? What appealed to you specifically about Zoox’s approach as compared to other companies in the industry?

Janek Hudecek Yeah, sure. So the first time I heard about Zoox is now almost five years ago when it was still in the early startup stages and it had less than 100 employees. And back then I was looking for a new job and I had to make a choice. I could either join one of the well-established OEMs in Germany. Yeah, surprise. I’m not from Texas. Or I move to the States and join a, quote, “crazy startup.”

So I decided to take my chances and to fly to the Bay Area, go through the whole on-site interviews. And there I was introduced to a lot of talented people. And during that time, I also learned more about Zoox’s mission reinventing personal transportation with a goal of making it safer and cleaner. And, okay, granted, back then it was more like a vision than a mission. But anyway, I also had the chance to talk to the founders and they just had such a clear vision on everything with lots of details. And I was just, I was really impressed. And so I decided to join Zoox, moved to California, and never regretted that decision.

Jim Freeze You’ve held a handful of different roles with the company leading up to your current job today as Director of Planning and Control. Can you talk a little bit about your role and what a day looks like?

Janek Hudecek Yeah. Of course. Yeah. It’s true. I did go through quite a few different roads here at Zoox. I started as an IC in the planning and controls team and I was working on our trajectory generation algorithms. And those define how exactly the vehicle moves with the environment, like how it steers, how it changes lanes, and defines the feeling of it. And I always had the goal to achieve a maximum level of confidence, smoothness, and I didn’t want to make it feel like a robot was driving, but more like a well-trained driver. So back then I also had the honor to be the safety driver in some of our demos. And I remember a few ones where the audience would get in and we would start driving. And after some time they would impatiently ask, “Hey, like when do we hand over the control to the AI?”

Because they were expecting a somewhat jerky and brake-tap-y ride. And so it always filled me with pride to respond, “We did this two minutes ago when you stepped into the vehicle,” and they just had the surprised look on their face and that told me, mission accomplished. So in order to achieve this level of comfort, you have to optimize for the big picture, and it’s really the weakest link in your stack that causes undesired behavior like brake taps and small unnecessary nudges. And so we started asking ourselves as a company, how do we identify those weakest links? And the answer for us was to work very closely together as a team across department boundaries, rather than this finger-pointing game and claiming it’s the other team’s fault. Like I remember I was always typing, “Oh, it’s perception’s fault.”

It’s like the wrong velocity we get here. And I truly believe that this cross-functional cooperation is one of the major reasons for Zoox’s success and it still dominates my day-to-day work today. And of course my work has changed since then. It’s less engineering and more staying on top of everything, making sure that everything is taken care of. But fortunately I do have a great team that I can rely on. And sometimes there’s even a little bit of time left for me to do a technical deep dive into something, or at least a quick code review.

Jim Freeze Interesting history. So Zoox has taken a unique approach to autonomy. I mean, we’re all familiar with the Teslas of the world. And every major manufacturer now is going down the path and talking about more autonomy in cars, but you guys have taken a unique approach to autonomy and purpose building a vehicle designed for riders instead of drivers. Can you tell us a little bit about this strategy and why you built your business around it?

Janek Hudecek Yeah, happy to. So maybe let me first take a stab at explaining what “built for riders” means. Today’s cars are built around the driver. Who’s basically the first-class citizen in almost every car, meaning the driver has the maximum comfort and control and the front passenger usually gets pretty similar, maybe slightly worse experience. Examples here are less comfortable access to infotainment systems and sometimes missing adjustment options for the seat that’s motors in the seat. But that’s all still complaining at a pretty higher level. And then you get into the second row of the seats, which usually offers a lot less comfort. It has less space, no infotainment control, limited AC control, and so forth. And you basically depend on the mercy of the driver at this point. And on top of that, the second row is usually also less safe in case of an accident.

So now if you look at our targeted business, which is the riders, that second row should really be the one that offers first-class safety and comfort. And we were able to achieve this goal by rethinking the interior concept and building a car where every seat is equally good and has full control over infotainment and climate control with the same driving experience on all the individual seats. And that’s why we call it “built for riders.” So it, of course, it doesn’t end just with a vehicle, it’s the whole ecosystem that we’re building around it. And the idea is that you don’t have to buy your own vehicle. You don’t have to worry about where to park it or bring it to a dealer for maintenance, fueling it up, or anything like that. You just call it, you know what you get, you enjoy your ride, and once you get out of it, you just forget that it’s even there until you need it again.

Jim Freeze It’s pretty amazing. And specifically on that point, we watched a video where you narrated an actual trip of one of Zoox’s vehicles driving fully autonomously from Menlo Park to downtown San Francisco. And I can tell you, having lived in San Francisco, I’ve done that drive many a time. And the fact that the car was able to actually do that, it’s just amazing technology. Can you pop the hood, pun intended, and tell us a little bit about the role that AI plays in making an autonomous trip like that possible?

Janek Hudecek Yeah. Nice pun. Sure. Let me try that. So the AI ultimately controls the vehicle by the same means as you and I, and that is by using the steering, the motor, and the brakes. And modern cars, like the Highlanders we use in our fleet, they allow for electronic controls of these actuators.

So instead of physically turning the steering wheel, the AI controls the input talk to the steering rack with the help of an electric motor, that is part of your power steering rack. And since our vehicle, like the L5 vehicle was designed from the ground up for our riders, it wasn’t born with a steering wheel, but instead with an enhanced ability to electronically control all those actuators and that in combination with the fact that it’s an all electric vehicle with four wheel steering enables us to control in with an even tighter tolerances than what we can control the Highlanders to. So let’s go back to steering actuation and brake commands. Those are used to track the trajectory that I mentioned before, which is a series of desired states over time, navigating the vehicle safely and comfortably through the environment. Does that make sense so far?

Jim Freeze It does make sense. And in particular, I’m just, maybe you’re going to embellish on this a little bit more, but I’m thinking about highway driving is one thing, but then when you get into a city where there are pedestrians and there’s ambulances, folks on bikes, a real-world environment, that’s highly unpredictable. Can you talk a little bit about how the technology deals with the unpredictable nature of city life?

Janek Hudecek So actually hold on for a second. I would just talk a little bit more about the high level approach, the perception system, and so on. So if you want, I can dive a little bit into that before we go into the unpredictability.

Jim Freeze Absolutely. Yeah. Tell the whole story there. That’s great.

Janek Hudecek Okay, cool. So, okay. The question now becomes how to generate those trajectories. So let me talk a little bit about that first. So at first, the AI perceives the environment around it, through the sensor stack and this sensor stack is actually shared between all vehicle platforms. And what that means is that the types of sensors, as well as the geometrical configuration, is the same on the Highlanders and on the other five vehicles, because, and this allows us to collect data and train our perception system with our test fleet and use the same algorithms in our production fleet, which is pretty neat.

So what do those perception algorithms detect? Well, first and foremost, they detect what is called dynamic agents, that includes other vehicles, bikes, and pedestrians, but also other things like birds or larger animals like dogs and deer. And it also detects static obstacles like trash cans and vegetations, parked vehicles, you name it, as well as the nonrigid things like fog and steam and exhaust. 

But that’s also not where it stops, like, to give the world even more color and to enable highly interactive driving, we further detect specific characteristics about other agents around us. So for example, is a pedestrian distracted because they are looking at their phone? Are they walking? Are they standing still? Which direction are they facing? And we also detect whether they are a construction worker or a police officer who controls the traffic, et cetera, et cetera.

Jim Freeze That’s amazing.

Janek Hudecek It is. And that always creates these nice and fancy videos. That’s the topic of perception. And then another important aspect is actually the localization, which is the process of mapping the weakest position precisely into a map so that it knows where it is in the world. And we developed algorithms that allow us to localize reliably and with a very high precision, even on the most challenging conditions. And one example of these challenging conditions are actually highways, and you named it before. Highways because they typically have very few features like trees or specific signs. So it always looks kind of similar and they also have other challenges like underpasses or tunnels where you might lose the GPS signal. So once you have detected all the environment around you, you know where you are on the map, you get this info to the AI, and it can all start planning its trajectory.

At a high level this works by first planning a route from your current position to your destination, very much like your navigation system. And then comes to the tricky part, which is this high-frequency planning of a short time horizon trajectory that defines how to interact with the immediate environment. And probably the most challenging aspect is to accurately predict interactions with the other traffic participants in situations where your behavior affects their behavior. So just to give you an example, you can imagine a low speed lane change in dense traffic, where you have to squeeze yourself in between two constantly adjacent lanes. And you kind of have to force your way in, and the reaction of the adjacent cars highly depends on how you execute that maneuver.

Jim Freeze Yeah. Having done that as a driver many times in San Francisco, I know exactly what you mean, and from my perspective that makes the technology even more amazing. So one of the things that I’m kind of curious about, and I’m sure you would, you deal with this is, a lot of cars today have things like lane centering in it. And the other day I was on a highway and I had a friend in the car with me and I took my hands off the wheel because I said, “Oh, look how this lane centering works.” And the car kind of stayed in the lane. And my friend was kind of freaked out. And so there’s this, he’s like, “No, put your hands on it.” And so there’s a kind of a consumer comfort that has to happen with the technology and some confidence. What do you think it’ll take for the kind of everyday person to get on board and feel confident with the technology?

Janek Hudecek Yeah, that is a great question. I think first and foremost, our opinion is that autonomous vehicles do not even deserve to be accepted if they are unsafe. So safe, as in safe for the occupants, as well as for other traffic participants. And they actually provide a huge potential for being safe. And let’s quickly look at today’s cars for comparison. Most of the safety systems in there are focused on protecting the occupants, and protecting the occupants in case of an accident. And only slowly do we see that systems that actively are avoiding accidents in very specific situations get introduced. So why is that? Like those systems, they have to be designed in a very strict way to avoid false positive braking, or even steering intervention, because otherwise they would not be accepted by the consumer who ultimately pays for them.

And AVs just offer a great potential here. And beside the software and the extensive testing, you can see the big role that safety plays at Zoox also by the fact that our purpose-built AV includes more than 100 safety innovations that do not exist in conventional cars today. And throughout the years, we will announce more and more of those. So that’s part of the safety aspect, and another key, and you mentioned that already, to this acceptance is the right quality and the rate of what we can call the mission accomplishment. And that is how often is the vehicle able to reach its target destination without getting stuck on its way. And as I mentioned earlier, our goal is to offer a smooth and pleasant ride without any surprises. So this is not only attractive to the passenger, but also to the traffic around you, because your driving becomes much more predictable and confident, and then people have a much easier time dealing with you.

So you’re not like this one special, crazy car there. So how do we achieve the desired level of comfort? Well, I talked a little bit about it earlier, but I guess I can summarize it as you obviously, you need the right set of algorithms and metrics, and you need a very close cooperation between the software and hardware engineers, and the vehicle operators who are ultimately riding those AVs every day. And we have always encouraged them to speak up if they don’t like a specific behavior or get honked at or whatever, to give us something we should look into and that has been very helpful so far.

Jim Freeze I suppose it’s no different than any other technology that people are apprehensive about because they’re unfamiliar with it. But I think the way you describe the built-in safety and quality it almost sounds like in reality, it potentially is more safe. It probably is more safe than somebody driving themselves, just because of all of the technology built into it. So it’s fascinating. I guess one last question for you, as you think about the future, look into the crystal ball, what do you think is next over the course of the next five years for the self-driving industry more broadly?

Janek Hudecek Yeah, let me polish my crystal ball real quick. Well, in the past few years, we’ve made great progress on our driving abilities and we’ve demonstrated how our AI can handle really complex situations in various cities, in our case, mostly San Francisco and Las Vegas, by the way. And feel free to check out the numerous posts about that online. And while we continue to work on improvements of our AI stack for denser urban environments, we now also focus on testing our own purpose-built vehicle. And as I mentioned before, this vehicle comes without a steering wheel and we just have to make sure that the AI stack confidently handles each and every edge case since there is no safety driver onboard who can simply take over.

And as our first step, we are testing it thoroughly on private roads and then iterating and improving on our software and validations before we will actually start testing it in driverless operation on public roads. To answer a question about what’s next for the industry, I think it’s about scaling. Of course, we first need to show that we can drive confidently in a specific geofence, a single city for example. But soon after we want to scale our business and all the development and validation that we invest in, it just needs to be able to scale to those new geofences as well.

Jim Freeze I haven’t ever ridden in a fully autonomous vehicle, but I’m anxious to try one. And in particular, I’m really anxious to try Zoox’s vehicles because once again—and I encourage the listeners to go to your website and look at some of the videos—it really, really is impressive technology. And we really very much appreciate you, Janek, taking time and making a connection between artificial intelligence and what you’re doing with autonomous vehicles. It’s fascinating stuff. Thank you so much for being a guest on The ConversAItion.

Janek Hudecek Yeah. Thanks again for having me and giving me the chance to speak.

Jim Freeze On the next episode of The ConversAItion, we’re joined by Tatsiana Maskalevich, Director of Data Science at Stitch Fix. Tatsiana will discuss how the company brings together AI and seasoned stylists to provide personalized clothing recommendations that align with a customer’s unique preferences and help them evolve their style.

This episode of The ConversAItion podcast was produced by Interactions, a Boston-area conversational AI company. I’m Jim Freeze, and we’ll see you next time.



Check out more episodes of The ConversAItion.