Robust Ugly Zone Scenarios For AI Autonomous Cars

Ugly zone stress testing pushes AI self-driving cars to the limits using simulations, with the aim of making the cars safer on the read roads.

By Lance Eliot, the AI Trends Insider

Is he a man or a machine?

That was asked about Francesco Molinari when he won the 2018 Open Golf Championship and earned himself nearly $2 million in prize money.

It was his first major golf victory and it was the first time that the now 147th annual golf tournament was won by an Italian (there was a lot of celebrating in Italy!).

How did he achieve the win?

You could say that it was years upon years of other golf competitions and smaller wins that led to this big win.

Or, you could say it was maybe the weather conditions and the mood and skills of the other golfers at the tournament that converged to let him be the best on that particular occasion.

Presumably, you could say it was random chance, or maybe that he had a lucky charm.

Would you be willing to say it was due to practice?

As they famous joke goes, how do you get to Carnegie Hall – via practice, practice, practice.

Francesco is known for being a slave to practicing.

Of course, the odds are that many of the other golfers there had put in as many hours practicing as he has. It stands to reason that golfers at that level of play are practicing all of the time, day and night. They likely dream about golf. They likely are mentally playing golf when they eat lunch or dinner. It’s an all-consuming passion for most of them.

If they are all practicing about the same amount of the time, perhaps the nature of how they practice might account for some of the differences in their playing levels. Just because I say that I practice, it doesn’t indicate in what manner I practice. For almost any kind of practices, you can take a varied approach to how you practice.

I used to play tennis when I was in college. My practices often involved hitting tennis balls against a wall for hours on end. When I could find someone to play against, I’d certainly do a practice game, but at other times it was solo practicing that took place. Is the potential outcome of the solo practicing as good as doing actual practice games? You can debate the matter. The practice games are certainly more akin to what will occur when playing a match game, and so it seems logical that the practice game is a better form of practice. On the other hand, the repetition of hitting hundreds of times back-and-forth against a wall does build-up your arm and body in a manner that a practice game cannot.

Francesco realized about two years ago that he needed to do something to boost his golf game.

He had been a professional golfer for more than a dozen years and had been an amateur champion before turning pro. But, he had not yet reached the top echelon of the winner’s circle of professional players. Would it take some kind of voodoo magic to push him to the top? Did he have to make changes to how he perceived golf and played golf.

He opted to radically change his practice routines.

He entered into the ugly zone.

Introducing The Notion Of An Ugly Zone

For those of you familiar with the Twilight Zone (the old TV series), I suppose the ugly zone sounds somewhat like it.

There’s nothing especially odd about the ugly zone though. The concept is relatively straightforward.

When practicing any kind of skill, you are to do so with a maximum amount of pressure, perhaps even more so than what you’ll experience during live competition play.

The goal is to make practices as rough and tough as a real match.

Maybe even more so.

When I used to help coach my son’s Little League baseball team, we often had rather acrimonious debates among the coaches and assistant coaches about whether the practices should be easy or hard.

There were some coaches that said we should be easy on the kids and provide a supportive environment for them to learn baseball and hone their skills. It was about fun. It was about falling in love with the sport. We knew in contrast that the actual games would be pressure cookers, so the practices would hopefully serve as a means to inspire them towards becoming proficient baseball players.

If you’ve not been to a Little League baseball game, allow me to open your eyes. You’ve got the doting parents that want their kids to win no matter what it takes. Many hope that their child will someday become a big league player, getting the fat paychecks and the out sized fame.

Some of the eager parents had a different but similarly high pressure perspective, namely they thought that winning was the key to life, and they didn’t care that it was a baseball game per se. Instead, it was that their child needed to discover that winning is good and losing is bad.

It wasn’t so important that the child was able to swing a bat — what was really paramount was that you must win however you can achieve it – this includes maybe swinging a bat, or catching a fly ball, or tricking the opposing team, screaming at the other team, spitting on the other team, you name it (all’s fair in love and war, and baseball).

Should the practices be like the games?

Would it be better to have the boys experience the crazed high pressures of a real game during their practices, or would that distract them from the needed step-at-a-time of learning their craft?

Maybe doing high pressure practices would make them emotionally upset and they would become disgruntled about playing the sport entirely. They’d also have no opportunity to try out new techniques. They’d be constantly under the gun, so to speak.

You’ll find this next anecdote amusing (or, maybe serious!).

One of the coaches suggested that we setup loudspeakers at practice that would blare out the sounds of a typical game audience, including a recorded cacophony of loudmouthed spectators yelling and screaming, doing so during the practices (side note, we opted to not do that). This would help re-create the setting of actual games, apparently.

Anyway, there are some that philosophically believe that practices need to be conducted in a high pressure manner that aligns with the pressures encountered during competitive matches.

Of course, maybe doing this with Little League kids is not the right audience. Perhaps we might say that this approach is more suitable to adults. Furthermore, adults that are already versed in their craft, rather than someone just starting out to gain a new skill.

Well, I realize that some of you that believe in the ugly zone approach will maybe disagree with me and my list of carve out exceptions, and you’ll insist that the ugly zone is always applicable, regardless of age, skill level, etc.

Fine, have it your way.

Let’s agree to disagree, and continue on, thanks.

Desirable Difficulty Is King

Francesco shifted his practices two years prior to his incredible win into becoming near torture tests.

His new coach embodied the ugly zone philosophy and emphasized that the frustration level had to be equal to a real game or possibly higher than a real game.

The more annoyed that Francesco became with his coach, the more the coach knew he was doing something right in terms of making practices hard. Every practice golf shot was considered vital. No more of the traditional hitting golf balls with your clubs for mindless hours on end. Instead, all sorts of complicated shots and series of shots were devised for practices.

There you are on the putting green, practicing. You are 8 feet away from the hole. You try to make the putt, but miss the hole. It’s practice, so you just shrug your shoulders, you try to figure out what went wrong, and you then casually setup to do the same shot again. Not so with the ugly zone. That 8-foot putt is for the golden trophy, every time. If you miss the hole, you are done for. You are a failure. You must take each and every putt with somber seriousness. If you happen to make the putt the first time, that’s not good enough. Do it again. Indeed, do it five times in a row, flawlessly.

Some psychologists suggest that adding challenges to practices tends to boost the long-term impacts of the practices.

It is often referred to as desirable difficulty.

As mentioned earlier, you might perceive that this challenges factor should be for all of the practices and all of the time of the practices, or you might believe that it should be done in a more measured fashion, just for some of the practices and maybe for just some of the time of those practices.

Let’s take a slightly different angle on this ugly zone notion.

Suppose you had practices that never were in the ugly zone.

So far, I’ve mentioned the belief by some that the practices should always and exclusively be in the ugly zone. The opposite tack perhaps would be to never use the ugly zone approach at all. I’ve seen this happen in some contexts.

For example, I was helping a group of middle school students learn about robotics as they were getting ready for a robotics competition.

A fellow mentor was purposely having them avoid encountering any problems while practicing writing code to program the robots for doing various tasks. I took him aside and gently pointed out that we ought to have the kids experience some issues or errors, so that they’d be ready during the live competition. He insisted that any kind of difficulty would mar their learning and rebuffed my suggestion. Sadly, things didn’t go very well for them during the live competition and they were baffled as to what to do when their robots faltered.

So, I’d generally argue that you need some amount of ugly zone involved in practicing.

I suppose that I’m the Goldilocks kind of practices person. It should be not too much ugly zone, and nor too little ugly zone. Just the right amount of ugly zone is the aim. And, crucially, having no ugly zone at all is likely an unfortunate and perhaps misguided omission that undermines the overall utility of the practices.

The ugly zone proponents contend that you need to learn how to think and act under pressure.

They say that if you are the type of person that gets butterflies in your stomach during live competitions, you need to hone your skills so that instead of expunging the butterflies that you instead learn to shape them so they fly in a formation. Use the pressure to overcome your fears. Use the pressure as a kind of high octane juice. That’s what the ugly zone is supposed to achieve.

AI Autonomous Cars And Ugly Zones

What does this have to do with AI self-driving driverless autonomous cars?

At the Cybernetic AI Self-Driving Car Institute, we are developing AI software for self-driving cars. In addition, we make use of a wide variety of techniques and one of those that we advocate is the use of the ugly zone.

Allow me to explain.

Many of the auto makers and tech firms that are making AI self-driving cars are doing testing in these ways:

  • Use of simulations
  • Use of proving grounds
  • Use on public roads

For my article about the use of simulations for AI self-driving cars, see:

For my article about providing grounds for AI self-driving cars, see:

When an AI self-driving car is being “tested” on public roads, this means it is being done in a relatively uncontrolled environment and that presumably just about anything can happen.

On the one hand, this is good because there might be that “unexpected” aspect that arises and for which it is then handy to see how well the AI can respond to the matter. On the other hand, you might go hundreds, thousands, or millions of miles using the AI self-driving car and not encounter these plausible rare occasions at all, thus, in that sense, the AI self-driving car will not be tested readily on such facets.

There’s also the rather obvious but worth stating point that doing “testing” of AI self-driving cars while on public roads is something of a dicey proposition. If the AI is unable to appropriately respond to something that occurs, the public at large could be endangered. Suppose a man on a pogo stick suddenly appears in front of the AI self-driving car and the AI does not know what to do, and perhaps hits and injures the man – that’s not good.

See my article about the Uber crash incident that killed a pedestrian:

And, my follow-up article about the Uber crash:

As I’ve mentioned many times, there are some AI developers that have an “egocentric” perspective about AI self-driving cars and seem to think that if someone does something “stupid” like pogoing in front of a self-driving car that they get what they deserve (this will doom the emergence AI self-driving cars, I assure you).

There is also some sense of false security by many of the auto makers and tech firms that having a human back-up driver during public roadway testing is a sure way of avoiding any adverse incidents. This is quite a myth or misunderstanding, and there is still a bona fide chance that even with a human back-up driver that things can go awry for an AI self-driving car.

See my article about egocentric designers for AI self-driving cars:

For my article about the dangers even with a human back-up driver, please see:

Another aspect of doing testing on public roadways is that it might be difficult to reproduce the instance of what happened. I mention this because trying to do Machine Learning (ML) via only one example of something is quite difficult to do. It would be handy to be able to undertake the situation a multitude of times in order to try and arrive at a “best” or at least better way to respond. I’ve stated in my industry speeches that we’re suffering from a kind of irreproducibility in the AI self-driving car realm and for which inhibits or staggers potential progress.

For more about irreproducibility, see my article:

For my overall framework about AI self-driving cars, see:

As perhaps is evident, doing testing on public roadways has some disadvantages.

That’s why it is vital to also do testing via the other means possible, including using simulations and using proving grounds.

For simulations, you can presumably run the AI through zillions of scenarios. There’s almost no limit to what you could try to test. The main constraint would be the computational cycles needed. Some auto makers and tech firms are even using supercomputers for their simulations, similar to how such high-powered computing is being used to gauge the impacts of climate change or other large-scale problems.

Not everyone though necessarily believes that the simulations are true to the real-world and thus the question is posed whether the AI reacting in a simulated environment is actually the same as it will react while on the roadways. If you are simulating climate change and your simulation is a bit off-base by estimates being made, this is likely Okay. But, if you are dealing with AI self-driving cars, which are multi-ton beasts that can produce instantaneous life-or-death consequences, a simulation that isn’t true to the real-world does not give one a full sense of confidence in the results.

In essence, if I told you that I had an AI self-driving car that has successfully passed a simulation of over one-hundred million miles of car driving, albeit only in a computer-based simulation, and never been on an actual road, would you be happy to see it now placed into public use, or unhappy, or disturbed, or what?

I think it’s fair to say that you’d be concerned.

There’s also the potential use of proving grounds.

See my article about proving grounds and self-driving cars:

This is usually private land or sometimes government land that is set aside for the purposes of testing AI self-driving cars.

You could say that in some ways it is better than simulations because it has a real-world aspect to it.

You could also say that this is safer than being on the public roadways since it is in an area that avoids potential harm to the general public.

I recently had a chance to closely explore a well-known proving ground, namely the American Center for Mobility (ACM) in Michigan, and spoke with the CEO and President, Michael Noblett, along with getting a specially guided tour of the facility by Angela Flood, Executive Director.

The ACM consists of over 500-acres, offering multiple test environments adjacent to the Willow Run Airport. There is about 2.5 miles of an extensive driving loop that contains high-speed usable highway roads and two tri-level overpasses. It is an impressive facility and available for commercial purposes, governmental purposes, and usable too by standards bodies and colleges.

For more info about the ACM, see:

Creating Ugly Zones In All Modes

Generally, it seems apparent that you’d want to use a combination of simulations, proving grounds, and public roadways for developing and testing of your AI self-driving car.

Each approach has its own merits, and each approach has its own drawbacks.

In combination, you can aim to get more kinds of testing that will hopefully lead to sounder AI self-driving cars.

Let’s now revisit the ugly zone.

For real-world driving of an AI self-driving car, as mentioned earlier, the AI might go for many miles without ever encountering some really difficult driving situations. Any such instances would presumably occur by happenstance, if at all. With a providing ground, you can possibly setup the AI for having to cope with quite ugly situations. Same goes for the use of simulations.

Regrettably, there are some auto makers and tech firms that are not pushing their AI to the limits via the use of the proving grounds and nor the simulations. They seem to believe that the focus should be the “normal” conditions of driving.

For example, at a proving ground, the AI self-driving car is driving on a road and all of a sudden a woman pushing a baby stroller carriage starts to walk across the street (this might be a stunt woman hired for this purpose, and the baby stroller is empty other than a fake doll). The AI self-driving car detects the motions and objects involved, i.e., the adult female and the stroller, and deftly swerves to avoid them. AI saves the day! Case closed, the AI is prepared for such a scenario.

This seems convincing as a test.

You might mark-off on your checklist and claim that the AI can detect a person with a baby stroller and take the right kind of action to avoid a calamity.

There are though additional considerations.

How many other cars were on the road with the AI self-driving car?

In this case, none.

Was there a car directly next to the AI self-driving car that would have been potentially in the way of the swerving action?

Not in this case.

Were there other pedestrians also trying to cross the street at the same time as the woman and the stroller?

No, just the woman and the stroller.

Were there any road signs warning about an upcoming hazard or perhaps any orange cones in the road due to roadway repairs being made? No.

And so on.

I think we would all feel a bit more confident in the testing of avoiding the woman with the baby stroller if we believed it was done in a more high-pressure situation.

Imagine if the AI self-driving car had other cars all around it, boxing it in, and meanwhile there were lots of other pedestrians near to or approaching the self-driving car, and the road itself was a mess, and a lot of things were happening all at once. That’s more telling about what the AI can cope with.

Having a simplified, stripped down situation with an otherwise barren road, and just the woman and the stroller, does not seem like much of a test per se.

It’s not anything close to being an ugly zone.

Don’t misunderstand my point. I’m fine with the stripped down test as one such test.

But, if that’s going to be the nature of the testing that’s taking place, it would seem like there’s no provision for the ugly zone.

Recall that I earlier mentioned that having a practice without any kind of ugly zone would seem to be a practice that has a substantial omission and we ought to question the validity of the practice overall.

For AI self-driving cars, we should definitely have ugly zone testing (or, if you prefer, we can say “practices” rather than “testing”).

Should you use only and always ugly zones?

Well, as I mentioned previously, I’m an advocate for a measured amount of practice time for sometimes having ugly zones and sometimes not.

My Goldilocks viewpoint is to have a combination of times with and without the ugly zones. But, however you allocate the time, there must be some amount of ugly zone practice.

Avoidance of using an ugly zone approach in undertaking practices for AI self-driving cars is a scary and understated form of practice and will pretty much “guarantee” the failure of AI self-driving cars in the real-world.

Per my framework, these are the key AI self-driving car driving tasks:

  • Sensor data collection and interpretation
  • Sensor fusion
  • Virtual world model updating
  • AI action planning
  • Cars control commands issuance

The ugly zone is a means to see how well each of those AI elements are able to perform. Furthermore, you want to see how well they each individually work as a semi-independent component, along with how they work in concert together to drive the self-driving car. Therefore, the ugly zone needs to have a varied and myriad of aspects that will put “pressure” on each of the components.

You might wonder how you can “pressure” an AI system, since it’s not like a human wherein you can pressure a human to get into a tizzy by throwing all sorts of things at them at once. Actually, in some ways, you can indeed pressure the AI system by doing likewise of what you’d do to a human, namely, pile-on as many things as you can, and see what the AI does. The internal timing of the AI system needs to be taxed to see that it can handle a multitude of simultaneous things happening on the roadway at the same time and in the same place.

For my article about the cognition timing of real-time AI systems, please see:


We believe in the ugly zone approach for AI self-driving cars.

Let’s create as tough an environment as feasible so that once the AI self-driving car is on the public roadways, it’s a piece of cake.

True stress testing should be done in all means feasible and not wait until the AI self-driving car is in a public place and for which public harm can occur.

Whether you want to put your own children into an ugly zone for their piano practices or for their art lessons, that’s up to you.

I think we can all agree that we’d believe more so in the potential of AI self-driving cars to be trustworthy on our streets if we knew that they had survived, learned from, and were adept at dealing with ugly zones.

Go, ugly zones, go.

Copyright 2019 Dr. Lance Eliot

This content is originally posted on AI Trends.

This UrIoTNews article is syndicated fromAITrends

About Post Author