Tesla is creating San Francisco in a simulation to help train Autopilot
Tesla
Tesla may be ramping up how it uses simulation to train its Autopilot system. A report by Electrek asserts that it has sources claiming that the company is concentrating on a reproduction of San Francisco. The article includes an image of the recreation and states that Tesla is working with Real Engine on its simulation.
According to Electrek, the image below is part of Tesla's simulation of San Francisco.
An image of Tesla's San Francisco simulation that was obtained by Electrek
Electrek
Tesla gave the world a look at how it uses simulation to advance the Autopilot program during the first AI Day in August of 2021 (recap).
AI Day
At the first AI Day Tesla talked about the use of using simulations to help train Autopilot. The video below is cued up to where they discuss a simulation.
Ashok Elluswamy, the Director of the Autopilot Program, showed a video that, at first glance, looked real other than an appearance by a Cybertruck. “I may say so myself. It looks very pretty,” said Elluswamy. He explained that the company is investing heavily in using simulation. “It helps when data is difficult to source. As large as our fleet is (FSD Beta users), it can still be hard to get some crazy scenes,” the director explained while showing a rendering of two people and a dog running in the middle of a busy highway. “This is a rare scene, but it can happen, and Autopilot still needs to handle it when it happens,” said Elluswamy.
It appears that Tesla has jumped on Fortnite’s Battle Bus by teaming up with Epic Games and its development platform — Unreal Engine. Fortnite is one of the most popular games of all time, with 80 million subscribers and 4 million daily users, and it was created with Unreal Engine. Epic flexed its creative muscles when it gathered experts to create The Matrix Awakens: An Unreal Engine 5 Experience. The goal was to “blur the boundaries between cinematic and game, inviting us to ask — what is real?” The project spotlight on Unreal Engine shows just how incredibly realistic a simulation can be.
After Elluswamy explained that the company is investing in simulation, it makes sense that Tesla would be hiring several positions with simulation in the job description. Electrek pointed out one posting for Autopilot Rendering Engineer. The posting states the successful candidate “will contribute to the development of Autopilot simulation by enabling and supporting the creation of photo-realistic 3D scenes that can accurately model the driving experience in a wide range of locales and conditions.” Tesla prefers the candidates have experience working with Unreal Engine.
While not new, this does show that Tesla is doubling down on efforts to improve Autopilot. It has recently rolled out Full Self Driving to 60,000 more users, bringing the FSD Beta program to 160,000 in North America.
We can only guess how many thousands of simulations the Autopilot team is conducting to add to the data the Beta testers are collecting. It seems unlikely that Tesla has only created the City by the Bay in its simulations. Perhaps Elluswamy will show more renderings at the second AI Day on September 30th.
Subscribe
Subscribe to our newsletter to stay up to date on the latest Tesla news, upcoming features and software updates.
Just over a week into the Robotaxi launch, Tesla began laying the groundwork for a more scalable remote supervision model, which will be key to achieving success with the Robotaxi Network.
About a week ago, Elon Musk posted on X that Tesla will likely reach the crucial safety threshold to enable this shift within a month or two. While that means at least another month of in-vehicle Safety Monitors, it does provide us with a timeline of what to expect.
As soon as we feel it is safe to do so.
Probably within a month or two. We continue to improve the Tesla AI with each mile driven.
This timeline came in response to a question about Tesla’s plans for the ratio of autonomous vehicles to remote supervisors. The more vehicles that a single human can supervise, the better, especially if that number can be reduced to something drastic, like a 100:1 ratio. A single human operator would be able to manage an entire city of Robotaxis, which will be critical to make the Robotaxi Network turn a profit.
While Tesla works towards that ambitious future, it is also taking immediate steps to improve the current user experience during the Austin pilot program, where 15-minute wait times have become the norm.
Solving for Wait Times
According to Eric E, one of Tesla’s principal engineers on Robotaxi, the current 15-minute wait times are a classic logistics challenge. The supply of vehicles is lower than the current demand for rides. To solve this, there’s a two-pronged solution for Tesla.
First, Tesla is directly increasing supply by hiring more Safety Monitors/Vehicle Operators in Austin, even hosting an on-site hiring event.
We're looking to hire more Vehicle Operators in Austin, TX to accelerate Robotaxi deployment. We will be hosting an onsite hiring event next Thursday. Please consider applying to the official job posting and completing this hiring event form:
Second, Tesla is working to make FSD and the Robotaxi fleet management software faster and smarter. This means they are utilizing the data from the pilot to better orchestrate the fleet by predicting demand and pre-positioning vehicles in prime locations to reduce wait times. After dropping someone off, the vehicle can start traveling to areas of higher demand, even if someone hasn’t booked a ride yet.
Next Up: Remote Supervision
These immediate fixes are all in service of that much larger goal. Scaling the Robotaxi Network isn’t just about having more cars; it’s about increasing the number of vehicles a single human can safely supervise remotely, which is a requirement for Robotaxi to turn a profit.
Elon’s comments give us this timeline. A more flexible and favorable ratio of 3:1 (although still far from the ideal 100:1) is likely to be achieved within a few months.
Tesla is committed to safety, as evidenced by the safety monitors in the vehicle. A single incident could not only tarnish the public’s view of the Robotaxi Network but could also halt Tesla’s operations altogether.
The data gathered from more Robotaxis on the road is crucial to the whole project. Tesla is gathering more data and issuing newer FSD builds specific to the Robotaxi.
As FSD requires less remote oversight per mile driven autonomously, Tesla can safely increase the number of vehicles per remote supervisor, moving the service closer to its ultimate goal.
Tesla has laid out an aggressive roadmap for the Robotaxi Network and its next few phases. We’ll have to wait and see just how this goes over the next few months, and whether they feel comfortable enough to increase the geo-fence and remove safety monitors.
Following the recent news of Grok being almost ready for Tesla vehicles, Elon Musk confirmed on X that the next major step is with Optimus, Tesla’s humanoid Robot. xAI’s advanced Grok models will eventually serve as the voice and brain for Optimus. This will be a convergence of Musk’s two biggest AI ventures — Tesla and xAI.
This will combine a physically humanoid robot - the brawn - with the new brains, Grok. This integration is more than just giving Optimus a voice - it suggests that Tesla is thinking ahead and possibly intends to use Grok to understand the environment around Optimus, while FSD will handle the robot’s movements.
The combination of Optimus and Grok creates a relationship where each component plays to its strengths.
For years, Tesla’s robotics team has been focused on the immense challenge of physical autonomy. Optimus learns complex tasks by observing humans, basically training itself through video by watching humans. This helps Optimus develop the physical dexterity needed to work in the real world. This is the brawn - the ability to navigate, manipulate objects, and perform useful work.
Grok provides the conversational brain. It adds a layer of natural language understanding, reasoning, and interaction. Instead of needing a computer, a specialized app, or pre-programming commands to give Optimus instructions, a user will be able to simply talk to it in a natural way. This makes Optimus infinitely more approachable and useful, especially for tasks in a dynamic environment, such as work or at home.
xAI and Tesla
Viewed from a different perspective, this move isn’t just about upgrading one product. It is the clearest evidence that xAI and Tesla are collaborating together to build a single, unified AI platform. Musk’s biographer, Walter Isaacson, believes Tesla and xAI will merge. Seeing Tesla and xAI both play critical roles in creating Optimus makes us believe that it may very well be the case.
Transformation to a Humanoid Robot
The confirmation of Grok in Optimus is one of the most significant milestones for the project to date. While Optimus’s ability to walk and work (and dance) is already an incredible engineering feat, it has all been physical abilities so far. Adding the ability to interact with Optimus in a human-like way will transform Grok from a machine to a true, general-purpose humanoid robot.
The ability to understand nuanced requests, ask clarifying questions, and respond intelligently is what will ultimately make Optimus a daily fixture in our lives.
![Tesla Updates Robotaxi App: Adds Adjustable Pick Up Locations, Shows Wait Time and More [VIDEO]](https://www.notateslaapp.com/img/containers/article_images/tesla-app/robotaxi-app/25-7-0/robotaxi-app-25.7.0.webp/4ac9ed40be870cfcf6e851fce21c43b9/robotaxi-app-25.7.0.jpg)
