How Tesla’s FSD Works - Part 2

By Karan Singh
Not a Tesla App

We previously dived into how FSD works based on Tesla’s patents back in November, and Tesla has recently filed for additional patents regarding its training of FSD.

This particular patent is titled “Predicting Three-Dimensional Features for Autonomous Driving” - and it’s all about using Tesla Vision to establish a ground truth - which enables the rest of FSD to make decisions and navigate through the environment. 

This patent essentially explains how FSD can generate a model of the environment around it and then analyze that information to create predictions.

Time Series

Creating a sequence of data over time - a Time Series - is the basis for how FSD understands the environment. Tesla Vision, in combination with the internal vehicle sensors (for speed, acceleration, position, etc.,) establishes data points over time. These data points come together to create the time series.

By analyzing that time series, the system establishes a “ground truth” - a highly accurate and precise representation of the road, its features, and what is around the vehicle. For example, FSD may observe a lane line from multiple angles and distances as the vehicle moves through time, allowing it to determine the line’s precise 3D shape in the world. This system helps FSD to maintain a coherent truth as it moves forward - and allows it to establish the location of things in space around it, even if they were initially hidden or unclear.

Author’s Note

Interestingly, Tesla’s patent actually mentions the use of sensors other than Tesla Vision. It goes on to mention radar, LiDAR, and ultrasonic sensors. While Tesla doesn’t use radar (despite HD radars being on the current Model S and Model X) or ultrasonic sensors anymore, it does use LiDAR for training.

However, this LIDAR use is for establishing accurate sensor data for FSD - for training purposes. No Tesla vehicle is actually shipped with any LiDAR sensors. You can read about Tesla’s use for its LIDAR training rigs here.

Associating the Ground Truth

Once the ground truth is established, it is linked to specific points in time within the time series - usually a single image or the amalgamation of a set of images. This association is critical - it allows the system to predict the complete 3D structure of the environment from just that single snapshot. In addition, they also serve as a learning tool to help FSD understand the environment around it.

Imagine FSD has figured out the exact curve of a lane line using data from the time series. Next, it connects this knowledge to the particular image in the sequence where the lane line was visible. Next, it applies what it has learned - the exact curve, and the image sequence and data - to predict the 3D shape of the line going forward - even if it may not know for sure what the line may look like in the future.

Author’s Note

This isn’t part of the patent, but when you combine that predictive knowledge with precise and effective map data, that means that FSD can better understand the lay of the road and plan its maneuvers ahead of time. We do know that FSD takes into account mapping information. However, live road information from the ground truth is taken as the priority - mapping is just context, after all.

That is why when roads are incorrectly mapped, such as the installation of a roundabout in a location where a 4-way stop previously existed, FSD is still capable of traversing the intersection.

Three Dimensional Features

Representing features that the system picks up in 3D is essential, too. This means that the lane lines, to continue our previous example, must move up and down, left and right, and through time. This 3D understanding is vital for accurate navigation and path planning, especially on roads with curves, hills, or any varying terrain.

Automated Training Data Generation

One of the major advantages of this entire 3D system is that it generates training data automatically. As the vehicle drives, it collects sensor data and creates time series associated with ground truths.

Tesla does exactly this when it uploads data from your vehicle and analyzes it with its supercomputers. The machine learning model uses all the information it gets to better improve its prediction capabilities. This is now becoming a more automated process, as Tesla is moving away from the need to manually label data and is instead automatically labeling data with AI.

Semantic Labelling

The patent also discusses the use of semantic labeling - a topic covered in our AI Labelling Patent. However, a quick nitty-gritty is that Tesla labels lane lines as “left lane” or “right lane,” depending on the 3D environment that is generated through the time series.

On top of that, vehicles and other objects can also be labelled, such as “merging” or “cutting in.” All of these automatically applied labels help FSD to prioritize how it will analyze information and what it expects the environment around it to do.

How and When Tesla Uploads Data

Tesla’s data upload isn’t just everything they may catch - even though they did draw an absolutely astounding 1.28 TB from the author’s Cybertruck once it received FSD V13.2.2. It is based on transmitting selective sensor information based on triggers. These triggers can include incorrect predictions, user interventions, or failures to correctly conduct path planning. 

Tesla can also request all data from certain vehicles based on the vehicle type and the location - hence the request for the absurd 1.28 TB coming from one of the first Canadian Cybertrucks. This allows Tesla to collect data from specific driving scenarios - which it needs to help build better models that are more adaptable to more circumstances while also keeping data collection focused, thereby making training more efficient.

How It Works

To wrap that all up, the model applies predictions to better navigate through the environment. It uses data collected through time and then encapsulated in a 3D environment around the vehicle. Using that 3D environment, Tesla’s FSD formulates predictions on what the environment ahead of it will look like.

This process provides a good portion of the context that is needed for FSD to actually make decisions. But there are quite a few more layers to the onion that is FSD.

Adding in Other Layers

The rest of the decision-making process lies in understanding moving and static objects on the road, as well as identifying and reducing risk to vulnerable road users. Tesla’s 3D mapping also identifies and predicts the pathing of other moving objects, which enables it to conduct its path planning. While this isn’t part of this particular patent per-say, it is still an essential element to the entire system.

If all that technical information is interesting to you, we recommend you check out the rest of our series on Tesla’s patents:

We’ll continue to dive deep into Tesla’s patents, as they provide a unique and interesting source to explain how FSD actually works behind the curtains. It’s an excellent chance to get a peek behind the silicon brains that make the decisions in your car, as well as a chance to see how Tesla’s engineers actually structure FSD.

Tesla Expected to Offer FSD Transfers in Europe

By Karan Singh
Not a Tesla App

It has been a long wait for FSD for European customers, many of whom paid for the feature years ago on now legacy hardware. While the FSD transfer program has come and gone multiple times, there’s something to be said about having it available in North America, where it can be used, and in Europe or other countries, where it still just remains the same as Enhanced Autopilot (differences between Autopilot, EAP, and FSD).

FSD Transfer is a nice goodwill gesture from Tesla that in theory doesn’t cost them anything. Instead, it keeps customers, especially those who have been waiting for years, loyal and happy. It also incentivizes them to upgrade to a newer Tesla with HW4, where FSD will hopefully be achieved.

In a reply to a post on X, Elon agreed with the suggestion that offering FSD transfers in Europe would be a fair solution for those who have already purchased FSD but can’t use its capabilities.

A Fair Solution

FSD Transfer directly addresses a growing concern for many long-term European Tesla owners. Thousands of customers purchased the full package, often many years ago, with the expectation that FSD would eventually be capable and approved for use. However, the reality is that FSD, even as an advanced driver assistance system (ADAS), continues to be pushed back in Europe.

As the regulatory process continues at a snail's pace, many of these early supporters are now reaching or have already passed the point where they’re ready to upgrade to a new Tesla.

Without the transfer program, it's a difficult choice: either throw away your original investment in FSD and pay for the package a second time (FSD price history), or subscribe to it in the future.

Offering FSD transfers is a good way for Tesla to meet them halfway. It's a difficult situation, and one that’s being hindered by processes beyond the control of both the customer and Tesla. However, a transfer helps both parties. Tesla sells another vehicle, and the customer gets to keep FSD.

When Will it Be Available?

Based on how FSD transfers have worked in the past following Elon’s announcements, this feature is likely to become available for a limited time period in the coming days or weeks. If it happens, we should expect an announcement from Tesla Europe on X and emails being sent out to Tesla customers.

Once the program is in place, all you need to do is complete your vehicle purchase and then inform your Tesla sales advisor that you’d like to transfer FSD. You don’t even have to sell or trade in your old Tesla; FSD will simply be removed from it as a feature.

Hopefully, Tesla enables FSD Transfers for everyone, regardless of region. It should be an ongoing offer until at least FSD is approved in the given country or region.

Tesla Increases Robotaxi Fare Fee, Up from $4.20

By Karan Singh
Not a Tesla App

The introductory price for Tesla’s Robotaxi Network has finally been updated. In a post on X, Elon Musk confirmed that the new fare would be rolling out to complement the new Robotaxi geofence expansion.

This change marks the first adjustment to Tesla’s fares since the initial $4.20 launch price 23 days ago. While the price increase may seem significant in terms of percentages, when compared to other options in the ride-hailing area, it is still drastically cheaper.

Context Matters

Robotaxi currently operates on a simple, flat-rate model. The new $6.90 fare gets you a ride to anywhere within the recently expanded geofence.

So far, this is the opposite approach compared to other services, such as Waymo or traditional ride-sharing options like Uber and Lyft. All these services use dynamic pricing based on distance, time of day, and demand. A comparable trip on any one of these services could cost anywhere from $30 to $65, and potentially even higher during peak hours.

That doesn’t even include the tip fees for human drivers either - another win for Robotaxi (can you tip a Robotaxi?).

Even with the adjustment, the flat $6.90 fare remains less than half the price of a typical competing ride, making Robotaxi the most affordable point-to-point transportation option in Austin, aside from mass transit, for now.

A “Maturing” Service

The price change, moving from one meme-worthy number to another, is a sign that Robotaxi is finally graduating from its initial pilot phase. Following the first major expansion of the service area, this adjustment is a logical next step towards finding a more sustainable flat price.

While the new fare is a 65% increase over the old fare, the key takeaway is that it is still far cheaper than other options, and still just as meme-worthy. Tesla is aiming to have its early access riders complete as many rides as possible during these early months, and this pricing is still very reflective of that.

Latest Tesla Update

Confirmed by Elon

Take a look at features that Elon Musk has said will be coming soon.

More Tesla News

Tesla Videos

Latest Tesla Update

Confirmed by Elon

Take a look at features that Elon Musk has said will be coming soon.

Subscribe

Subscribe to our weekly newsletter