Tesla FSD will be pure vision and not rely on radar use

Elon tweeted that v9 of the FSD beta would remove its reliance on radar completely and instead determine decisions based purely on vision. Humans don’t have radar after all, so it seems like a logical solution and tells us Tesla is feeling much more confident in their vision AI.

Radar and vision each have their advantages, but radar has thus far been much more reliable in detecting objects and determining speed. If you’ve ever noticed your Tesla being able to detect two vehicles in front of you when you can only see the one directly ahead of you, that’s radar at work.

In this situation the radio waves from the radar sensor are bouncing underneath the car in front of you and are able to continue traveling and detect that there is another object ahead even though it could never “see” it.

It really is one of those wow moments where you can feel the future and the ability for AI-powered cars to drive better than humans one day. It’s baby steps and slowly we’ll see more and more of these situations where the vehicle simply sees or does something we could never do.

There’s no doubting that more sensors could provide a more reliable and accurate interpretation of the real world as they each have their own advantages. In an ideal world a vehicle with radar, lidar, vision, ultrasonic sensors and even audio processing would provide the best solution. However, more sensors and systems come at a price, resulting in increased vehicle cost and system complexity.

After all humans are relatively safe drivers with two “cameras” and vision alone. If Tesla can completely solve vision, they’ll easily be able to achieve superhuman driving capabilities. Teslas have eight cameras, facing in all directions. They’re able to analyze all of them concurrently and make much more accurate interpretations then we ever could in the same amount of time.

Tristan on Twitter recently had some great insight into Tesla vision AI and how they’re going to replace radar. Here’s what Tristan had to say:

"We recently got some insight into how Tesla is going to replace radar in the recent firmware updates + some nifty ML model techniques

From the binaries we can see that they've added velocity and acceleration outputs. These predictions in addition to the existing xyz outputs give much of the same information that radar traditionally provides (distance + velocity + acceleration).

For autosteer on city streets, you need to know the velocity and acceleration of cars in all directions but radar is only pointing forward. If it's accurate enough to make a left turn, radar is probably unnecessary for the most part.

How can a neural network figure out velocity and acceleration from static images you ask?

They can't!

They've recently switched to something that appears to be styled on an Recurrent Neural Network.

Net structure is unknown (LSTM?) but they're providing the net with a queue of the 15 most recent hidden states. Seems quite a bit easier to train than normal RNNs which need to learn to encode historical data and can have issues like vanishing gradients for longer time windows.

The velocity and acceleration predictions is new, by giving the last 15 frames (~1s) of data I'd expect you can train a highly accurate net to predict velocity + acceleration based off of the learned time series.

They've already been using these queue based RNNs with the normal position nets for a few months presumably to improve stability of the predictions.

This matches with the recent public statements from Tesla about new models training on video instead of static images.

To evaluate the performance compared to radar, I bet Tesla has run some feature importance techniques on the models and radar importance has probably dropped quite a bit with the new nets. See tools like https://captum.ai for more info.

I still think that radar is going to stick around for quite a while for highway usage since the current camera performance in rain and snow isn't great.

NoA often disables in mild rain. City streets might behave better since the relative rain speed is lower.

One other nifty trick they've recently added is a task to rectify the images before feeding them into the neural nets.

This is a common in classical CV applications so surprised it only popped up in the last couple of months.

This makes a lot of sense since it means that the nets don't need to learn the lens distortion. It also likely makes it a lot easier for the nets to correlate objects across multiple cameras since the movement is now much more linear.

For more background on LSTMs (Long Short-Term Memory) see https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21

They're tricky to train because they need to encode history which is fed into future runs. The more times you pass the state, the more the earlier frames is diluted hence "vanishing gradients".

Tesla’s FSD beta v9 will be a big improvement forward from what FSD beta users have been using where the system was still relying on radar. And it’ll be an even bigger leap from what non-beta testers currently have access to. We can’t wait. Now where’s that button?

Almost ready with FSD Beta V9.0. Step change improvement is massive, especially for weird corner cases & bad weather. Pure vision, no radar.

— Elon Musk (@elonmusk) April 9, 2021

Follow

Follow

Follow

First Look at Tesla's V12 User Interface, Full Screen Visualizations and New Media Player [Updated: Photos and Video]

The_Alfather / Edited by Not a Tesla App

Tesla's latest V12 user interface will change the look and feel of some of the vehicle’s operations. This new interface, announced on X, is already partially available on the Cybertruck but will now be rolled out to the Model 3 and Model Y, equipped with AMD Ryzen processors according to Tesla. However, the new Model S and Model X will likely receive it as well, although possibly not at the same time. It integrates several new features and aesthetics that set new standards in vehicle interface design.

The new interface will become available in Tesla update 2024.14, which started rolling out to employees yesterday.

New Parked Vehicle Visualization

A standout feature of the v12 UI update, not to be confused with FSD v12, is the centralized vehicle visualization, which dominates the display while parked (video below). This design choice enhances visual appeal and improves functionality by placing critical vehicle performance metrics, and status updates front and center. Similar to the layout in the Cybertruck, this feature provides drivers with a clear and immediate view of their vehicle's status and shows off the gorgeous 3D model.

V12 Ansicht (Frühlingsupdate)
👉Kein Autopilot, EAP, FSD aktiv.
👉Bremslichter werden angezeigt.
👉Blinken wird angezeigt.
👉Geile Darstellung
❤️Danke liebes Tesla Team pic.twitter.com/lmzTjKIq1x

— EFIEBER (@EFIEBER_ANDRE) April 18, 2024

New Media Player

There’s a new media player that’s larger and easier to use. By increasing the size of the media player, Tesla is now able to fit additional options that were hidden before, such as EQ and audio settings, the search icon and shuffle and repeat options.

The new media player appears while the vehicle is parked, driving or while the visualizations are in full-screen mode.

The media player is available on the Model 3 and Model Y and according to Tesla it’ll be limited to vehicles with the Ryzen-based infotainment center.

Tesla adds a new media player in update 2024.14

Improved Navigation

The navigation system will see several improvements. You’ll now see a little trip progress bar that lets you visually see how far along you are on your route.

If your vehicle has a rear screen, as in the new Model 3, the redesigned Model S or Model X, then trip information such as ETA will also be displayed on the rear screen.

Tesla already has the ability to update your route if a faster route becomes available. You can change some of these settings under Controls > Navigate. However, now the vehicle will show you if a faster route becomes available and gives you a chance to cancel the updated route if needed.

Expanded Autopilot Visualizations

Tesla is now bringing its full-screen visualizations outside of North America. However, there will be some improvements as well. In addition to being able to have the visualizations go full screen, there will now be a small map displayed in the corner as well.

That’s one of the issues with the full-screen FSD visualizations right now. If you make them full-screen, you lose your navigation map completely and only have the next turn available.

This will be the first time full-screen visualizations are available outside North America. It’s not clear whether all the FSD visualizations will be available such as traffic lights, curbs and more will be displayed, but Tesla has slowly been adding additional visualizations for non-FSD users, so there’s a chance that this feature will finally display all FSD visualizations to users outside of North America.

It’s not immediately clear whether this feature will require Enhanced Autopilot (EAP) or FSD.

Update: The full-screen visualizations do not require FSD or EAP, but unfortunately the visualizations displayed are still the same ones as in previous updates, so it won’t display the surrounding environment and curbs.

Full-Screen Browser Support

With this update, Tesla will finally let you manually make the browser full-screen. While this will make Tesla’s Theater apps a little redundant since they just loaded the website of the selected streaming service, you’ll now be able to stream any video service full screen, as long as the service supports Tesla’s browser.

As expected, the full-screen button will only be available while the vehicle is parked.

Checking Compatibility

Owners can verify their vehicle’s compatibility with the new full-screen visualizations while parked and driving by navigating to Controls > Software > Additional Vehicle Info on their Tesla’s touchscreen. This update is tailored for Tesla vehicles equipped with the AMD Ryzen processor.

Follow

Follow

Follow

Tesla's Next FSD v12 Update Will Be Available For Users on 2024.8

Tesla will soon roll out a FSD v12 which is compatible with update 2024.8 Tesla began its public rollout of FSD v12 last month, which brings a much more refined, human-like experience to the self-driving software.

To celebrate the achievement and show owners how big of an improvement FSD v12 is, Tesla offered a free month of FSD to all owners with compatible vehicles in the U.S. and Canada.

However, there was a caveat. The latest FSD v12 (v12.3.4) is Tesla update 2024.3.15. Since Tesla doesn't roll back updates since the process isn't rigorously tested, that means many owners who had already received Tesla update 2024.8.9 were left without a free trial and access to FSD v12.

Subscribing to FSD on Update 2024.8

In fact, if you're in the U.S. or Canada and subscribe to FSD while on update 2024.8.x, you'll actually receive the older, and quite a bit worse FSD v11.4.9.

The FSD v12 update has been highly anticipated and aims to finally upgrade Tesla’s ADAS level 2 system to something higher. Musk has talked about level 4 or 5, and even recently said Tesla is going all-in on FSD.

FSD v12 for Update 2024.8 Is Coming

Rohan Patel, who recently left Tesla, recently said that Tesla is already working on an FSD v12 update that is compatible with users on update 2024.8.x. This means that the next FSD update may finally be based on the 2024.8 codebase.

However, just this morning Tesla released the next major update to the platform, update 2024.14 for employee testing. The update is massive and includes an updated UI, hands-free trunk opening, a full-screen browser and more.

However, Patel never specified which codebase the next FSD update would be based on. If it's based on update 2024.8, then users who are upgraded to 2024.14 in the next couple of weeks may face a similar fate as users on 2024.8 today.

Release Date

Patel said to expect the release of the next version of FSD v12 this week. Since update 2024.14 is still being tested with "wave 1" employees, we'll likely see one of two things happen. We may see the next FSD update be based on 2024.8 and released very soon, or it'll be based on update 2024.14, but it may not be available for another couple of weeks.

Once this update arrives, owners with a Tesla on 2024.8.x will finally be able to update and receive their one-month trial of FSD v12 and finally experience what everyone has been excited about.

Follow

Follow

Follow