My lightning got BRICKED with the software update. WILL NOT START.

[Jim Lewis]

However, due to the fact that FDRS now has a requirement where an NFTSA password is needed for non-dealer users to update the BCM (which is the module responsible for putting a vehicle in inhibit mode for an OTA and releasing it when's done), it seems like even with FDRS, non-dealers FDRS users are screwed if this happens to them. The sample size for this is very small......that I know of, only one owner (who is on this forum) had his truck bricked and was not able to get it out of inhibit mode with FDRS.

I might be that "one owner." An OTA update similarly bricked my truck, and I got the SYNC screen advisory to call Ford Roadside Assistance and have my truck towed to a dealer. But my truck was bricked in my home garage on a hill with a very tight >90° turn and a danger of rolling into a nearby neighbor's house. My dealer said it would take at least 24 to 48 hours to even look at the truck, and four other San Antonio dealers said it would take at least seven to ten days, during which time my truck might be sitting out in an asphalt parking lot that might reach 130° to 140° F in San Antonio summertime.

I think what's going on with the dealer wait times is nicely explained in the following post:

https://www.f150lightningforum.com/...riat-powertrain-service-now.20509/post-407539

My dealer's CSA basically told me the same thing: they had a long waiting list of people and only one EV tech who could not do mobile service calls because he was so busy. It wouldn't be right to bump me ahead of the waiting list.

So, I refused to have my truck potentially bake on a dealer's lot for an indefinite amount of time. I got a Mongoose Plus Ford2 cable and a Clore PL6100 power supply on Amazon (roughly a $1K purchase), but I hit the NFTSA password requirement for non-dealers that rugedraw mentioned.

With help from expert forum members, I located someone with the necessary credentials to use FDRS to unlock Inhibit Mode on my truck, and just like rugedraw said, it took ~5 minutes, and my truck never had to leave my garage. IIRC, the APIM module had to be reset, too, as although the truck would turn on and off normally after the Inhibit Mode reset, as judged by the dash screen, the SYNC screen was non-functional until that module was reset.

Too many unknown factors, I guess, but it seems like the BEV wizards at FMC (or the dealer's tech) should be able to unlock a truck over-the-air if a customer is willing to sign off on the responsibility for having the right equipment and the liability if anything goes wrong.

It's crazy to have to have your truck towed, sit for an indefinite time in the broiling sun of the American Southwest for something that only takes a few minutes to fix without moving the truck an inch. The other thing would be to have a more foolproof OTA update procedure that's capable of fully rolling back a failed update.

It's crazy to have such an inefficient rescue plan that wastes the customer's, the dealer's, and FMC's time and resources. No wonder Ford Motor Company has a big money-losing operation in electric vehicles. They're making that loss happen with their software and inefficient operational policies...

Sponsored

 

[rugedraw]

@Jim Lewis Yea, you're the owner I was referring to; just didn't want to put you out there.

 

[Newton]

This is unacceptable, anything automatically pushed must be able to be reverted. This could literally kill somebody in inclement weather. A young woman had a minor disabling crash in the rural area that I live during the winter, wandered off to get help and died of hypothermia. If this happens to me on the ferry I could trap 50 cars for an hour (I have seen it happen.)

How do you turn off automatic updates?

 

[FordEVr DAD]

I might be that "one owner." An OTA update similarly bricked my truck, and I got the SYNC screen advisory to call Ford Roadside Assistance and have my truck towed to a dealer. But my truck was bricked in my home garage on a hill with a very tight >90° turn and a danger of rolling into a nearby neighbor's house. My dealer said it would take at least 24 to 48 hours to even look at the truck, and four other San Antonio dealers said it would take at least seven to ten days, during which time my truck might be sitting out in an asphalt parking lot that might reach 130° to 140° F in San Antonio summertime.

I think what's going on with the dealer wait times is nicely explained in the following post:

https://www.f150lightningforum.com/...riat-powertrain-service-now.20509/post-407539

My dealer's CSA basically told me the same thing: they had a long waiting list of people and only one EV tech who could not do mobile service calls because he was so busy. It wouldn't be right to bump me ahead of the waiting list.

So, I refused to have my truck potentially bake on a dealer's lot for an indefinite amount of time. I got a Mongoose Plus Ford2 cable and a Clore PL6100 power supply on Amazon (roughly a $1K purchase), but I hit the NFTSA password requirement for non-dealers that rugedraw mentioned.

With help from expert forum members, I located someone with the necessary credentials to use FDRS to unlock Inhibit Mode on my truck, and just like rugedraw said, it took ~5 minutes, and my truck never had to leave my garage. IIRC, the APIM module had to be reset, too, as although the truck would turn on and off normally after the Inhibit Mode reset, as judged by the dash screen, the SYNC screen was non-functional until that module was reset.

Too many unknown factors, I guess, but it seems like the BEV wizards at FMC (or the dealer's tech) should be able to unlock a truck over-the-air if a customer is willing to sign off on the responsibility for having the right equipment and the liability if anything goes wrong.

It's crazy to have to have your truck towed, sit for an indefinite time in the broiling sun of the American Southwest for something that only takes a few minutes to fix without moving the truck an inch. The other thing would be to have a more foolproof OTA update procedure that's capable of fully rolling back a failed update.

It's crazy to have such an inefficient rescue plan that wastes the customer's, the dealer's, and FMC's time and resources. No wonder Ford Motor Company has a big money-losing operation in electric vehicles. They're making that loss happen with their software and inefficient operational policies...

***********************

Sorry to hear about your ordeal as well. This was really a shock for me especially since this was my first EV and everything was working well. I had to even say I was loving driving this truck every day and how quiet and efficient it was until the day it wound not turn on. I am glad you had yours fixed with the help of forum members, that is just awesome. Super glad I joined this. Glad you didn't have to have your new truck sit and bake in the scorching heat. I am going to have to save up for the Mongoose Plus Ford2 cable and a Clore PL6100 power since I recently had to buy the Ford Charge Station pro for $1,300 and installation for $2,300 with Qmerit. Cost that was never mentioned until I got the vehicle hahaha. I should have inquired and researched all this before I got it so that was on me.

 

[Jim Lewis]

With help from expert forum members

@rugedraw. Thanks, same here. I was vague on exactly how my Inhibit Mode problem was cured as I didn't want to put anyone else on the spot if they didn't want to be. I hope anyone who needs help like I did will contact you to see what else might be done. I presume anyone who uses FDRS or PTS on my vehicle can eventually see who did what when to my vehicle, as you could see with PTS my own failed FDRS efforts.

 

[Jim Lewis]

I am going to have to save up for the Mongoose Plus Ford2 cable and a Clore PL6100 power

Read up on the forum first on power supplies. There are less expensive power supplies than the Clore PL6100. It's really nice, though, if you don't have to break the piggy bank to buy it.

The other thing is that as long as Ford has the BCM lockout of independent FDRS users, you will be limited in what you can do. The general feeling is that Ford created this security requirement to thwart would-be hackers from stealing your vehicle by forcing a rigged BCM update. One would hope Ford could split off a security module for vehicle locks, etc., and still allow independent operators to access other main BCM functions.

For me, I'm happy to have the equipment, as if my truck manages to get bricked again somehow, perhaps I'll be able to get it restored without the "Get it towed and start waiting..." hassle. Being a geek, I was always interested in learning about FDRS and FORScan somewhere down the road. Thanks to Ford for giving me a reason to dive in that much sooner! When I described to my wife what Ford and its dealers were telling me, she didn't bat an eyelid about the price of the Mongoose cable and the PL6100! Good Luck in that department, too!

Despite all the ups and downs, including those with the Home Integration System and unreliable backup power transfer, I love my truck, too, and am really glad I got it. Would do it again without hesitation. Am 78 years old and figured there wasn't time to wait for the perfect EV with the long-range battery, etc. With luck, Ford will even sort all its truck software problems out while I'm still on the planet!

 

[Jim Lewis]

How do you turn off automatic updates?

The toggle is at the top of the Software Updates tab on the SYNC screen.

 

[TaxmanHog]

How do you turn off automatic updates?

Go to the updates screen, first switch.

 

[rugedraw]

@rugedraw. Thanks, same here. I was vague on exactly how my Inhibit Mode problem was cured as I didn't want to put anyone else on the spot if they didn't want to be. I hope anyone who needs help like I did will contact you to see what else might be done. I presume anyone who uses FDRS or PTS on my vehicle can eventually see who did what when to my vehicle, as you could see with PTS my own failed FDRS efforts.

Now that I know what the fix entails, I can do it, too, and I would totally do it for anyone that has FDRS if this were to happen for them.

 

[PrimeRisk]

This is unacceptable, anything automatically pushed must be able to be reverted. This could literally kill somebody in inclement weather. A young woman had a minor disabling crash in the rural area that I live during the winter, wandered off to get help and died of hypothermia. If this happens to me on the ferry I could trap 50 cars for an hour (I have seen it happen.)

How do you turn off automatic updates?

This is a joke that Ford hasn't thought this through. It is a very simple software configuration management process for updating remote devices. You have two booting partitions and the device boots to alternate partitions EVERY SINGLE TIME.

To update the software, you are booted on Partition A, a software package is downloaded, and installed on partition B. A flag is set that partition B is getting an update and if partition A gets booted again with this flag set, it phones home to say the update is FUBAR. Then someone can look to see what happened and decide to roll-back the update or use automation that if it boots to the partition with the flag set you automatically roll back partition B and phone home to say the update failed and we rolled back automatically.

Then it reboots, this time to Partition B. If everything comes up and looks ok (either Post Implementation Test scripts to automate or a phone home saying "I'm running on the update, check me out to make sure everything is cool"), then it copies the update to Partition A and you now have a good update on both partitions. This also prevents issues with partition corruption due to other issues. Sure, the device fails to boot correctly every other reboot, but easy to figure out and rewrite the corrupted partition.

SMH

RTUs and PLCs have been doing this since the late 80s. TiVo has been doing this on consumer DVRs since 1999. Why? Because having to send recovery technicians to the field is slow, expensive, and makes for very pissed off customers. Even worse if you have to pull the device to depot, or in this case dealership, to fix it.

Decades later, Ford looks like a monkey f*cking a football with software updates.

 

[Jim Lewis]

This is a joke that Ford hasn't thought this through. It is a very simple software configuration management process for updating remote devices. You have two booting partitions and the device boots to alternate partitions EVERY SINGLE TIME.

Microsoft Copilot's answer as to why Ford may not use dual boot partitions:

You
Why doesn't Ford use dual boot partitions in its software to make rolling back corrupt updates easier?
Copilot
Ford's current approach to software updates, particularly for their SYNC systems, doesn't typically involve dual boot partitions. Instead, they rely on a single partition system, which can sometimes lead to issues when updates fail or become corrupted [1] [2].
There are a few reasons why dual boot partitions might not be used:

1. Complexity and Cost: Implementing dual boot partitions would increase the complexity of the system. This could lead to higher development and maintenance costs, which might not be justified given the relatively low frequency of update failures.
2. Storage Constraints: Automotive systems often have limited storage capacity. Allocating space for a second partition could reduce the available space for other critical functions and data.
3. Existing Recovery Methods: Ford already has recovery methods in place, such as master resets and USB-based reinstallation processes [1] [2]. While these methods can be cumbersome, they are generally effective in restoring functionality.
4. User Experience: Dual boot systems can be confusing for the average user. Ford aims to provide a seamless and straightforward user experience, and adding the complexity of managing dual boot partitions might detract from this goal.

If you're experiencing issues with a SYNC update, there are community guides and forums that can help with downgrading or reinstalling the software [1] [2].
Do you think dual boot partitions would be a good solution despite these challenges?
Source: Conversation with Copilot, 8/30/2024

1. HOWTO Downgrade Sync3 v3.0 back to Ford Official v2.2 (build 17011)
2. Sync 3 stuck at "Updating system software" - CyanLabs Official Community
3. Sync 3 update problems - Ford Puma Club - Ford Owners Club

 

[PrimeRisk]

Microsoft Copilot's answer as to why Ford may not use dual boot partitions:

100% of those are absolute BS and nothing but excuses for shoddy development.

1) Fixing things in the field after you FUBAR a deployment is at least an order of magnitude more expensive than architecting resilient fault-tolerant systems. Why don't you ask a company that used CrowdStrike how much it cost them to fix their little deployment faux pas that impacted 8.5 million machines that had to be manually repaired.

2) Absolute BS. These things are rolling computers and storage is DIRT CHEAP. Commercial grade embedded SSDs are <$99/TB. I'd share the price per GB, but no one is even pricing at sub TB levels since it is so cheap.

3) Sure, sounds great. Pay to tow the immobilized vehicle to the closest dealership, provide a free rental to the consumer for a week or 3 because this is your f-up, and then pay the dealership tech to roll it back.

4) End user would never ever see this or know about it with automation. The only thing they might see is a screen that details that a software update was attempted, but didn't work, so contact Ford customer service with the update code.

Let's say that Ford pulls a CrowdStrike and bricks 100k+ Lightnings and Mach-Es, how many years do you think it would take them to recover them all? I'd love to see the backlash for these shoddy practices then.

Look, that answer you dug up seems to really be about just the Sync infotainment system. I don't care if Sync is bricked if I can still drive the vehicle, but we're talking about system updates that are actually bricking end-user vehicles in the field. Ford needs to rethink this and fast.

 

[Jim Lewis]

@PrimeRisk, the AI forces are aligned against you! <<<just kidding>>>

Here's what Google Gemini says about dual boot systems in automotive modules:
(Seems like some of the answers might be slight AI hallucinations, like #1, #3).

One thing to keep in mind about things like storage costs is that the development of the Lightning was probably initiated quite a few years before 1 TB NVMe drives were $99. And Ford probably knew it would be running a money-losing operation for a few years (and still is), so every penny it could save counted. I am certainly not backing the end result in terms of OTA update reliability!

Why doesn't Ford uses a dual boot partition in its automotive software to offer better protection against failed over-the-air updates?

Why Ford Might Not Use Dual Boot for Over-the-Air Updates
While dual booting is a common practice in personal computers for redundancy and safety, it might not be the optimal solution for automotive software due to several factors:

1. Real-time Requirements: Automotive software often has strict real-time requirements. Switching between partitions during a dual boot can introduce latency, which might be critical for safety-critical functions like braking or steering.
2. Complexity and Cost: Implementing a dual boot system in a car's software can increase complexity and development costs. It requires additional hardware, software, and testing to ensure seamless switching between partitions.
3. Security Risks: While dual booting can provide some protection against failed updates, it also introduces new security vulnerabilities. If an attacker can compromise one partition, they might be able to gain access to both.
4. Limited Space: Automotive software is typically running on embedded systems with limited storage space. Allocating space for two partitions could be inefficient and reduce available storage for other critical functions.
5. Alternative Solutions:Ford and other automakers might be using other methods to mitigate the risks of failed over-the-air updates, such as:
   • Rollback mechanisms: These allow the system to revert to a previous working state if an update fails.
   • Incremental updates: Smaller, more frequent updates can reduce the risk of a single update causing major problems.
   • Thorough testing and validation: Rigorous testing and validation processes can help identify and fix potential issues before updates are released.
   • Redundancy and fault tolerance: Designing systems with redundancy and fault tolerance can help ensure that the vehicle can continue to operate safely even if a component fails.

While dual booting might have some benefits, the potential drawbacks and the availability of other effective solutions make it less likely to be widely adopted in automotive software.

 

[PrimeRisk]

@PrimeRisk, the AI forces are aligned against you! <<<just kidding>>>

Here's what Google Gemini says about dual boot systems in automotive modules:
(Seems like some of the answers might be slight AI hallucinations, like #1, #3).

I've been an IT professional for 35 years and specialize in Deep Field technologies for energy where all of these constraints exist. Most of them are excuses and the rest are complete BS. The difference is, if we f-up, we could not just strand a single family in a life threatening situation, we could leave a million people with no heat in -25F weather (A real-life situation I experienced during 2021's Winter Storm Yuri in Minneapolis/St Paul). We literally would have loss of life, potentially 100s or even 1000s on top of billions in damage for frozen pipes and subsequent flooding.

Maybe stop listening to an AI that is grasping to provide a coherent answer that fits the leading question. Literally, AIs are giving you Confirmation Bias answers.

So, let's ask AI a different leading question:

You:
"Why would car manufacturers use a dual boot partition in its automotive software to offer better protection against failed over-the-air updates?"


Sent by Copilot:
Car manufacturers use a dual boot partition in automotive software to enhance protection against failed over-the-air (OTA) updates for several reasons:

1. Redundancy: A dual boot system typically has two separate partitions, often referred to as the “active” and “inactive” partitions. When an OTA update is received, it is first installed on the inactive partition. This ensures that the current, stable version of the software remains untouched and operational during the update process1.
2. Rollback Capability: If the update fails or the new software version is found to be faulty, the system can easily revert to the previous stable version stored on the active partition. This rollback capability minimizes the risk of the vehicle becoming inoperable due to a software issue1.
3. Continuous Operation: The dual boot system allows the vehicle to continue operating normally while the update is being downloaded and installed on the inactive partition. This means there is no downtime or interruption in the vehicle’s functionality during the update process1.
4. Safety and Reliability: In safety-critical environments like automotive systems, ensuring that updates do not compromise the vehicle’s operation is crucial. The dual boot system provides a safeguard against “bricking” the vehicle, which could occur if a single partition system fails during an update1.

Overall, the dual boot partition system provides a robust and reliable method for managing OTA updates, ensuring that vehicles remain safe and operational even if an update encounters issues.

When I ask the AI my leading question, I get answers that fit my professional experience and conclusion.

 

[Newton]

The fact that it is possible for Ford to fix it at a dealer means that it would be possible to provide a way for the owner to do it themselves - or some way for a tech to ‘phone into’ the truck to do the fix.

Back in the day I would literally dial into customer’s databases using our debugger and twiddle bits to get them going again. Usually at 1AM. It was rather fun when it was a large bank and the fix was to delete a few large financial transactions, I always wondered how that worked out…

Sponsored