Guy and Seth on SimulinkFebruary 18th, 2011If you can put a man on the moon, can you take a look at my car?Today I welcome my colleague Paul Barnard to guest post about the final report from the NHTSA-NASA Study of Unintended Acceleration in Toyota Vehicles.
The report focuses its investigation on two possible electronic failures. One was potential failures in the pedal position sensors and the other concerned a software malfunction in the main CPU. For the software malfunction, NESC looked for both coding defects and for algorithm flaws. This is where it gets interesting. They looked at more than 280,000 lines of code from a representative Camry engine control unit. In order to verify the logic and understand the functionality of the code, the team used Model-Based Design. They built a high-fidelity model, using MATLAB and Simulink, for the software and the hardware to explore system design and implementation, failure causes and effects. Simulation scenarios were run and model responses were then compared to the actual vehicle responses to verify that the models were accurate. The software models provided insight into how the ECM was supposed to work. In effect, they used MATLAB to discover how the system is supposed to work and how it can fail. This allowed them to look for system vulnerabilities. Beyond that, it was interesting to note that they found another benefit in using Model-Based Design. “The MBD approach also supported the dissemination of the software functions and behaviors to the team as a whole. Presentations of the software in this manner efficiently communicated the software within the MY 2005 Camry microcontrollers without exposing the native source code.” Improving team communication is often one of the reasons cited for adoption of Model-Based Design. Simulating the system models and using these models for design communication are two of the key values that Model-Based Design provides. In the end, no major flaws in the control unit were detected, but now, there is a lot more understanding. What do you think? How does your organization use Model-Based Design to improve communication and understanding? Leave a comment here with your ideas. 
25 Responses to “If you can put a man on the moon, can you take a look at my car?”Leave a Reply |
|
|||
|
These postings are the author's and don't necessarily represent the opinions of The MathWorks. |
By 
Seth, this is cool to know, but how much of NASA’s use of Matlab in the investigation was because thats how they do all of their investigation, and how much of it was due to the fact that Toyota develops the control software for their vehicles with Mathworks products in the first place?
Hi Matt, from what I read in the report, the selection of MATLAB for use in the investigation was driven by the need to understand the system, “at a level of abstraction suitable for control system insight and analysis”. (Appendix A, A.10.1 Modeling Effort Overview) I didn’t see anything in the report referring to Toyota’s use of MathWorks tools in their own development process. It seems like Toyota shared only source code and specs with the investigators, and then NASA built models from scratch to understand the system.
NASA’s review appeared to be pretty exhaustive, but should they have performed dynamic analysis on the code as well as static analysis?
The question of performing dynamic analysis is a good one. Whether or not they did dynamic analysis, I did see quite a lot of dynamic testing performed. NASA did dynamic testing with the models, running a total of 114,244 test cases. Source code was also incorporated into the models to increase model fidelity. Because the source code was brought into Simulink through S-functions, it allowed for reuse of the same test framework and exploration of different scenarios.
@Paul, NASA’s report is still sketchy. Do you aware if any Tech Reports being published in this area? It would be nice if TMW can produce a demo or seminar.
@Wei, Sorry, but I’m not aware of more detailed public reports. I’ll post if I find some follow-ups that go into more depth.
I was hoping to learn from this article what were the specific advantages to using simulink. One can build a model-based design testing system in any language or platform, so what are the advantages to using mathworks? It was stated in previous comments that Toyota did not design any of their control software using matlab, so why the fuss over simulink? The acceleration and breaking system must be implemented in micro controller technology. I’m pretty confident that Toyota has proprietary testing software for these devices. The fact that NASA chose to use matlab is not so surprising because it’s a great tool overall, but I didn’t learn anything about the specific details and advantages that were leveraged, which presumably I can also use in my own research. This article didn’t teach me anything. It’s just an advertisement that tags NASA and mathworks in the same webpage.
This statement of the NHTSA-NASA study illustrates the short comings of simulation testing.
I own a 2009 model and this acceleration has occurred one time on my vehicle!
I was approaching a stop signal, the car suddenly accelerated and buried the tachometer, I stood on the brake, and immediately killed the engine which resets the controller and the problem is solved. (for the time being)
The on going problem is that it is difficult to re-appear, one occurance in 2 years of driving makes the goverment and Toyota hard to convince that this is happening.
Based on this description of NASA’s activity, what they determined is that with a high degree of confidence that the design is sound given a wide variety of scenarios. However as a reimplementation in high level languages, their Matlab/Simulink re-implementation of the design would gloss over and not test any low level design errors, coding errors, typos, compiler bugs (though these are rare these days), or a whole separate universe that doesn’t seem to be recognized any more, analog effects such as noise, timing margin, temperature and voltage variations, and our new one, tin whiskers. Heck, what about radiation induced soft memory errors? Our society is too used to shrugging our shoulders and pusshing the “reset” button when something flakes-out. All of this renders their effort, however thorough and well thought out at the level it was performed, meaningless as an exhoneration of Toyota. It is quite an impressive statement that they have no high level design flaws, but it is completely insufficient toward the goal of determining if the controller was at fault, both on a software and a hardware level. I had the opposite experience with another make of car. Every few weeks it would NOT go for 4, 5, 6 seconds when the accelerator pedel was pushed. This too can result in some very dangerous situations. Needless to say, the computer reported no faults, must not have really happenned. Right? Wrong. I was driving, my wife was driving, and my son was driving. It was terrifying. Automotive engineering teams had better learn to make reliable and failsafe systems, and management and marketing had better recognize that it is not acceptable to compromise in this area. Otherwise we need to get used to shrugging our shoulders and saying “Oh well, just a spurious failure. We’ll have the funeral on Thursday.”
When the first “Fly by wire” airplane came out, Triple Mode redundancy was required in an effort to prevent transient life ending errors. For the human space program, Quad redundancy is usually required. For some reason, fly by wire cars have escaped this requirement. As soon as they disconnected the foot from the throttle, they were asking for trouble. Technology is great, but cannot be made 100% failure free.
mark
I reviewed the report and see no details of the PID or H bridge, other than brief summaries of what they do. Was this done? A ‘wound up’ integral would give unintended acceleration.
Paul,
Sorry, but the NASA report only displays the intentional subterfuge of the players. The analysis required to “exhonerate” a mfr of responsibility in a system like this one is exponentially untenable and every scientist there knows that. Doing 114,000 experiments on a system that has a virtually unlimited number of input combinations (not even including the highly probable unknown externally induced inputs) is plain technobabble, designed to provide pseudo-respectability to the defense’s case.
Given that systems of this complexity are inherently unmanagable from a reliability analysis standpoint, it is unreasonable to expect Toyota to to something no other man-made system can currently do. If there is anything they (and any other seller in the same position) are responsible for it’s telling the buyer clearly that their accelerator pedal is not connected to the throttle body directly, like they might assume it is, and therefore may occasionally exhibit unexpected behavior. Then the buyer can decide if they want to buy that car. Then Toyota can realize what Mark Hall adeptly pointed out above, that the way to solve the problem of customer skepticism (and federal regulation) is redundancy and long-term demonstration of reliability. AND THE MARKET WILL DECIDE IF THE PRODUCT SURVIVES.
But to publish a report that says “NASA said it wasn’t Toyota’s fault because their smart people did a bunch of tests and couldn’t find a failure mode like that” is literally a slap in the face to the many people who have experienced first hand the horrors of having their vehicle decide for itself to accelerate.
Good connection to MBD though – which was your original intention anyway :)
While the phrase “exonerate Toyota electronics” may be correct – please note it does not exonerate the design or execution of software. If there was a brake over-ride in the Toyota software (when the brake is pressed the accelerator is reduced – like many other manufacturers implemented) Toyota would not have been in this situation. That software design also would accomodate a confused driver that had his feet on both pedals and also provide a method to prevent catastrophic failure if a car mat was stuck in the accelerator.
I would guess that the drive-by-wire systems are probably more reliable than the old mechanical linkage to a carburetor system. I have been in older cars where the throttle stuck open. But people didn’t call it unintended acceleration back then, they just called it a stuck throttle. Probably because they could understand what was happening. This doesn’t exonerate Toyota, they still need to understand any problems in the field and try to make their systems as safe as they can. But the general public needs to be educated that no man-made system is ever going to be perfect. Even triple or quadruple redundancy can still fail, just look at the Challenger or the nuclear reactors in Japan.
I know my 2005 tundra has a run away condition with cruise control that happens on any hilly terrain. I have two of them and they both do it. I reported to toyota and they ignored the problem. You can find my complaint on the ntsb web site.
If the input potentiometer fails, I bet the car will run away full open throttle. Anybody brave enough to try it?
I don’t really trust the current NASA scientists…After all, they did not design the shuttle…it was designed in the 60′s, 70′s, and 80′s (The Enterprise was featured in my 6th grade weekly reader in ’68) …and those folks are retired and the kids these days just do not have the know how to do that sort of thing anymore. I wish I could get a kid out of school that knows anything at all….American Education really sucks today.
No program based analysis can do as much analysis as the human brain. The computer does not evaluate the assembler produced by the compiler. So there may be a problem there that no one looked at….software that rides on software that rides on more software.
I suggest you buy a car with a dead man switch on the dash.
Before the Toyota/Lexus incidents a co-worker had an “event” with a Mercedes Benz drive-by-wire car briefly going to WOT while backing out of a parking place! I am sure the problem is real, its not limited to just Toyota, and will never be “proven” by NASA or anyone else. That is just what happens when emission regulations and production cost pressures team-up to override basic safety issues. Maybe the car has a dozen air bags in it, but it one low-cost computer and one low-cost actuator with full authority over engine power!
My gut feeling was that the idea of having NASA inspect Toyota software for design flaws was a fabulous waste of taxpayer funds. Given the huge number of cars on the road, the numbers of drivers, operating conditions, etc, and the tiny number of problems reported it was a forgone conclusion nothing meaningful would be found.
Funny, I can remember installing double throttle return springs (one on the linkage, one on the carb.) to make sure the butterfly was pulled closed if one spring or the linkage broke!
I think it is very unfair and libelous to attack the authors of this report as an attempt to whitewash Toyota. They have used the best tools at their disposal to analyze and look for software issues. They found none.
As with the Audi issues of the 1980′s the problems appear to be
- elderly (or not so elderly) people hitting the accelerator pedal instead of the brake.
- a mechanical defect in the design of the accelerator pedal
- floor mats catching under the accelerator pedal.
@Edward Ryklin – I think the advantages of using Simulink for a Model-Based Design testing system are the ease of understanding the functional components and the ability to simulate the specification. I think obtaining a greater understanding of the operation is what the NASA engineers were looking for. Enhanced communication between team members is also a key benefit of using Simulink models as executable specifications.
@Dan Bintz – Simulation based testing relies on the creativity of the test designers to produce test cases that exercise the system. In complex systems, property proving and formal methods can find edge cases not covered by design test cases.
@Ray Clark – You make a good point… the report states that they found no design flaws. It is easy to prove a flaw when one exists, however, proving none exist is very difficult.
@Tom Bunker – I don’t remember seeing specific mention of implementations of an H bridge or PID.
As a safety engineerr NASA’s conclusions piqued my curiousity, so I searched for the NASA-NHTSA report,and while they used simulation software. My first discovery was that the core NASA team did not comprise anyone with expertise in Safety! (page 11), and yet this is clearly a catastrophic safety issue. another major disconnect that I observed on page 16 of the NASA report where it states: “Based on postulated failure modes and predicted system responses, numerous electrical *system hardware failure modes were tested on benchtop simulators”. Page 22 adds: “Test scenarios were developed based on analysis of software and hardware documentation” and later states: “Model responses were compared to the hardware external responses. Monitoring of actual responses inside the ECM hardware was not possible; however, the software model and ASIC block diagrams did give a level of insight into system function”.
In simple terms, this means that NASA took the existing Toyota documentation and assumed that the Toyota data was complete as well as accurate. NASA conducted their analyses and test scenarios (both hardware and software) at a functional block level, as can be seen by doing a deep dive into the NASA report. This approach is only as good as the data input into the software simulation as well as the test scenarios emulating the faults listed in the functional block analyses. This methodology relies on the premis that the existing data from Toyota completely describes all possible failure nodes. Software Engineers should recall the expression “garbage in, garbage out”. Note: a functional level failure mode analysis is only a preliminary top-down analysis, and should be followed by a more detailed bottoms-up component level analysis.
We have two Toyota cars – a 2009 Camry Hybrid & a 2007 RAV4. Both vehicles have experienced one each sudden acceleration event while coming to a stop. The Camry had over 14500 miles. The RAV4 over 50,000 miles. I don’t believe driver error was the cause, but I would say that – wouldn’t I!
Anyway, could the Simulink model by NASA be expanded to include motor-torque and tire-friction effects? Our Camry incident left distinct acceleration marks on the curb in from of our house; and brake skid marks as my wife worked to regain control of the vehicle. (Those skid marks show where the car left the sidewalk from our driveway entrance and stopped across the street between our neighbors car, truck and a rock wall.)
If the NASA model could also be used to replicate the sound from the electric motor/gas engine; and screeching tires, that would be great, too! We had a neighbor come running out from his home three houses away to see what the commotion was all about from our otherwise “quiet” hybrid.
The only difference between this incidence and my computer crashing, but working again after it gets rebooted – is nothing! It wasn’t driver error – it was driver skill that averted disaster in front of our home – and in the thirty some starts-and-stops that my wife makes in her 50-mile commute to her critical-care nurse job, passing through small towns and school crossings. If Simulink could be used to identify the real cause of these two events, that would be really a great thing to read about!
I find the concept of using proven tools (Matlab and Simulink) in the hands of knowledgeable and well-known users, an excellent way to try and prove or disprove a control system problem for the failure modes they think of.
A FACT that I think is ignored (probably because of the “magic” surrounding fly by wire systems) is that mechanically coupled accelerator linkages are also prone to failure. And has probably been so since before the Model T Ford. This has happened to me (and my wife) on 4 occasions (throttle stuck in 75% demand) before the root cause could be identified. I don’t think the failure mode we experienced, although very simple, could have been thought out by any one –not even a “Rocket Scientist”.
The main startpoint in model based design is the model. If your model does not contain the failure- mode/component/concept it wont simulate the effect of the failure.
If I recall correctly at the press release the NASA representative specifically said that this study does NOT exonerate the system design or implementation. He also stated something about not being able to prove a negative. I get the same sense from reading the NASA report, to paraphrase: “here’s where we looked for problems and how we evaluated those problems, we didn’t find a smoking gun”.
From page 17 of the NASA report: “Because proof that the ETCS-i caused the reported UAs was not found does not mean it could not occur. However, the testing and analysis described in this report did not find that TMC ETCS-i electronics are a likely cause of the large throttle openings as described in the VOQs.”
I have to weigh in with the analysis of the entire system, not just the code. I do look forward to fully automatic drive by wire (not even needing a driver) as the technology is already at hand as it is available for aircraft use today (but not yet accepted). Three micros talking together (voting circuits) would be a good start toward the goal, but no number of lines of code can substitute for a solid interface between the micro and the world.
My other thought: 280,000 lines of code? Is that excluding comments? Excluding lookup tables? Is it assembly language? The number of lines of code just seems to be abnormal for a simple engine controller. Betcha 5 bucks I could do it in fewer than 2,000 lines of Pascal or even C+ (the language of the computationally promiscuous)!
Al
Why was the report released? That was purely rhetorical.
Seeing the world through a keyhole is better than seeing the world through the eyes of computer mouse (bad analogy). An interesting collection of critical professional view “ports”. Maybe someone will open the door and we will have an expanded perspective. It all sounds familiar only the names have been changed.
Why?