Kuro5hin.org: technology and culture, from the trenches
create account | help/FAQ | contact | links | search | IRC | site news
[ Everything | Diaries | Technology | Science | Culture | Politics | Media | News | Internet | Op-Ed | Fiction | Meta | MLP ]
We need your support: buy an ad | premium membership

[P]
Cars, Value Engineering, and Bugs

By imon2nd in Technology
Fri Mar 05, 2010 at 02:18:46 PM EST
Tags: cars, automobiles, recalls, software, toyota, recall, gas pedal, value engineering, engineering, business (all tags)
Software

Two trends in automotive development, value engineering and drive-by-wire have the potential to reduce driver safety. Toyota's recent travails provide some food for thought.


The recent brouhaha over Toyota's 2 billion dollar, 8 million vehicle recall raises some difficult questions about the underlying causes for apparently obvious problems like interfering floor mats, sticky pedal assemblies and buggy software. For me, the most interesting issue is the relationship between the value engineering of mechanical systems and the reliability of control software.

I first learned about value engineering in the automotive industry from an electronics engineer employed by one of the Big Three in Detroit. We were chatting between sessions at a computer industry conference and I asked him what his job entailed. He said his main task every year was to reduce the cost of the car radio by 5 cents. It was not a simple matter of cost cutting. The finished product still had to meet its specifications, which included mean time to failure. According to him and other automotive engineers I've talked to since, the concept of setting performance specifications and then incrementally reducing the cost of each component until it just barely met the specs is standard practice for every part of all mass produced cars.

The benefits of value engineering are clear: cars cost less today than ever before. Based on U.S. Census Bureau data, in 2006 a family with the U.S. median annual income of $58,407 would spend 22% of that to buy a new, $13,000 Chevrolet Cobalt. The same median income family in 1960 would have had to spend 36% of their $5,620 annual income to buy a new Chevrolet Corvair for $2,000. Arguably, the Cobalt is a better vehicle than the Corvair by almost any metric one might choose. In fact, I can't think of a single feature of the Corvair that is superior to the Cobalt unless it's styling, and that's subjective.

However, there are at least two drawbacks to value engineering: the margin between operational and failed is reduced, and quality assurance requirements are increased. In the case of the Toyota gas pedal assembly, there are two parts that could easily be adversely affected by value engineering: the pedal return spring, and the pin hinge that the pedal arm rotates around. We can examine it on the CTS Specifications page, here...

http://www.ctscorp.com/automotive/datasheets/703.pdf

The spring appears to be a six-turn coil of steel wire. The return to idle position time is controlled by federal specifications, which you can read here...

http://www.nhtsa.dot.gov/staticfiles/DOT/NHTSA/Vehicle%20Safety/Test%20Procedure s/Associated%20Files/TP-124-06.pdf

The relevant part of the specification states that the pedal must return to its idle position in one second or less. No doubt, the pedals submitted for certification by CTS met that requirement. According to industry sources, CTS was under pressure from Toyota to value engineer the components of the pedal assembly such as the materials, spring and the hinge pin.

Of course, one way to save money on components is to look offshore for a supplier. The supplier of springs and/or pins might start out as American and end up as Chinese or Brazilian. Those suppliers would have to certify that the parts they propose meet the original materials and performance specifications. Unfortunately, as McDonald Douglas and Boeing found out the hard way, offshore suppliers are sometimes all too happy to fill out the compliance paperwork without much actual testing of the parts. The specification for the CTS pedal spring was for 3 million cycles. A cheap spring might not break after 100,000 cycles, but it could get very tired and slow.

The simple hinge pin at the heart of the CTS pedal assembly will probably never break. But what if the plastic around it isn't dimensionally stable? Toyota has admitted that the plastic guide fins and grooves on the pedal arm behind and above the hinge can develop an interference problem if the plastic has absorbed water. Does that indicate the specs for the plastic were weak or that the supplier didn't meet them? Whatever the reason, the increased friction affects the pedal return to idle position speed and adds stress to the spring, possibly shortening its useful life and making it more likely the pedal will stick.

At first glance, it would seem that these mechanical gremlins should have nothing to do with control software. The drive-by-wire system shouldn't be affected by a stuck pedal. The computer connected to it would consider the depressed pedal a valid go-fast command. But real-time control software is never that simple. The rate of change of the pedal position, both up and down was also factored in. The pedal position was sampled continuously and then processed to determine what the fuel system for the engine should do next.

Take samples, calculate a vector, look up a table, apply a control algorithm - it's pretty straightforward coding as software goes. But, there are pitfalls. For one thing, automotive systems that rely on a computer to manage the brakes and accelerator must meet a very high standard of reliability; ideally, a one in one trillion operations error rate. That is not an easy goal with software. The problem is verification. While there are strict, published test regimes defined for computer controlled weapons systems, digital media systems, telecom equipment and medical devices, the transportation industry keeps its methods a secret. The Society of Automotive Engineers has published a number of papers on this topic, but the extent of any manufacturer's implementation and their test results are not available.

My own experience with a subtle bug illustrates what the automotive software engineers are up against. In one of the digital audio effects programs I wrote, the left and right stereo channels would occasionally, less than once a day, swap sides. The bug was random and unprovoked.  Why would real-time high-speed software driven by slow-speed human input swap audio channels for no reason?  After a week of tedious debugging with a logic analyzer I had my answer: a software induced "race condition" in which two signals were so close in clock timing that one or the other might be in the lead at any given point, depending on random system noise or jitter.

The software I'd written used buffers to store sets of audio samples for processing left and right raw data. The digital signal processor could operate on both buffers in parallel, but it had to load its internal coefficient memories sequentially, not in parallel. The timing was so close that one in about 15 billion times the right channel audio sample buffer would be ready to feed the engine first, causing a swap. Though I'd studied the manual to learn the chip, I didn't realize I had to allow for that slight asymmetry.  

Perhaps Toyota's on-board computers have hardware idiosyncrasies, too. Equally likely, changes in mechanical parameters resulting from value engineering were never communicated to the software engineers. What if a gas pedal that operates slowly due to mechanical problems causes exposes a bug in the control algorithm, causing it to fail under certain conditions? Or, Toyota's software engineers may simply have overlooked a rare, random and difficult to test condition. It wouldn't surprise me a bit if that were the case, and it shouldn't surprise the National Highway Transportation Safety Administration either.

Soon, all of the controls in our cars will be inputs to computers, including the steering. To ensure that the ongoing value engineering process and inevitable programming errors don't reduce safety, regulatory agencies should change the rules for certification. Instead of relying entirely on testing cars as mechanical subsystems and finished objects, NHTSA should require manufacturers to comply with ISO 90003 procedures for verifying software quality in all real-time control systems. The car manufacturers' means and methods as well as test results would become visible to experts who could comment before a new system was released for production. Although no quality assurance method can guarantee failure-proof software and value engineering can go astray, there's a good chance fail-safe systems can be achieved.

Sponsors

Voxel dot net
o Managed Hosting
o VoxCAST Content Delivery
o Raw Infrastructure

Login

Related Links
o http://www .ctscorp.com/automotive/datasheets/703.pdf
o http://www .nhtsa.dot.gov/staticfiles/DOT/NHTSA/Vehicle%20Safety/Test%20Procedure s/Associated%20Files/TP-124-06.pdf
o Also by imon2nd


Display: Sort:
Cars, Value Engineering, and Bugs | 24 comments (18 topical, 6 editorial, 0 hidden)
This turd won't flush. Looks like a floater. (2.69 / 13) (#2)
by Chino Ginelli on Thu Mar 04, 2010 at 03:08:28 AM EST


_____
"Can you recommend a Chino vendor?" - MichaelCrawford, Nov 2008

"[President Obama] likes em a little loose but not saggy. He likes a wool and cashmere blend. He rolls with an inch and a quarter cuff." - tdillo, Nov 2008

"I think they look sharp. But it's important that they be pressed, or at least hung properly, so as to maintain the crease." - MichaelCrawford, Sep 2009

"I find a bit of well-cut tailoring quite hot." - TDS, Jun 2010

IAWTP (none / 0) (#19)
by BottleRocket on Fri Mar 05, 2010 at 02:18:41 PM EST

+1 FP then.

$ . . . . . $ . . . . . $ . . . . . $
. ₩ . . . . . . . . . . . . . . . . . . . .
. . . . * . . . . . * . . . . . * . . . . . * . . . . . *
$ . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . $
Yes I do download [child pornography], but I don't keep it any longer than I need to, so it can yield insight as to how to find more. --MDC
$ . . . . .
. . . . . . . . . . . . . . . . . . . . . . . . . $
. . . . * . . . . . * . . . . . * . . . . . * . . . . . *
. ₩ . . . . . . . . . . . . . . . . . . . .
$ . . . . . $ . . . . . $ . . . . . $
$B R Σ III$

[ Parent ]

You may as well get your $5 worth (none / 0) (#3)
by Del Griffith on Thu Mar 04, 2010 at 03:14:52 AM EST


-------
I...I like me. My wife likes me. My customers like me. Because I'm the real article. What you see is what you get. - Me


WHAT IS THIS FUCKING SHIT (3.00 / 4) (#4)
by schlouse on Thu Mar 04, 2010 at 04:19:03 AM EST

+1FP

Fraudulent Engineering, or, Counterfeit Bolts (2.00 / 7) (#5)
by Michael David Crawford on Thu Mar 04, 2010 at 08:15:38 AM EST

The Counterfeit Bolt problem has been around for many years. It's been a while since I've read any updates about it, so perhaps it is under control now, but I would be surprised if it really was.

Regular bolts made of ordinary steel have unmarked heads. High-strength bolts made of specially formulated and tempered steel have three raised lines on their heads. The only really easy way to tell the difference between a regular bolt and a high-strength bolt is to look for those raised lines.

Unfortunately, some unethical bolt manufacturers got the idea that they could earn some extra coin by marking regular, non-high-strength bolts with those raised lines. These bolts are quite a lot cheaper to manufacture than high strength bolts, but can be sold at a premium.

I first learned about the Counterfeit Bolt problem in a magazine article, which told of a construction worker who tried to torque such a counterfeit bolt while building what was to be a Saturn automobile plant. The bolt's head sheared off, the poor fucker lost his balance and fell to his death.

They also quoted a US Army general who said that when his tank battalions went out on maneuveur in the desert, the tanks would leave a trail of broken-off bolts wherever they went.

There is a way to test bolts to see whether they really are high strength, but at least at the time I read that magazine article, to perform the test cost hundreds of dollars per bolt tested. Basically one puts it in a machine that pulls and twists on the bolt with prodigious force, to determine just how much force is required to make it break.

The relevance to this here article is that when one manufactures a car, one really can't be sure that the parts suppliers have lived up to their contract specifications. Simply to assume that all the parts are within spec is likely to lead to deaths on the highways, because the suppliers are under tremendous pressure to cut costs, pressure which is readily relieved by cutting corners.



+1FP, as geeked about cars as (3.00 / 2) (#6)
by Enlarged to Show Texture on Thu Mar 04, 2010 at 09:33:15 AM EST

I am about planespotting


"Those people who think they know everything are a great annoyance to those of us who do." -- Isaac Asimov
NOW I MAKE FUN OF YOU. WELCOME TO K5. (3.00 / 2) (#7)
by GhostOfTiber on Thu Mar 04, 2010 at 11:22:36 AM EST

This is after the fact but I didn't expect you to include them in the article. Toyota stuff universally samples at eight times a second. There's exceptions, but for the most part it's eight times a second.

For the second part, Toyota stuff has always been sensor + "guess map" for redundency. The Toyota MAF sensor is the stuff of legend in terms of design problems. It's got a Fuel Air map it uses when it's busted (and on an MR2 you want to bust it so you can supply your own table) but in terms of actual fixability and durability, it's crap in both places. It work by shining a laser through the air stream off a mirror to determine the deviation. It's really insanely accurate and works great - until it gets filled with road grime. Fortunately for me and the MR2 crowd, the map toyota included in the ECU works just as well if not better when the sensor goes to pot.

That being said, it sounds to me like the map is wrong.

[Nimey's] wife's ass is my cocksheath. - undermyne

A Dr.Gilbert may have reproduced acceleration. (none / 0) (#8)
by Pentashagon on Thu Mar 04, 2010 at 01:48:18 PM EST

He testified to some house committee and some news station had something about him.  Basically, he shorted the two sensor wires to the pedal position sensor and was able to get some acceleration without triggering a trouble code.

I haven't seen any other verification of his experiment at this point.

IN FUTURE IM GOING TO RIDE THE BUS (2.33 / 3) (#9)
by I Did It All For The Horse Cock on Thu Mar 04, 2010 at 02:41:39 PM EST

CHOOOO CHOO!


\\\
  \ \        ^.^._______  This comment brought to you by the penis-nosed fox!
    \\______/_________|_)
    / /    \ \
    \_\_    \ \

Good luck on your submission (none / 1) (#10)
by nostalgiphile on Thu Mar 04, 2010 at 02:58:00 PM EST

but the editors of K5 only vote up stories submitted with embedded links like this.

Also, I feel sure I'm not the only one who found the paragraph

The software I'd written used buffers to store sets of audio samples for processing left and right raw data. The digital signal processor could operate on both buffers in parallel, but it had to load its internal coefficient memories sequentially, not in parallel. The timing was so close that one in about 15 billion times the right channel audio sample buffer would be ready to feed the engine first, causing a swap. Though I'd studied the manual to learn the chip, I didn't realize I had to allow for that slight asymmetry.  
disturbingly Crawfordesque. I believe a more metallic writing style would be appropriate here.

"Depending on your perspective you are an optimist or a pessimist[,] and a hopeless one too." --trhurler
omg (none / 0) (#20)
by my gold bling shines on Sat Mar 06, 2010 at 07:58:49 PM EST

http://www.allasiansexygirls.com/data/media/7/korean_race_queen_paddock_girls.jp g


Stop your blubbering prima donna - BottleRocket
[ Parent ]
I find that (none / 1) (#13)
by achievingfluidity on Thu Mar 04, 2010 at 07:21:45 PM EST

the links and article quite disturbing.

--
ANNOY A LIBERAL USE FACTS AND LOGIC


good show (none / 0) (#14)
by zombie lonelyhobo on Fri Mar 05, 2010 at 05:05:18 AM EST

you retitled for the third time and fooled most of the idiots here
NIGGA YOU DUMB
wtf (2.60 / 5) (#15)
by GrubbyBeardedHermit on Fri Mar 05, 2010 at 05:07:08 AM EST

haven't we binned this shit once before?

GBH

twice $ (none / 1) (#18)
by cockskin horsesuit on Fri Mar 05, 2010 at 01:46:39 PM EST

SHIT SHIT SHIT SHIT SHIT

[ Parent ]
great analysis (none / 1) (#21)
by nr1 on Sun Mar 07, 2010 at 06:33:45 AM EST

excellent article...this is what K5 used to be

Thanks (none / 1) (#22)
by imon2nd on Sun Mar 07, 2010 at 08:01:11 PM EST

Hello, nr1;

I appreciated your positive comment.

K5 seems to have a small war going on among some of its members. Maybe the site owner should create a K5 Wars section of the site for those inclined to do battle.

Best Regards,

Dave


[ Parent ]

K5's Small War (none / 1) (#23)
by Michael David Crawford on Mon Mar 08, 2010 at 02:08:35 AM EST

A friend of mine asked me how I could tolerate all the abuse and hatred that is flung around so incessantly here.

I replied that despite the apparent anger, that we are really a tight-knit community, with many members who have been around for years and years.

I said we were just like a family, and my friend understood completely.



[ Parent ]

Cars, Value Engineering, and Bugs | 24 comments (18 topical, 6 editorial, 0 hidden)
Display: Sort:

kuro5hin.org

[XML]
All trademarks and copyrights on this page are owned by their respective companies. The Rest 2000 - Present Kuro5hin.org Inc.
See our legalese page for copyright policies. Please also read our Privacy Policy.
Kuro5hin.org is powered by Free Software, including Apache, Perl, and Linux, The Scoop Engine that runs this site is freely available, under the terms of the GPL.
Need some help? Email help@kuro5hin.org.
My heart's the long stairs.

Powered by Scoop create account | help/FAQ | mission | links | search | IRC | YOU choose the stories!