Boeing 737 MAX Discussion

[Fat_ninja]

Question I have is how long will we be running at 42 build rates

[moyski]

Originally Posted by Cryten

You are mistaken if you think you ever have control of the Airbus. You get different control laws which give you differing levels of control, but be under no illusion, all inputs still go through the computer.

The 737 however is directly controlled by the pilots, it even has a cable back up system for when everything goes wrong. All control inputs affect the aircraft in a predictable and linear manner. No one had to design the 737 to do beat ups, but because its controlled by the pilots its a simple manoeuvre.

These problems always occur when someone puts a computer between the pilot and the controls. Boeing always gave the pilot the final authority, the question now is, has this changed? They seem to be going down the Airbus route and using a computer to programme out design flaws.

Sometimes we should just know when to stop commenting on topics we know nothing about or have no experience of. Not saying you should stop, because you clearly know everything there is to know about planes and probably have decades of experience flying different aircraft.

[RW16610]

Not sure if I missed it earlier in the thread but has anyone seen this on Bloomberg? The 737 Max software was apparently outsourced to $9-an-Hour Engineers:

Boeing’s 737 Max Software Outsourced to $9-an-Hour Engineers

It remains the mystery at the heart of Boeing Co.’s 737 Max crisis: how a company renowned for meticulous design made seemingly basic software mistakes leading to a pair of deadly crashes. Longtime Boeing engineers say the effort was complicated by a push to outsource work to lower-paid contractors.

The Max software -- plagued by issues that could keep the planes grounded months longer after U.S. regulators this week revealed a new flaw -- was developed at a time Boeing was laying off experienced engineers and pressing suppliers to cut costs.

Increasingly, the iconic American planemaker and its subcontractors have relied on temporary workers making as little as $9 an hour to develop and test software, often from countries lacking a deep background in aerospace -- notably India.


In offices across from Seattle’s Boeing Field, recent college graduates employed by the Indian software developer HCL Technologies Ltd. occupied several rows of desks, said Mark Rabin, a former Boeing software engineer who worked in a flight-test group that supported the Max.

The coders from HCL were typically designing to specifications set by Boeing. Still, “it was controversial because it was far less efficient than Boeing engineers just writing the code,” Rabin said. Frequently, he recalled, “it took many rounds going back and forth because the code was not done correctly.”

Boeing’s cultivation of Indian companies appeared to pay other dividends. In recent years, it has won several orders for Indian military and commercial aircraft, such as a $22 billion one in January 2017 to supply SpiceJet Ltd. That order included 100 737-Max 8 jets and represented Boeing’s largest order ever from an Indian airline, a coup in a country dominated by Airbus.

Based on resumes posted on social media, HCL engineers helped develop and test the Max’s flight-display software, while employees from another Indian company, Cyient Ltd., handled software for flight-test equipment.

Costly Delay

In one post, an HCL employee summarized his duties with a reference to the now-infamous model, which started flight tests in January 2016: “Provided quick workaround to resolve production issue which resulted in not delaying flight test of 737-Max (delay in each flight test will cost very big amount for Boeing).”

Boeing said the company did not rely on engineers from HCL and Cyient for the Maneuvering Characteristics Augmentation System, which has been linked to the Lion Air crash last October and the Ethiopian Airlines disaster in March. The Chicago-based planemaker also said it didn’t rely on either firm for another software issue disclosed after the crashes: a cockpit warning light that wasn’t working for most buyers.

“Boeing has many decades of experience working with supplier/partners around the world,” a company spokesman said. “Our primary focus is on always ensuring that our products and services are safe, of the highest quality and comply with all applicable regulations.”

In a statement, HCL said it “has a strong and long-standing business relationship with The Boeing Company, and we take pride in the work we do for all our customers. However, HCL does not comment on specific work we do for our customers. HCL is not associated with any ongoing issues with 737 Max.”

Recent simulator tests by the Federal Aviation Administration suggest the software issues on Boeing’s best-selling model run deeper. The company’s shares fell this week after the regulator found a further problem with a computer chip that experienced a lag in emergency response when it was overwhelmed with data.

Engineers who worked on the Max, which Boeing began developing eight years ago to match a rival Airbus SE plane, have complained of pressure from managers to limit changes that might introduce extra time or cost.

“Boeing was doing all kinds of things, everything you can imagine, to reduce cost, including moving work from Puget Sound, because we’d become very expensive here,” said Rick Ludtke, a former Boeing flight controls engineer laid off in 2017. “All that’s very understandable if you think of it from a business perspective. Slowly over time it appears that’s eroded the ability for Puget Sound designers to design.”

Rabin, the former software engineer, recalled one manager saying at an all-hands meeting that Boeing didn’t need senior engineers because its products were mature. “I was shocked that in a room full of a couple hundred mostly senior engineers we were being told that we weren’t needed,” said Rabin, who was laid off in 2015.

The typical jetliner has millions of parts -- and millions of lines of code -- and Boeing has long turned over large portions of the work to suppliers who follow its detailed design blueprints.

Starting with the 787 Dreamliner, launched in 2004, it sought to increase profits by instead providing high-level specifications and then asking suppliers to design more parts themselves. The thinking was “they’re the experts, you see, and they will take care of all of this stuff for us,” said Frank McCormick, a former Boeing flight-controls software engineer who later worked as a consultant to regulators and manufacturers. “This was just nonsense.”

Sales are another reason to send the work overseas. In exchange for an $11 billion order in 2005 from Air India, Boeing promised to invest $1.7 billion in Indian companies. That was a boon for HCL and other software developers from India, such as Cyient, whose engineers were widely used in computer-services industries but not yet prominent in aerospace.

Rockwell Collins, which makes cockpit electronics, had been among the first aerospace companies to source significant work in India in 2000, when HCL began testing software there for the Cedar Rapids, Iowa-based company. By 2010, HCL employed more than 400 people at design, development and verification centers for Rockwell Collins in Chennai and Bangalore.

That same year, Boeing opened what it called a “center of excellence” with HCL in Chennai, saying the companies would partner “to create software critical for flight test.” In 2011, Boeing named Cyient, then known as Infotech, to a list of its “suppliers of the year” for design, stress analysis and software engineering on the 787 and the 747-8 at another center in Hyderabad.

The Boeing rival also relies in part on offshore engineers. In addition to supporting sales, the planemakers say global design teams add efficiency as they work around the clock. But outsourcing has long been a sore point for some Boeing engineers, who, in addition to fearing job losses say it has led to communications issues and mistakes.

Moscow Mistakes

Boeing has also expanded a design center in Moscow. At a meeting with a chief 787 engineer in 2008, one staffer complained about sending drawings back to a team in Russia 18 times before they understood that the smoke detectors needed to be connected to the electrical system, said Cynthia Cole, a former Boeing engineer who headed the engineers’ union from 2006 to 2010.

“Engineering started becoming a commodity,” said Vance Hilderman, who co-founded a company called TekSci that supplied aerospace contract engineers and began losing work to overseas competitors in the early 2000s.

U.S.-based avionics companies in particular moved aggressively, shifting more than 30% of their software engineering offshore versus 10% for European-based firms in recent years, said Hilderman, an avionics safety consultant with three decades of experience whose recent clients include most of the major Boeing suppliers.

With a strong dollar, a big part of the attraction was price. Engineers in India made around $5 an hour; it’s now $9 or $10, compared with $35 to $40 for those in the U.S. on an H1B visa, he said. But he’d tell clients the cheaper hourly wage equated to more like $80 because of the need for supervision, and he said his firm won back some business to fix mistakes.

HCL, once known as Hindustan Computers, was founded in 1976 by billionaire Shiv Nadar and now has more than $8.6 billion in annual sales. With 18,000 employees in the U.S. and 15,000 in Europe, HCL is a global company and has deep expertise in computing, said Sukamal Banerjee, a vice president. It has won business from Boeing on that basis, not on price, he said: “We came from a strong R&D background.”

Still, for the 787, HCL gave Boeing a remarkable price – free, according to Sam Swaro, an associate vice president who pitched HCL’s services at a San Diego conference sponsored by Avionics International magazine in June. He said the company took no up-front payments on the 787 and only started collecting payments based on sales years later, an “innovative business model” he offered to extend to others in the industry.

The 787 entered service three years late and billions of dollars over budget in 2011, in part because of confusion introduced by the outsourcing strategy. Under Dennis Muilenburg, a longtime Boeing engineer who became chief executive in 2015, the company has said that it planned to bring more work back in-house for its newest planes.

Engineer Backwater


The Max became Boeing’s top seller soon after it was offered in 2011. But for ambitious engineers, it was something of a “backwater,” said Peter Lemme, who designed the 767’s automated flight controls and is now a consultant. The Max was an update of a 50-year-old design, and the changes needed to be limited enough that Boeing could produce the new planes like cookie cutters, with few changes for either the assembly line or airlines. “As an engineer that’s not the greatest job,” he said.

Rockwell Collins, now a unit of United Technologies Corp., won the Max contract for cockpit displays, and it has relied in part on HCL engineers in India, Iowa and the Seattle area. A United Technologies spokeswoman didn’t respond to a request for comment.

Contract engineers from Cyient helped test flight test equipment. Charles LoveJoy, a former flight-test instrumentation design engineer at the company, said engineers in the U.S. would review drawings done overnight in India every morning at 7:30 a.m. “We did have our challenges with the India team,” he said. “They met the requirements, per se, but you could do it better.”

Multiple investigations – including a Justice Department criminal probe – are trying to unravel how and when critical decisions were made about the Max’s software. During the crashes of Lion Air and Ethiopian Airlines planes that killed 346 people, investigators suspect, the MCAS system pushed the planes into uncontrollable dives because of bad data from a single sensor.

That design violated basic principles of redundancy for generations of Boeing engineers, and the company apparently never tested to see how the software would respond, Lemme said. “It was a stunning fail,” he said. “A lot of people should have thought of this problem – not one person – and asked about it.”

Boeing also has disclosed that it learned soon after Max deliveries began in 2017 that a warning light that might have alerted crews to the issue with the sensor wasn’t installed correctly in the flight-display software. A Boeing statement in May, explaining why the company didn’t inform regulators at the time, said engineers had determined it wasn’t a safety issue.

“Senior company leadership,” the statement added, “was not involved in the review.”

https://www.bloomberg.com/news/artic...hour-engineers

[M45]

Originally Posted by RW16610

Not sure if I missed it earlier in the thread but has anyone seen this on Bloomberg? The 737 Max software was apparently outsourced to $9-an-Hour Engineers:





https://www.bloomberg.com/news/artic...hour-engineers

Corporations: we save billions, you and your family might die.

[RW16610]

Business process outsourcing simply shouldn't be applied to certain things. This would be one of those things, especially at the rate these "Engineers" were paid. Unbelievable...

[John45]

Showing my age here, I've been wondering if the Max problem is in any way related to the early 707 stab trim problems. I seem to remember that AA was doing barber pole descents early on in the 707 but stopped after an accident. My memory is a bit foggy as I checked out in the 707 in 1968.....

There's a discussion online relating to this but unfortunately I can't post link.

[TheVTCGuy]

$9 per hour Engineering? Kind of reminds me of a story we used to jab around in the squadron, the talk was if you ever had to reach for the D-ring (Ejection pull-handle on the Martin-Baker seat) the thought will be going through your head that the lowest-bid won the contract to manufacture this seat.

[RW16610]

Originally Posted by TheVTCGuy

$9 per hour Engineering? Kind of reminds me of a story we used to jab around in the squadron, the talk was if you ever had to reach for the D-ring (Ejection pull-handle on the Martin-Baker seat) the thought will be going through your head that the lowest-bid won the contract to manufacture this seat.

Oh man, I can't even begin to imagine! I plan on going up in a Cessna C172 again soon and this will now be a thought in my mind. Albeit on a different level it's a bit similar, nonetheless.

I'm actually flabbergasted by this still. Boeing is / was my favourite aerospace company. Case studies about Alan Mulally and a relatively good track record before this, etc.

This for sure is an industry where corners can't be cut.

[Hawksfan]

The lesson that we should be earning from all this is that 2 well trained pilots are the ultimate safety device. I have heard engineers explain to me how this and that "can't happen" in an airplane and have then seen it happen. I guarantee every airplane in existence has software/hardware glitches that are undiscovered. When the MAX comes back it will be the safest airplane in the sky.

As an aside...outsourcing is a major problem in the airplane industry. The 787 had many manufacturing problems because Boeing did so much outsourcing. A call center is one thing...engineering and manufacturing is another.

[Abdullah71601]

Originally Posted by RW16610

Business process outsourcing simply shouldn't be applied to certain things. This would be one of those things, especially at the rate these "Engineers" were paid. Unbelievable...

Originally Posted by TheVTCGuy

$9 per hour Engineering? Kind of reminds me of a story we used to jab around in the squadron, the talk was if you ever had to reach for the D-ring (Ejection pull-handle on the Martin-Baker seat) the thought will be going through your head that the lowest-bid won the contract to manufacture this seat.

India produces excellent engineers. These engineers were paid $9 an hour because they were in India, not in the US. Cost of living there is much lower and wage scales reflect that. That would be about a middle class wage in India.

Lower local pay doesn't mean they weren't qualified engineers. The reporting is one sided, but it doesn't sound like Boeing managed the project efficiently.

[Jojoshop]

They outsourced their software to low paying programmers, $9 per hour. What other shortcuts did they take to reduce cost. They didn’t disclose the issue to their customers and didn’t make it part of training.
My non technical comment, they got a governance and leadership issue that needs to be fixed ASAP.

[bayerische]

Originally Posted by Cryten

These problems always occur when someone puts a computer between the pilot and the controls. Boeing always gave the pilot the final authority, the question now is, has this changed? They seem to be going down the Airbus route and using a computer to programme out design flaws.

Planes never crashed before computers, those where the days. We are so silly to have put computers in planes.

[oldman2005]

Pay peanut get monkey - You get what you pay for!

[oldman2005]

Originally Posted by Abdullah71601

India produces excellent engineers. These engineers were paid $9 an hour because they were in India, not in the US. Cost of living there is much lower and wage scales reflect that. That would be about a middle class wage in India.

Lower local pay doesn't mean they weren't qualified engineers. The reporting is one sided, but it doesn't sound like Boeing managed the project efficiently.

Obviously you're not from the Silicon Valley, you get what paid for. Why not buy a Rolex made in China?

With a background in engineering, I was stunned that how could Boeing missed such a basic simple test. Now that explained it.

[Cryten]

Originally Posted by moyski

Sometimes we should just know when to stop commenting on topics we know nothing about or have no experience of. Not saying you should stop, because you clearly know everything there is to know about planes and probably have decades of experience flying different aircraft.

I feel the burn. Someone's feelings are hurt. Sorry, it's just an airplane.

Airbus make aeroplanes loved by pilots the world over.

There. I fixed it.

[Cryten]

Originally Posted by bayerische

Planes never crashed before computers, those where the days. We are so silly to have put computers in planes.

In a way you're right in what you say, safety has been improved with the implementation of computers, but new problems have also arisen. Computers are only as good as the programme you run (and therefore person programming it) and info you give it.

Check out the Mt Erebus crash. The pilots flew a perfectly serviceable aircraft into a mountain because the navigation computer was programmed incorrectly. It didn't look right to the pilots, but the computer must be right, surely? Blind faith in a computer can be as dangerous as a rouge pilot.

[The Libertine]

Originally Posted by oldman2005

Pay peanut get monkey - You get what you pay for!

“pay peanut get monkey” - made me laugh, but it is very true.

[Abdullah71601]

Originally Posted by oldman2005

Obviously you're not from the Silicon Valley, you get what paid for. Why not buy a Rolex made in China?

With a background in engineering, I was stunned that how could Boeing missed such a basic simple test. Now that explained it.

Strawman argument.

Boeing outsourced the programming to a firm they believed competent for the work. The price was right, and it was right because programmers and engineers in India aren't afforded the lavish salaries and benefits one finds in California. The salary has no bearing on the qualification, this is a red herring.

If the firm (and engineers) are determined to be qualified to write the code during the contracting process, the burden falls on Boeing to ensure the end product meets their contract specs. It doesn't look like Boeing effectively managed the contract work, which is a significant management issue. The engineers could have been billing at $100 an hour and Boeing would have hosed this up.

[Andad]

I agree with Abdullah.

The firm contracted to supply would have had their own internal test engineers sign off on the product.

Boeing would not necessarily manage the contracted work but would also have had their own internal test engineers sign off on the final product.

Under set guidelines the product may have performed to spec.

In actual use it failed.

This is not peanuts and monkeys.

This is serious computer engineering and mechanical interfacing.

[TheVTCGuy]

Oh yeah, you guys are right, this is BOEING’s issue, not the programmers. Regardless of who got paid what, it is Boing’s responsibility for quality control. If the code allowed a non-controllable and deadly situation to occur, Boeing should have caught this LONG before deaths occurred.

[oldman2005]

Originally Posted by directioneng

..
This is not peanuts and monkeys.

This is serious computer engineering and mechanical interfacing.

Basic function was not tested: They never tested the failure of the sensor so there's no serious engineering in this. Again pay peanut get monkey, otherwise I have my friendly shady mechanic work on my Ferrari and save tons of money.

[Abdullah71601]

Originally Posted by oldman2005

Basic function was not tested: They never tested the failure of the sensor so there's no serious engineering in this. Again pay peanut get monkey, otherwise I have my friendly shady mechanic work on my Ferrari and save tons of money.

You’re missing the point. The failure was in Boeing’s software design and testing, not in the programmers who banged out the code to meet Boeing specs. Boeing could have hired all your silicon valley pals at hundreds of dollars per hour and the outcome would be the same.

[Bigblu10]

Bottom line. When and if this plane ever gets back into the sky there is no way in He!! I will ever fly on one. Those who don't have a death wish will likely agree. This plane is a lame duck if I ever seen one.

[breitlings]

Originally Posted by Abdullah71601

Strawman argument.

Boeing outsourced the programming to a firm they believed competent for the work. The price was right, and it was right because programmers and engineers in India aren't afforded the lavish salaries and benefits one finds in California. The salary has no bearing on the qualification, this is a red herring.

If the firm (and engineers) are determined to be qualified to write the code during the contracting process, the burden falls on Boeing to ensure the end product meets their contract specs. It doesn't look like Boeing effectively managed the contract work, which is a significant management issue. The engineers could have been billing at $100 an hour and Boeing would have hosed this up.

You're the man A, but I am afraid I have to disagree. There are a lot of projects I have led that I simply cannot get done in India. I've tried. Used referrals from networks, discussed with companies that have resources in both countries. On new tasks and heavily recursive functions there can be some "deer in the headlights" moments.

I do a lot of new architecture and heuristics and really the best places for new heuristics and programming are the US, Russia, and Israel. Between all those the US I believe is best at new heuristics (and there are some things Russians and Israelis do better I think) but I could see arguments against and for.

Not to say there isn't a place for Indian programmers in the value chain. I just don't think it was smart to use in this instance, programming a new flight law control system. I mentioned it to a colleague that I was impressed they even programmed it, her response was "apparently they didn't".

[breitlings]

Originally Posted by Abdullah71601

You’re missing the point. The failure was in Boeing’s software design and testing, not in the programmers who banged out the code to meet Boeing specs. Boeing could have hired all your silicon valley pals at hundreds of dollars per hour and the outcome would be the same.

Now you may be right here. Using one sensor instead of two, and updating an old plane design for one customer. At some point there needs to be a fine big enough levied that the stick is bigger than the carrot of saving money on FAA approval.

I just don't know that world well enough to comment beyond I wish Toyota made passenger planes.

[breitlings]

Originally Posted by TheVTCGuy

Oh yeah, you guys are right, this is BOEING’s issue, not the programmers. Regardless of who got paid what, it is Boing’s responsibility for quality control. If the code allowed a non-controllable and deadly situation to occur, Boeing should have caught this LONG before deaths occurred.

I agree. And the decision to outsource such a critical piece of software is imho penny smart and dollar foolish.

[oldman2005]

Originally Posted by Abdullah71601

You’re missing the point. The failure was in Boeing’s software design and testing, not in the programmers who banged out the code to meet Boeing specs. Boeing could have hired all your silicon valley pals at hundreds of dollars per hour and the outcome would be the same.

No, I'm not missing the point. Sorry but I don't think you're familiar with typical software engineering process or how an offshore engineering unit is typically working with a parent company in US.

[drumminj]

Originally Posted by breitlings

You're the man A, but I am afraid I have to disagree. There are a lot of projects I have led that I simply cannot get done in India. I've tried. Used referrals from networks, discussed with companies that have resources in both countries. On new tasks and heavily recursive functions there can be some "deer in the headlights" moments.

I do a lot of new architecture and heuristics and really the best places for new heuristics and programming are the US, Russia, and Israel. Between all those the US I believe is best at new heuristics (and there are some things Russians and Israelis do better I think) but I could see arguments against and for.

My experience is culture plays a role here too. Having worked with contractors, outsourced teams in other countries, etc, I find it's important to have a team that feels invested in and ownership over the project/product. When they do, they will speak up when they feel requirements have been missed, or there's a better/safer/more efficient way to do something.

Without that sense of ownership, your requirements (and whomever is validating the work product) better be incredibly strong, and detailed. Personally, I'd rather rely on a team of people to speak up when they see something that's missed, rather than rely on a single person who may not be into the intricate details.

Culturally, I've found domestic teams, especially those who are culturally American (regardless of nationality/origin) are better about exhibiting the above.

[TheVTCGuy]

Originally Posted by drumminj

My experience is culture plays a role here too. Having worked with contractors, outsourced teams in other countries, etc, I find it's important to have a team that feels invested in and ownership over the project/product. When they do, they will speak up when they feel requirements have been missed, or there's a better/safer/more efficient way to do something.

Without that sense of ownership, your requirements (and whomever is validating the work product) better be incredibly strong, and detailed. Personally, I'd rather rely on a team of people to speak up when they see something that's missed, rather than rely on a single person who may not be into the intricate details.

Culturally, I've found domestic teams, especially those who are culturally American (regardless of nationality/origin) are better about exhibiting the above.

I understand what you’re saying... and in a way I agree with you (IF I am interpreting correctly). You are saying that programmers separate from the whole process, are just a bunch of people doing a bunch of lines in code. BUT, if you have programmers that are somehow attached to Boeing, either by area or contract or some familiarity, then they feel a part of the time and might be more diligent in their product (software code).

IF, I got it correctly, I agree to a small degree, you might be right. But, IMHO, that still does not acquit Boeing of their ultimate responsibility. They should have installed ultimate safety checks and testing, we ARE discussing people’s lives here.

[Abdullah71601]

Originally Posted by oldman2005

No, I'm not missing the point. Sorry but I don't think you're familiar with typical software engineering process or how an offshore engineering unit is typically working with a parent company in US.

I am quite familiar with contracting offshore technical work. The onus was on Boeing to provide the necessary oversight for the project. Your "peanuts" argument is predicated on the programmer salary defining the programmer's talent, which has no merit in this context.

Originally Posted by TheVTCGuy

I understand what you’re saying... and in a way I agree with you (IF I am interpreting correctly). You are saying that programmers separate from the whole process, are just a bunch of people doing a bunch of lines in code. BUT, if you have programmers that are somehow attached to Boeing, either by area or contract or some familiarity, then they feel a part of the time and might be more diligent in their product (software code).

IF, I got it correctly, I agree to a small degree, you might be right. But, IMHO, that still does not acquit Boeing of their ultimate responsibility. They should have installed ultimate safety checks and testing, we ARE discussing people’s lives here.

Agree. This whole discussion about programmer salaries is a distraction from the real issue, which is that Boeing failed design a robust system.

Originally Posted by drumminj

My experience is culture plays a role here too. Having worked with contractors, outsourced teams in other countries, etc, I find it's important to have a team that feels invested in and ownership over the project/product. When they do, they will speak up when they feel requirements have been missed, or there's a better/safer/more efficient way to do something.

Without that sense of ownership, your requirements (and whomever is validating the work product) better be incredibly strong, and detailed. Personally, I'd rather rely on a team of people to speak up when they see something that's missed, rather than rely on a single person who may not be into the intricate details.

Culturally, I've found domestic teams, especially those who are culturally American (regardless of nationality/origin) are better about exhibiting the above.

Excellent observation. Indian culture is important. The older generation often will not speak up. The millennials will shout concerns from the mountain top, but they will not do so in a patriarchal firm dominated by old school management. I don't know this firm, or how they are managed, but Boeing should have been prepared for it.