Vital Relays Rebuilt Incorrectly at a Cost of $10 million

Vital relays are a relatively simple but very critical part of the automatic train control (ATC) system. They have been used by the railroad industry since the 1800’s and haven’t changed much in decades. See the photo below:

The relay labeled A2-942 TR in the photo is a “track relay” that is used to indicate whether the platform #2 track circuit at Shady Grove (A99) is occupied or vacant. It appears as though there was a train at the platform because the relay is ‘dropped’ (de-energized). The way you can tell is that the ‘silver’ contacts (called “heels”) are not touching the contacts above them (called “top contacts”). The circuit is numbered 942 because the inbound end of the platform at A15 is approximately 94,200 feet from Metro Center (A01).

A relay is essentially a remote controlled switch. They are used in everything from cars to washing machines. A relay uses electromagnetism to move the contacts. When current passes through the coil(s) of wire in the relay it creates a magnetic force that lifts the relay contacts up (or “picks” the relay), ‘opening’ and/or ‘closing’ those contacts. If a set of contacts are referred to as ‘closed’ they are making physical contact with each other – it is the equivalent of turning a switch on. If they are open there is no contact and current does not flow.

At Metro vital relays control speed commands, signals, track switches, etc. They indicate whether or not track circuits are occupied. For that reason they are designed to be fail safe. The idea is that if there is no (or insufficient) current passing through the relay coil(s) gravity will pull the contacts down (or “drop” the relay). A dropped relay will almost always create a more safe (or safer) condition than if it were picked – zero (or at least slower) speed commands, signals go red, switches are locked and will not throw, associated track circuits appear occupied, etc. Any failure – a blown fuse, a broken wire, a bad contact, power failure, etc, that interrupts current flow to the relay coil(s) should cause it to drop.

They are supposed to be very reliable and have a long service life – 50 years or more. That turned out not to be the case at Metro. A few vital relays were found that would remain picked with no voltage at the coils – even when removed from the rack and set on a desk they remained picked! Luckily no one was killed as a result but there were a few potentially dangerous incidents – one train received much higher speed commands than it should have while crossing over from one track to the other. I believe in that case the operator noticed the excessive speed and slowed the train. In another case a train received speed commands even though the controlling signal was red.

It was determined that all 20,000+ relays should be rebuilt or replaced with new relays. The superintendent of ATC at the time decided he knew better than the engineers at GRS (the mfr) and decided to rebuild the relays in-house. He was warned that rebuilding vital relays required special skills, knowledge, parts, and test equipment but he plowed ahead anyway.

The relay project took well over a year and cost $10 million, IIRC. During that time, trains had to be operated in manual which meant longer, slower, jerkier rides for passengers and more mental fatigue for operators.

After all of the relays were rebuilt it was determined that – surprise – they were rebuilt incorrectly. The ATC superintendent had recruited just about any tech that was willing to do the job – ATC, COMM, AFC, etc. They were not properly trained and did not have the correct parts, tools, or test equipment. The project was an abject failure. All 20,000 relays had to be replaced a second time with new relays (which of course should have been done in the first place).

Then a few of the new relays began to fail (remain picked with no voltage at the coils), so many/most of them had to be tested and replaced if necessary.

The relay project is another example of hubris on the part of Metro management – risking lives, wasting money, and/or inconveniencing passengers. It reminds me of our former “engineer” (the one who only has a GED) deciding he knew better than the mfr and ordering ATC to mix new impedance bonds with old track circuit modules. That is what caused the June 2009 train wreck that killed 9 and injured dozens more.

Maybe it’s something in the water but many people in management at WMATA seem incapable of listening to ATC equipment mfrs and/or learning from other transit systems (let alone employees). They act as if Metro exists in a bubble and must invent new procedures and policies as they go – as if there isn’t 150+ years of railroad experience and knowledge to draw from.

“We don’t care how they do things in (NYC, San Francisco, Moscow, London, etc) – this is METRO!!”

Advertisements
This entry was posted in Management Follies, Safety Incidents and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s