By Ron Munden — started 11/28/2021 -> finished 7/5/2022
Have you ever dreamed you failed to pay 10,000 people? I had this nightmare and I was awake.
Today I received an email from my bank. It said that my December retirement check had been deposited into my account by the Federal Reserve one day early.
This email sent old memories flooding back into my mind.
In the mid-1980s I was moved from the Head of Technical Support for the Design Division position into the Head of Information Resources Management (IRM) for Mare Island Naval Shipyard. Two weeks into my new job an incident occured that remains crystal clear in my mind today.
One of the responsibilities of IRM was processing payroll data and generating a computer tape that could be delivered to the Federal Reserve (FR) in San Francisco. The Federal Reserve processed the tape and sent electronic transactions into the country’s banking system and deposited funds into the accounts of the 10,000+ people working at Mare Island Naval Shipyard.
The process is repeated every two weeks. On Monday the payroll for the previous two weeks was processed on swing shift at Mare Island. Tuesday morning the tape was hand delivered to the FR. That evening the FR used the tape to generate the transactions that would deposit the pay into MI employees’ banking accounts. On Thursday night all 10,000 employees’ bank accounts were credited with the pay for the previous two weeks. On Friday morning the funds were available for use.
This process had been used for the prior couple of years without a problem.
This time we ran payroll and the Comptroller Department took the tape to the FR Tuesday as scheduled.
On Tuesday night I was at home having dinner when I received a call from a computer operator at the FR. He told me that our magnetic tape had failed validation at the FR and they could not process Mare Island’s payroll.
I was not really worried when I got off the phone. I knew magnetic tapes were not completely reliable and we have to rerun jobs from time to time.
On Wednesday morning I got to the shipyard at about 6:30am. I noticed the schedulers in operations that we were going to need to re-run payroll and generate a new tape to take to the FS. My next call was to the Comptroller, a Navy Captain. I explained the situation. He said that he would have the Deputy Comptroller dive it to the FR and wait to ensure it passed validation.
We re-ran payroll and got it to the FR by mid-morning.
Just before noon the Comptroller walked into my office and told me that his Deputy had just called and said the new tape had failed validation. This was the point that the Comptroller and I both realized we had a serious problem. If we could not pay people on Friday, there were going to be a lot of bounced checks Friday night.
Our problem was complicated by the fact that our office did not write the software that was failing. It was written by a central design agent on the East Coast. We reported the problem to them but the group was not known for quick responses so we could not count on them to fix our problem.
We started looking for solutions. The shipyard had been using this software for about 6 months. Prior to that we printed checks and distributed them on Friday at the Shipyard. We thought we could go back to the old software and print checks bypassing the FR electronic processing. A quick look at check form inventory showed that we did not have enough checks in stock to print 10,000 checks.
That left us with only two alternative:
- Find the error in the payroll software and fix it or
- Modify the magnetic tape going to the FR so it would pass validation.
We also recognized we had to develop a plan of action to address the possibility that we could not get the people paid.
I took the lead on 1 and 2 approaches. The Comptroller finalized his plan of action if alternative 1 and 2 both failed.
It is now mid-afternoon on Wednesday and the drop-dead for FR processing was late-afternoon Thursday without special processing by the FR.
I used the Admiral Rickover approach by assigning different teams to work in parallel and trying to solve a problem using two different approaches.
We assigned the two best Cobol coders to read the payroll source code listing and try to find any coding errors that might be causing the problem.
A second team of engineers started writing computer C code to read the payroll output tape and recalculate the tallies that the FR expected so the modified tape would pass validation. I worried a little about if this was legal but the vision of the Comptroller and I hanging from a crane on the waterfront trumped the legal concern at this point.
Meanwhile the Comptroller scoped out the size of the problem. As I mentioned at that time the shipyard employed about 10,000 people. Those people used 54 banks in the Bay Area.
The Comptroller wrote a letter that would be hand delivered to each of the 54 banks first thing on Friday if our efforts to fix the problem failed. Each letter had an attachment listing the name and bank account number for each person doing business with that bank. The letter quickly outlined the problem and asked the banks to honor all checks listed in the attachment. It also committed the Shipyard to pay all the bank charges resulting from any account overdraws.
At about 6pm the Comptroller stopped by the office. I briefed him on my status. I calculated we had about 12 to 14 hours to fix the problem and deliver a good magnetic tape to the FR. I told the Comptroller we would work through the night and with luck we would have a tape that would pass validation before 8am Friday morning. The Comptroller said he could not be much help but he could keep us supplied with coffee and donuts.
He was true to his word. Every couple of hours he would drop by our office with more coffee and donuts.
At about 3am Thursday morning the Cobol programmers came to my office and said that they thought they may have found the bug. They showed another manager and me the code. There was a problem but was that the only problem? They ask if they should continue looking? I did a quick calculation in my head and decided we had run out of time. I told them to stop looking and start fixin. I also allerted computer operations to prepare to re-run payroll again.
The programmers made the corrections to the source code and recompiled the corrected code. They were very careful because everyone knew we only had one change to get it right.
By 6am we were ready to re-run payroll and cut a new tape. By 10am the Deputy Comptroller had the tape and was driving to San Francisco.
Then the hard part – we wait to see if the tape would pass validation. Before 2pm we got the call. The tape passed validation at the FR. The transaction would be loaded into the banking system on its next run. Money would be in employee bank accounts Friday morning.
It had been a long two days. We were all very happy people. The image of the Comptroller and yours truly hanging from a crane on the buildways slowly faded from my memory, replaced by a team working together.
Footnote:
After more research we knew what had happened. The software had worked and paid personnel in all the shipyards for 6 months. That was about 80,000 people. Why did it stop working at Mare Island?
There are 8 Federal Reserve banks in the country. The software should have been able to handle payments going through more than one FR bank but for the previous 6 month all shipyards had all their employees using a single FR bank.
For years Mare Island supplied employees to work at a prototype nuclear reactor site in New York state. The assignments were usually 3 to 6 months. During these assignments no employee had ever opened a bank account in NY and had their paycheck deposited into that account. The week before our payroll run failed validation, one employee did exactly that. The payroll software had a bug. It only worked when all the deposits went to a single FR bank. Therefore, our payroll contained data for two FR banks and the central design agents software generated tallies that could not pass validation.
We notified the central design agent in Washington of the problem. They issued a software fix to all of the shipyards prior to the next pay date.
I stayed in the IRM job for another 7 or more years. We ever had another nail-biter moment like this during that period. What a way to start a new job.
***********************************
GIVE US YOUR FEEDBACK. CLICK ON “COMMENT” TO TELL US WHAT YOU THINK or use one of the alternative methods for providing feedback.
Click here to submit feedback. Let us know what you think.
—————————————————————–
Click here to go to the iExposed.us youtube channel
Click here to go to the iExposed.us website
—————————————————————–
iExposed Us
P.O. Box 721
Scottsville Texas
75688
iEXPOSED.us Blog 
