Royal Bank of Scotland collapse details revealed: Arrow points to defective part
The tech problems at the RBS banking group that left millions of people unable to access money for four days last week were caused by a failure in a piece of batch scheduling software, sources have told The Register.
And at least some of the support staff for that software have been outsourced to India - as recently as February.
Batch scheduling software is used to process routine jobs on a computer without the need for manual input: jobs are prioritised, scheduled and performed automatically - saving human time and using computer resources more efficiently. However the batch scheduling software needs to be maintained and overseen and it's there that our two sources believe the error occurred.
The main batch scheduling software used by RBS is CA-7, said one source, a former RBS employee who left the company recently.
They have more than one type of batch processing software. But the vast majority is run on CA-7.
A second former employee, familiar with RBS's use of the CA-7 tool, confirmed that the majority of batch processing at RBS was performed using the software and fingered a failed update as the cause of the problem:
RBS do use CA-7 and do update all accounts overnight on a mainframe via thousands of batch jobs scheduled by CA-7 ... Backing out of a failed update to CA-7 really ought to have been a trivial matter for experienced operations and systems programming staff, especially if they knew that an update had been made. That this was not the case tends to imply that the criticisms of the policy to "off-shore" also hold some water.
http://www.theregister.co.uk/2012/06/25/rbs_natwest_what_went_wrong/
Millions of British consumers haven't been able to access money or accounts at the National Westminster bank subsidiary of Royal Bank of Scotland.
Like other big banks, their master accounts system is a mainframe batch system consisting of thousands of programs which run overnight to post transactions, update accounts, generate reports, check for fraud, etc. The Computer Associates CA-7 software orchestrates the running of these programs. If the processing goes awry in a way not anticipated by the writer of the CA-7 scripts, manual interventions are needed to recover files, rerun jobs, etc. but if it is not caught in time this can be excruciatingly complex and tedious.
RBS, now controlled by the UK government, has apparently outsourced the maintenance and oversight of CA-7 to India. Likely these new folks have little insight into the business purpose of the processing steps being controlled.
http://www.theregister.co.uk/2012/06/25/banking_fail_rbs_natwest/
http://www.theregister.co.uk/2012/06/22/rbs_natwest_outage_fourth_day/
http://www.theregister.co.uk/2012/06/21/rbs_natwest_tech_glitch_banking_freeze/
