HSBC Online Banking is Down
As of 9:47 on 5th January, 2016, HSBC UK's Online Banking service has been down for AT LEAST 18 hours. Last night, there was NO service (I was getting 500 errors). This morning, still no service, but this is due to "a higher volume of traffic than usual".
Wow, HSBC, wow.
Have you heard of "Load Testing" or "Quality Assurance"? How about "Performance Monitoring"? I look forward to you VERY PUBLICLY publishing your Root Cause Analysis and Post Mortem, HSBC UK.
Update at 10:00
It looks from Paul85D's screencap from yesterday that HSBC UK are using an Oracle Application server (note the un-configured "you@your.address"). I assume that you are lending them a hand right now, Oracle?
Update at 10:05
DownDetector provide an excellent view of the situation from the user's perspective. A great demonstration of crowd-sourcing data. I hope that HSBC UK are a little more proactive than this for fault monitoring, though.
Update at 10:22
The BBC News guys are on to it. This will, of course, just result in anyone and everon who has an HSBC account trying to log in to see if their service is OK.
Short term: "Damn. Now no-one will be able to connect for days."
Long term: "Good. This will kick HSBC into action to a) sack whoever caused this nightmare and b) treat their Internet Banking service team a lot better."
Update at 10:30
Twitterfall is a great way to watch HSBC's nightmare unfold.
Update at 10:34
Unrelated, but this video give a glimpse of how HSBC UK IT might look right now (from "All of our data is GONE!"):
Update at 11:11
You can track the HSBC Share Price here
Update at 11:13
You can complain online here: https://www.hsbc.co.uk/1/2/contact-and-support/feedback-and-complaints . At least that part of HSBC's website is working right now.
Update at 11:23
Xhalr is something that will help all HSBC customers trying to use online banking right now:
Update at 11:25
Dear HSBC you may be interested in monitoring your IT infrastructure using a Performance Management System like LogicMonitor.
Update at 11:30
...and it's back! (applies to Business banking, though running a little slowly)
Update at 11:49
I spoke too soon. This is what I get when I log in:
Update at 11:52
OK, so THIS is new. DNS failures now... Perhaps they're reverting back to the old system?
Update at 12:00
Yes, Business Banking is back, and looking very much like the old system to me. I'll sign off now, but will update further if appropriate.
ARGH! Started making payments and it's back down again.
Update at 12:12
Update at 13:12
Still down. Apologies have been sent to all those my company owes money to.
...and tomorrow... the finger pointing starts...
Update at 13:25
Back up for now. I expect HSBC IT people are now staring intently at their screens with fingers crossed.
...and tomorrow... the finger pointing starts...
Update at 16:49
One thing worth pointing out. The relevant A record has a TTL of (randomly) between 0 and 20 seconds. OUCH. This is REALLY not good. HSBC, are you SURE people need to re-fetch an authoritative answer with each query? Also, are you SURE that this is not an attack? I sincerely hope that you are in control of that DNS record, because if you are not, customer traffic could potentially be re-routed elsewhere on the Internet.
Update at 17:11
There's an interesting discussion about DNS going on at The Register. Also, a lot of HSBC abuse from unhappy customers.
