Outage 26-07-17


#1

Looks like Starling are having problems with card payments at the moment:

https://starlingbank.statuspage.io/incidents/4q5dpnq18yy4

Just in time for lunch :unamused:


#2

I’d be curious to see what went wrong. Doesn’t look like it’s their card processor as competitors using the same processor work fine for me.


#3

Hi all,

As per our status page we have been experiencing an issue with our cards – this has now been downgraded and all services have resumed again. You should all be able to buy your lunch still! In fact the testing would be very useful.

We apologise for any inconvenience.
Sarah


#4

What do you mean by “downgraded”? Was a code update pushed to production which caused the issue?


#5

Yeah had two declines over lunch :slightly_frowning_face: Last outage i received a useful text message to say there was a problem so used a different card. This time I wasn’t notified.


#6

I got the text message


#7

Downgraded generally means we have identified the cause and resolved the issue but we are still monitoring. The status on our status page has now been updated to operational and those who subscribe to the status page will have received a message.
We are very sorry for those who had their card declined within the last half an hour.


#8

Got email on both thanks


#9

Sorry but I thought downgrade means “switch back to a lower version”. If you have fixed the problem I’d call that a resolution and not a downgrade. Also, would you mind sharing details on what went wrong? Did someone spill their beer on the servers or deployed a bad version of the code? :joy:


#10

Could you please look into app notifications to let us know of issues would be really useful.


#11

Ah good point. Downgrade of the incident.
When we “switch back to a lower version” I guess we say roll back.


#12

Hi, My first day with Starling and both Apple Pay and my card were declined in a store ;(


#13

I am really sorry to hear that Deano, and hope we can make it up to you as we continue to improve the product.


#14

This is becoming unacceptable. How are we supposed to trust starling if we keep having outages. I accept that it’s usually the payment processor however there is no plans to bring this in house so it’ll Just keep happening and that’s okay is it???

I also find it unacceptable that I don’t get a notification when card payments aren’t working. I shouldn’t have to sign up to a notifications page there should be push notifications from the app or a text message.


#15

We are prioritising sending communications to all customers when their card is affected, and we apologise for anyone who was affected yesterday for the brief time our cards might have been declined. We are in beta, and are continuously improving our product and service – and we hope to reduce the impact on customers going forward. If you were affected, we are very very sorry.


#16

What makes me sad is that you gave zero details on what went wrong, as if it was your fault and you were ashamed to admit it - you shouldn’t be. If you messed up the best outcome is to admit it and tell us what you’ve done to make sure it won’t happen in the future.


#17

Firstly, let me state for the record I do not believe any bank should suffer unscheduled downtime: their processes and systems should be reliable and robust - designed such that automatic failover occurs when things go wrong (and we know things go wrong occasionally - be it comms lines, hardware, software, or human error).

That said, the “big banks” have the scale, technologies, resources and procedures in-house and entirely under their control to maintain service reliability. Whereas upstarts like Starling just don’t have that scale or maturity yet. They rely on business partners to deliver some services on their behalf. Being a respectable business, I would not expect Starling to throw anyone under the bus - be it an organisation or individual.

On these occasion, we don’t know if it was a parters’ fault, or something that went wrong within Starling themselves. But they have apologised, and I am unsure we can expect anything more from them.

All we can expect is that Starling provide the service we want from them. It’s up to them to ensure their processes, systems, and partners are fit for purpose and robust in their delivery. Because if they’re not, we will go elsewhere.

Monzo take the same approach - while they apologise for cock-ups, they too do not name and shame other parties when things go wrong. These are normal grown-up business ethics.


#18

I am not saying they should blame it on a particular person - Mr X spilled his coffee on the server and that’s why it’s gone down.

However if someone did indeed spill coffee I expect them to admit it and explain how they plan to avoid that in the future.


#19

That’s not quite true, they blamed external providers when it’s been their fault (without naming them) and explained they were removing their dependence on them as a long term solution (which they’ve done with the launch of the preview current account).


#20

Lets be honest - if I get a push notification saying someone spilt coffee and an explanation on how they’ll stop it happening then I’d be more concerned that they’ve got nothing better to do.

30 mins downtime is fairly insignificant when you look at major banks, with the in-house software and resourcing, have had outages last days.

Again, it’s in a beta stage - you wouldn’t expect a initial concept car to be ploughing through 200k miles before the kinds were ironed out. Same applies. Albeit frustrating that it’s your money, think a little slack is due. There is the option to sign up for notifications via both via text and email, which I have done. if you can’t be bothered to do that, then more fool you. No right to complain you weren’t notified if you can’t be bothered to sign up to notifications :joy: