Sign into an existing account

Im in IT now as part of a team that finds bugs in a proprietary software.

For the past 5 months we've been dealing with a problem a customer had because of reasons unknown until recently. There were many, many several hour long phone calls with many highly paid people (each of the calls alone were in the 5 figure range for the combined pay rate). I spent 2 weeks straight going through our system compiling a gigantic spreadsheet of information. All of this was happening while our normal workload was coming through and created a heavily degraded quality of service. And the normal red alerts were going off from time to time, compounding the issue greatly. During this period the company gave very large credits to the customer to retain them.

It turns out in the end, at least 35% of the problem was the customer submitting improperly formatted data to us, and a simple pre conditions check would have caught all of this the first time they tried it. They recently overhauled the software in the past 4-5 years or less and didn't put these checks in there. Not only would these checks have stop this catastrophic incident, but it would reduced our ticket load HOURS per instance, at a minimum. These things happen every day, or at the very least several times per week. I guess this makes job security for me, but wow. How did a program written in 2017-18 not have pre conditions when the stakes are this high? I don't actually know the dev team. Is it possible there are boomers there who don't know modern programming practices (yeah yeah preconditions have been around but they were preached like they are now).

What is going on that these enormous costs are incurred?

Update: there is an amount of outsourcing going on, but every time I've been on a call with a programmer they're American.