Entrepreneur David Heinemeier Hansson tweeted Thursday that he had 20 times the credit limit his wife had received from the Goldman Sachs Apple Card.
“My wife and I filed joint tax returns, live in a community-property state, and have been married for a long time. Yet Apple’s black box algorithm thinks I deserve 20x the credit limit she does,” Hansson wrote in his tweet.
Furthermore, he and his wife found that her credit score was higher than his, he continued in the Twitter thread.
And he’s not the only one to notice that some husbands have been approved for notably more credit than their similarly-qualified wives. Apple co-founder Steve Wozniak reported the same issue.
Now New York state has opened an investigation to figure out why.
New York’s Department of Financial Services' investigation is looking into allegations that the algorithm used by Apple Card is discriminatory. Linda Lacewell, New York State DFS superintendent, announced in a Medium post, the department will examine “whether the algorithm used to make these credit limit decisions violates state laws that prohibit discrimination on the basis of sex.”
When asked in an email if the company was investigating the matter internally, Goldman Sachs spokesperson Andrew Williams pointed back to the company’s statement: “In all cases, we have not and will not make decisions based on factors like gender.”
Credit limits are generally determined by algorithms, which Women Leading in AI co-founder Allison Gardner said is inherently biased.
"[Algorithms] have to be trained on a massive data set,” she said. She said those data sets are based on historical data and previous customers’ records.
“If a woman has had a career break for having two or three children, she wouldn’t have worked the same number of years as her male partner,” she said. “Just by virtue that she’s worked [fewer] years, even if you don’t say what gender she is, [an algorithm] could identify that and downgrade it, even if she makes more money now.”
She said even when algorithms don’t take gender into account, other “proxy features” will affect the ways algorithms interpret data.
Gardner said the problem is widespread in data collection and automated algorithms worldwide, but technology is moving faster than regulation.
Meanwhile, Wozniak tweeted “the same thing happened to us (10x) despite not having any assets or accounts. Some say the blame is on Goldman Sachs but the way Apple is attached, they should share the responsibility.”
Wozniak also told Bloomberg “we don’t have transparency on how these companies set these things up and operate.”
Apple’s customer service eventually intervened and raised Hansson’s wife’s credit score, but said it could not change that algorithm.
