How much money do you deserve for your work? How much money do Wall Street executives deserve for what they do? How much money do workers in China deserve for making your iPod? Many of us are tempted to answer that they all deserve exactly what they are getting now. But do they really?

There is a well-known psychological phenomenon known as the “just world hypothesis.” People have a desire to believe that all is right in the world, and that everyone gets what they deserve. I suppose it’s a comforting delusion that makes us sleep better at night. So when we see injustice, we often try to find reasons why the victim “deserved it” or why unjust rewards were “earned.”

It should come as no surprise that we apply this idea to our views on the rich and poor in our capitalist society. We assume everyone has earned what they have, but are there any objective standards of merit by which we could say that the rich have actually earned their status?

Let’s consider all the possible arguments. First, there is the idea that the rich work harder than everyone else, and that anyone who works hard can get rich. That is blatantly false. Do Warren Buffett and Donald Trump work harder than a single mother with two jobs? Do Wall Street bankers work harder than coal miners? Even if you think the answer is yes; do you believe they work so much harder that it justifies the enormous difference in income? Remember, the rich don’t make five, 10 or even 20 times the amount of the average American. They make thousands of times that amount. The average income of the top 0.1 percent is $31 million per year. Assuming they work 24 hours a day, every day, that comes down to over $3,500 per hour – or a little under $1 per second. If we are to believe that this is all due to hard work, we’d have to believe that your average billionaire works as hard as entire cities, and never sleeps or plays golf.

But maybe it’s not about hard work. There’s another argument you hear very often: That the rich are brilliant innovators that bring us an array of wonderful new products. Except that this is not true either.

Some of the wealthiest started out by inventing things – Bill Gates for example – but most did not. In fact, many of them inherited a fortune from their parents. Examples include Mitt Romney, the Koch brothers, the Walton family (of Walmart fame) and Gina Rinehart, the richest woman in the world (who recently complained about lazy poor people).

Many of the super-wealthy made their money by gambling on the stock market, often with borrowed funds. Warren Buffett is the chief example. Others, like Donald Trump, made billions simply through buying and selling real estate. Then, there are people who got rich by finding ways to make the same product with lower-paid workers. Even the few celebrated “innovators” like Steve Jobs actually got their innovations from hired staff.

If it’s not hard work, and it’s not being particularly innovative, then what else could it be? Well, the rich do risk their money on new businesses and projects. This is the argument of last resort: That the rich are braver than us, because they take risks we don’t. But the main reason why ordinary people don’t risk millions of dollars on new projects is because they don’t have millions of dollars in the first place. We are not unwilling to take those kinds of risks; we are unable. The rich take more risks with their money not because they’re braver, but simply because they can afford it. To risk money, you must first have money. And besides, is it really that brave to risk a million dollars when you have hundreds of millions more? It would be like me risking $10. To say that people deserve to be rewarded for taking risks with their money is basically saying they deserve more money because they already have money.

But why does it matter if the rich deserve their wealth? Many capitalists claim that “a rising tide lifts all boats,” and that it’s not a problem when the super-rich get thousands of dollars every hour because we’re all getting wealthier at the same time. If the pie is growing and everyone is getting a growing slice, then why should we care if the slices are unequal?

Well, first of all, it’s simply not true that accumulation of wealth by the rich helps anyone but themselves. For the past few decades, the income of the average American has not grown at all. Adjusted for inflation, the average taxpayer had an income of $33,400 in 1988, and $33,000 in 2008. In the same time period, the incomes of the top 1 percent grew by 33 percent. The pie has been growing (until the recession, anyway), but the only slices that got any bigger were those of the rich.

More importantly, money doesn’t grow on trees. If some people get rich without deserving it any more than the rest of us – if some people get billions of dollars when they only deserve thousands – then where did all their extra money come from? If it didn’t come from their own work, it must have come from someone else’s.

This is why it matters who deserves what. Because for every dollar that goes to someone who didn’t deserve it, there is someone else who deserved that dollar and didn’t get it.

We have two choices: Either we believe, without any evidence, that the rich are radically better than everyone else in some way, or we accept that they are getting a lot of wealth that should rightfully belong to others who worked for it.