Tax Tips for Tycoons

The affluent with enough savvy to manipulate the tax code to their benefit can reduce their income tax rate to a level lower than a middle class family. That was the trick Roy M. Speer, the millionaire founder of the Clearwater-based Home Shopping Network, was able to turn. Along the way, he and his son wrote off at taxpayer expense a security fence and an architect's fees for redesigning a luxury home. In 1981, Speer set up Home Shopping Channels, a company that sold retail merchandise over the airwaves to the Tampa Bay area market.

Home Shopping Network went live on July 1, 1985, broadcasting nationally for five hours each day via satellite. Speer owned 60 percent of the company, and he held the reins tightly. I have a principle in my business: If I don't control it, I don't own it, he once testified in a tax case.

Speer's hard work, along with the cable and credit card booms of the 1980s that fueled impulse buying, made the Home Shopping Network a huge financial winner. When Speer cashed out in December 1992, he was paid $160-million for his controlling stake by Liberty Media Corporation, a cable-television programmer. He sold the rest of his shares in 1993 for another $100-million, and relinquished his position as chairman.

It wasn't the profits Speer reaped from the sale of his stake in Home Shopping Network that caught the attention of the IRS, but rather, the way he ran the firm. Pioneer Data Processing, a company that Speer once owned, developed the software program that Home Shopping Channel — the Tampa Bay-area predecessor to the national Home Shopping Network — used to take its orders from the throngs of cable TV viewers.

When Speer launched the Home Shopping Network in 1985, he hired Burroughs Corp. to rewrite the program in a different computer language, and he paid the computer company $55,250 for its efforts. He agreed to pay Pioneer, whose program he was no longer using, a far more generous sum. Speer signed a licensing agreement with the firm guaranteeing it 1 percent of the annual gross profits of Home Shopping Network. The agreement was open-ended.

Speer signed the document for Pioneer even though he didn't own the software firm. He was just exercising his parental prerogatives: His son, Richard Speer, owned Pioneer.

After auditing the returns of Pioneer software and Roy and Richard Speer, the IRS concluded that the licensing agreement was a sham — that it was merely an attempt to divert income from Home Shopping to Pioneer and Speer's son, Richard. Pioneer, in turn, paid hundreds of thousands of dollars in consulting fees to companies owned by Roy Speer, which the IRS discovered in a 1991 audit of the closely held software developer.

On Dec. 2, 1988, Pioneer wrote a $300,000 check to Tahitian Investments, Inc., for consulting fees, and later deducted it from its tax returns as an ordinary and necessary business expense. Roy Speer, it turned out, owned Tahitian Investments, and the $300,000 in consulting fees was for his time. Roy Speer stated that as a matter of family tradition, an IRS agent who audited the Pioneer tax returns wrote in his report, he and his family would discuss business on the weekends, which is when he discussed many of Pioneer Data Processing's business affairs with his son, Richard.

Unlike most fathers, however, Roy Speer charged his son for his weekend advice. He stated that he believes in making his children pay for the things that they get, and that they should work for their money. The IRS auditor decided to disallow the deduction from Pioneer's tax returns.

The audit of Pioneer led the IRS to audit the Speers as well. In 1994, the Service sent Roy Speer and his wife, Lynnda, received Notices of Deficiency for more than $9.9-million. The IRS challenged the licensing agreement between Home Shopping Network and Pioneer, arguing that Speer was funneling money from Home Shopping through Pioneer and back to himself.

Speer assembled a team of high-priced lawyers, including James A. Bruton III, who had worked in the IRS chief counsel's office. Testimony in the Tax Court case lasted five days. Among the witnesses called by Speer's half-dozen attorneys and the IRS were experts on computer programming; among the exhibits entered into the record was the 1981 program itself. Speer also took the stand. Senior attorney Francis C. Mucciolo, who tried the case for the IRS, asked Speer about the $300,000 his son paid him for his advice.

Probably should have charged him half a million instead, he replied. He got a break.

Among the documents entered into the record were joint tax returns filed by Roy and Lynnda Speer. Speer reaped rewards from his cable business that put him, in terms of income, among the top 1 percent of all Americans. In 1989, for example, his salary from Home Shopping Network was more than $400,000. Tahitian Development — the company to which Pioneer Data Processing paid $300,000 in 1988 — paid Speer a salary of $48,000. Altogether, Speer and his wife earned more than $550,000 in salary. They also received just over $1-million in taxable interest income, another $157,000 in capital gains, and $490,000 in tax-exempt interest income from investments in municipal bonds and the like.

Altogether, the Speers' total income amounted to $1.2-million.

Not everything Speer touched turned to gold, as his tax returns show. In 1989, for example, he claimed losses of $1.1-million from Maximo Marina, a full-service marina operating in St. Petersburg. Speer owned a controlling interest in Maximo and could claim those losses on his Form 1040. He claimed losses of $78,000 from another of his companies, Gateway Marine, a tug and barge business. Altogether, he claimed losses from his other business ventures of nearly $1.2-million.

That left the Speers with adjusted gross income of $578,000 — still not a bad year. The personal exemption that year was $2,000; Speer claimed one for himself and his wife. The standard deduction for a married couple filing a joint return in 1989 was $5,200; the Speers instead claimed itemized deductions of more than $400,000. They deducted $61,000 paid in local and state taxes, another $142,000 for interest paid on loans they took out to invest in various businesses, and nearly $200,000 for donations made to various charities.

Altogether, the Speers were left with a little less than $174,000 in taxable income, or about 15 percent of their $1.2-million in gross income. They claimed they owed $89,600 in federal income taxes. In a year in which the Speers, by their own admission, received more than a million dollars in income, they paid taxes at an effective rate of 7.4 percent.

A family of four earning the median income of $42,300 and taking the standard deduction would pay about 9 percent of their earnings in federal income taxes. Tack on their Social Security and Medicare taxes, and the middle class family ends up paying 17 percent of their income to the federal treasury. The Speers, among the wealthiest Americans, paid $7,200 for Social Security and Medicare taxes and another $61,000 in state and local taxes. Altogether, the Speers' total tax burden amounted to a little more than 13 percent of their total income.

In the end, Judge Robert P. Ruwe ruled that the program owned by Pioneer had value, it was the basis of the program Burroughs later wrote, and that the licensing agreement was legitimate. As a result, Speer did not have to pay the bulk of the deficiencies assessed against him.

That's largely because the tax code treats different kinds of money differently. Money spent to make more money is tax deductible. Money spent to meet one's basic needs is not. A young couple that buys a cradle, bassinet, changing table, and so forth on an installment plan may not deduct the interest payments on the furniture. By contrast, a millionaire like Roy Speer, who borrows to invest in a slew of ventures, may write off on his taxes the interest he pays on those loans.

In 1995, more than 39,700 millionaires deducted more than $4-billion from their tax returns for interest paid to invest in everything from the latest high-tech company to the most old-fashioned speculative property, real estate. Those millionaires, incidentally, represented a mere three of every 10,000 taxpayers; collectively, they claimed 38 cents of every dollar in deductions for investment interest expenses.

Corporations get an even sweeter deal, as Speer discovered and used for his benefit. Consider the IRS audit of Pioneer Data Processing. After discovering questionable items on its returns from the 1980s, agents interviewed the Speers — part of the audit process, in which a taxpayer can give his side of the story.

In 1987, Pioneer paid $525 in country club dues for Roy Speer and deducted the cost from its taxes as a necessary business expense. Pioneer paid $20,304 for a security fence for Richard Speer's home and deducted that; the IRS chose not to challenge either expense. Pioneer also paid an architect $5,798 to design a home for Roy Speer and his wife on land owned by the company; the IRS allowed Pioneer to deduct the architect's fees.

And so it goes for the wealthy. If they make money, it's theirs to keep; if they lose it, it's a tax write-off.

This article was adapted from the recently published book, The Cheating of America, by Charles Lewis and Bill Allison (William Morrow, $25). The book was commissioned by the Center for Public Integrity in Washington, D.C. Allison can be reached at 202-466-1300 or at ballison@publicintegrity.org.

This article appears in Apr 19-26, 2001.