Walter is the older of two sons of James W. Walter, Sr., a Tampa, Florida home builder and industrialist. He excelled at math and science in school, completing high school at the Asheville School in North Carolina, and was offered a Morehead scholarship to the University of North Carolina, but turned it down at his father's urging in order to enter business school.

Walter earned a B.S. in Business Administration from the University of North Carolina in 1969, and worked in his father's business for two years. His father's political influence helped Walter join the Air Force Reserves to stay out of active combat in the Vietnam War; he earned an officer's commission and served for four years.

After leaving Walter Industries, Walter lived in New York, Hong Kong, and Tampa. He worked in a number of jobs and industries, including financial public relations in New York City, and founded several corporations..

On May 28, 1991, Walter and Jack Edwards co-wrote Banzai, You Bastards (ISBN 028563027X), the story of Edwards's abuse in a Japanese World War II prisoner-of-war camp.

When his father died in January 2000, Walter inherited a fortune reported between $7–14.3 million; he says $11 million. Much of that has been spent on his campaigns. He no longer receives a salary, but receives income from investments.

Since January 2005, Walter has been living in self-imposed exile in Vienna, Austria, after being attacked and threatened due to his campaign.

In the early 1990s, Walter used outside funding and US$3 million of his own money to found the Life Skills Foundation, an organization that taught skills and goal setting to Florida prison inmates. The project produced results, but was shut down after the governor cut state financial backing.

Walter is also the founder of "Walden Three", a non-profit educational foundation in Santa Barbara, California that researches ideas for sustainable, environmentally friendly urban development. The foundation has developed a computer model for the ideal sustainable living, carfree and fossil fuel-free city or society that produces almost all of the consumables, durable goods, structures, mass transportation and social security needed by its citizens. The model uses rational-emotive therapy (REBT), developed by Dr. Albert Ellis. Walter considers Walden Three his "day job" when he is not involved with campaigning.

Walter had been opposed to the Vietnam War, a source of contention with his father, but had not used his money to act on his convictions. On February 27, 2003, Walter spent US$125,000 to take out a full-page advertisement in the New York Times attacking Secretary of State Colin Powell's justification for the impending 2003 invasion of Iraq. It was headlined "Powell Lied?" describing Powell's February 5, 2003 speech to the United Nations Security Council—the New York Times insisted on adding the question mark. Walter took out similar advertisements in English and Spanish in local newspapers in Santa Barbara, California, where he was living. His stance was unpopular; posters he attached to his house were torn down, and a rock was put through his car's window. Walter did not repeat the advertisement.

Walter also created a "Chicken-Hawk-In-Chief" design mocking George W. Bush which he put on t-shirts, sweatshirts, and a full costume. In May 2004, a Williamstown, Vermont middle school student gained national attention when he wore the t-shirt and was required to cover parts of the design depicting Bush drinking and snorting cocaine. On August 30, 2006, the U.S. Second Circuit Court of Appeals ruled the school was wrong to censor the shirt, and on June 29, 2007 the US Supreme Court rejected an appeal, allowing that ruling to stand.