(Bloomberg) -- From the harbor on St. Thomas, the boat skims eastward across the crystalline Caribbean, takes a turn and there it is: the palm-fringed paradise that was the private redoubt of Jeffrey Epstein.

An American flag on a towering pole flutters in the breeze. A blue-and-white building that resembles a temple sits atop one of the hills. The pool and cabanas are visible in the distance. There’s no traffic on the winding dirt roads, no people on the dock or the beach.

It’s quiet now on the island of Little St. James. Epstein dubbed it Little St. Jeff’s. Locals have other names for it: Pedophile Island and Orgy Island.

This is where Epstein –- convicted of sex crimes a decade ago in Florida and now charged in New York with trafficking girls as young as 14 –- repaired, his escape from the toil of cultivating the rich and powerful.

Over the years, the 66-year-old crossed paths with former and future presidents, as well as a Who’s Who of wealthy business figures and celebrities. On St. Thomas, he has been a subject of lurid speculation for as long as anyone can remember. Tourists still take boats out to get a glimpse of the island, where, according to a former employee, Epstein hosted young women who flew into St. Thomas with him and were ferried over in groups aboard a 38-foot vessel called the “Lady Ghislaine,” apparently named for his friend Ghislaine Maxwell.

“It’s part of the tour and has been forever,” said Kristi Query, owner of Virgin Islands Yacht Charters in Compass Point Marina on St. Thomas, the gateway isle of the U.S. Virgin Islands.

Few here doubt that Epstein is wealthy. He routinely touched down on St. Thomas aboard his private jet before being whisked by helicopter to his 72-acre retreat. He spent many millions after buying it for $7.95 million in 1998, carving roads, planting scores of 40-foot palms, building several villas and the temple structure, which was topped by a gold-colored dome until Hurricane Irma blew it off, according to locals.

Yet the size and source of Epstein’s fortune are as much a source of speculation here as they are on Wall Street. Bankers and money managers wonder exactly what his business entailed, with theories ranging from helping the ultra-rich reduce their taxes to buying and selling currencies.

Reid Weingarten, a lawyer for Epstein, didn’t immediately return a voicemail message seeking comment for this story.

The properties Epstein owns certainly suggest he knew his way around the world of big money. He has a second private island, Great St. James, mansions in Palm Beach and Manhattan, an apartment in Paris and a ranch in New Mexico that, according to an October 2009 deposition in a civil lawsuit, has its own 4,500-foot airstrip.

The New York townhouse alone is worth more than $100 million, according to Dolly Lenz, a luxury real estate agent who has viewed it. “Literally, only billionaires have these kind of assets,” she said.

Epstein didn’t spend much time on St. Thomas, according to locals, but his business is headquartered there in a commercial plaza that’s also home to a Sami’s Mini-Mart and the Happy Nails salon. There’s no map to guide a visitor to the office and no signs on any door, though at least five Epstein entities are or have been registered in a suite there, according to public records.

Among them are companies that hold his Gulfstream jets, Financial Trust Co. and Southern Trust Co., whose business is described as “DNA database and data mining” on a government website that lists Cecile deJongh as the office manager. DeJongh, the wife of a former governor of the territory, couldn’t be reached for comment.

Before he transferred his operations two decades ago to the U.S. Virgin Islands, Epstein was based in New York. Financial Trust was then called J. Epstein & Co., a money-management firm he famously touted as catering to billionaires only. The primary client, it appears, was Victoria’s Secret mogul Les Wexner.

Wexner visited Little St. Jeff’s at least once, the former Epstein employee said, and Victoria’s Secret models were among the guests he saw there.

The move to the U.S. territory could have been for privacy, or the juicy tax benefits. Qualifying companies are eligible for a 90% reduction in corporate income tax and personal income tax if they meet certain conditions, including employing at least 10 residents and investing $100,000 in local industries. Southern Trust is listed at the Economic Development Authority as one of the beneficiaries of local tax incentives.

Back in his heyday, before his 2008 guilty plea in Florida, Epstein would visit his piece of the Virgin Islands archipelago about two or three times a month for stays of three or four days, according to the former staffer, who asked not to be named because Epstein insisted on secrecy from his employees. He described it as a Zen-like retreat when Epstein was there, padding around shirtless in shorts and flip-flops, with meditative music piped into the area around the main house, the cabanas and the pool, where women would sometimes sunbathe topless or in the nude.

The crew of groundskeepers and other workers sported black or white polo shirts when the master was in residence -- and darted away when they spied him. The former employee said they had to obey one ironclad rule: Epstein could never catch sight of them.

Among the other oddities of being employed there, the former staffer said, was Epstein’s interest in “pirate treasure.” Not gold doubloons, but old rum bottles and crockery that workers would come across. If you found an old rum bottle, Epstein would pay $100 for it. An unbroken plate was worth $1,000. A broken plate, as long as it was more than half intact, would fetch $500.

The only unusual aspect of the main residence the former worker said he was aware of were the security boxes in two offices. The level of secrecy around a steel safe in Epstein’s office, in particular, suggested it contained much more than just money, he said. Outside of an occasional visit by a housekeeper, no one was allowed in those rooms.

Whether Epstein will ever return to Little St. Jeff’s is unknown. He is incarcerated at Metropolitan Correctional Center in New York, where a federal judge is scheduled to consider his request for bail at a hearing on Monday.

--With assistance from Zeke Faux, Tom Maloney, David Voreacos, Caleb Melby, Patricia Hurtado and Erik Schatzker.

To contact the reporters on this story: Jonathan Levin in Miami at jlevin20@bloomberg.net;Greg Farrell in New York at gregfarrell@bloomberg.net;Tom Metcalf in London at tmetcalf7@bloomberg.net

To contact the editors responsible for this story: Pierre Paulden at ppaulden@bloomberg.net, ;Winnie O'Kelley at wokelley@bloomberg.net, Ambereen Choudhury

©2019 Bloomberg L.P.