The Fed's economic stimulus has been a key driver of the stock market's 25 percent surge this year, along with rising corporate profits and a recovering U.S. economy.
The Dow Jones industrial average lost 66.21 points, or 0.4 percent, to 15,900.82. It was up 20 points shortly before the minutes were released at 2 p.m. Eastern time.
The Standard & Poor's 500 index was lost 6.50 points, or 0.4 percent, to 1,781.37. The Nasdaq lost 10.28 points, or 0.3 percent, to 3,921.27.
The market began the day higher after an encouraging report on retail sales and better news from long-struggling J.C. Penney.
Investors already know the Fed will reduce its economic stimulus eventually, yet they remain highly sensitive to concrete signals that a pullback is imminent and worry that the Fed might withdraw its support before the economy is ready.
Bond prices also declined. The yield on the benchmark 10-year Treasury note rose sharply, to 2.80 percent from 2.71 percent just before the minutes were released. That's the highest since Sept. 17. Bond yields rise when demand for them falls.
The Fed's next policy meeting is scheduled for Dec. 17-18. Investors are split on whether the bank will vote to pull back its bond purchases, or "taper" them, as it is sometimes called on Wall Street. The Fed surprised investors at its Sept. 17-18 meeting by keeping the bond purchases in place, despite widespread predictions that it would start to wind the program down.
With Wednesday's decline, the S&P 500 is down roughly 1 percent for the week. The index hasn't had a weekly loss since the week ending Oct. 4.
Hooper and other market watchers said they would not be surprised if the market continued to fall. "It would not be unreasonable for investors to step back here," said Ron Florance, deputy chief investment officer for Wells Fargo Private Bank.
J.C. Penney led the S&P 500 index higher with an 8 percent gain. Despite reporting a loss in the third quarter, there were hopeful signs for the long-struggling store chain heading into the holiday shopping season. The company said its sales increased in October for the first time since December 2011. The stock rose 73 cents to $9.44. It's still down 52 percent this year.
Lowe's, the home improvement store chain, fell $3.11, or 6 percent, to $47.33. Lowe's earned 47 cents per share in the latest quarter, a penny short of what analysts were looking for. Lowe's was outshone by competitor Home Depot, which reported a 26 percent surge in net income the day before.
The holiday shopping season is a make-or-break time for U.S. retailers, and more broadly the U.S. economy. Sales during November and December can account for up to 40 percent of the annual revenue for store operators.
So far, there's reason to be hopeful. The National Retail Federation, the nation's largest retail trade group, expects holiday sales to increase 3.9 percent to $602.1 billion this year.
Separately, the U.S. Commerce Department said Wednesday that U.S. retail sales rose 0.4 percent in October. That was much better than the 0.1 percent increase economists had predicted, according to FactSet, a financial information provider.
"Consumers seem to be in a better mood, which will be good for the overall economy," Florance said.