The responses come as two states have legalized recreational marijuana, with more than 20 states and Washington D.C. allowing some medical use of the drug.
"It's just a matter of time before it's in more states," said Steve Pratley of Denver, a 51-year-old pipefitter who voted for legalization in Colorado in 2012.
Pratley, who did not participate in the Pew survey, agreed with 76 percent of respondents who said people who use small amounts of marijuana shouldn't go to jail.
"If marijuana isn't legalized, it fills up the jails, and that's just stupid," Pratley said.
Legalization opponents, however, drew a distinction between making pot legal for all and thinking that pot users belong in jail.
"It's an illegal drug, period. I don't see it spreading," said Laura Sanchez, a 55-year-old retiree in Denver who voted against legalization. She agreed that pot smokers don't belong in jail, but she disagreed with legalization.
"I've seen no proof that it's good for anybody," said Sanchez, who also did not participate in the survey.
The poll suggested that despite shifting attitudes on legalization, the public remains concerned about drug abuse, with 32 percent of those surveyed calling it a crisis and 55 percent of respondents viewing it as a serious national problem.
And a narrow majority, 54 percent, said that marijuana legalization would lead to more underage people trying it.
As for mandatory minimum sentences, public attitudes have been shifting for years.
In 2001, the survey was about evenly divided on whether it was a good thing or bad thing for states to move away from mandatory minimum sentences for non-violent drug offenders. In 2014, poll respondents favored the move by a nearly 2-to-1 margin, or 63 percent to 32 percent. The other 5 percent either didn't respond or said they didn't know.
Public officials are well aware of the public's shifting attitudes on drug penalties.
Just last month, U.S. Attorney General Eric Holder testified in support of proposed sentence reductions for some non-violent drug traffickers in an effort to reserve the "the harshest penalties for the most serious drug offenders."
"Certain types of cases result in too many Americans going to prison for too long, and at times for no truly good public safety reason," Holder said last month at the U.S. Sentencing Commission.