Schumer said the Port Authority of New York and New Jersey, which owns the 16-acre site, is responsible for any security breaches.
"They're the public authority in charge of this; I'm holding them responsible," Schumer said during a news conference on Sunday.
In a pair of statements Sunday, Port Authority security chief Joseph Dunne called the lapses "unacceptable" and invited Homeland Security officials to tour the site again "in light of recent security breaches."
Dunne said the Port Authority and the Durst Organization, which manages 1 World Trade Center and provides private security for the building still under construction, "have taken significant steps" to address the lapses.
David Velazquez, the assistant security director for Durst, resigned on Friday.
Dunne noted that the two reported breaches are unrelated. "Each involved different means of accessing the site," he said, without elaborating.
A "Phase 1" review of site security was submitted to DHS in November and is currently under review, Dunne said. A "Phase 2" application with measures including fences, closed-circuit television, perimeter patrol and access control is being prepared, he said.
Schumer was not satisfied.
"I do not feel we can leave it up to the Port Authority to say, 'Oh, we've corrected it, never mind,'" the senator said. "There have been too many breaches."
Schumer said he's pleased the New York Police Department is also participating in the security review and that Dunne, the PA security chief, is a former NYPD First Deputy Commissioner.
The New York Post published a photo Wednesday that showed a guard in a bright-green security vest leaning back in a chair, legs crossed and head tilted back. A Durst Organization spokesman told the paper the guard was fired immediately.
NYPD Commissioner Bill Bratton said the image of the sleeping guard is an "extraordinary embarrassment."
"You'll see security levels enhanced significantly on that site," Bratton told WABC television on Sunday.
The commissioner said police were looking for possible accomplices who may have helped the parachuters enter the high-rise.
"They didn't walk up," Bratton said. "They had assistance getting in and out of there, and we're continuing our investigations as to who helped them get up there."