A built-in safety feature returned the Bluefin to the surface and it was not damaged, they said.
The data collected by the sub was later analyzed and no sign of the missing plane was found, the U.S. Navy said.
Crews shifted the search zone away from the deepest water before sending the Bluefin back for Tuesday's mission, the U.S. Navy said.
Search authorities had known the primary search area for Flight 370 was near the limit of the Bluefin's dive capabilities. Deeper-diving submersibles have been evaluated, but none is yet available to help.
A safety margin would have been included in the Bluefin's program to protect the device from harm if it went a bit deeper than its 4,500-meter limit, said Stefan Williams, a professor of marine robotics at the University of Sydney.
"Maybe some areas where they are doing the survey are a little bit deeper than they are expecting," he said. "They may not have very reliable prior data for the area."
Meanwhile, officials were investigating an oil slick about 5,500 meters (3.4 miles) from the area where the last underwater sounds were detected.
Crews collected an oil sample and sent it back to Perth in western Australia for analysis, a process that will take several days, said Angus Houston, the head of the joint agency coordinating the search off Australia's west coast.
He said it does not appear to be from any of the ships in the area, but cautioned against jumping to conclusions about its source.
The submarine is programmed to take 24 hours to complete each mission: two hours to dive to the bottom, 16 hours to search the seafloor, two hours to return to the surface, and four hours to upload the data.
The Bluefin can create a three-dimensional sonar map of any debris on the ocean floor. But the search is more challenging in this area because the seabed is covered in silt that could potentially cover part of the plane.
"What they're going to have to be looking for is contrast between hard objects, like bits of a fuselage, and that silty bottom," Williams said. "With the types of sonars they are using, if stuff is sitting up on top of the silt, say a wing was there, you could likely see that ... but small items might sink down into the silt and be covered and then it's going to be a lot more challenging."
The search moved below the surface after a U.S. Navy device towed by an Australian ship detected underwater sounds consistent with signals from an aircraft's black boxes, which record flight data and cockpit conversations. The devices emit "pings" so they can be more easily found, but their batteries last only about a month and are now believed to have failed.
Australian Prime Minister Tony Abbott raised hopes last week when he said authorities were "very confident" the signals were from the black boxes on Flight 370, which disappeared March 8 during a flight from Kuala Lumpur, Malaysia, to Beijing with 239 people on board, mostly Chinese.
Houston said the signals were a promising lead, but that finding aircraft wreckage in the remote, deep patch of ocean remains extremely difficult.