The pre-dawn operation was carried out at a compound known to Muslims as Noble Sanctuary and Jews as Temple Mount.
It was ordered by Prime Minister Benjamin Netanyahu after he convened his security cabinet for a second day’s talks.
They concluded with a vote to scrap the walk-through gates and replace them less obtrusive surveillance.
Local media reported that high resolution cameras capable of detecting hidden objects would be deployed.
Israel announced: “The security cabinet accepted the recommendation of all of the security bodies to incorporate security measures based on advanced technologies (smart checks) and other measures instead of metal detectors.”
It said the measure would “ensure the security of visitors and worshippers”.
The metal detectors were installed after Arab gunmen killed two policemen from inside the shrine, holy to both Muslims and Jews, earlier this month.
The 37-acre walled compound is the third holiest site of Islam, after Mecca and Medina in Saudi Arabia.
On Sunday, Israeli minister for regional development Tzachi Hanegbi was defiant the metal detectors would stay
“They will remain,” he said. “The murderers will never tell us how to search the murderers.
“If they (Palestinians) do not want to enter the mosque, then let them not enter the mosque.”
Palestinian President Mahmoud Abbas also said he would halt security ties with Israel until the gates were removed.