Police say the attackers were later shot dead at the entrance to the Temple Mount/Noble Sanctuary complex – an area of huge religious importance to Muslims and Jews.
The gunmen had opened fire on the patrol near the Lion’s Gate area of the Old City before fleeing to the holy site.
Police spokeswoman Luba Simri told the Reuters news agency: “When they saw policemen they shot toward them and then escaped toward one of the mosques in the Temple Mount compound.
“A chase ensued and the three terrorists were killed by police.”
She also said that firearms were found on their bodies.
Dramatic mobile phone footage, which has been circulated widely on social media, shows several Israeli police chasing a man and then shooting him dead inside the compound.
The Israeli authorities are still trying to work out the identities of the attackers.
Israeli President Reuven Rivlin said: “The murderous terror attack at the Temple Mount was stopped by the courageous Israeli policemen who threw themselves in the way, and prevented an even greater disaster.
“We cannot allow for agents of murder, who desecrate the name of God, to drag us into a bloody war, and we will deal with a heavy hand against all the arms of terror, and its perpetrators.”
The incident is the second attack at the Old City in the past month and has resulted in the closure of the area for Friday prayers whilst searches and an investigation is carried out; a move which could raise tensions further.
Israeli Prime Minister Benjamin Netanyahu said the incident would not lead to a change in prayer rights at the site.
Security services are now on high alert, with the possibility of violent protests in the coming hours.
The Temple Mount/Noble Sanctuary site is frequently a place of conflict and tension.
It is managed by the Jordanian authorities and is adjacent to the Western Wall – the holiest prayer site in Judaism.
Israel annexed East Jerusalem, where the Old City and the holy sites are located, following the 1967 Six-Day War and now claims sovereignty over the entire city – a move that is not recognised by the international community.
This latest round of violence – sometimes referred to as the “Stabbing Intifada” – began in 2015 and although the attacks have become less frequent they have not stopped.
The perpetrators – who use knives, guns and even vehicles as weapons – often appear to be acting autonomously as “lone wolves”.
Israel blames the “terror” on incitement by the Palestinian leadership, but the Palestinian Authority claims Israeli military occupation is the driving force behind the violence.