Francois Fillon, 62, is currently seen as the favourite ahead of 71-year-old Alain Juppe to secure the nomination of the centre-right Republicans party.
Whoever wins the nomination is likely to face a showdown with far-right National Front leader Marine Le Pen in the general election next spring.
Ms Le Pen will be seeking to build on anti-immigrant, anti-Muslim and anti-establishment feeling in France to sweep her to power.
The current Socialist President Francois Hollande is expected to announce in the coming weeks whether he will stand for re-election.
His deep unpopularity has undermined the position of the country’s Left and there have been calls for his prime minister Manuel Valls to contest the party’s primary in January.
Mr Fillon, who is married to a Welsh woman, has enjoyed a boost in the polls in recent weeks.
He has promised to focus on tackling Islamic extremism and has promised to reduce immigration to France “to a minimum”.
Mr Juppe has promoted a more liberal stance with respect for religious freedom and ethnic diversity. He has attacked the “brutality” of his rival’s manifesto.
The pair also have opposing views on dealing with Russian President Vladimir Putin.
Mr Fillon is in favour of forging closer links with Moscow and dropping sanctions over the Kremlin’s actions in Ukraine.
He wants to join Russia in fighting against Islamic State and insists Moscow “poses no threat” to the West.
Mr Juppe wants France to continue putting pressure on Mr Putin.
Mr Fillon was the prime minister from 2007 to 2012 under President Nicolas Sarkozy, who was eliminated in the primary’s first round a week ago and now is supporting him.
Mr Juppe was prime minister from 1995 to 1997 under President Jacques Chirac.
In the first round of primary voting on 20 November, Mr Fillon won 44.1% of the votes, Mr Juppe 28.6% and Mr Sarkozy 20.7%.
A second round is being held because no candidate secured a majority.
All French citizens over 18 – whether they are members of the Republicans party or not – can vote in the primary if they pay €2 (£1.70) in fees and sign a pledge stating they “share the republican values of the right and the centre”.
Polling stations opened at 8am local time and will close at 7pm.
More than 4.2 million people voted in the first round.
Results are expected on Sunday night.