Mr Fillon had won 67% of votes and Mr Juppe was on 33%, according to results from about 90% of polling stations.
The pair, both former prime ministers, were vying to become the centre-right Les Republicains party candidate in the election.
Speaking after his victory, Mr Fillon, 62, said: “I must now convince the whole country our project is the only one that can lift us up.
“My approach has been understood: France can’t bear its decline.
“It wants truth and it wants action.
“I will take up an unusual challenge for France – tell the truth and completely change its software.”
In Paris, 71-year-old Mr Juppe congratulated Mr Fillon on the “large victory”, adding: “I finish this campaign as I began it – as a free man who did not compromise what he is or what he thinks”.
He called for unity and calm after a campaign during which he had accused Mr Fillon of pandering to anti-immigrant and anti-Muslim feeling.
Mr Fillon is now likely to face a spring showdown with far-right National Front leader Marine Le Pen, who will be seeking to build on that same anti-immigrant, anti-Muslim and anti-establishment feeling.
Ms Le Pen has also promised to lower the retirement age and guarantee France’s welfare safety net.
A Harris Interactive poll published on Sunday night showed Mr Fillon would beat Ms Le Pen by 67% to 33%.
The victory for Mr Fillon, who is married to a Welsh woman, comes against a national unemployment rate of 10%, weak economic growth, worries about immigration and globalisation and concern about the future of a costly but valued welfare state.
In response, Mr Fillon has proposed spending cuts, increasing sales tax, scrapping a tax on the wealthy, fewer restrictions on the working week and raising the retirement age to 65.
He also wants to limit the adoption rights of gay couples, push for closer ties with Russia and focus on tackling Islamic extremism and reducing immigration to France “to a minimum”.
Mr Juppe had promoted a more liberal stance with respect for religious freedom and ethnic diversity, attacking the “brutality” of his rival’s manifesto.
Mr Fillon, whose wife Penelope Clarke is British, was the prime minister from 2007 to 2012 under President Nicolas Sarkozy, who was eliminated in the primary’s first round a week ago and now is supporting him.
Mr Juppe was prime minister from 1995 to 1997 under President Jacques Chirac.
In the first round of primary voting on 20 November, Mr Fillon won 44.1% of the votes, Mr Juppe 28.6% and Mr Sarkozy 20.7%.
A second round was held because no candidate secured a majority.
All French citizens over 18 – whether they are members of the Republicans party or not – can vote in the primary if they pay €2 (£1.70) in fees and sign a pledge stating they “share the republican values of the right and the centre”.
Socialist Party candidates now have to announce their intention to run before 15 December and their first primary is held on 22 January.
If no candidate gains more than 50%, there is a second round a week later.
The general election is set for 23 April, with a possible second round of voting on 7 May.
The current Socialist President Francois Hollande is expected to announce in the coming weeks whether he will stand for re-election.
His deep unpopularity has undermined the position of the country’s Left and there have been calls for his prime minister Manuel Valls to contest the party’s primary.