People travelling into Scotland on international flights have to self-isolate in a hotel for 10 days as of Monday, while in England that only applies to travellers from 33 "red list" countries.
It has led to accusations of a "loophole" that could allow visitors to arrive in England and then travel up to Scotland to avoid quarantining in a hotel.
On Monday, Sturgeon was asked if she would shut the border to prevent travellers getting around the quarantine.
"I don’t rule it out," the Scottish first minister told a press briefing.
"I would like to have the most effective system in place here. I'm not sure it is the best solution... if it's the only one I can put into place then it may come to that."
However, it is unclear how a border closure would be enforced.
Watch: UK Hotel Quarantine Begins - What Are the New Rules?
Anyone arriving in Scotland has to pay £1,750 to quarantine in one of six hotels in Aberdeen, Edinburgh and Glasgow, unless they are exempt.
But Scottish transport secretary Michael Matheson said on Sunday that the differing policies between Scotland and England create a "loophole".
It could allow someone who is not in one of England's 33 red list countries to arrive there without having to go into a hotel and then travel up over the border.
They would still have to self-isolate for 10 days but could avoid doing so at a designated hotel.
Matheson said it would "potentially undermine the public health approach here in Scotland".
Sturgeon added: "It would be better if we had that four nations approach, or at least a three nations approach where the border of the island that Scotland, England and Wales share had the same provisions in place.
"My responsibility is to try and protect Scotland as much as possible so if it just proves impossible to do that then we have to look at alternatives, but it would be far better to try to come to a reasonable agreement on a four nations, or at least a three nations, basis."
UK health secretary Matt Hancock said he would be happy to discuss the issue with the Scottish government.
The first international flight subject to Scotland's new rules arrived from Istanbul on Monday morning, carrying only five passengers instead of the planned 65 due to departure issues.
Last summer, Sturgeon also refused to rule out introducing quarantine for travellers from the rest of the UK if doing so was supported by health evidence, sparking a war of words with Westminster.
Watch: Nicola Sturgeon tells footballers - ‘make sure your house is in order’