The patrol boat Infanta Cristina was told to exit the area by the Royal Navy unit stationed in the small British overseas territory – home to just over 30,000 people.
A Gibraltar government spokeswoman said: "The ship entered British/Gibraltarian territorial waters. It was met by the Gibraltar Squadron and invited to leave."
She was unable to say if Gibraltar would make a formal protest to Spain over the incident.
The Spanish naval incursion follows the country's foreign minister Alfonso Dastis telling the UK to calm down after former Tory leader Lord Howard suggested Prime Minister Theresa May might be ready to go to war to keep Gibraltar British.
Tensions increased after an EU document suggested that Spain would be given a veto on post-Brexit agreements governing the Gibraltar.
This prompted Lord Howard to say he believed Mrs May would defend the Rock as Margaret Thatcher did the Falklands.
While the EU Brexit negotiating guidelines released by European Council president Donald Tusk last week made no reference to Spain's claim of sovereignty over Gibraltar, the decision to give Madrid a key role in deciding if a trade deal will apply to the Rock caused deep unease in Whitehall.
Incursions from Spanish ships in British territorial waters are a regular occurrence, as the Spanish dispute Gibraltar’s claim to the waters.
The Rock’s residents are patriotic supporters of their British identity, but questions were raised after Gibraltar voted 98 per cent for Remain in the Brexit referendum.
