The biggest group the ancient Greeks could feel patriotic about was the polis, the city-state - that is, the city with its supporting countryside. Bigger than that just didn't feel real to most Greeks.
For some Greeks, even the polis didn't quite feel real. For them, only the family and tribe was real, and many city-states in times of stress broke down into factions along tribal lines. For most Greeks, though, patriotism meant loving, serving, and defending your city-state.
The Greeks knew they shared a language, and a religion, and a culture, and a homeland, but somehow that still wasn't enough for them to feel they were a single people, nor that they were a nation. When the Persians attacked the Greeks, they were able to unite to fight their common enemy, but whenever they weren't under that kind of pressure they tended to fall back apart into city-states. The idea of Greece, of all the city-states united and working together, just wouldn't stick with them for long, just couldn't compete with the simpler idea of one's own home polis.
As long as the relationship between people and polises remained Which side are you on? nobody could transcend the polis, and the history of Greece remained an endless civil war interrupted by periods of uneasy peace.
Although the Greeks had most of the makings of a mighty people, they couldn't stop fighting each other, and so their astonishing energies and innovations went into tearing each other down, leaving them vulnerable to conquest.
And indeed, eventually conquest found them, when Alexander the Great and his armies swept down from Macedonia and resolved all their conflicts for them.