Yes, it worth a trip over but I don't think there's enough to see to make it worthwhile stopping overnight, unless you want to take time to go on a boat trip or swim out to the blue caves.
Bear in mind the currency is Euros and the restaurants are quite a bit more expensive than, for example, Rhodes. Everything is brought in by the twice-weekly ferry from Rhodes, or one of the ferries from Kas.
I challenge you to climb the 400 odd steps up to the plateau and back again.
If you do it, you've got to post pictures of the view from the top as proof.