Down at the southern tip of Manhattan you find Battery Park. Inside the park you can see one of the forts which used to protect Manhattan against attacks from various enemies of the USA. The location of Castle Clinton is only a couple of blocks west of the old Fort Amsterdam.
Originally the fort was built on a small artificial island to the south west of Manhattan and was designated as the south west battery. Thanks to landfill the fort is now located on land inside Battery Park. Even if you have no interest in the fort you might still go here to buy your ticket for a trip to Liberty Island and Ellis Island.
The construction of the fort in 1808 and was finished in 1811 which was just in time for the war against Britain in 1812. But the fort never saw any action in this war or any other war. Already in 1821 the US Army stopped using the fort as a military base.
It had many different uses after the military use of the building. It was a beer hall, theater and aquarium. More interestingly it was the first immigration office in New York from 1855 and 1890 predating Ellis Island and the location where the immigrants would go to America.
Today most people might go to the fort to get the ticket for a cruise to Liberty Island. But if you go to lower Manhattan you can easily go in and have a look at the old building since it is close by the last metro stop on Manhattan Island before the subway goes across the water to Brooklyn.