Battle of Santa Rosa Island

Harvey E. Brown, a rock on which the Confederacy broke

Harvey Brown was born September 6, 1795 in Rahway, New Jersey. He was a disciplined and studious child, entering West Point military academy in 1814, and graduating in 1819, with a specialty in artillery. He was posted to Boston, New London Ct., and St. Augustine, Fl. He progressed rapidly through the ranks, achieving first lieutenant in 1821, where he attracted the attention of General of the Army Jacob Brown (no relation), and then was made Captain, participating in the First Seminole War. During his time in the north, he met and married Anne Eliza Rodman. Together they had five children – one was Dr. Harvey E Brown Jr., later a celebrated military physician. Anne must have had the patience of a saint, managing five lively children with Harvey’s long absences, never knowing for sure that he would return.

Harvey fought in the Black Hawk War, the Second Seminole War, a series of disputes on the US-Canadian border, and the Mexican War – like many Civil War officers. He is best remembered for his role as commander of Fort Pickens in the Battle of Santa Rosa Island.

Near the beginning of the Civil War, a priority for the new Confederate nation was to secure its ports, and assure a life line of goods shipping into and out of the South. Santa Rosa Island is a forty mile long spit of sand guarding the port of Pensecola, Florida, and only forty miles from Mobile, Alabama. The Confederacy was anxious to take Fort Pickens at this location, and make southern shipping safe. Harvey was just as determined that this wasn’t going to happen. He commanded the 5th United States Artillery, and the 6th New York Zouve Infantry volunteeers under Col. William Wilson was camped around the fort to repel invasion. As long as Fort Pickens remained in Union hands, the railroad terminal at Pensecola would be unable to ship valuable supplies to waiting Confederate troops. Unoccupied as late as December, 1860, the Union commander in the area recognized the strategic importance of Fort Pickens, and deemed it the easiest of the three forts in the area to defend. He moved his small force of sixty there with ammunition, and sent an SOS to Washington as hostilities with the South loomed.

The Confederates had a score to settle – they were looking for revenge for the Union attack on the privateer Jonah two weeks earlier. The Confederate commander, under Braxton Bragg, Col. Richard Anderson conceived a bold plan to land on three points of the island at once. The Confederates crossed over by steamers at night, October 9, 1861, surprising and overwhelming the 6th New York. Bragg seems to have withheld from Anderson that he didn’t think the fort could be taken – he wrote to his wife on Oct. 10 “that it was Billy Wilson and his crowd that I fondly hoped to destroy. “

In that, Anderson was successful – the 6th New York’s camp was mostly destroyed, and Wilson’s 220 men, outnumbered by an assault of almost 1000, were routed. To enable themselves to distinguish friend from foe in the darkness, the Confederates tied strips of white cloth around their left arms. The entire plan hinged on surprise, though, and as often happens, the human element causes a plan to derail. Elated by their easy victory over the 6th, the Confederates stopped to loot the camp, and lost precious time and momentum. By the time Anderson united his units and got them moving, it was near light, and he realized that the fort was alert and ready for them. With the dawn, the Federals would bring in gunboats, and his steamers back to the mainland would be the targets. It was time to go home.

The Confederates began their retreat, but Harvey sent a detachment in company strength, around 100, to support the 6th New York, under Major Israel Vogdes. In the dark, without intending to, Vogdes slipped past the Confederate artillery, and got between the Confederates and their steamers, cutting off their retreat. Heavily outnumbered, Vogdes fell back to the Gulf (south) side of the island, but was himself captured in the process. Harvey sent more reinforcements including artillery, and soon the Confederates found themselves flanked, with Union rifle fire sweeping the decks of the steamers. Anderson himself was wounded – a minie ball hit his arm at the elbow.

Harvey’s quick actions in sending the reinforcements and the bravery of Captain John Hildt, Vogdes second in command, had largely saved the day. Anderson and Bragg still claimed victory, in spite of not capturing the fort. Harvey Brown claimed victory as well. Anderson reported 18 killed, 39 wounded and 30 captured in his command. Harvey and the Union reported 14 killed, 29 wounded, 24 captured or missing.

The Union held Fort Pickens for the entire Civil War. On May 10, 1862, the last Confederate forces in Pensecola surrendered to Fort Pickens. Abraham Lincoln nominated Brown for appointment to the grade of brevet brigadier general in the regular army, to rank from November 23, 1861, for “gallantry and good conduct during the engagement between Fort Pickens and the rebel batteries”, and the United States Senate confirmed the appointment on March May 12, 1862.

Harvey got a post closer to home and family, becoming commander of New York Harbor in April 1862. In that capacity, he opposed the mayor of New York City in the draft riots of July 13, 1863. New York City had significant business ties to Southern cotton, and many in the city did not favor war with the Confederacy. After the battle of Gettysburg, Lincoln diverted troops to the city to handle the problem, with Harvey Brown in command. Congress passed the Enrollment act in March, 1863, to draft men to fight in the Civil War for the north, requiring every male citizen between twenty-five and forty-five to register for the draft. The law contained a provision that a draftee could pay $300 and get a substitute.

The riots started out as a protest of the draft, but rapidly degenerated into a race riot, Irish against blacks. The mob beat, tortured and/or killed numerous black people, including one man who was attacked by a crowd of 400 with clubs and paving stones, then lynched, hanged from a tree and set alight. Throughout the areas of rioting, mobs attacked and killed at least 120 black people and destroyed their known homes and businesses. Harvey Brown lacked sufficient men to contain the violence. The New York Times defended itself by setting its own Gatling guns in front of the building. Troops began to return from Gettysburg. By July 16, there were several thousand Federal troops in the city, and order was restored. But Harvey had had enough – He retired from active duty on August 1, 1863.

In 1866, Harvey again served briefly as head of recruitment for the US Army. He died at his home in Clifton, New York March 31, 1874.