Robert was born a slave in Princeton, New Jersey in 1769 or 1770. His father was an English gentleman. At the age of four, he was given to his owner₂s son-in-law and moved to Georgetown, the District of Columbia. As an adult, Robert fell in love with a woman in Maryland and arranged for a friend to buy him, with the understanding that Robert would live as a free man and would pay back the friend, with interest. He married and had two children. Robert₂s friend betrayed him and sold him to slave traders in Charleston, South Carolina. Robert escaped once and was captured. He escaped a second time to Salem, Massachusetts, where he became a sailor. Over the next 20 years he sailed from Salem and then Providence, Rhode Island. He returned to Maryland to find his wife and children and found that she had committed suicide shortly after he was sold, and that his children had died. Robert returned to Providence, and became a hermit in a cave just across the state line in Massachusetts.