Scarborough moved to South Carolina, arriving in 1836 in Charleston, where he would have faced competition with well-established painters. He spent the early years of his career as an itinerant, advertising in local newspapers and often staying with his clients while he painted their portraits.