Rainbow Row and Slave Market.

Definitely take a tour of some of the old homes in the Historic District, e.g. the Edmondston-Alston House or Nathaniel Russell House.  Go for a long walk through the Historic District and just explore.  Take the ferry out to Fort Sumter National Monument.  I'd avoid Charles Towne Landing - it's not all that authentic, it's very touristy and if you go on a weekday it'll be inundated with kids on school trips. 

For food, definitely try to eat one meal at Husk or its sister McCrady's, although reservations are not easy to get.  If you have time, make the drive out to Bowen's Island Restaurant for oysters.  Gaulart & Maliclet Fast & French Inc. is a great place for breakfast or an inexpensive lunch