We love the water. Don't like guided tours... Enjoy nice food and would appreciate local specialty "cuisine".
The older, historic part of Annapolis is a wonderful place to walk and easy to see in just one day.
I usually park in one of two garages on Calvert Street, an easy left off of Spa Road near Church Circle. It's a convenient location and a great starting point for a walking tour. The Annapolis and Anne Arundel County Conference and Visitors Bureau is right there in that block of Spa Road. It has all the information for anything you may be interested in, friendly staff, and clean bathrooms.
From here, walk towards St Annes Parish House in the center of Church Circle. It's a beautiful old church that is often open for visitors.
From here, you can walk down the more touristy Main Street. This is where you'll find tons of shops from the cute boutique style to the tacky tourist stuff, several ice cream and fudge shops, and some good restaurants. One of the local favorites (and mine) is Chick & Ruth's Delly. It is small, old fashioned, and NOT FANCY AT ALL, but the food is great and it's just a good family-owned place. I love the pastrami on rye, but their Maryland Crab soup is really delicious and people rave about their crab cakes. Around here, crab is the local food and it's good.
If you go in this direction, you'll end up at Ego Alley which is where people park boats. To the left you can find the entrance to the United States Naval Academy which is free to enter and tour. The visitor's center is on this side. You just have to show id, and can enter and do your own tour or join a guided one. I usually enter at the end of Maryland Avenue and wander around there. That will get you closer to the Naval Academy Chapel which is absolutely gorgeous inside and out and has the U.S. Naval Academy Chapel & Crypt of John Paul Jones below.
I often end my day on Main Street and head toward the Maryland State House on State Circle first. It's also open and free to do your own short tour. It will only take you 5-15 minutes to see, but it is the first capitol of the US. The little streets off of this one are adorable and fun to walk through. Cornhill and East are residential. You'll get a feel for the history of the town. There are tons of old historical homes around here with many tours available. I'm not a fan of tours either, but grabbing a brochure at the information center will give you a glimpse of what you're seeing at least.
Maryland Ave, off of State Circle, has several antique shops and art galleries. They also have Galway Bay Irish Restaurant which is excellent. This street ends at my favorite Naval Academy entrance near the Chapel.
The Hammond Harwood House and William Paca House & Garden are both decent tours, but I think the latter is too expensive and too long. I think just a nice walk through the streets off of State Circle give you a good feel for Annapolis.
If you love the water, either catch a cruise from one of the tours off of Ego Alley or rent a kayak. The kayak rentals are in a different area at Truxtun Park, but it's an easy drive and you can kayak from there all the way into Ego Alley and get a whole new perspective. (I put more info and photos here.)
Mentioned in this answer: