Butterfly Beach by the Four Seasons Resort The Biltmore Santa Barbara is pretty nice and quiet for Santa Barbara.  Just need to go when it is low tide.  Another recommendation for Santa Barbara County is Guadalupe Beach in the northwest part of Santa Barbara County.  This beach is a nature preserve for the Snowy Plover.  You will drive through a ranch, and the Guadalupe-Nipomo Dunes State Reserve to get to the beach.  It is very rustic and quiet.  There are usually a couple fishermen, maybe a surfer or two.  No lifeguards will be found at this beach.  There are amazing views and great waves.  It is not a recommended place for swimming as the currents here are very rough.

Darren,

If you're willing to walk up to a mile and carry your beach gear, I recommend Ellwood Beach (Ellwood Beach). This is between Coal Oil Point at the edge of Isla Vista and the Bacara Resort. Because it doesn't have a big parking lot or restaurants, it's lightly used. You'll see cool birds. And I often spot dolphins in the water. You could also stop by the nearby Monarch butterfly preserve. They'll be here a bit longer. Technically, this is Goleta, not Santa Barbara.

For Santa Barbara, I recommend Arroyo Burro Beach County Park. 101 to Las Positas. Turn away from the mountains and follow Las Positas until it comes to a T. Turn right at Cliff and then left into the parking lot. The Boathouse Restaurant is there if you'd like a cocktail or meal. From the parking lot, walk to the right for the people beach or to the left for the dog beach. You'll see LOTS of off-leash dogs to the left. The further you walk to the right from the parking lot, the more privacy you'll typically get. If you don't like dogs, just go to Butterfly Beach in Montecito by the Four Seasons Resort The Biltmore Santa Barbara. It's quintessential Santa Barbara.

Tip: No matter which SB beach you got to, bring baby oil and a rag to get the tar off your feet afterward.

Great suggestions already.  I will just add Loon Point Beach, just south of Summerland.  There is a small parking lot on Padaro Lane from which you walk along the railroad tracks then under the freeway off ramp, and down a short path to the beach.  Very scenic and quiet beach!  You'll sometimes see people riding horses here.

Hammond's beach, which is near Montecito. Small, compact, quiet, you have to walk to it via a beach path. Its off of San Ysidro Rd, before you get into Santa Barbara (from the South). Hammonds Beach.

Hendry's beach (also called Arroyo Burro Beach County Park) is my favorite.