Take a look at the local Ordnance Survey map (No.199 is the 1:50000 one) and you'll see loads of places to explore that are just a short hop by bus or car. A few miles along the coast to the east is Bexhill, which amongst its other attractions is probably best known for the De La Warr Pavilion, opened in 1935 and seen as one of the best examples of the International Modernist style. There's always something on here to excite the artistically inclined members of your party, and the distinctive architecture makes a visit doubly worthwhile.
Head north a few miles from the Harbour and you'll reach 15th century Herstmonceux Castle. Plenty of history to enjoy here, as well as the spectacular sight of what is the oldest brick building of any note still standing in England, surrounded by its beautiful moat. A summer's day spent here is a memorable event, and of course Herstmonceux won't miss the opportunity to sell you a tea and cake or two.
Herstmonceux Castle
Heading westwards from Sovereign Harbour, and beyond Eastbourne, the lovely village of East Dean is worth a trip, and just down the road you'll find the Seven Sisters Sheep Centre, with apparently one of the largest collections of sheep in the world: 40 different breeds including some of the older ones no longer seen on modern farms, plus of course child-friendly elements like "rabbits and chicks to feed and cuddle".
A closer look at the OS map will reveal a staggering array of footpaths splattering the map like tangled red cotton. A walk from the Harbour along the prom to Eastbourne is an obvious start which will take half an hour or more, but keen walkers will want to get their boots on and start at the other end of Eastbourne, where the walk up to Beachy Head is a must. Better still, take the car or bus a bit further to Birling Gap, from which a walk along the headland offers spectacular views of the Seven Sisters, and you can make it circular by heading inland after a while and once again finding East Dean, with its popular Tiger Inn pub.
Fabulous coastal views west of Birling Gap 
Walking east from the harbour you'll come fairly rapidly to Pevensey, and once more the red dots on the OS map will take you on a bendy route to the castle, crossing and following the lazy river......here you're certainly very much in what the county council insists on calling '1066 country'.
Back home there's plenty more walking to be done around the Harbour, and you can spend quite a while at the locks watching boats of all sizes come and go. These locks are mighty big, and operate 24/7 all year round.