To get there is Easy

A) Walk to the end of the pier, cross the road via the pedestrian crossing beside the Little Clock Tower and get a Taxi for less than a 10 minute ride.

B) Or catch a bus. Instead of crossing the road, turn left and stroll along the sea front with its busy little harbour passing the town church and The Albert Memorial on your right and arrive at the pretty little bus terminus amongst the trees
Get a no.5,6 or 7 bus for a leisurely 8-12 minute drive for only a £1. Ask the driver to drop you off at the Manor.

The Return Trip

All the busses 5a, 6a, and 7a, all take the same direct route back to the town in 10 mins and will pick you up at the gates of the Manor. This means that you have only a few minutes to wait as 7a  has 3 an hour and the other 2 make it up to 5 an hour during the day
Note during the term time busses 5a and 6a do the school run between 3 O'clock and 4 O'clock so there are only the three no 7a's running then.
A walk back along the coast for the fitter person will take around a couple of hours and is breathtakingly beautiful; but as the Path is very up and down in parts, it is only for the fit or the unhurried.
Or take the no 7 again for an enchanting 2 hour trip round the rest of the island back to the harbour for £1