I personally would not kick down a Sand Castle whether nobody is around or not and I also would not want to wait until the tide reaches it.
I would admire it and continue on with my day. :/
-
I would clear out the castle, and fortify it as a settlement.