I personally would not kick down a Sand Castle whether nobody is around or not and I also would not want to wait until the tide reaches it.
I would admire it and continue on with my day. :/
-
Only if I (or someone I know who's ok with it being destroyed) made it will I destroy it.