Looks tricky. What about current coming IN to neutral through the earth? If there is a fault next door for example your earth might actually be a higher voltage than the incoming neutral, assuming TN supply. And of course this could be changing all the time.
Other than equipment issues I can't think of anything else at the moment.
