I'm not sure. We should test this.
I speculate the following:
Having all the pubKeys means a person can generate a local wallet which will give them view access to the wallet.
They don't have is the privKeys to any of the pubKeys. I am not sure if they would would be able to brute force their way to the privKey or Seed with this info. @Nikolai posted a useful article which I have yet to digest: