I had the same problem in my cave in GH. A couple of times, I came home to find staff jumping around in my yard. After that was fixed, the portal started to spawn further to the left, but I guess now it has reverted to its wallflower days.

I never eneded up filing a bug report because I just moved away from the portal, got a second invite, and when I clicked yes it took me right to the party with no additional portal-ness required.

I've been trapped this way. Now I try to remember to move away from the wall before accepting an invite!

Portal gun.