Do you hear a chime before it pops up? Maybe the "talk to assistant" gesture is being invoked accidentally/unintentionally: https://support.google.com/assistant/answer/7513985?visit_id=638282594601460963-2105321338&rd=1#zippy=