That’s a bunnybee!

What’s probably happening here is your adapter is signaling some button press when you connect it that’s popping up the Assistant. Only way to fix that is to get a new adapter since you can’t manually disable that input from triggering the Assistant without root.

However, you can completely disable the Google Assistant from appearing at all via:

Settings > Apps > Default Apps > Digital Assistant App > Default Digital Assistant App > None

That should fully disable it in all regards and it’ll never pop up again.