I am totally ignorant of code but a thought I had was maybe there would be a way to alter the current shout box to include a dropdown menu like the one for 'send message to' that lists all the players and then the conversation would just appear in the shoutbox but not have the pop-up. I suppose that would also mean that the area where the team and public shouts are listed would also have to be adjusted to include a private area.
If totally impossible, just keep in mind that I know nothing about any of the processes of game making