In the lobby, tap on the 'Friends' icon and then switch to the 'Game Buddies' tab. Tap on any of the game buddy's photo to see their profile. The profile will have a red button beside their name. Tap on that button to remove them from the game buddy list.