If nothing else if friends could be ordered (or somehow marked) by how recently they played the game that would even be helpful. I don't know how many times I've been unfriended by people just because I send them requests from a game they no longer play, but haven't dropped/blocked even if we both play other games actively.
Or maybe have friends drop off the list if they haven't played in the last month?

It actually does affect game play as it's better to send requests to people who actively play the game(s) than to have them wasted going to someone who no longer plays (the giant in BW is one example of that).