That’s definitely not how it should work. You do need to be generally in front of the Game Frame, but the buttons should work 100% consistently and from up to 15-20 feet away.
Can you do me a favor and shine a light at the center top of the front screen? You should see the IR receiver there through a hole in the black grid. There should also be no white vellum visible in the IR port.
I’ve attached an example photo. It’s from a prototype where the port doesn’t exactly line up with the interior but you get the idea.