Inside-out tracking of human body pose using wearable sensors has great potential in AR/VR applications, for instance, remote communication using 3D avatars with full-body expressions. Existing inside-out systems typically adopt vision-based methods with hand-held controllers or employ densely distributed body-worn IMU sensors. The former prohibits hands-free and occlusion-robust interactions, while the latter suffers from insufficient accuracy. We propose MI-Poser, a body tracking system with AR glasses and two wrist-worn electromagnetic fields (EMF) sensors that achieves high-fidelity pose estimation using deep neural networks for inverse kinematics. Our lightweight system achieves high accuracy while running efficiently at 60Hz. Moreover, by leveraging an IMU sensor collocated with the EMF sensor, we propose solutions to mitigate the effects of metal interference that corrupts EMF signal, an inherent issue in EMF tracking. Our EMF-IMU fusion approach demonstrates significant tracking improvement in environments with different metal profiles.