The rendering engine itself obviously transforms and projects all triangles onto the screen in order to render them. I don't know if the N64 does this in software or hardware. Is it possible to access these transformed&projected coordinates from the emulator using lua script? That would be an easy solution.
If it's not possible, and you only have access to the enemy model 3D world coordinates and the camera parameters, you'll have to do the transformation&projection yourself.
It's difficult to give a concrete answer in code without knowing exactly what camera parameters are available. You'll probably have to give more detail about this.
But in general, for each point you want to project, you need to convert that point from world coordinates to camera coordinates (there are several possible ways of doing this, one of them is using a transformation matrix, which might even be directly in the camera data, depending on how the rendering engine works) and then project it to the screen (again, there are a couple of ways of doing this, one of them also being a transformation matrix, which might be in the camera data itself.) Essentially you would be doing a few point-matrix multiplications. What those matrices are exactly, it depends on how the camera parameters are defined in the game.
(Edit: Oh, and I forgot that there are a few more steps you need to do in order to clip coordinates that are behind the camera, else they would be mirrored in front of it. But this is relatively easy to do.)
