ears back, listening to the rider!

I know I have heard it many times over the years, that if the horse has the ears back, he is listening to his rider. But is he, or is he listening for a threat? Dudley has had a girthing issue, just happened to take girthing the wrong way, and at this the stage we were at at overcoming this, I noticed that even while being lead around, no rider, just a saddle,lightly girthed, and bridle, that he spent most of the time with his ears, as though he was listening to his rider, but no rider.