D&C 121
I think it is vastly important to note that in his frustration here Joseph is turning to the Lord for answers. The feeling of frustration is neither evidence of nor cause for sin and despair. Instead, he turns to the Lord and supplicates for His intervention. That is the proper response to frustration – as opposed to my frequent response of getting sullen or starting to despair – and that is why he could have he Spirit with him so that the Lord could answer him.
Remember why so many are not chosen even after they have been called – “because their hearts are set so much on the things of this world . . . [that they do not learn how to exercise the priesthood properly].” All who have the priesthood have been called but only those who learn to exercise it in the way ordained by the Lord become chosen.