This is called flower abortion that are caused by various factors such as nutritional deficiency that is boron,zinc and potassium.
Also due to pest and disease infestation. Probably you can said a picture of the tree, the leaves and the fruits for better diagnosis.