I use the ROSIE server, and I want to know, how can I calculate the free energy in my new models of protein-protein docking?
Rosetta typically works in arbitrary energy units (commonly known as "Rosetta Energy Units" or REUs, though those values aren't necessarily numerically consistent from protocol to protocol).
If one were to attempt to convert those values into a wet-lab-measurable unit, the way to do it would be to run the particular protocol you're using against a set of similar systems with known values. (e.g. a set of protein-protein interactions with known binding affinity.) You can then make a graph of experimental energies against the Rosetta predicted values, and derive a conversion function (as well as a sense of likely accuracy) from the correlation. The more similar your reference cases are to your desired experimental case, the more likely the conversion is going to be applicable.
Keep in mind that predicting experimental energies can be difficult. Off hand, I can't recall anyone doing it for protein-protein interaction, though I do know that Kellogg et al. did it for monomer stability prediction (http://www.ncbi.nlm.nih.gov/pubmed/21287615) and Meiler et al. have done it for protein-small molecule interactions (http://www.ncbi.nlm.nih.gov/pubmed/16972285).
You can look at the graphs in those two papers to see what sort of accuracy you're likely to get. Generally, you do much better with qualitative metrics ("is this conformation/mutant better than this other one" or "is this going to be a good binder, a weak binder, or a non-binder") than you do with quantitative how-many-kcals-type predictions.