What's the unit of Rosetta energy function?

Because Rosetta uses a mix of statistical and physical potentials (and score3 is mostly a statistical potential based scorefunction) it doesn't attempt to to match up with actual physical energy units (be it kcal/mol or kJ/mol or what have you). That's particularly the case with simplified scorefunctions like score3, which are used in early stages, and provide an approximated representation of the energy landscape to assist in fast sampling.

However, it sometimes turns out that, depending on system, you can often create a mapping of Rosetta energies to actual physical energies. Kellogg et al. (http://www.ncbi.nlm.nih.gov/pubmed/21287615) is a good example of that. The caveat here is that there are a number of different energy functions being used in Rosetta, and there isn't necessarily any consistency in the numbers which are output by them. I don't have evidence for it, but I personally suspect that even within a single energy function you'll have slightly different conversion factors between different experimental systems. So a conversion factor which is valid for monomer stability ddG will not necessarily be applicable to protein docking ddGs, although it might be close. (One reason I believe this is because the energy you get from a Rosetta simulation can vary depending on your sampling procedure - different sampling procedures can result in different energies, although standard procedures tend to be in a similar ballpark.)

Most of the time Rosetta energies are used as a "non-physical" metric. Lower Rosetta energies equate to more stable structures, even if you don't necessarily have an equivalence or even a known linear relationship between Rosetta energies and experimental values.

If you really want to convert Rosetta energies into experimentally equivalent values, the recommended procedure is to come up with a benchmark set of systems where the experimental value you're interested in is known. You then run the Rosetta protocol you're interested in on those systems, and match up the Rosetta-produced values with the previously determined experimental ones. You can then apply this conversion factor to your unknown system, keeping in mind that the conversion might change if you change the scorefunction used or the sampling procedure.