fix backbone design of interface residue

Given your resfile, it's not too surprising that you are getting very little or no sequence diversity in your output. Your resfile is fixing everything except ~18 amino acids on a helix.

The packer, which is used to do fixed-backbone design, can be very convergent if the search space is small enough. And the thing that expands the search space is not so much the number of residues, but the combinatorial assortment of residues. With just the 18 residues spread out along the helix, you have rather little combinatorial complexity. Each residue only sees a few residues on either side of it. In addition, the fixed residues surrounding the helix further cut down on the complexity, by limiting the types of amino acids and number of rotamers which can actually fit at each position. I'm guessing that you're only getting a few output sequences because the packer can efficiently search the possibility space, and runs into the same lowest energy structure each time.

If you're looking for diversity, there's several ways to do it.

The first is to expand the flexibility in the sidechains. Instead of having NATRO for the residues surrounding the designed residues, try making them NATAA. This will increase the flexibility of the designed pocket, allowing other possibilities to show themselves.

I'm not sure how much it will gain you, but you can also increase the rotamer sampling. You can add -ex3 and -ex4 flags, to also diversify the chi3 and chi4 rotamers, for those residues which have them. You could also increase how finely you diversify the rotamers. Right now you're diversifying the chi1 and chi2 rotamers by plus and minus one standard deviation. You can use the -ex_:level options to change that. For example, "-ex1:level 5" will sample at plus or minus 0.5, 1, 1.5 and 2.0 standard deviations. (The levels are slightly non-obvious. See the full options documentation https://www.rosettacommons.org/manuals/archive/rosetta3.5_user_guide/d5/... for details.) You could also add in some of the rare rotamers which are typically not sampled. The options "-dunbrack_prob_buried 1.0" and "-dunbrack_prob_nonburied 1.0" will add in all of the rare rotamers. (Note that these options, especially in combination, can greatly slow down your runs. Add them in slowly, and check that performance is not too compromised.)

Another option is forgo the diversification in the design phase, but attempt to put it into the preparation stage. Instead of having a single starting structure from which you generate multiple output models, have a set of input structures from each of which you only generate a few output models. There's a number of ways to get the input diversity, but one way is to use the relax protocol (particularly a constrained relax protocol if you don't want to move the structure too much) and use the multiple output structures. The slight diversity of backbones and sidechains in the starting structures may result in slightly different output structures for the fixedbb design portion.