how do i unding properly?

Undocking a soluble protein is much simpler than docking it. Basically, all you have to do is take all the residues you want to undock and translate them away from the others. Typically we do this by just translating them some really large displacement.

Since you're in PyRosetta, the simple, naive way to do it would be simply to iterate through all the atoms in the protein and add some large, fixed offset (e.g. [200,200,200]) to their coordinates. This way they'll no longer be bound, and will be far enough apart that they won't interact.

The way it's typically done within Rosetta itself is by finding the FoldTree jump where all the residues are downstream, and then using something like the protocols.rigid.RigidBodyTransMover. The constructor takes a core.Vector of the translation axis and the jump number you want to translate. Then you'd also want to use the step_size() method to set the size of the step (the vector only controls the direction, not the magnitude).  The apply() will then move eveything downstream of the jump in the direction of the vector, with the displacement specified by step_size().

The interfaceAnalyzerMover is quite straightforward to use here is a snippet (from here)

# given
# pose: pyrosetta.Pose
# interface: str
ia = pyrosetta.rosetta.protocols.analysis.InterfaceAnalyzerMover(interface)
ia.apply(pose)
print({'complex_energy': ia.get_complex_energy(),
       'separated_interface_energy': ia.get_separated_interface_energy(),
       'complexed_sasa': ia.get_complexed_sasa(),
       'crossterm_interface_energy': ia.get_crossterm_interface_energy(),
       'interface_dG': ia.get_interface_dG(),
       'interface_delta_sasa': ia.get_interface_delta_sasa()})

However, it is very blackbox and if you have say a protein that wraps around something via a normally disordered loop (for example cadherin, TFs etc.), you may want to customise what/how you repack for example allowing backbone movement doing what Roco said. Here is a snippet that does that (from same link).

class ModVariant(Variant):
    """
    This reintroduces the old way of ∆∆G interface calculation.
    """
    
    def score_interface(self, pose: pyrosetta.Pose, interface: str) -> Dict[str, float]:
        if pose is None:
            pose = self.pose
        assert self.has_interface(pose, interface), f'There is no {interface}'
        shift_pose = pyrosetta.rosetta.core.pose.Pose()
        shift_pose.assign(pose)
        # Shift
        chains = interface.split('_')[1]
        self.shift_chains(shift_pose, chains)
        #self.repack_interface(shift_pose, chains)
        # Score
        scorefxn = pyrosetta.get_fa_scorefxn()
        holoscore = scorefxn(pose)
        aposcore = scorefxn(shift_pose)
        #     print('holoscore', holoscore)
        #     print('aposcore', aposcore)
        #     shift_pose.dump_pdb('shifted.pdb')
        #     pose.dump_pdb('unshifted.pdb')
        #     raise ValueError
        return {'holo_dG': holoscore,
                'apo_dG': aposcore,
                'interface_dG': holoscore - aposcore,
                'holo_pose': pose,
                'apo_pose': shift_pose}


    def shift_residue(self, pose: pyrosetta.rosetta.core.pose.Pose, r: int) -> None:
        """
        inspired from https://graylab.jhu.edu/pyrosetta/downloads/scripts/demo/D090_Ala_scan.py

        :param pose: changed in place
        :param r: pose numbering
        :return:
        """
        xyz = pyrosetta.rosetta.numeric.xyzVector_double_t()
        xyz.x = 500.0
        xyz.y = 0.0
        xyz.z = 0.0
        for a in range(1, pose.residue(r).natoms() + 1):
            pose.residue(r).set_xyz(a,
                                    pose.residue(r).xyz(a) + xyz)


    def shift_chains(self, pose: pyrosetta.rosetta.core.pose.Pose, chains: str) -> None:
        """
            inspired from https://graylab.jhu.edu/pyrosetta/downloads/scripts/demo/D090_Ala_scan.py

            :param pose: changed in place
            :param chains: iterable of chain
            :return:
            """
        pdb2pose = pose.pdb_info().pdb2pose
        for chain in chains:
            for i in range(1, 999):
                r = pdb2pose(res=i, chain=chain)
                if r != 0:
                    self.shift_residue(pose, r)


    def repack_interface(self, pose: pyrosetta.rosetta.core.pose.Pose, interface: str) -> None:
        scorefxn = pyrosetta.get_fa_scorefxn()
        # packer_task = pyrosetta.rosetta.core.pack.task.TaskFactory.create_packer_task(pose)
        # packer_task.restrict_to_repacking()
        # pyrosetta.rosetta.core.pack.pack_rotamers(pose, scorefxn, packer_task)
        operation = pyrosetta.rosetta.core.pack.task.operation
        allow = operation.RestrictToRepackingRLT()
        chainB_sele = pyrosetta.rosetta.core.select.residue_selector.ChainSelector(interface.split('_')[1])
        not_chainB_sele = pyrosetta.rosetta.core.select.residue_selector.NotResidueSelector(chainB_sele)
        sele = pyrosetta.rosetta.core.select.residue_selector.InterGroupInterfaceByVectorSelector(chainB_sele, not_chainB_sele)
        sele.nearby_atom_cut(6)
        sele.cb_dist_cut(8)
        restrict_to_focus = operation.OperateOnResidueSubset(allow, sele, True)
        tf = pyrosetta.rosetta.core.pack.task.TaskFactory()
        tf.push_back(operation.PreventRepacking())
        tf.push_back(restrict_to_focus)
        packer = pyrosetta.rosetta.protocols.minimization_packing.PackRotamersMover(scorefxn)
        packer.task_factory(tf)
        packer.apply(pose)
        print('done')

Sorry for not cutting it up, but I think it's fairly intuitive. The main thing is the shift_residue method and residue_selector in the last method.

If you are using membranes x translation is still membrane bound if in OMP orientation.

The translation does not work for DNA or ligands, so make use the interface peptide first (say `AB_C` iwhere AB is the protein).