For the first step enter setup and ensure #093 is set to USB, if not the radio will expect input from separate data port on the back when in RTTY mode.
Next adjust setting #096 to 10, the default of 50 is just way to hot and DM780 will report RX level overload. As is the case this setting and the input level on the microphone for the radio’s sound card will interact with this. Adjust both to get DM780 to a happy place RX level wise.
While in the menu system record the value in #065, if you followed my PSK setup instructions this will be set to DTR.