Datasheet говорит, что :
"In case a [DLE] character needs to be transmitted as data, it should be preceded by another [DLE] character."
Т.е. пропатчить нужно не только ETX, но и DLE:
Код
void USART_TransmitDLE( unsigned char data )
{
if(data==DLE || data==ETX) {
while ( !( UCSR1A & (1<<UDRE1)) ); /* Wait for empty transmit buffer */
UDR1=DLE;
}
while ( !( UCSR1A & (1<<UDRE1)) ); /* Wait for empty transmit buffer */
UDR1 = data; /* Put data into buffer, sends the data */
}