Я никак не пойму под какое волновое сопротивление рассчитывать линию под эти самые квадратуры.

Тут смотрели (апп. ноут, tutorial, описание отладочной платы, модели и проч.) ?
http://www.maximintegrated.com/datasheet/i...vp/id/5508/t/do

Ставите MAX4453 и получаете 50 Ом не дифференциальных. И потом, частоты там на выходе до 40 Мгц. И ...

Yes, the nominal differential impedance would be around 60ohms if
| you want to match the line impedance to the source. I would
| recommend adding a 20ohm resistor in series with each output and
| using 100ohm differential line if you really want to match the
| source and line impedance. This will enable you to use a more
| typical 100ohm stucture and reduce sensitivity to variation in
| the internal 30ohm resistance. Note that the load of the line
| cannot be terminated in 60ohms (or 100ohms). The MAX2112 is
| specified to drive a 2kohm load, so the far end of the line will
| be mismatched.
|
| If you simply keep the interconnect length short on the baseband
| outputs, then you do not need to worry about controlled
| impedances. In most cases, customer treat the output as a
| voltage mode output with a high impedance load. Just try to keep
| the traces reasonably short.
|
| Thank you,
| Maxim Support