Please recommend good books on this topic.

This could be good start:
http://www.amazon.com/Microelectronics-Pre...s/dp/0137134738

I need something more advanced.

And for advanced readers:
http://www.amazon.de/System-Design-Transce...s/dp/0387241612

Do you have some software to recommend.

Agilent systemvue is very good choice.

Thank you!

It has very good interface to measurement equipment and Golden Gate simulator.

How it can be used with GG?

You can produce Fast Envelope model using GG and than simulate your block in SV on system level.
Have a look:
https://www.youtube.com/watch?v=_i2OyBvPjTI

Wow thanks. This is great.

Yes, this is the most complete design flow for on chip communication system design.
It is very fast.