Can I run Trilian side by side with Trilogy?

Trilian and Trilogy can be run side by side, since they are separate instruments. Keep in mind that Trilian is a completely new instrument that also includes the Trilogy library. Trilogy is no longer supported on the newest operating systems and hardware, so may not perform as expected.

Do I need to uninstall Trilogy before installing Trilian?

No, this isn't necessary unless you are no longer going to use Trilogy. All of Trilogy is included in Trilian, so you can uninstall if you like. However, it's useful to keep Trilogy installed to recall older projects.

Does it replace Trilogy on those sessions with all of their settings automatically?

Trilian will not automatically replace Trilogy in older sessions. You can reselect the same Trilogy patches in Trilian, since it includes the original Trilogy library and patches.

Will I be able to open custom patches that I made in Trilogy?

No. Since Trilian is a completely new instrument, it's not possible to open custom saved Trilogy patches in Trilian.
However, you will be able to recreate your custom patches with much more choice and flexibility, and also trade and share patches with other users! This is possible because Trilian is based on our own technology, and will allow for song compatibility with future updates, just like all the updates with Omnisphere and RMX have been song compatible since their introduction.