How do I hook up surround sound to my Smart TV?

First, you need to establish whether your TV has the HDMI-ARC port. In Samsung TVs, this port is usually labeled ARC on the back of the TV. Now turn off your TV as well as the home theater system. Next, connect the HDMI cable to both the HDMI-ARC/output of your home theater, and the HDMI port of your TV.

What is audio out on Samsung TV?

The optical audio out looks like the original TOSLINK square port, which never carries any electrical connectors (unlike the 3.5mm round optical out which usually carries both optical and eletrical connectors. Your options are using a optical-to-analogue converter, or replacing the TV.