A correctly fitting jacket should button easily without pulling or creasing, and without the lapels bowing outward from the body. Sleeves, which are a subtle signifier of a correctly (or incorrectly) ...
Besides smooth woven fabric, you also have the option of wearing more casual knit shirts. These can be polos (in short or long sleeves), henleys, quarter zips and turtlenecks. All of these look fine ...