The line at Gen 3:17, “Because thou hast hearkened unto the voice of thy wife,” makes it sound as if Adam is being punished for listening to his wife. Is that really correct?

Yes, but not, or not only, because he should have exercised some authority over her. His first and most important sin was that he should not have trusted anyone else, but instead used his own judgment, when it came time to decide whether to eat some fruit. Whether she persuaded him to sin, or simply negligently accepted fruit he should have inquired critically about, he placed both himself and Eve in danger of sin by simply accepting her judgment. After all, if his individual judgment had turned out different, he would have had an opportunity to overrule Eve. In addition, probably the text does imply that he had the right and responsibility, of the elder and stronger, to rule over the younger and weaker; but (see above) this was not made explicit until Eve’s punishment.