I think you a both right. Historically, Semites referred to a large cultural group.
Over time, it has become a nonsense word because those cultural groups have become so dilute and diverse that you can’t point at someone and say they are part of that group.
More recently, the label has become misappropriated by some sort of whacky religious nutbaggery so they can oppress other people.