Neelix was awful, and probably made worse because I wanted to like him from the start.
Without getting into Kes - which compounded the Neelix problem for me - I started to hate him less in the last few seasons when he went all dark sad clown. I was still viscerally annoyed, but he turned into almost a Michael Scott type character. He was horrible, but you could see the the pain that made him that way.