Job you love for a little money, or a job you hate for a lot

Yesterday, I was telling my younger brother about the Andre Agassi book I read.  Agassi was a tennis player that absolutely hated the game.  He was very good and made millions of dollars.  He retired at a very young age, as most athletes do.  He is obviously set for life with all of the money he made.  I asked my brother if he would rather work in a job he loved to do for a little bit of money, or if he would rather be miserable at work making a ton of money.  He said that he would rather work at a job he hated for a ton of money because then he could retire young.  I would much rather work at a job I am happy in rather than a job I hate.  I'm sure that I will be able to live off of my teacher salary with no problems.  Money doesn't buy happiness.  Naturally, my brother argued the point with me and said I was wrong (because obviously every view he has is right).  How many stories have you heard of people winning the lottery and being broke shortly after?  Also, if you win the lottery, everyone will be asking you for money.  Lots of people that have a lot of money are not necessarily happy.  I would much rather be happy with less money than miserable with a lot of money.  I really think that most people would feel the same way as me.  I could be completely wrong, though.