lol, funny how you just proclaimed that poker was all about luck at my table and then I stumble upon this thread!
A hand in hold em is made out of 5 cards; the best possible combination of 5 cards made from both your hole cards and the cards on the board.
There were 2 pairs on the board, 55 and 66. This means that you both had 2 pair. Your 44 was meaningless here because it's a smaller pair than the other two, so it's not part of the best possible combination.
Considering you both have 2 pair, it's the kicker that makes the difference. The ace being the highest kicker, your opponent wins the pot.
If, for example, you were holding 77, then you would have won the hand with 2 pair, sevens and sixes, five kicker.