How do you lose karma points when you havnt been downvoted?

If you open your user profile (by clicking on your username) there will be a tab called "karma history". This shows you where you most recently gained or lost any karma.

As far as I know, the only ways your karma points can go down is as follows:

• One of your posts is downvoted (-2)
• One of your posts is reported (-2)
• You downvote somebody else's post (-1)
• You award your points to another user

There is also a 24-hour window where a vote can be revoked; a user can upvote one of your posts and your karma would go up like normal, but they can then take that vote back and your karma would go back down to its previous level. Accepting answers may work the same, but I've not tested it.

And as Al Ummat says, it takes a while for changes to save through the system, so any karma gained or lost within the last half-hour to an hour probably won't be reflected properly.