treasury room puzzle

I'm in the treasury room, at the puzzle where blocks are moved to match colored blocks to the colored buttons on top of three columns. The blocks either don't move, move after 20 or 30 tries, or in a direction that I don't want. How do I control these blocks? they won't slide, or respond to clicking on the blocks.

You have to hold a piece, then drag slightly to the direction you want it to go. It's sometimes hard to control and annoying.

Basically, just take it slowly.