One downside of moving from 32-bit to 64-bit is that memory pointers are twice their size. In other words programs in memory require more RAM. But does it matter in practice?
To find out I ran the 32-bit version of Ubuntu 14.04 from a USB stick and compared the used and available memory in different situations on the same system when booting from a different memory stick with the 64-bit version of the OS.