What i'd do is turn off the bluetooth on your iPod touch when you're not using it. If you don't do that it'll stay connected and slowly but surely drain itself and the wireless speakers.

Here's some other tips :

https://www.lifewire.com/improve-battery-life-ipod-touch-1999759

Also it might be worth checking to see if there's a software update for either device that may well fix the issue if it's a known bug.