changed browser.cache.memory.capacity value from -1 to 1000000 and changed media.cache_readahead_limit / media.cache_resume_threshold to 999999 but I didn't notice any difference. Rebuffering still happens.
What I did notice was that after hitting the replay button, if I press right arrow quickly enough to skip forward (40% or more), the buffer reppears and my network activity reflects that.
So it seems to me is that Firefox caches about 60% of the video (latter half) and when you replay it, it starts rebuffering the lost 40% of the first half of the video. I reverted my settings back to default values (the 3 prefs I mentioned above) and it still behaves this way.