Sounds like interference to me too, especially given the time aspect; people get home from work and start turning things on...
wifi can have a pretty decent range on it so it might not just be the immediate neighbors (given the right antennas and nothing in the way you can get 100s of miles, granted that's probably not the case with your problem though

)
Could also be anything else in the 2.4GHz spectrum too - Bluetooth devices, cordless telephones, microwave ovens(!), wireless security cameras, cordless video extenders, etc. etc. -- anything that transmits and doesn't require a license is likely in the same area of the spectrum.
Have you tried setting the wifi base station to use a different channel, just to see if it helps ? You could try relocating it too.
Course if it's a real problem you could just get a couple of mates, give 'em RF power meters with directional antennas, triangluate the, er, problem area, and pop over there with a non-negotiable offer to reconfigure the offending device using the DIY tool of your choice; hammers tend to fix most of these problems.