This is how my network, I have 2 routers and a cable modem. The first router is Linksys with phone port(which required to use VOIP service) connected to the cable modem, the second wireless router D-Link 524 is connected to WAN from the first router.

The internet runs smooth without Bit torrent, when i run bit torrent, with Maximum upload speed set to 5kbytes per second, the whole network will down, including the Wired LAN connection between a computer with the First Linksys router. It doesn't matter who use bit torrent, as long as people try to use bit torrent, the internet will runs really slow. What's the problem? Do it have to do with the QOS ?