QoS with PfSense

Alright heres our setup.

We are running Trixbox 2.4.

There are approx 20 ip phones in the building. Including Aastra 57i's, Polycom 4000's, and softphones.

We use SIP protocol with Vitelity for all of our incoming and outgoing calls.

We have several SIP trunks setup between our other locations "also using trixbox 2.4-2.6".

We have a 3/3 internet connection comming into the building.

We have 2 gigabit switches, one for data and one for phones, both are D-Link only the phone switch is PoE.

We are using Pfsense as the router. We only have 2 Nic's int he pfsense. Lan and Wan.

We preformed several speed test to verify that we were reciving 3/3 or better.

We enabled traffic shaping on the pfsense. We set our bandwidth up as 2500/2500. We want VoIP to have a min of 1024Kbps available at all times.

Here is how our queue setup
########################
WAN->LAN UDP * 192.168.1.164 qVOIPDown/qVOIPUp VOIP Adapter
LAN->WAN UDP 192.168.1.164 * qVOIPUp/qVOIPDown VOIP Adapter
########################
"192.168.1.164 being our Trixbox"

In our Service Curve, we have the following set for each voip queue
Real Time: m1=1500Kb d=5000 m2=1024kb

Note that I am in a different office
That being said, I call up a coworker to test the call "via interoffice". This is going over standard SIP. I start a speed test and I begin counting slowly to him.

While the speed test in running the "Download" test, he hears every number perfect. However, once upload begins he says I become very choppy, not even heard some numbers.

When we have him count, I hear everything perfect.

This is a little confusing to me. Since im calling into the system, shouldn't the upload affect what I hear?

How can I make this QoS better?

"update"
I have noticed this when looking at the traffic graph when I run the speed test. When upload starts, Download dies horribly. see attached photo.

http://speekgeek.com/images/see.JPG