Normally it is US to Canada that takes a long period of time. Canada to US being slow is actually rarer but it does happen. The longest was 2-3 weeks for me.
Canada has a slower processing system so US to Canada is slower. When it enters Canada it takes them forever to clear customs, processed, and sent out. However, the only good thing about Canada is every province has a large enough city and is centralized enough for their own international sorting facility. US is different, they have specific ones. I know of packages traveling across US states just to be processed and sent out internationally. Fortunately, they do it a lot quicker than CP.
One thing I noticed is that it truly matters what city you send your item in. Here in Vancouver, the best place to send stuff internationally is actually in Richmond. We ship all Air here due to geographic reasons and Richmond is actually where the airport is located. It saves me about 1-2 days in delivery time. It gets processed and shipped the same day I mail cause they take all the US mail send it to the airport and they are off. Vancouver mail has be processed in Vancouver, then goes to Richmond, sort again and then go out.
