In a Google hangout video, at 18 minutes and 50 seconds in, Google's John Mueller answered a question around which desktop signals are used for mobile ranking as well.
John said that many are, including the page speed of your desktop web site, the rel canonical signal and many other signals are used for your mobile ranking. John said this may change over time as Google obtains more mobile signals through their crawling but for now, the page speed of your desktop site actually plays more importance than the page speed of your mobile site, when it comes to your mobile ranking.
Here is the video:
It is reported that in SMX Milan you told "Pagespeed of desktop version is used as ranking signal for mobile version. If desktop is fast enough and mobile is too slow, it doesn't affect ranking." Is it correct? I feel this is an odd decision.
I think at the moment, this is correct.
So we need do focus on the desktop page for the search results for the most part. That's also the one that you use with the rel canonical.
As we pick up more information from mobile friendly pages or from mobile pages in general, then I would expect that to flow into the rankings as well. So that's something to keep in mind there.
I'd still make sure that your mobile friendly pages are as fast as possible, that they work really well on mobile devices, that you're going past just essentially the required minimum that we had with the mobile friendly tool, and really providing a great experience on mobile. Because lots of people are using mobile to kind of make their decisions, to read content, and if your site is kind of minimally usable on mobile, but really a bad user experience, really, really slow, then that's something that users will notice as well and they'll jump off and do something else or go to a different site.
Forum discussion at Google+.