Gary Illyes from Google was asked on Twitter how SEOs and webmasters should measure their page speed. The question was "when ranking pages on speed does google base it on factors such as those in page speed insights or pure response speed?"
Gary responded, he'd look at both.
Here are the tweets:
@Cr21Craig I'll go with both.— Gary Illyes ᕕ( ᐛ )ᕗ (@methode) December 9, 2016
Now, of course, Google seems to use page speed insights for measuring this stuff but there many be something they don't share with us in terms of speed.
The overall thing - I believe - your rankings will only suffer if your site is very very slow. If you are average or even a little slow, I don't think you'd be hurt in terms of rankings.
Of course, your web site conversions may suffer if you are even a little slow - but rankings, that is a different story.
Forum discussion at Twitter.