We've talked about page speed here a lot, but page speed and server response time are really two different things. According to Google's documentation for their PageSpeed insights tool, they say "this rule triggers when PageSpeed Insights detects that your server response time is above 200 ms."
This specific tool measures how long it takes to load the necessary HTML to begin rendering the page from your server, subtracting out the network latency between Google and your server. If it takes over 200 milliseconds the tool triggers an issue.
Pranshu Kumar asked Google about this and Google said on Twitter "we don't define a fixed time requirement," for search but often Google uses "milliseconds" as "the unit used for this though," John added.
Here are the tweets:
We don't define a fixed time requirement. Milliseconds is usually the unit used for this though.
— Google Webmasters (@googlewmc) October 24, 2017
I wouldn't be concerned that your rankings will drop if it takes over 200 milliseconds here but it might not be good for your users.
Forum discussion at Twitter.
