We know Google has an HTTPS Ranking Boost. Initially when it launched, all it checked was to see if HTTPS was in the URL, but it didn't check if it was valid or working. A couple months ago, Google began sending out notifications of SSL certificate issues, which shows they may start validating it before giving the ranking boost.
John Mueller said on Twitter that self-signed certificate, which is an identity certificate that is signed by the same entity whose identity it certifies, would typically be flagged as invalid by Google's detection tools. He said, if it works in a browser, than you are safe, if not, then not.
Bottom line is, if it doesn't generate an error in a browser, it should work for Google as well.
To be safe, don't go with self-signed certificates.
Forum discussion at Twitter.