[Solved]: Connect-SrmServer Unable to connect to the remote server

Summary:

Unable to connect to the remote server - SRM

Description:

Unable to connect to the srm server using powercli cmdlet. After connecting to vcenter server when I try to connect to the srm server it throws the below error,

PS C:\Users\rkalaiselvan> $srm = Connect-SrmServer -Port 443
Connect-SrmServer : 4/8/2020 9:10:37 PM Connect-SrmServer Unable to connect to the remote server
At line:1 char:8
+ $srm = Connect-SrmServer -Port 443
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Connect-SrmServer], VimException
+ FullyQualifiedErrorId : ConnectServer_UnhandledException,VMware.VimAutomation.Srm.Commands.Commands.ConnectSrmServer

When I debugged further, could see this exceptions,

PS C:\Users\rkalaiselvan> $exception = $error[0].Exception
$exception | select *
ErrorId : ConnectServer_UnhandledException
ErrorCategory : NotSpecified
TargetObject :
RecommendedAction :
SessionId :
ConnectionId :
Severity : Error
Message : 4/8/2020 9:10:37 PM Connect-SrmServer Unable to connect to the remote server
Data : {ParameterValues}
InnerException : System.NullReferenceException: Object reference not set to an instance of an object.
at VMware.VimAutomation.Srm.Impl.V1.SrmServerImpl.ConstructSrmServer(String srmServerId, SrmClient srmClient, String loggedInUser)
at VMware.VimAutomation.Srm.Impl.V1.SrmServiceImpl.ConnectSrm(SrmClientConnectAction connectAction, String hostName, Nullable`1 portOverride, String protocol, String username, CookieContainer cookieContainer, IServerCertificateValidationHandler certificateValidationHandler)
at VMware.VimAutomation.Srm.Impl.V1.RelatedSrmServiceImpl.<ConnectSrmServers>d__5.MoveNext()
at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
at VMware.VimAutomation.Srm.Commands.Commands.ConnectSrmServer.<>c__DisplayClass64_0.<ConnectCertificateValidationHandledb__0()
TargetSite : Void ConnectCertificateValidationHandled(System.Func`2[VMware.VimAutomation.Common.Interop.V1.Certificate.IServerCertificateValidationHandler,VMware.VimAutomation.Srm.Interop.V1.SrmServerInterop[]])
StackTrace : at VMware.VimAutomation.Srm.Commands.Commands.ConnectSrmServer.ConnectCertificateValidationHandled(Func`2 connectAction)
at VMware.VimAutomation.Srm.Commands.Commands.ConnectSrmServer.DoWork(VIAutomation client, List`1 moList)
at VMware.VimAutomation.ViCore.Util10Ps.BaseCmdlet.BaseCmdlet.DoWork(VIConnection connection, List`1 moList)
at VMware.VimAutomation.Sdk.Util10Ps.BaseCmdlet.BaseCmdlet.DoWorkEx(VIConnection connection, List`1 moList)
at VMware.VimAutomation.Sdk.Util10Ps.BaseCmdlet.ConnectionDispatchHelper.ExecuteRunlist(Dictionary`2 runlist,DoWorkDelegate doWorkDelegate,UidDispatchHelper uidDispatchHelper, List`1 result)
HelpLink :
Source : VMware.VimAutomation.Srm.Commands
HResult : -2146232832

Note: I have tried this in the latest powercli version 12

PS C:\Users\rkalaiselvan> Get-Module VMware.PowerCLI -ListAvailable
Directory: D:\Documents\WindowsPowerShell\Modules
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Manifest 12.0.0.... VMware.PowerCLI

PS C:\Users\rkalaiselvan> (Get-Module VMware.PowerCLI -ListAvailable).RequiredModules
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Script 12.0.0.... VMware.VimAutomation.Sdk
Script 12.0.0.... VMware.VimAutomation.Common
Script 7.0.0.1... VMware.Vim
Script 12.0.0.... VMware.VimAutomation.Core
Script 11.5.0.... VMware.VimAutomation.Srm
Script 12.0.0.... VMware.VimAutomation.License
Script 12.0.0.... VMware.VimAutomation.Vds
Script 12.0.0.... VMware.CloudServices
Script 12.0.0.... VMware.VimAutomation.Vmc
Script 12.0.0.... VMware.VimAutomation.Nsxt
Script 12.0.0.... VMware.VimAutomation.vROps
Script 12.0.0.... VMware.VimAutomation.Cis.Core
Script 7.12.0.... VMware.VimAutomation.HorizonView
Script 12.0.0.... VMware.VimAutomation.Cloud
Script 7.0.0.1... VMware.DeployAutomation
Script 7.0.0.1... VMware.ImageBuilder
Script 12.0.0.... VMware.VimAutomation.Storage
Script 1.3.0.0 VMware.VimAutomation.StorageUtility
Script 6.5.1.7... VMware.VumAutomation
Script 12.0.0.... VMware.VimAutomation.Security
Script 12.0.0.... VMware.VimAutomation.Hcx
Script 12.0.0.... VMware.VimAutomation.WorkloadMan...

Since I couldn't find any major issues from my end, I have filed a DCPN request to VMware (suspecting it to be a VMware bug)

Later I have received a reply stating to check the certificate

Logs:

Below is the SSL connection that seems to be failing :

2020-04-08T06:45:31.524Z verbose drconfig[05196] [SRM@6876 sub=ProbeSsl.Url.DrConfigSslCertificateManager] Established TCP connection to 'prod-srm-vcsa.dev.storage.com:443'
2020-04-08T06:45:31.528Z warning drconfig[05196] [SRM@6876 sub=ProbeSsl.Url.DrConfigSslCertificateManager] SSL client handshake to 'prod-srm-vcsa.dev.storage.com:443' failed.
--> N7Vmacore3Ssl18SSLVerifyExceptionE SSL Exception: Verification parameters:
--> PeerThumbprint: 28:2B:1A:5D:95:4E:96:9F:C1:7B:F4:5F:2E:9B:B8:F1:3E:46:BD:C7
--> ExpectedThumbprint:
--> ExpectedPeerName: prod-srm-vcsa.dev.storage.com
--> The remote host certificate has these problems:
-->
--> * unable to get local issuer certificate
2020-04-08T06:45:31.553Z verbose drconfig[05195] [SRM@6876 sub=Default.SiteConnections opID=1b61a273-8870-411b-abb8-f5d61b2bbef7-listVcServices] Connection to LS++ 'https://prod-srm-vcsa.dev.storage.com:443/lookupservice/sdk' created
2020-04-08T06:45:31.576Z verbose drconfig[05195] [SRM@6876 sub=Default.SiteConnections opID=1b61a273-8870-411b-abb8-f5d61b2bbef7-listVcServices] Search for service type 'com.vmware.cis.vcenterserver' returned 1 endpoints:
--> https://prod-srm-vcsa.dev.storage.com:443/sdk

I checked again and could confirm both the storage providers were online and in fact I have re-registered it again at the dr site but still the problem persists and I have shared them the new support bundles and screenshot for reference.

After few days, VMware team advised to try with 11.1 PowerCLI modules which I did try but no luck

Solution:

After a long back and forth communication with VMware team found that the powercli version I use is the actual culprit !

PS C:\Users\rkalaiselvan> Get-Module VMware.* -ListAvailable
Directory: D:\Documents\WindowsPowerShell\Modules
ModuleType Version Name ExportedCommands
---------- ------- ---- ----------------
Script 12.0.0.... VMware.CloudServices {Connect-Vcs, Get-VcsOrganizationRole, Get-VcsService, Get-VcsServiceRole...}
Script 7.0.0.1... VMware.DeployAutomation {Add-DeployRule, Add-ProxyServer, Add-ScriptBundle, Copy-DeployRule...}
Script 7.0.0.1... VMware.ImageBuilder {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare-EsxImageProfile, Export-EsxImageProfile...}
Manifest 11.1.0.... VMware.PowerCLI
Script 7.0.0.1... VMware.Vim
Script 12.0.0.... VMware.VimAutomation.Cis.Core {Connect-CisServer, Disconnect-CisServer, Get-CisService}
Script 12.0.0.... VMware.VimAutomation.Cloud {Add-CIDatastore, Connect-CIServer, Disconnect-CIServer, Get-Catalog...}
Script 12.0.0.... VMware.VimAutomation.Common {Get-Task, New-OAuthSecurityContext, Stop-Task, Wait-Task}
Script 12.0.0.... VMware.VimAutomation.Core {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAdapter, Add-VMHost, Add-VMHostNtpServe...
Script 7.9.0.1... VMware.VimAutomation.HorizonView {Connect-HVServer,       Disconnect-HVServer}
Script 12.0.0.... VMware.VimAutomation.License Get-LicenseDataManager
Script 12.0.0.... VMware.VimAutomation.Nsxt {Connect-NsxtServer, Disconnect-NsxtServer, Get-NsxtPolicyService, Get-NsxtService}
Script 12.0.0.... VMware.VimAutomation.Sdk {Get-ErrorReport, Get-PSVersion, Get-InstallPath}
Script 12.0.0.... VMware.VimAutomation.Security {Add-AttestationServiceInfo, Add-KeyProviderServiceInfo, Add-TrustAuthorityKeyProviderServer, A...
Script 11.5.0.... VMware.VimAutomation.Srm {Connect-SrmServer, Disconnect-SrmServer}
Script 12.0.0.... VMware.VimAutomation.Storage {Add-EntityDefaultKeyProvider, Add-KeyManagementServer, Add-VsanFileServiceOvf, Add-VsanObjectT...
Script 1.3.0.0 VMware.VimAutomation.StorageUtility Update-VmfsDatastore
Script 12.0.0.... VMware.VimAutomation.Vds {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHost, Export-VDPortGroup, Export-VDSwitch...}
Script 12.0.0.... VMware.VimAutomation.Vmc {Add-VmcSddcHost, Connect-Vmc, Disconnect-Vmc, Get-AwsAccount...}
Script 12.0.0.... VMware.VimAutomation.vROps {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, Get-OMAlertDefinition...}
Script 6.5.1.7... VMware.VumAutomation {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Compliance...}

From the above output it is clear that we are consuming 12.0 version as well this is a corrupt installation, so we need to try on a fresh 11.1 installation and in fact VMware team shared me a Save-PowerCLI module for installation but before that I must uninstall the existing modules,

(Get-Module VMware.PowerCLI -ListAvailable).RequiredModules | Uninstall-Module -Force

Get-Module VMware.PowerCLI -ListAvailable | Uninstall-Module -Force

or instead just delete the module folders from the location where they are installed, once done try,

Get-Module VMware.* -ListAvailable

Make sure no modules get listed here, then make use of the Save-PowerCLI module to install the desired version,

Save-PowerCLI -RequiredVersion 11.1.0.11289667 -Path <path to modules folder or save location>

Once this is done, tried Connect-Srmserver command and it worked like a charm,

PS D:\Documents\WindowsPowerShell\Scripts> Connect-SrmServer -Port 443
ServiceUri : https://prod-srm.dev.storage.com/vcdr/extapi/sdk
SessionSecret : "fc4bcbb63fbc762d712180f72b256c7294e4781a"
User : VSPHERE.LOCAL\Administrator
IsConnected : True
Port : 443
Version : 8.3.0
Build : 15928802
ProductLine : srm
InstanceUuid : e7b674a0-75b7-484a-2301-3dc755eada31
RefCount : 1
ExtensionData : VMware.VimAutomation.Srm.Views.SrmServiceInstance
Uid : /SrmServer=vsphere.local\administrator@prod-srm.dev.storage.com:443/
Id : /SrmServer=vsphere.local\administrator@prod-srm.dev.storage.com:443/
Name : prod-srm.dev.storage.com
IsInUse : True

Also the latest version of PowerCLI is not yet supported with 8.3 and hence will not work.

Whenever there is a release in future or when we want to use any particular version of PowerCLI with SRM, we can check for version compatibilities here:

https://www.vmware.com/resources/compatibility/sim/interop_matrix.php#interop&299=&505=

Thank you