Unable to install ServiceControl error instance remotely using powershell New-ServiceControlInstance

[PhilBastian]

Describe the bug

Description

When attempting to remotely install ServiceControl error instances using Invoke-Command (WSManagement), the attempt fails with different errors, depending on the method used to supply the installation script.

Method 1: Directly supply the script in -ScriptBlock

Invoke-Command -Session $session -ScriptBlock {
Import-Module -Force "Particular.ServiceControl.Management" -RequiredVersion 6.6.0
$Script:ErrorRetention = New-TimeSpan -Days 10
New-ServiceControlInstance -Name "Particular.ServiceControl" -InstallPath "C:\Program Files (x86)\ParticularSoftware\Particular.ServiceControl" -DBPath "C:\ProgramData\Particular\ServiceControl\Particular.ServiceControl\DB" -LogPath "C:\ProgramData\Particular\ServiceControl\Particular.ServiceControl\Logs" -Port "33333" -DatabaseMaintenancePort "33334" -Transport "RabbitMQ.QuorumConventionalRouting" -ErrorQueue "Errors" -ErrorRetentionPeriod $ErrorRetention -ConnectionString "host=localhost" -ServiceAccount "LOCAL SYSTEM" -HostName "localhost" -Acknowledgements "RabbitMQBrokerVersion310","RabbitMQManagementApi" -ErrorAction SilentlyContinue -Force
}

The error in this case is

OperationStopped: Processing data for a remote command failed with the following error message: The WSMan provider host process did not return a proper response. A provider in the host process may have behaved improperly. For more information, see the about_Remote_Troubleshooting Help topic.

which a web search suggests is due to MaxMemoryPerShellMB

Get-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB

WSManConfig: Microsoft.WSMan.Management\WSMan::localhost\Shell

 Type            Name                           SourceOfValue   Value
 ----            ----                           -------------   -----
 System.String   MaxMemoryPerShellMB                            2147483647

Method 2: Run a ps1 script from the -ScriptBlock

Invoke-Command -Session $session -ScriptBlock { C:\temp\installSC.ps1 }

where installSC.ps1 contains the same contents as the ScriptBlock above. In this case the installation starts, and the service is installed on the target machine, but errors out when ServiceControl.exe (started from Process.Start in the New-ServiceControlInstance powershell command) attempts to start RavenDB for setting up the SC queues.

fail: Microsoft.Extensions.Hosting.Internal.Host[11]
      Hosting failed to start
      System.InvalidOperationException: Unable to start the RavenDB Server
      Output:
             _____                       _____  ____
            |  __ \                     |  __ \|  _ \
            | |__) |__ ___   _____ _ __ | |  | | |_) |
            |  _  // _` \ \ / / _ \ '_ \| |  | |  _ <
            | | \ \ (_| |\ V /  __/ | | | |__| | |_) |
            |_|  \_\__,_| \_/ \___|_| |_|_____/|____/


            Safe by default, optimized for efficiency

       Build 62028, Version 6.2, SemVer 6.2.4, Commit 63d1300
       PID 3344, 64 bits, 10 Cores, Phys Mem 8.745 GBytes, Arch: X64
       Source Code (git repo): https://github.com/ravendb/ravendb
       Built with love by Hibernating Rhinos and awesome contributors!
      +---------------------------------------------------------------+


      System.TypeInitializationException: The type initializer for 'InternalUpgrader' threw an exception.
       ---> System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation.
       ---> System.TypeInitializationException: The type initializer for 'Raven.Server.Storage.Schema.Updates.Server.From40011' threw an exception.
       ---> System.TypeInitializationException: The type initializer for 'Voron.Slices' threw an exception.
       ---> Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 2,047.9999 TBytes / 2,048 TBytes, Memory: 6.627 GBytes / 8.745 GBytes, Available memory for processing: 2,047.9999 TBytes, Dirty memory: 0 Bytes, Managed memory: 43.54 MBytes, Unmanaged allocations: 8 KBytes, Lucene managed: 0 Bytes, Lucene unmanaged term cache: 0 Bytes, Lucene unmanaged sorted terms: 0 Bytes
         at Sparrow.Server.LowMemory.MemoryInformation.ThrowInsufficientMemory(MemoryInfoResult memInfo) in /_/62028/src/Sparrow.Server/LowMemory/MemoryInformation.cs:line 181
         at Sparrow.Server.LowMemory.MemoryInformation.AssertNotAboutToRunOutOfMemory() in /_/62028/src/Sparrow.Server/LowMemory/MemoryInformation.cs:line 120
         at Sparrow.Server.LowMemory.LowMemoryMonitor.AssertNotAboutToRunOutOfMemory() in /_/62028/src/Sparrow.Server/LowMemory/LowMemoryMonitor.cs:line 47
         at Sparrow.LowMemory.LowMemoryNotification.AssertNotAboutToRunOutOfMemory() in /_/62028/src/Sparrow/LowMemory/LowMemoryNotification.cs:line 470
         at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size) in /_/62028/src/Sparrow/Utils/NativeMemory.cs:line 138
         at Sparrow.Utils.NativeMemory.AllocateMemory(Int64 size, ThreadStats& thread) in /_/62028/src/Sparrow/Utils/NativeMemory.cs:line 155
         at Sparrow.Server.UnmanagedGlobalSegment..ctor(Int32 size) in /_/62028/src/Sparrow.Server/ByteString.cs:line 477
         at Sparrow.Server.ByteStringMemoryCache.Allocate(Int32 size, Action allocationFailure) in /_/62028/src/Sparrow.Server/ByteString.cs:line 618
         at Sparrow.Server.ByteStringContext`1.AllocateSegment(Int32 size) in /_/62028/src/Sparrow.Server/ByteString.cs:line 1415
         at Sparrow.Server.ByteStringContext`1..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in /_/62028/src/Sparrow.Server/ByteString.cs:line 799
         at Sparrow.Server.ByteStringContext..ctor(SharedMultipleUseFlag lowMemoryFlag, Int32 allocationBlockSize) in /_/62028/src/Sparrow.Server/ByteString.cs:line 691
         at Voron.Slices..cctor() in /_/62028/src/Voron/Slice.cs:line 343
         --- End of inner exception stack trace ---
         at Voron.Data.Tables.TableSchema.DefineKey(IndexDef index) in /_/62028/src/Voron/Data/Tables/TableSchema.cs:line 119
         at Raven.Server.Storage.Schema.Updates.Server.From40011..cctor() in /_/62028/src/Raven.Server/Storage/Schema/Updates/Server/42012/From40011.cs:line 60
         --- End of inner exception stack trace ---
         at Raven.Server.Storage.Schema.Updates.Server.From40011..ctor()
         at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)
         --- End of inner exception stack trace ---
         at System.RuntimeType.CreateInstanceDefaultCtor(Boolean publicOnly, Boolean wrapExceptions)
         at Raven.Server.Storage.Schema.SchemaUpgrader.InternalUpgrader..cctor() in /_/62028/src/Raven.Server/Storage/Schema/SchemaUpgrader.cs:line 66
         --- End of inner exception stack trace ---
         at Raven.Server.Storage.Schema.SchemaUpgrader.InternalUpgrader..ctor(StorageType storageType, ConfigurationStorage configurationStorage, DocumentsStorage documentsStorage, ServerStore serverStore) in /_/62028/src/Raven.Server/Storage/Schema/SchemaUpgrader.cs:line 75
         at Raven.Server.Storage.Schema.SchemaUpgrader.Upgrader(StorageType storageType, ConfigurationStorage configurationStorage, DocumentsStorage documentsStorage, ServerStore serverStore) in /_/62028/src/Raven.Server/Storage/Schema/SchemaUpgrader.cs:line 185
         at Raven.Server.ServerWide.ServerStore.PreInitialize() in /_/62028/src/Raven.Server/ServerWide/ServerStore.cs:line 797
         at Raven.Server.RavenServer.Initialize() in /_/62028/src/Raven.Server/RavenServer.cs:line 208
         at Raven.Server.Program.Main(String[] args) in /_/62028/src/Raven.Server/Program.cs:line 249


         at Raven.Embedded.EmbeddedServer.RunServer() in /_/62028/src/Raven.Embedded/EmbeddedServer.cs:line 304
         at Raven.Client.Extensions.TaskExtensions.<>c__3`1.<WithCancellation>b__3_0(Task`1 t) in /_/62028/src/Raven.Client/Extensions/TaskExtensions.cs:line 36
         at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
      --- End of stack trace from previous location ---
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
      --- End of stack trace from previous location ---
         at Raven.Embedded.EmbeddedServer.GetServerUriAsync(CancellationToken token) in /_/62028/src/Raven.Embedded/EmbeddedServer.cs:line 191
         at Raven.Embedded.EmbeddedServer.<>c__DisplayClass13_0.<<GetDocumentStoreAsync>b__0>d.MoveNext() in /_/62028/src/Raven.Embedded/EmbeddedServer.cs:line 147
      --- End of stack trace from previous location ---
         at Raven.Client.Extensions.TaskExtensions.<>c__3`1.<WithCancellation>b__3_0(Task`1 t) in /_/62028/src/Raven.Client/Extensions/TaskExtensions.cs:line 36
         at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
      --- End of stack trace from previous location ---
         at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
         at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
      --- End of stack trace from previous location ---
         at Raven.Embedded.EmbeddedServer.GetDocumentStoreAsync(DatabaseOptions options, CancellationToken token) in /_/62028/src/Raven.Embedded/EmbeddedServer.cs:line 167
         at ServiceControl.RavenDB.EmbeddedDatabase.Connect(CancellationToken cancellationToken) in C:\Projects\ServiceControl\src\ServiceControl.RavenDB\EmbeddedDatabase.cs:line 175
         at ServiceControl.Persistence.RavenDB.RavenEmbeddedPersistenceLifecycle.Initialize(CancellationToken cancellationToken) in C:\Projects\ServiceControl\src\ServiceControl.Persistence.RavenDB\RavenEmbeddedPersistenceLifecycle.cs:line 52
         at Microsoft.Extensions.Hosting.Internal.Host.<StartAsync>b__15_1(IHostedService service, CancellationToken token)
         at Microsoft.Extensions.Hosting.Internal.Host.ForeachService[T](IEnumerable`1 services, CancellationToken token, Boolean concurrent, Boolean abortOnFirstException, List`1 exceptions, Func`3 operation)

The actual exception of note here is

Sparrow.Server.LowMemory.EarlyOutOfMemoryException: The amount of available memory to commit on the system is low. Commit charge: 2,047.9999 TBytes / 2,048 TBytes, Memory: 6.627 GBytes / 8.745 GBytes, Available memory for processing: 2,047.9999 TBytes, Dirty memory: 0 Bytes, Managed memory: 43.54 MBytes, Unmanaged allocations: 8 KBytes, Lucene managed: 0 Bytes, Lucene unmanaged term cache: 0 Bytes, Lucene unmanaged sorted terms: 0 Bytes

which is noteworthy, since 2,048 TBytes is the same size as the MaxMemoryPerShellMB from Method 1 (and modifying the value of WSMan:\localhost\Shell\MaxMemoryPerShellMB causes this value in the exception to change). Note that this is Tebibytes, not Terabytes

Also worthy of note is that ServiceControl audit instances do not have this exception, despite also setting up RavenDB

Expected behavior

ServiceControl error instances can be installed remotely

Actual behavior

ServiceControl error instances cannot be installed remotely

Versions

Please list the version of the relevant packages or applications in which the bug exists.

Particular.ServiceControl.Management 6.6.0

Steps to reproduce

The remote machine needs to be setup to allow Invoke-Command scripts to be run against PowerShell v7+. The Invoke-Command can be pointed at v7 by first starting a session

$session = New-PSSession -ComputerName [RemoteMachine] -ConfigurationName PowerShell.7 -credential [local credential is required if not using a domain]

If there are errors running the script, enable more permission execution policies (I had to do this in both PS7 and PS5 for some reason) on the remote machine, and ensure the firewall is allowing the request through (or temporarily disable the firewall)

Set-ExecutionPolicy -ExecutionPolicy RemoteSigned

Relevant log output

Additional Information

Workarounds

Possible solutions

Additional information

[PhilBastian]

I was able to successfully get the RavenDB initialised by using powershell remoting via SSH rather than WSMan

Invoke-Command -HostName user@MyHost -ScriptBlock { C:\temp\installSC.ps1 }

[PhilBastian]

v4.33.5 (latest v4) works fine using WSMan, v5.0.0 and above doesn't.
One possible cause could be the upgrade from RavenDB 3.5 to 5 in this version of the error instance, however audit instances are also on v5 and they work fine.

[Marcin7878]

Hi

Have the same issue with Particular.ServiceControl.Management 6.11.0, when remotly call Invoke-ServiceControlInstanceUpgrade, on Windows Server 2022 Standard.

Any update here ?

Edit:
on target machine there are event logs:

Faulting application name: wsmprovhost.exe, version: 10.0.20348.2849, time stamp: 0xae54153e
Faulting module name: ntdll.dll, version: 10.0.20348.4767, time stamp: 0x76a3704d
Exception code: 0xc00000fd

which indicates a stack overflow exception

[danielmarbach]

@Marcin7878 Thanks for reporting. It seems there is a related bug in RavenDB that the Hibernating Rhinos team hasn't been able to reproduce yet. It seems the server sets the check to be enabled which then, in some cases during the installation, causes this memory exception to occur.

It would quite likely be possible to set the RAVEN_DISABLE_EARLY_OOM environment variable to true which then bypasses this check during the installation.

https://github.com/ravendb/ravendb/blob/release/v5.4/src/Sparrow.Server/LowMemory/MemoryInformation.cs#L79

Which is something we could do during the setup phase if this turns out to be the fix.

@Marcin7878 could you try invoking the commands with $env:RAVEN_DISABLE_EARLY_OOM = "true" and report back?

[danielmarbach]

The instance setup code is https://github.com/Particular/ServiceControl/blob/master/src/ServiceControlInstaller.Engine/Setup/InstanceSetup.cs#L53 and we could add processStartupInfo.EnvironmentVariables.Add("RAVEN_DISABLE_EARLY_OOM", "true");

[Marcin7878]

@danielmarbach thanks for replay.

Setting the RAVEN_DISABLE_EARLY_OOM variable didn't help in the Invoke-ServiceControlInstanceUpgrade scenario.

Still error:
The WSMan provider host process did not return a proper response. A provider in the host process may have behaved improperly.

I'm upgrading a fresh (dataless) service control instance from version 5.11.11 to 6.11.0.

I haven't found any logs from the upgrade process, so I can't verify if the problem is with RavenDB.

[danielmarbach]

@Marcin7878 Did you make sure the environment variable is set on the remote machine when the script is executed (in that shell)?

[Marcin7878]

Yes, im sure:

$s = New-psSession -ComputerName *******  -ConfigurationName PowerShell.7

Invoke-Command -Session $s -ScriptBlock { Import-Module Particular.ServiceControl.Management; $env:RAVEN_DISABLE_EARLY_OOM; Invoke-ServiceControlInstanceUpgrade *********  -verbose }

result:

VERBOSE: Use sc-help to list ServiceControl management commands
true
OperationStopped: Processing data for a remote command failed with the following error message: The WSMan provider host process did not return a proper response.  A provider in the host process may have behaved improperly. For more information, see the about_Remote_Troubleshooting Help topic.

[danielmarbach]

Hmmm... I wonder whether it has no effect or it does not get correctly set to the child processes. Could you try setting it temporarily machine wide?

[System.Environment]::SetEnvironmentVariable("RAVEN_DISABLE_EARLY_OOM", "true", "Machine")

FYI I have also reached out to the RavenDB folks to double-check things, and I'll keep you posted.

[Marcin7878]

I set this variable manualy on the target machine, double checked, restarted it.

in the example above, $env:RAVEN_DISABLE_EARLY_OOM displays this value (second line of output 'true')

it seems that this has no impact on this case.

Are there any update logs saved anywhere? I couldn't find any.

[danielmarbach]

@Marcin7878 Is your PowerShell profile -ConfigurationName PowerShell.7 doing something special? I'm not a powershell expert but as far as I do understand, a custom remoting configuration could behave differently from a standard Windows PowerShell remoting.

Have you tried using Windows PowerShell instead?

$s = New-PSSession -ComputerName <machine> 
Invoke-Command -Session $s -ScriptBlock { 
    $env:RAVEN_DISABLE_EARLY_OOM = "true"
    # upgrade command
}

Regular logs are usually written to

%ProgramData%\Particular\ServiceControl\<instanceName>\logs\

Did you see anything there? I guess, like you mentioned, the event logs have nothing else

Another option you could try to verify it is indeed the same issue is by invoking the ServiceControl binaries manually with the --setup flag either remotely or directly on the machine.

[Marcin7878]

@danielmarbach This PowerShell.7 is standard profile, created during PowerShellCore instalation. I have not made any changes to the configuration of this profile.

approach

$s = New-PSSession -ComputerName <machine> 
Invoke-Command -Session $s -ScriptBlock { 
    $env:RAVEN_DISABLE_EARLY_OOM = "true"
    # upgrade command
}

was tested.

On target machine exists only these paths in %ProgramData%:
C:\ProgramData\Particular\ServiceControlInstaller\ConfigBackups
C:\ProgramData\ParticularSoftware
with no logs

run with --setup looks fine:

Invoke-Command -Session $s -ScriptBlock { Import-Module Particular.ServiceControl.Management; $env:RAVEN_DISABLE_EARLY_OOM; cd C:\ParticularServices\ServiceControl\Bin; .\ServiceControl.exe --setup }

VERBOSE: Use sc-help to list ServiceControl management commands
true
info: Microsoft.Hosting.Lifetime[0]
      Application started. Press Ctrl+C to shut down.
info: Microsoft.Hosting.Lifetime[0]
      Hosting environment: Production
info: Microsoft.Hosting.Lifetime[0]
      Content root path: C:\ParticularServices\ServiceControl\Bin
info: Microsoft.Hosting.Lifetime[0]
      Application is shutting down...