
Service stopping after .NET Runtime exception
Hello,
I just want to report we're seeing the Glim service crash every couple of days on our setup, when checking Event Viewer it appears to be an exception in the .NET Runtime with a 'The process was terminated due to an unhandled exception.' error. The exception traces vary and I've copied several examples from the past few weeks below.
The Log files at the corresponding times show the app shutting down then coming back up 10 seconds later (as we've set the services recovery options to restart on failure)
[10:26:42 INF Microsoft.Hosting.Lifetime] Application is shutting down... [10:26:53 INF Telestream.Glimpse.Startup] Authentication: "None" [10:26:54 INF Microsoft.Hosting.Lifetime] Now listening on: http://[::]:5000 [10:26:54 INF Microsoft.Hosting.Lifetime] Now listening on: https://[::]:5001 [10:26:54 INF Microsoft.Hosting.Lifetime] Application started. Hosting environment: Production; Content root path: C:\Program Files\Telestream\GLIM
We're running Glim Version 1.1.546 on an EC2 (m5n.8xlarge) running Windows Server 2016 and are predominantly using Glim to stream ProRes files from S3 Buckets. The nature of the errors leads me to believe that there is an issue interfacing with the S3 bucket but was wondering if this had been seen elsewhere or if there is perhaps a known bug or something we can look into to prevent this from occurring?
Example stack traces from event viewer below:
Faulting application name: GlimpseService.exe, version: 1.1.546.0, time stamp: 0x5ec598f1 Faulting module name: KERNELBASE.dll, version: 10.0.14393.3659, time stamp: 0x5e9140ed Exception code: 0xe0434352 Fault offset: 0x0000000000034f38 Faulting process id: 0x1488 Faulting application start time: 0x01d69b02088aa532 Faulting application path: C:\Program Files\Telestream\GLIM\GlimpseService.exe Faulting module path: C:\Windows\System32\KERNELBASE.dll Report Id: 82ec5886-1eeb-4797-ba07-4b057e8a88e5 Faulting package full name: Faulting package-relative application ID:
Events including the Stack Trace from the .Net Runtime Error event
Application: GlimpseService.exe CoreCLR Version: 4.700.20.26901 .NET Core Version: 3.1.5 Description: The process was terminated due to an unhandled exception. Exception Info: System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Telestream.Glimpse.Program.Main(String[] args) at Telestream.Glimpse.Program.<Main>(String[] args)
Application: GlimpseService.exe CoreCLR Version: 4.700.20.26901 .NET Core Version: 3.1.5 Description: The process was terminated due to an unhandled exception. Exception Info: System.OperationCanceledException: The operation was canceled. at System.Threading.CancellationToken.ThrowOperationCanceledException() at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token) at Telestream.Glimpse.Program.Main(String[] args) at Telestream.Glimpse.Program.<Main>(String[] args)
Application: GlimpseService.exe CoreCLR Version: 4.700.20.26901 .NET Core Version: 3.1.5 Description: The process was terminated due to an internal error in the .NET Runtime at IP 00007FFDEBB41D5B (00007FFDEB9B0000) with exit code c0000005.
Application: GlimpseService.exe CoreCLR Version: 4.700.20.26901 .NET Core Version: 3.1.5 Description: The process was terminated due to an unhandled exception. Exception Info: System.FormatException: Could not parse the JSON file. ---> System.Text.Json.JsonReaderException: '{' is invalid after a value. Expected either ',', '}', or ']'. LineNumber: 142 | BytePositionInLine: 3. at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes) at System.Text.Json.Utf8JsonReader.ConsumeNextTokenUntilAfterAllCommentsAreSkipped(Byte marker) at System.Text.Json.Utf8JsonReader.ConsumeNextToken(Byte marker) at System.Text.Json.Utf8JsonReader.ConsumeNextTokenOrRollback(Byte marker) at System.Text.Json.Utf8JsonReader.ReadSingleSegment() at System.Text.Json.Utf8JsonReader.Read() at System.Text.Json.JsonDocument.Parse(ReadOnlySpan`1 utf8JsonSpan, Utf8JsonReader reader, MetadataDb& database, StackRowStack& stack) at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedBytes) at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 json, JsonDocumentOptions options) at System.Text.Json.JsonDocument.Parse(String json, JsonDocumentOptions options) at Microsoft.Extensions.Configuration.Json.JsonConfigurationFileParser.ParseStream(Stream input) at Microsoft.Extensions.Configuration.Json.JsonConfigurationProvider.Load(Stream stream) --- End of inner exception stack trace --- at Microsoft.Extensions.Configuration.Json.JsonConfigurationProvider.Load(Stream stream) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload) --- End of stack trace from previous location where exception was thrown --- at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload) at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration() at Microsoft.Extensions.Hosting.HostBuilder.Build() at Telestream.Glimpse.Program.Main(String[] args) at Telestream.Glimpse.Program.<Main>(String[] args)
Application: GlimpseService.exe CoreCLR Version: 4.700.20.26901 .NET Core Version: 3.1.5 Description: The process was terminated due to an unhandled exception. Exception Info: System.ObjectDisposedException: Cannot write to the response body, the response has completed. Object name: 'HttpResponseStream'. at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponsePipeWriter.<ValidateState>g__ThrowObjectDisposedException|14_0() at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponsePipeWriter.Advance(Int32 bytes) at Telestream.Glimpse.Core.Models.MediaWriter.Write(ReadOnlySpan`1 source) at Telestream.Glimpse.Core.Models.MediaWriter.NativeWrite(IntPtr data, Int32 length, IntPtr bytesWritten)
Many Thanks,
Mitch
-
Some services stop automatically if they are not in use by other services or programs". Once the log on account for the Node Manager service is changed from the default to a custom Windows account, starting the service throws the following.
regards: shakemods