0

Service stopping after .NET Runtime exception

Hello,

 

I just want to report we're seeing the Glim service crash every couple of days on our setup, when checking Event Viewer it appears to be an exception in the .NET Runtime with a 'The process was terminated due to an unhandled exception.' error. The exception traces vary and I've copied several examples from the past few weeks below. 

The Log files at the corresponding times show the app shutting down then coming back up 10 seconds later (as we've set the services recovery options to restart on failure)

[10:26:42 INF Microsoft.Hosting.Lifetime] Application is shutting down...
[10:26:53 INF Telestream.Glimpse.Startup] Authentication: "None"
[10:26:54 INF Microsoft.Hosting.Lifetime] Now listening on: http://[::]:5000
[10:26:54 INF Microsoft.Hosting.Lifetime] Now listening on: https://[::]:5001
[10:26:54 INF Microsoft.Hosting.Lifetime] Application started. Hosting environment: Production; Content root path: C:\Program Files\Telestream\GLIM

We're running Glim Version 1.1.546 on an EC2 (m5n.8xlarge) running Windows Server 2016 and are predominantly using Glim to stream ProRes files from S3 Buckets. The nature of the errors leads me to believe that there is an issue interfacing with the S3 bucket but was wondering if this had been seen elsewhere or if there is perhaps a known bug or something we can look into to prevent this from occurring?

Example stack traces from event viewer below:

Faulting application name: GlimpseService.exe, version: 1.1.546.0, time stamp: 0x5ec598f1
Faulting module name: KERNELBASE.dll, version: 10.0.14393.3659, time stamp: 0x5e9140ed
Exception code: 0xe0434352
Fault offset: 0x0000000000034f38
Faulting process id: 0x1488
Faulting application start time: 0x01d69b02088aa532
Faulting application path: C:\Program Files\Telestream\GLIM\GlimpseService.exe
Faulting module path: C:\Windows\System32\KERNELBASE.dll
Report Id: 82ec5886-1eeb-4797-ba07-4b057e8a88e5
Faulting package full name:
Faulting package-relative application ID: 

Events including the Stack Trace from the .Net Runtime Error event

Application: GlimpseService.exe
CoreCLR Version: 4.700.20.26901
.NET Core Version: 3.1.5
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Telestream.Glimpse.Program.Main(String[] args)
   at Telestream.Glimpse.Program.<Main>(String[] args)
Application: GlimpseService.exe
CoreCLR Version: 4.700.20.26901
.NET Core Version: 3.1.5
Description: The process was terminated due to an unhandled exception.
Exception Info: System.OperationCanceledException: The operation was canceled.
   at System.Threading.CancellationToken.ThrowOperationCanceledException()
   at Microsoft.Extensions.Hosting.Internal.Host.StopAsync(CancellationToken cancellationToken)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.WaitForShutdownAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Microsoft.Extensions.Hosting.HostingAbstractionsHostExtensions.RunAsync(IHost host, CancellationToken token)
   at Telestream.Glimpse.Program.Main(String[] args)
   at Telestream.Glimpse.Program.<Main>(String[] args)

Application: GlimpseService.exe
CoreCLR Version: 4.700.20.26901
.NET Core Version: 3.1.5
Description: The process was terminated due to an internal error in the .NET Runtime at IP 00007FFDEBB41D5B (00007FFDEB9B0000) with exit code c0000005.
Application: GlimpseService.exe
CoreCLR Version: 4.700.20.26901
.NET Core Version: 3.1.5
Description: The process was terminated due to an unhandled exception.
Exception Info: System.FormatException: Could not parse the JSON file.
 ---> System.Text.Json.JsonReaderException: '{' is invalid after a value. Expected either ',', '}', or ']'. LineNumber: 142 | BytePositionInLine: 3.
   at System.Text.Json.ThrowHelper.ThrowJsonReaderException(Utf8JsonReader& json, ExceptionResource resource, Byte nextByte, ReadOnlySpan`1 bytes)
   at System.Text.Json.Utf8JsonReader.ConsumeNextTokenUntilAfterAllCommentsAreSkipped(Byte marker)
   at System.Text.Json.Utf8JsonReader.ConsumeNextToken(Byte marker)
   at System.Text.Json.Utf8JsonReader.ConsumeNextTokenOrRollback(Byte marker)
   at System.Text.Json.Utf8JsonReader.ReadSingleSegment()
   at System.Text.Json.Utf8JsonReader.Read()
   at System.Text.Json.JsonDocument.Parse(ReadOnlySpan`1 utf8JsonSpan, Utf8JsonReader reader, MetadataDb& database, StackRowStack& stack)
   at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 utf8Json, JsonReaderOptions readerOptions, Byte[] extraRentedBytes)
   at System.Text.Json.JsonDocument.Parse(ReadOnlyMemory`1 json, JsonDocumentOptions options)
   at System.Text.Json.JsonDocument.Parse(String json, JsonDocumentOptions options)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationFileParser.ParseStream(Stream input)
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationProvider.Load(Stream stream)
   --- End of inner exception stack trace ---
   at Microsoft.Extensions.Configuration.Json.JsonConfigurationProvider.Load(Stream stream)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.HandleException(ExceptionDispatchInfo info)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load(Boolean reload)
   at Microsoft.Extensions.Configuration.FileConfigurationProvider.Load()
   at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers)
   at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build()
   at Microsoft.Extensions.Hosting.HostBuilder.BuildAppConfiguration()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Telestream.Glimpse.Program.Main(String[] args)
   at Telestream.Glimpse.Program.<Main>(String[] args)
Application: GlimpseService.exe
CoreCLR Version: 4.700.20.26901
.NET Core Version: 3.1.5
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ObjectDisposedException: Cannot write to the response body, the response has completed.
Object name: 'HttpResponseStream'.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponsePipeWriter.<ValidateState>g__ThrowObjectDisposedException|14_0()
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpResponsePipeWriter.Advance(Int32 bytes)
   at Telestream.Glimpse.Core.Models.MediaWriter.Write(ReadOnlySpan`1 source)
   at Telestream.Glimpse.Core.Models.MediaWriter.NativeWrite(IntPtr data, Int32 length, IntPtr bytesWritten)

Many Thanks,

Mitch

1reply Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
Like Follow
  • 2 wk agoLast active
  • 1Replies
  • 12Views
  • 2 Following