Sometimes all the threads of the server fall and the server stops responding to requests

[khv2online]

• GNAT version: 12.2.1 (alire)
• AWS version: 23 (alire)
• OS: Debian GNU/Linux 11 (bullseye) x86_64
• Host: KVM/QEMU (Standard PC (i440FX + PIIX, 1996) pc-i440fx-6.2)
• CPU: AMD EPYC 7282 (4) @ 2.794GHz

Nginx (HTTP2) is used as a reverse proxy server.

Process logs

Fatal error, slot 5 is dead now.
raised AWS.NET.SOCKET_ERROR : Send : [32] Broken pipe
0x6bb1a5 Aws.Net.Std.Raise_Exception at aws-net-std__gnat.adb:632
0x6bb615 Aws.Net.Std.Send at aws-net-std__gnat.adb:695
0x752463 Aws.Net.Send at aws-net.adb:480
0x66929e Aws.Net.Buffered.Flush at aws-net-buffered.adb:76
0x5dad81 Aws.Http2.Frame.Send at aws-http2-frame.adb:209
0x7da122 Aws.Server.Protocol_Handler at aws-server-protocol_handler.adb:295
0x7d5500 Aws.Server.LineT at aws-server.adb:395
0x9d1b39 System.Tasking.Stages.Task_Wrapper at s-tassta.adb:1211
[/lib/x86_64-linux-gnu/libpthread.so.0]
0x7fea3b8d7ea5
[/lib/x86_64-linux-gnu/libc.so.6]
0x7fea3b6b3a2d
0xfffffffffffffffe

[AJ-Ianozi]

I'm running into this exact same problem with AWS 24 (alire), gnat 13.2.1 (alire). Server is running fine for hours and hours all day, but overnight I come back to this in my log; same files as @khv2online 's:

Fatal error, slot 5 is dead now.

raised AWS.NET.SOCKET_ERROR : Send : [32] Broken pipe
0x62eedc Aws.Net.Std.Raise_Exception at aws-net-std__gnat.adb:632
0x62f31f Aws.Net.Std.Send at aws-net-std__gnat.adb:695
0x6bb3f4 Aws.Net.Send at aws-net.adb:480
0x5e4a67 Aws.Net.Buffered.Flush at aws-net-buffered.adb:76
0x56282d Aws.Http2.Frame.Send at aws-http2-frame.adb:209
0x736b90 Aws.Server.Protocol_Handler at aws-server-protocol_handler.adb:295
0x7322ab Aws.Server.LineT at aws-server.adb:395
0x8ec059 System.Tasking.Stages.Task_Wrapper at s-tassta.adb:1211
[/lib/x86_64-linux-gnu/libpthread.so.0]
0x7fbc4393eea5
[/lib/x86_64-linux-gnu/libc.so.6]
0x7fbc43718a6d
0xfffffffffffffffe

Fatal error, slot 3 is dead now.

raised AWS.NET.SOCKET_ERROR : Send : [32] Broken pipe
0x62eedc Aws.Net.Std.Raise_Exception at aws-net-std__gnat.adb:632
0x62f31f Aws.Net.Std.Send at aws-net-std__gnat.adb:695
0x6bb3f4 Aws.Net.Send at aws-net.adb:480
0x5e4a67 Aws.Net.Buffered.Flush at aws-net-buffered.adb:76
0x56282d Aws.Http2.Frame.Send at aws-http2-frame.adb:209
0x736b90 Aws.Server.Protocol_Handler at aws-server-protocol_handler.adb:295
0x7322ab Aws.Server.LineT at aws-server.adb:395
0x8ec059 System.Tasking.Stages.Task_Wrapper at s-tassta.adb:1211
[/lib/x86_64-linux-gnu/libpthread.so.0]
0x7fbc4393eea5
[/lib/x86_64-linux-gnu/libc.so.6]
0x7fbc43718a6d
0xfffffffffffffffe

Fatal error, slot 2 is dead now.

raised AWS.NET.SOCKET_ERROR : Send : [32] Broken pipe
0x62eedc Aws.Net.Std.Raise_Exception at aws-net-std__gnat.adb:632
0x62f31f Aws.Net.Std.Send at aws-net-std__gnat.adb:695
0x6bb3f4 Aws.Net.Send at aws-net.adb:480
0x5e4a67 Aws.Net.Buffered.Flush at aws-net-buffered.adb:76
0x56282d Aws.Http2.Frame.Send at aws-http2-frame.adb:209
0x736b90 Aws.Server.Protocol_Handler at aws-server-protocol_handler.adb:295
0x7322ab Aws.Server.LineT at aws-server.adb:395
0x8ec059 System.Tasking.Stages.Task_Wrapper at s-tassta.adb:1211
[/lib/x86_64-linux-gnu/libpthread.so.0]
0x7fbc4393eea5
[/lib/x86_64-linux-gnu/libc.so.6]
0x7fbc43718a6d
0xfffffffffffffffe

Fatal error, slot 4 is dead now.

raised AWS.NET.SOCKET_ERROR : Send : [32] Broken pipe
0x62eedc Aws.Net.Std.Raise_Exception at aws-net-std__gnat.adb:632
0x62f31f Aws.Net.Std.Send at aws-net-std__gnat.adb:695
0x6bb3f4 Aws.Net.Send at aws-net.adb:480
0x5e4a67 Aws.Net.Buffered.Flush at aws-net-buffered.adb:76
0x56282d Aws.Http2.Frame.Send at aws-http2-frame.adb:209
0x736b90 Aws.Server.Protocol_Handler at aws-server-protocol_handler.adb:295
0x7322ab Aws.Server.LineT at aws-server.adb:395
0x8ec059 System.Tasking.Stages.Task_Wrapper at s-tassta.adb:1211
[/lib/x86_64-linux-gnu/libpthread.so.0]
0x7fbc4393eea5
[/lib/x86_64-linux-gnu/libc.so.6]
0x7fbc43718a6d
0xfffffffffffffffe

Fatal error, slot 1 is dead now.

raised AWS.NET.SOCKET_ERROR : Send : [32] Broken pipe
0x62eedc Aws.Net.Std.Raise_Exception at aws-net-std__gnat.adb:632
0x62f31f Aws.Net.Std.Send at aws-net-std__gnat.adb:695
0x6bb3f4 Aws.Net.Send at aws-net.adb:480
0x5e4a67 Aws.Net.Buffered.Flush at aws-net-buffered.adb:76
0x56282d Aws.Http2.Frame.Send at aws-http2-frame.adb:209
0x736b90 Aws.Server.Protocol_Handler at aws-server-protocol_handler.adb:295
0x7322ab Aws.Server.LineT at aws-server.adb:395
0x8ec059 System.Tasking.Stages.Task_Wrapper at s-tassta.adb:1211
[/lib/x86_64-linux-gnu/libpthread.so.0]
0x7fbc4393eea5
[/lib/x86_64-linux-gnu/libc.so.6]
0x7fbc43718a6d
0xfffffffffffffffe

[TurboGit]

@AJ-Ianozi : Would you be able to test a patch?

[AJ-Ianozi]

@TurboGit Absolutely.

[TurboGit]

Here it is, you can do a simple edit of the aws-server-protocol_handler.adb file based on the change below:

diff --git a/src/core/aws-server-protocol_handler.adb b/src/core/aws-server-protocol_handler.adb
index 5cb54e481..63d9fca73 100644
--- a/src/core/aws-server-protocol_handler.adb
+++ b/src/core/aws-server-protocol_handler.adb
@@ -290,9 +290,17 @@ begin
               or else (CNF.HTTP2_Activated (LA.Server.Properties)
                        and then not CNF.Security (LA.Server.Properties))
             then
-               HTTP2.Frame.GoAway.Create
-                 (Stream_Id => 0,
-                  Error     => HTTP2.C_Protocol_Error).Send (Sock_Ptr.all);
+               --  We don't want to fail, if the socket is closed already
+               --  we ignore the socket error.
+
+               begin
+                  HTTP2.Frame.GoAway.Create
+                    (Stream_Id => 0,
+                     Error     => HTTP2.C_Protocol_Error).Send (Sock_Ptr.all);
+               exception
+                  when Net.Socket_Error =>
+                     null;
+               end;
 
                exit For_Every_Request;

[AJ-Ianozi]

Thanks, I'll let you know how it goes!

[AJ-Ianozi]

@TurboGit Just wanted to follow back up, it's been a week and the server has not crashed yet. I think the patch is the solution.

[khv2online]

@TurboGit Just wanted to follow back up, it's been a week and the server has not crashed yet. I think the patch is the solution.

I tried the patch - broken pipe errors still happen.

I think it might be related to either AWS.templates or AWS.client. I have one server without external network requests and without html templates and it works for a year without a patch and without these errors.

[TurboGit]

Maybe, but then we'll need more data to be able to reproduce and fix this issue.