socket support

[pottendo]

Hi,
thanks for the library and the effort of keeping this alive.
I've used circle-stdlib now to do this: https://github.com/pottendo/pottendo-Pi1541

Today I started writing on a socket layer to wrap Circle CSocket. The background would be to port 'mongoose' to Circle as it features a few nice functions.
Now I stumbled over your 'circe-socket' branch, which seems not to be continued since a while.

What's the status on this project? Any plans to merge it into the master tree?
(Haven't tried this branch, yet - does it make sense?)

thanks for any feedback!
pottendo

[smuehlst]

Hi,

I had the intention to implement the socket layer a while ago. After I had started, something more important stalled this work, and I didn't pick it up since then again.

What is currently published on the https://github.com/smuehlst/circle-stdlib/tree/circle-socket branch is some initial infrastructure code without any functionality. I don't remember exactly but I may have more code locally, I can check that tomorrow.

At the moment I won't promise anything, but potentially I will pick this work up again if there is interest.

Best regards
Stephan

[pottendo]

Hi,
sure, I'd be interested. Not only for my project above but other other ideas I have in mind - e.g. a later libcurl support on circle-stdlib could enable quite a bunch of apps to be ported to circle then benefiting from the rapid startup-time, etc.
I can support with testing and even contributing code (as my time & programming skills permit).
thanks for quick response, pottendo

PS: I got mongoose compiling already against circle-stdlib with a layer of dummy socket functions - so that would be my first test-case, to enable a web-server with some mongoose funcs (e.g. serve a directory, etc.).

[smuehlst]

I see now that there is first some work required to merge the most recent circle-stdlib release into the circle-socket branch. After that I will think about the next steps.

[pottendo]

Great, thanks! pottendoAm 30.01.2024 21:51 schrieb Stephan Mühlstrasser ***@***.***>: I see now that there is first some work required to merge the most recent circle-stdlib release into the circle-socket branch. After that I will think about the next steps. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[smuehlst]

@pottendo The most important socket functions are now implemented on the circle-socket branch: socket(), bind(), listen(), accept() and connect().

It is clear that this set of functions alone is not sufficient for implementing or porting real applications that use the Posix socket API, but some initial tests should be possible.

The "05-smoketest" sample demonstrates what is working so far:

circle-stdlib/samples/05-smoketest/kernel.cpp

Lines 766 to 960 in 9f68ece

	void CKernel::SocketTest(void)
	{
	Report("Test socket operations...");
	struct basic_socket_test
	{
	basic_socket_test(int d, int t, int p, bool s) : domain(d), type(t), protocol(p), expect_success(s)
	{
	}
	int domain;
	int type;
	int protocol;
	bool expect_success;
	};
	basic_socket_test const basic_socket_tests[] =
	{
	{AF_INET, SOCK_DGRAM, IPPROTO_UDP, true},
	{AF_INET, SOCK_DGRAM, 0, true},
	{AF_INET, SOCK_STREAM, IPPROTO_TCP, true},
	{AF_INET, SOCK_STREAM, 0, true},
	{AF_INET, SOCK_DGRAM, IPPROTO_IP, false},
	{AF_INET, SOCK_STREAM, IPPROTO_UDP, false},
	{AF_UNIX, SOCK_STREAM, IPPROTO_TCP, false}};
	auto const socket_creation_test = [&](basic_socket_test const &t)
	{
	int const fd = socket(t.domain, t.type, t.protocol);
	unsigned int const testnum = &t - basic_socket_tests;
	if (t.expect_success)
	{
	if (fd == -1)
	{
	Report("error in basic socket test %u", testnum);
	PErrorExit("socket() failed");
	}
	if (close(fd) < 0)
	{
	PErrorExit("close (fd) failed");
	}
	}
	else
	{
	if (fd != -1)
	{
	Report("socket() succeeded unexpectedly");
	exit(1);
	}
	}
	};
	for (auto const &t : basic_socket_tests)
	{
	socket_creation_test(t);
	}
	Report("Basic socket() tests successful");
	Report("Basic socket lifecycle test");
	{
	int fd = socket(AF_INET, SOCK_STREAM, 0);
	if (fd == -1)
	{
	PErrorExit("socket() failed");
	}
	struct sockaddr_in server_address;
	/* The sin_port and sin_addr members shall be in network byte order. */
	server_address.sin_family = AF_INET;
	server_address.sin_addr.s_addr = htonl(INADDR_ANY);
	server_address.sin_port = htons(5000);
	int const bind_result = bind(fd, reinterpret_cast<struct sockaddr *>(&server_address), sizeof(server_address));
	if (bind_result == -1)
	{
	PErrorExit("bind() failed");
	}
	Report("bind() on file descriptor %d succeeded", fd);
	int const listen_result = listen(fd, 3);
	if (listen_result == -1)
	{
	PErrorExit("listen() failed");
	}
	Report("listen() on file descriptor %d succeeded", fd);
	int connected_fd = accept(fd, nullptr, nullptr);
	if (connected_fd == -1)
	{
	PErrorExit("accept (fd) failed");
	}
	Report("accept() on file descriptor %d returned %d", fd, connected_fd);
	char buffer[100];
	size_t const bufsiz = sizeof(buffer);
	auto read_result = read(connected_fd, buffer, bufsiz);
	if (read_result < 0)
	{
	PErrorExit("read (connected_fd) failed");
	}
	// Terminate string, strip newline if present
	if (read_result == bufsiz)
	{
	read_result -= 1;
	}
	if (read_result > 0 && buffer[read_result - 1] == '\n')
	{
	read_result -= 1;
	}
	buffer[read_result] = 0;
	Report("Read from socket succeeded: \"%s\"\n", buffer);
	strcpy(buffer, "Response from server\n");
	auto const write_result = write(connected_fd, buffer, strlen(buffer));
	if (write_result < 0)
	{
	PErrorExit("write (connected_fd) failed");
	}
	Report("Write of %d bytes to socket succeeded\n", static_cast<int>(write_result));
	if (close(connected_fd) < 0)
	{
	PErrorExit("close (connected_fd) failed");
	}
	if (close(fd) < 0)
	{
	PErrorExit("close (fd) failed");
	}
	fd = socket(AF_INET, SOCK_STREAM, 0);
	if (fd == -1)
	{
	PErrorExit("socket() failed");
	}
	// QEMU connects via guestfwd to a simulated server on IP address 10.0.2.100 port 5001
	uint32_t const simulated_server_ip = (10U << 24) | (0U << 16) | (2U << 8) | 100U;
	/* The sin_port and sin_addr members shall be in network byte order. */
	server_address.sin_family = AF_INET;
	server_address.sin_addr.s_addr = htonl(simulated_server_ip);
	server_address.sin_port = htons(5001);
	int const connect_result =
	connect(fd, reinterpret_cast<struct sockaddr *>(&server_address), sizeof(server_address));
	if (connect_result < 0)
	{
	PErrorExit("connect() failed");
	}
	read_result = read(fd, buffer, sizeof(buffer));
	if (read_result < 0)
	{
	PErrorExit("read() failed after connect");
	}
	Report("Read %d bytes from server", read_result);
	char const expected_data[] = "simulated server";
	if (strncmp(buffer, expected_data, sizeof(expected_data) - 1) != 0)
	{
	Report("read() from socket returned unexpected data");
	exit(1);
	}
	Report("Received expected data from server");
	if (close(fd) < 0)
	{
	PErrorExit("close (fd) failed");
	}
	Report("Read from client connection successful");
	}
	Report("Basic socket lifecycle test successful");
	}

[pottendo]

Thanks for the update.
I'm traveling and can't test now... will do later this week.
Looking forward to see all this new stuff on the shiny new RP5 circle lib... lots of progress lately happening. Thanks for the efforts!
pottendo

[pottendo]

hi,
could try your work now and I can confirm that your smoketest works on my setup (RPi4, Ethernet) - at least the first half with listening and connecting via telnet from my Linux host. The second part (addr 10.0.2.100:5001) may work, but I've not figured out how to do it in my setup.

I jumped further to get a mongoose webserver doing something and made some little progress.
What's noticed so far:

• some build tweaking was necessary
  • fcnt() seems to be coming from 2 parts of the library and conflicts then. So I built my own which just does nothing (for now).
  • The follow macros are needed to compile mongoose:

#define FD_CLOEXEC	1
#define F_SETFD		2
#define F_GETFL		4
#define F_SETFL		4
#define O_NONBLOCK	  04000
#define TCP_NODELAY 1
#define FD_SETSIZE 128

of course the values may be incorrect and functions behind are missing, but for a first connection, these are probably not needed.

• setsockopt(...SO_REUSEADDR...) needs to be implemented, so I just let it succeed, probably OK for a first test
• bind(...) barfs as it get's the local IP passed in sa_in->sin_addr.s_addr coming from mongoose, so I overruled to pass INADDR_ANY to make the assertion happy in your code
• select(...) is now missing (ENOSYS) and there I'm stuck.

Not sure if this is helpful for you, but I at least gives you thumbs up! Looking forward to any feeback!
bye, pottendo

[smuehlst]

Hi @pottendo ,

thanks for giving it a first try. I will look into the problems regarding fcntl(), it is not clear to me why it can be defined twice.

At the moment it is expected that any serious application that makes use of the full socket API will not work. select() is probably the next thing that will be implemented. But that will take a while, I first have to do some work to catch up with the new Circle release.

[pottendo]

Great, looking forward to test further.
Once cleaned up, I'll publish my mongoose experiments, so anyone can benefit.

I did some pre-testing on @rsta2's RPi5 branch (before the release) with my application, which is actually a circle-stdlib app. 'just' updating circle within circle-stdlib did the trick for me.

Bye, pottendo

[pottendo]

...forgot that I needed to do one more change - for even compiling this branch of circle-stdlib:
in libgloss/circle/stat.cpp I needed to add #include <fatfs/ff.h>.
bye, pottendo

[pottendo]

ad the fcntl():

  LD    kernel8-rpi4.elf
aarch64-none-elf-ld: ../circle-stdlib/install/aarch64-none-circle/lib/libcirclenewlib.a(io.o): in function `fcntl':
io.cpp:(.text+0xc1c): multiple definition of `fcntl'; ../circle-stdlib/install/aarch64-none-circle/lib/libc.a(lib_a-sysfcntl.o):sysfcntl.c:(.text+0x0): first defined here

[smuehlst]

ad the fcntl():

  LD    kernel8-rpi4.elf
aarch64-none-elf-ld: ../circle-stdlib/install/aarch64-none-circle/lib/libcirclenewlib.a(io.o): in function `fcntl':
io.cpp:(.text+0xc1c): multiple definition of `fcntl'; ../circle-stdlib/install/aarch64-none-circle/lib/libc.a(lib_a-sysfcntl.o):sysfcntl.c:(.text+0x0): first defined here

Ok, I see that this is a bug, thanks for reporting this. I will fix that with the upcoming circle-stdlib release that syncs with the recent Circle release step 46.

[smuehlst]

@pottendo Tbe duplicate definition of fcntl() is now fixed in circle-stdlib v16.3. Note that the restriction remains that the only fcntl() cmd implemented is F_DUPFD.

[smuehlst]

@pottendo I now picked up development of the socket feature again. @rsta2 supported this with an important addition to Circle's socket implementation, so that it was now possible to implement the select() and poll() functions.

The new development takes place on the circle-socket-2 branch.

https://github.com/smuehlst/circle-stdlib/tree/circle-socket-2

The implementation is still in a very early stage, and lots of functions from the socket API are not yet implemented. But the implementation is functional enough to run a slightly modified version of the Mongoose web server that you mentioned earlier. This is demonstrated in a corresponding sample application:

https://github.com/smuehlst/circle-stdlib/tree/circle-socket-2/samples/06-socket

[pottendo]

Hi,
great to see circle improving also on the networking side.
My project moved on without full socket support and I used the (at the time) existing webserver infrastructure: see here https://github.com/pottendo/pottendo-Pi1541

I may give it a try - not to change my application; that's too much change - but in general.
I'll keep you informed - you decide when to close this ticket.

thanks for efforts you're investing - great work, pottendo

[pottendo]

hi,
wanted to checkout your latest socket work - I cloned the repo and switched to the branch,
but it fails to build (configure -r 3 and make all build-stdlib-samples):

/work/src/cbm-retro/scratch/circle-stdlib/libs/circle-newlib/libgloss/circle/stat.cpp: In function 'int _stat(const char*, stat*)':
/work/src/cbm-retro/scratch/circle-stdlib/libs/circle-newlib/libgloss/circle/stat.cpp:19:9: error: 'FILINFO' was not declared in this scope
   19 |         FILINFO finfo;
      |         ^~~~~~~

when building libgloss.

Further, I'm not sure what to checkout for building the 06-samples.

minimum I think these:
git clone https://github.com/nlohmann/json.git
git clone https://github.com/cesanta/mongoose.git
other than that I've downloaded the pre-build web-stuff - but that's not yet used before I succeed in building...

thx, pottendo

[smuehlst]

Hi @pottendo,

I verified that the following steps result in a successful build from the current circle-socket-2 branch (commit e351e56):

git clone -b circle-socket-2 --recursive https://github.com/smuehlst/circle-stdlib.git circle-socket-2
cd circle-socket-2
make all
make build-stdlib-samples

This also builds the 06-socket sample.

Maybe you didn't clone all the repositories recursively? This is the safest way to get a working configuration.

[pottendo]

hi,
I did recursively checkout, but not the the branch but the repo and switched to the branch. Silly me.
...builds now! thanks as for the quick help, pottendo