Continuing from last time, let's continue the Zephyr RTOS programming story (3rd time). This time's topic is Zephyr TCP/IP Network and WireGuard Protocol (how to port to Zephyr). 😎
Index
1. STM32 Nucleo-F207ZG board2. Zephyr TCP/IP Stack3. Create a Zephyr Virtual Network Interface4. Implementing WireGuard in Linux user space (C language version)5. Porting WireGuard in Zephyr Environment I - Ethernet6. TODO: Porting WireGuard to work in Zephyr environment II - Wi-Fi7. References
Zephyr is a general purpose RTOS that would be great to teach to students in college. Linux is also a great OS, but it's a dinosaur now, so I think Zephyr, a baby dinosaur, would be easier to learn. 😗
November 27, 2024 (the day the first snow/heavy snow fell)
Task & Memory Management (aka Kernel Services), Device Driver models... So what should we look at next? It's Network. Network-related code contains a lot of content (network stack and socket applications), including wired (Ethernet, CAN) and wireless (Wi-Fi, Bluetooth, ZigBee, Thread, LoraWAN, LTE-M, NB-IoT, etc.).
1. STM32 Nucleo-F207ZG boardI bought a STM32 board with an Ethernet port. For resource-limited devices, Wi-Fi is a more realistic (common) option than Ethernet, but Ethernet also has its own charm in analysis. 😙 In this posting, I will use the board below to understand the structure of Zephyr's Ethernet & TCP/IP stack.
[Figure 1.1] STM32 Nucleo F207ZG board - top view [Source - Reference 2]
[Figure 1.2] STM32 Nucleo F207ZG board - top layout [Source - Reference 2]
<STM32 Nucleo F207ZG board h/w features>- STM32F207ZGT6 in LQFP144 package
- ARM® 32-bit Cortex® -M3 CPU
- 120 MHz max CPU frequency
- VDD from 1.7 V to 3.6 V
- 1 MB Flash
- 128 KB SRAM
- GPIO with external interrupt capability
- 12-bit ADC with 24 channels
- RTC
- 17 General purpose timers
- 2 watchdog timers (independent and window)
- SysTick timer
- USART/UART (6)
- I2C (3)
- SPI (3)
- SDIO
- USB 2.0 OTG FS
- DMA Controller
- 10/100 Ethernet MAC with dedicated DMA
- CRC calculation unit
- True random number generator
The STM32 board is equipped with an ST-Link adapter, making programming (flash writing) and debugging very easy. In addition, it has a well-equipped development environment (such as STM32CubeIDE) and has the advantage of having an appropriate user manual and abundant sample codes. In fact, the conventional method up until now has been STM32 SDK + FreeRTOS + 3rd party library (e.g. lwip), but in the future (although it will not be easy), I expect the working method to change based on Zephyr. 🙏
[Figure 1.3] STM32 Nucleo F207ZG Board Block Diagram [Source - Reference 2]📌 Honestly, it may not be easy to abandon the existing familiar method and switch to Zephyr in the industry reality. 😂
The Nucleo F207ZG board uses the RMII interface for Ethernet. The schematic, pinmap diagram, and device tree below show the RMII connection between the Ethernet controller (= MAC) and the PHY. Due to the characteristics of RMII, there are two RX pins and two TX pins, and a 50 MHz clock is applied, allowing a data rate of up to 100 Mbps (2 x 50).
[Figure 1.4] Excerpt from STM32 Nucleo F207ZG board schematic [Source - Reference 3]
[Figure 1.5] STM32 Nucleo F207ZG board - ethernet pins [Source - Reference 2]
[Figure 1.6] STM32 Nucleo F207ZG board device tree - excerpt from the mac section
[Figure 1.7] Description of RMII Signals [Source - https://en.wikipedia.org/wiki/Media-independent_interface#RMII ]
For more detailed information regarding this board, please refer to the site below.
[Figure 1.1] STM32 Nucleo F207ZG board - top view [Source - Reference 2]
[Figure 1.2] STM32 Nucleo F207ZG board - top layout [Source - Reference 2]
<STM32 Nucleo F207ZG board h/w features>
- STM32F207ZGT6 in LQFP144 package
- ARM® 32-bit Cortex® -M3 CPU
- 120 MHz max CPU frequency
- VDD from 1.7 V to 3.6 V
- 1 MB Flash
- 128 KB SRAM
- GPIO with external interrupt capability
- 12-bit ADC with 24 channels
- RTC
- 17 General purpose timers
- 2 watchdog timers (independent and window)
- SysTick timer
- USART/UART (6)
- I2C (3)
- SPI (3)
- SDIO
- USB 2.0 OTG FS
- DMA Controller
- 10/100 Ethernet MAC with dedicated DMA
- CRC calculation unit
- True random number generator
The STM32 board is equipped with an ST-Link adapter, making programming (flash writing) and debugging very easy. In addition, it has a well-equipped development environment (such as STM32CubeIDE) and has the advantage of having an appropriate user manual and abundant sample codes. In fact, the conventional method up until now has been STM32 SDK + FreeRTOS + 3rd party library (e.g. lwip), but in the future (although it will not be easy), I expect the working method to change based on Zephyr. 🙏
[Figure 1.3] STM32 Nucleo F207ZG Board Block Diagram [Source - Reference 2]
📌 Honestly, it may not be easy to abandon the existing familiar method and switch to Zephyr in the industry reality. 😂
The Nucleo F207ZG board uses the RMII interface for Ethernet. The schematic, pinmap diagram, and device tree below show the RMII connection between the Ethernet controller (= MAC) and the PHY. Due to the characteristics of RMII, there are two RX pins and two TX pins, and a 50 MHz clock is applied, allowing a data rate of up to 100 Mbps (2 x 50).
[Figure 1.4] Excerpt from STM32 Nucleo F207ZG board schematic [Source - Reference 3]
[Figure 1.5] STM32 Nucleo F207ZG board - ethernet pins [Source - Reference 2]
[Figure 1.6] STM32 Nucleo F207ZG board device tree - excerpt from the mac section
[Figure 1.7] Description of RMII Signals [Source - https://en.wikipedia.org/wiki/Media-independent_interface#RMII ]
2. Zephyr TCP/IP StackIn this chapter, we will look at the structure of zephyr's TCP/IP stack and prepare the basics for wireguard porting through a TCP/UDP echo server example.
2.1 Zephyr Ethernet Driver & TCP/IP Stack
The two pictures below show the process of sending and receiving UDP packets on a typical tcp/ip stack (which is nothing new). In a general-purpose OS environment like Linux, this is so familiar that you may not want to look into it, but it is different in Zephyr. 🚀 Let's check it out step by step, starting from the STM32 ethernet device driver, to the tcp/ip stack and socket API. 💪
[Figure 2.1] Zephyr network tx data flow [Source - Reference 4]
[Figure 2.1] Zephyr network tx data flow [Source - Reference 4]
📌 Packet transmission is a process of sequential function calls.
[Figure 2.2] Zephyr network rx data flow [Source - Reference 4]📌 Usually, packet reception is done using an interrupt (sometimes using the polling method) and a callback function (or thread function).
[Figure 2.2] Zephyr network rx data flow [Source - Reference 4]
📌 Usually, packet reception is done using an interrupt (sometimes using the polling method) and a callback function (or thread function).
In the figure above, the code below is actually used in relation to the ethernet device driver.
<stm32 ethernet driver>- zephyr/drivers/ethernet/eth_stm32_hal.c
- zephyr/subsys/net/l2/ethernet/ethernet.c
Meanwhile, the tcp/ip stack code is located below (subsys/net/ip directory).
chyi@earth:~/zephyrproject/zephyr/subsys/net/ip$ ls -laTotal 1116drwxrwxr-x 2 chyi chyi 4096 November 24 17:37 .drwxrwxr-x 7 chyi chyi 4096 July 12 12:40 ..-rw-rw-r-- 1 chyi chyi 40605 Jul 12 12:40 6lo.c-rw-rw-r-- 1 chyi chyi 2321 Jul 12 12:40 6lo.h-rw-rw-r-- 1 chyi chyi 4625 Jul 12 12:40 6lo_private.h-rw-rw-r-- 1 chyi chyi 2573 Jul 12 12:40 CMakeLists.txt-rw-rw-r-- 1 chyi chyi 29941 Jul 12 12:40 Kconfig-rw-rw-r-- 1 chyi chyi 3121 Jul 12 12:40 Kconfig.debug-rw-rw-r-- 1 chyi chyi 5693 Jul 12 12:40 Kconfig.ipv4-rw-rw-r-- 1 chyi chyi 9325 Jul 12 12:40 Kconfig.ipv6-rw-rw-r-- 1 chyi chyi 4247 Jul 12 12:40 Kconfig.mgmt-rw-rw-r-- 1 chyi chyi 833 Jul 12 12:40 Kconfig.stack-rw-rw-r-- 1 chyi chyi 3505 Jul 12 12:40 Kconfig.stats-rw-rw-r-- 1 chyi chyi 9102 Jul 12 12:40 Kconfig.tcp-rw-rw-r-- 1 chyi chyi 659 Jul 12 12:40 canbus_socket.c-rw-rw-r-- 1 chyi chyi 765 Jul 12 12:40 canbus_socket.h-rw-rw-r-- 1 chyi chyi 25768 Jul 12 12:40 connection.c-rw-rw-r-- 1 chyi chyi 6248 Jul 12 12:40 connection.h-rw-rw-r-- 1 chyi chyi 13076 Jul 12 12:40 icmp.c-rw-rw-r-- 1 chyi chyi 14913 Jul 12 12:40 icmpv4.c-rw-rw-r-- 1 chyi chyi 1613 Jul 12 12:40 icmpv4.h-rw-rw-r-- 1 chyi chyi 10116 Jul 12 12:40 icmpv6.c-rw-rw-r-- 1 chyi chyi 5641 Jul 12 12:40 icmpv6.h-rw-rw-r-- 1 chyi chyi 18070 Jul 12 12:40 igmp.c-rw-rw-r-- 1 chyi chyi 11073 Jul 12 12:40 ipv4.c-rw-rw-r-- 1 chyi chyi 12062 Jul 12 12:40 ipv4.h-rw-rw-r-- 1 chyi chyi 10476 Jul 12 12:40 ipv4_acd.c-rw-rw-r-- 1 chyi chyi 3823 Jul 12 12:40 ipv4_autoconf.c-rw-rw-r-- 1 chyi chyi 16979 Jul 12 12:40 ipv4_fragment.c-rw-rw-r-- 1 chyi chyi 21331 Jul 12 12:40 ipv6.c-rw-rw-r-- 1 chyi chyi 19461 July 12 12:40 ipv6.h-rw-rw-r-- 1 chyi chyi 18709 Jul 12 12:40 ipv6_fragment.c-rw-rw-r-- 1 chyi chyi 10727 Jul 12 12:40 ipv6_mld.c-rw-rw-r-- 1 chyi chyi 67825 Jul 12 12:40 ipv6_nbr.c-rw-rw-r-- 1 chyi chyi 19618 July 12 12:40 ipv6_pe.c-rw-rw-r-- 1 chyi chyi 5038 Jul 12 12:40 nbr.c-rw-rw-r-- 1 chyi chyi 5552 Jul 12 12:40 nbr.h-rw-rw-r-- 1 chyi chyi 77929 Jul 12 12:40 net_context.c-rw-rw-r-- 1 chyi chyi 13999 Jul 12 12:40 net_core.c-rw-rw-r-- 1 chyi chyi 127194 Jul 12 12:40 net_if.c-rw-rw-r-- 1 chyi chyi 11776 Jul 12 12:40 net_mgmt.c-rw-rw-r-- 1 chyi chyi 51640 Jul 12 12:40 net_pkt.c-rw-rw-r-- 1 chyi chyi 10351 Jul 12 12:40 net_private.h-rw-rw-r-- 1 chyi chyi 10342 Jul 12 12:40 net_stats.c-rw-rw-r-- 1 chyi chyi 18856 Jul 12 12:40 net_stats.h-rw-rw-r-- 1 chyi chyi 9243 Jul 12 12:40 net_tc.c-rw-rw-r-- 1 chyi chyi 5368 Jul 12 12:40 net_tc_mapping.h-rw-rw-r-- 1 chyi chyi 3775 Jul 12 12:40 net_timeout.c-rw-rw-r-- 1 chyi chyi 1071 Jul 12 12:40 packet_socket.c-rw-rw-r-- 1 chyi chyi 904 Jul 12 12:40 packet_socket.h-rw-rw-r-- 1 chyi chyi 1617 Jul 12 12:40 promiscuous.c-rw-rw-r-- 1 chyi chyi 24999 July 12 12:40 route.c-rw-rw-r-- 1 chyi chyi 8900 July 12 12:40 route.h-rw-rw-r-- 1 chyi chyi 113323 Jul 12 12:40 tcp.c-rw-rw-r-- 1 chyi chyi 3211 Jul 12 12:40 tcp.h-rw-rw-r-- 1 chyi chyi 10672 Jul 12 12:40 tcp_internal.h-rw-rw-r-- 1 chyi chyi 9637 Jul 12 12:40 tcp_private.h-rw-rw-r-- 1 chyi chyi 12709 Jul 12 12:40 tp.c-rw-rw-r-- 1 chyi chyi 4598 Jul 12 12:40 tp.h-rw-rw-r-- 1 chyi chyi 1422 Jul 12 12:40 tp_priv.h-rw-rw-r-- 1 chyi chyi 4712 Jul 12 12:40 udp.c-rw-rw-r-- 1 chyi chyi 3370 Jul 12 12:40 udp_internal.h-rw-rw-r-- 1 chyi chyi 20343 Jul 12 12:40 utils.c📌 Zephyr is IPv6 based, but here we will look at IPv4.
There isn't much code, so there's no problem looking through it all. For reference, if you follow the packet reception process, it's roughly as follows.
[Figure 2.3] Zephyr network rx flow (packet reception flow diagram)
2.2 TCP/UDP echo server
- zephyr/drivers/ethernet/eth_stm32_hal.c
- zephyr/subsys/net/l2/ethernet/ethernet.c
Meanwhile, the tcp/ip stack code is located below (subsys/net/ip directory).
chyi@earth:~/zephyrproject/zephyr/subsys/net/ip$ ls -la
Total 1116
drwxrwxr-x 2 chyi chyi 4096 November 24 17:37 .
drwxrwxr-x 7 chyi chyi 4096 July 12 12:40 ..
-rw-rw-r-- 1 chyi chyi 40605 Jul 12 12:40 6lo.c
-rw-rw-r-- 1 chyi chyi 2321 Jul 12 12:40 6lo.h
-rw-rw-r-- 1 chyi chyi 4625 Jul 12 12:40 6lo_private.h
-rw-rw-r-- 1 chyi chyi 2573 Jul 12 12:40 CMakeLists.txt
-rw-rw-r-- 1 chyi chyi 29941 Jul 12 12:40 Kconfig
-rw-rw-r-- 1 chyi chyi 3121 Jul 12 12:40 Kconfig.debug
-rw-rw-r-- 1 chyi chyi 5693 Jul 12 12:40 Kconfig.ipv4
-rw-rw-r-- 1 chyi chyi 9325 Jul 12 12:40 Kconfig.ipv6
-rw-rw-r-- 1 chyi chyi 4247 Jul 12 12:40 Kconfig.mgmt
-rw-rw-r-- 1 chyi chyi 833 Jul 12 12:40 Kconfig.stack
-rw-rw-r-- 1 chyi chyi 3505 Jul 12 12:40 Kconfig.stats
-rw-rw-r-- 1 chyi chyi 9102 Jul 12 12:40 Kconfig.tcp
-rw-rw-r-- 1 chyi chyi 659 Jul 12 12:40 canbus_socket.c
-rw-rw-r-- 1 chyi chyi 765 Jul 12 12:40 canbus_socket.h
-rw-rw-r-- 1 chyi chyi 25768 Jul 12 12:40 connection.c
-rw-rw-r-- 1 chyi chyi 6248 Jul 12 12:40 connection.h
-rw-rw-r-- 1 chyi chyi 13076 Jul 12 12:40 icmp.c
-rw-rw-r-- 1 chyi chyi 14913 Jul 12 12:40 icmpv4.c
-rw-rw-r-- 1 chyi chyi 1613 Jul 12 12:40 icmpv4.h
-rw-rw-r-- 1 chyi chyi 10116 Jul 12 12:40 icmpv6.c
-rw-rw-r-- 1 chyi chyi 5641 Jul 12 12:40 icmpv6.h
-rw-rw-r-- 1 chyi chyi 18070 Jul 12 12:40 igmp.c
-rw-rw-r-- 1 chyi chyi 11073 Jul 12 12:40 ipv4.c
-rw-rw-r-- 1 chyi chyi 12062 Jul 12 12:40 ipv4.h
-rw-rw-r-- 1 chyi chyi 10476 Jul 12 12:40 ipv4_acd.c
-rw-rw-r-- 1 chyi chyi 3823 Jul 12 12:40 ipv4_autoconf.c
-rw-rw-r-- 1 chyi chyi 16979 Jul 12 12:40 ipv4_fragment.c
-rw-rw-r-- 1 chyi chyi 21331 Jul 12 12:40 ipv6.c
-rw-rw-r-- 1 chyi chyi 19461 July 12 12:40 ipv6.h
-rw-rw-r-- 1 chyi chyi 18709 Jul 12 12:40 ipv6_fragment.c
-rw-rw-r-- 1 chyi chyi 10727 Jul 12 12:40 ipv6_mld.c
-rw-rw-r-- 1 chyi chyi 67825 Jul 12 12:40 ipv6_nbr.c
-rw-rw-r-- 1 chyi chyi 19618 July 12 12:40 ipv6_pe.c
-rw-rw-r-- 1 chyi chyi 5038 Jul 12 12:40 nbr.c
-rw-rw-r-- 1 chyi chyi 5552 Jul 12 12:40 nbr.h
-rw-rw-r-- 1 chyi chyi 77929 Jul 12 12:40 net_context.c
-rw-rw-r-- 1 chyi chyi 13999 Jul 12 12:40 net_core.c
-rw-rw-r-- 1 chyi chyi 127194 Jul 12 12:40 net_if.c
-rw-rw-r-- 1 chyi chyi 11776 Jul 12 12:40 net_mgmt.c
-rw-rw-r-- 1 chyi chyi 51640 Jul 12 12:40 net_pkt.c
-rw-rw-r-- 1 chyi chyi 10351 Jul 12 12:40 net_private.h
-rw-rw-r-- 1 chyi chyi 10342 Jul 12 12:40 net_stats.c
-rw-rw-r-- 1 chyi chyi 18856 Jul 12 12:40 net_stats.h
-rw-rw-r-- 1 chyi chyi 9243 Jul 12 12:40 net_tc.c
-rw-rw-r-- 1 chyi chyi 5368 Jul 12 12:40 net_tc_mapping.h
-rw-rw-r-- 1 chyi chyi 3775 Jul 12 12:40 net_timeout.c
-rw-rw-r-- 1 chyi chyi 1071 Jul 12 12:40 packet_socket.c
-rw-rw-r-- 1 chyi chyi 904 Jul 12 12:40 packet_socket.h
-rw-rw-r-- 1 chyi chyi 1617 Jul 12 12:40 promiscuous.c
-rw-rw-r-- 1 chyi chyi 24999 July 12 12:40 route.c
-rw-rw-r-- 1 chyi chyi 8900 July 12 12:40 route.h
-rw-rw-r-- 1 chyi chyi 113323 Jul 12 12:40 tcp.c
-rw-rw-r-- 1 chyi chyi 3211 Jul 12 12:40 tcp.h
-rw-rw-r-- 1 chyi chyi 10672 Jul 12 12:40 tcp_internal.h
-rw-rw-r-- 1 chyi chyi 9637 Jul 12 12:40 tcp_private.h
-rw-rw-r-- 1 chyi chyi 12709 Jul 12 12:40 tp.c
-rw-rw-r-- 1 chyi chyi 4598 Jul 12 12:40 tp.h
-rw-rw-r-- 1 chyi chyi 1422 Jul 12 12:40 tp_priv.h
-rw-rw-r-- 1 chyi chyi 4712 Jul 12 12:40 udp.c
-rw-rw-r-- 1 chyi chyi 3370 Jul 12 12:40 udp_internal.h
-rw-rw-r-- 1 chyi chyi 20343 Jul 12 12:40 utils.c
📌 Zephyr is IPv6 based, but here we will look at IPv4.
There isn't much code, so there's no problem looking through it all. For reference, if you follow the packet reception process, it's roughly as follows.
[Figure 2.3] Zephyr network rx flow (packet reception flow diagram)
2.2 TCP/UDP echo server
Okay, let's run an example of a TCP/UDP echo server using port 4242.
[Figure 2.4] STM32 Nucleo F207ZG board (ethernet port connection)
[Figure 2.4] STM32 Nucleo F207ZG board (ethernet port connection)
<Introduction to Echo Server Example>chyi@earth:~/zephyrproject/zephyr$ west build -b nucleo_f207zg samples/net/sockets/ echo_server --pristine...chyi@earth:~/zephyrproject/zephyr$ west flash...
$ sudo minicom -D /dev/ttyACM0
[Figure 2.5] Serial console execution
In the case of networks, there is something called a network shell, so you can directly enter commands as follows on the serial console .
[Figure 2.6] Serial console execution - network shell📌 Even if it's a small system, it has everything you need. 😋
uart:~$ net iface Hostname: zephyr Interface eth0 (0x20002a1c) (Ethernet) [1] =================================================== Link addr : 00:80:E1:4F:AC:DA MTU : 1500 Flags: AUTO_START,IPv4,IPv6 Device: ethernet@40028000 (0x8024de4) Ethernet capabilities supported: 10 Mbits 100 Mbits IPv6 unicast addresses (max 3): fe80::280:e1ff:fe4f:acda autoconf preferred infinite 2001:db8::1 manual preferred infinite IPv6 multicast addresses (max 4): ff02::1 ff02::1:ff4f:acda ff02::1:ff00:1 IPv6 prefixes (max 2): <none> IPv6 hop limit: 64 IPv6 base reachable time: 30000 IPv6 reachable time: 36131 IPv6 retransmit timer : 0 IPv4 unicast addresses (max 1): 192.168.8.101/255.255.255.0 manual preferred infinite IPv4 multicast addresses (max 2): 224.0.0.1 IPv4 gateway : 0.0.0.0
[Figure 2.5] Serial console execution
In the case of networks, there is something called a network shell, so you can directly enter commands as follows on the serial console .
[Figure 2.6] Serial console execution - network shell
📌 Even if it's a small system, it has everything you need. 😋
uart:~$ net iface
Hostname: zephyr
Interface eth0 (0x20002a1c) (Ethernet) [1]
===================================================
Link addr : 00:80:E1:4F:AC:DA
MTU : 1500
Flags: AUTO_START,IPv4,IPv6
Device: ethernet@40028000 (0x8024de4)
Ethernet capabilities supported:
10 Mbits
100 Mbits
IPv6 unicast addresses (max 3):
fe80::280:e1ff:fe4f:acda autoconf preferred infinite
2001:db8::1 manual preferred infinite
IPv6 multicast addresses (max 4):
ff02::1
ff02::1:ff4f:acda
ff02::1:ff00:1
IPv6 prefixes (max 2):
<none>
IPv6 hop limit: 64
IPv6 base reachable time: 30000
IPv6 reachable time: 36131
IPv6 retransmit timer : 0
IPv4 unicast addresses (max 1):
192.168.8.101/255.255.255.0 manual preferred infinite
IPv4 multicast addresses (max 2):
224.0.0.1
IPv4 gateway : 0.0.0.0
uart:~$ net ping 192.168.8.1 PING 192.168.8.1 28 bytes from 192.168.8.1 to 192.168.8.101: icmp_seq=1 ttl=64 time=1 ms 28 bytes from 192.168.8.1 to 192.168.8.101: icmp_seq=2 ttl=64 time=1 ms 28 bytes from 192.168.8.1 to 192.168.8.101: icmp_seq=3 ttl=64 time=0 ms
The code snippet below summarizes the general flow of a TCP/UDP echo server. What's unique is that both TCP and UDP process packet reception as threads (actually, that's not unique at all). 😋
[Figure 2.7] Rough flow of TCP/UDP echo-server code
uart:~$ net ping 192.168.8.1
PING 192.168.8.1
28 bytes from 192.168.8.1 to 192.168.8.101: icmp_seq=1 ttl=64 time=1 ms
28 bytes from 192.168.8.1 to 192.168.8.101: icmp_seq=2 ttl=64 time=1 ms
28 bytes from 192.168.8.1 to 192.168.8.101: icmp_seq=3 ttl=64 time=0 ms
[Figure 2.7] Rough flow of TCP/UDP echo-server code
Among the above contents, for wireguard porting, you need to pay attention to the udp process_udp4 thread and the recvfrom( ) function call part. After decrypting and decapsulating the packet received with udp recvfrom( ) (you get the internal original ip packet), you only need to throw the packet back to the ip stack using the function below (Check Point#1).
recvfrom ( ) -> wireguardif_network_rx () -> wireguardif_process_data_message ( ) -> decrypt_wg_packet () -> net_ipv4_input (struct net_pkt *pkt, bool is_loopback)
[Figure 2.8] WireGuard Packet Input Flow
So far, we have briefly looked at the structure of Zephyr's TCP/IP stack and analyzed the echo server example. In the next chapter, we will create a virtual network driver and think about how to utilize it when porting Wireguard.
[Figure 2.8] WireGuard Packet Input Flow
3. Create a Zephyr Virtual Network InterfaceIn this chapter, we will introduce how to create a virtual interface based on the zephyr/samples/net/virtual example.
The reason why we need to use virtual interface code is because we need something like tun device of linux. That is, since vpn ip is set in virtual interface, when sending packet to peer vpn ip, send function of virtual interface will be called naturally (by routing), and after wireguard encryption processing in this function, when sending packet through actual physical interface, desired processing (encryption and tunneling) will be performed (Check Point#2).
The reason why we need to use virtual interface code is because we need something like tun device of linux. That is, since vpn ip is set in virtual interface, when sending packet to peer vpn ip, send function of virtual interface will be called naturally (by routing), and after wireguard encryption processing in this function, when sending packet through actual physical interface, desired processing (encryption and tunneling) will be performed (Check Point#2).
📌 Please note that IP forwarding must be enabled for smooth communication.
[Figure 3.1] Wireguard communication concept using Linux's tun device📌 In reality, in the case of Linux, the structure is not such that the tun driver's send function is called, but rather, when packets are accumulated in the kernel buffer, they are read and used by the user space application.
[Figure 3.1] Wireguard communication concept using Linux's tun device
📌 In reality, in the case of Linux, the structure is not such that the tun driver's send function is called, but rather, when packets are accumulated in the kernel buffer, they are read and used by the user space application.
virtual_wg_interface_send( ) -> wireguardif_output( ) -> wireguardif_output_to_peer( ) -> wireguard_encrypt_packet( ) -> wireguardif_peer_output( ) -> sendto( )
[Figure 3.2] WireGuard Packet Output Flow
[Figure 3.2] WireGuard Packet Output Flow
First, let's run some example code (samples/net/virtual).
3.1 Virtual Interface Examplechyi@earth:~/zephyrproject/zephyr$ west build -b nucleo_f207zg samples/net/ virtual --pristine
chyi@earth:~/zephyrproject/zephyr$ west build -b nucleo_f207zg samples/net/ virtual --pristine
...chyi@earth:~/zephyrproject/zephyr$ west flash...
$ sudo minicom -D /dev/ttyACM0
[Figure 3.3] Serial console execution
*** Booting Zephyr OS build v3.7.0-rc2-417-g1e20f58c17c1 ***[00:00:01.551,000] <inf> net_virtual_interface_sample: Start application (dev VIRTUAL_TEST/0x8021d2c)[00:00:01.551,000] <inf> net_virtual_interface_sample: My example tunnel interface 3 (VirtualTest-1 / 0x2000142c)[00:00:01.551,000] <inf> net_virtual_interface_sample: Tunnel interface 2 (/ 0x2000131c)[00:00:01.551,000] <inf> net_virtual_interface_sample: Tunnel interface 4 (VirtualTest-2 / 0x2000153c)[00:00:01.551,000] <inf> net_virtual_interface_sample: IPIP interface -1 (0)[00:00:01.551,000] <inf> net_virtual_interface_sample: Ethernet interface 1 (0x2000120c)[00:00:01.552,000] <inf> net_virtual_interface_sample: This interface 3/0x2000142c attached to 2/0x2000131c
uart:~$ net iface -> Prints the currently operating network interface.Interface eth0 (0x2000120c) (Ethernet) [1]===================================================Virtual interfaces attached to this : 2 Link addr : 00:80:E1:4F:AC:DAMTU : 1500Flags: AUTO_START,IPv4,IPv6Device: ethernet@40028000 (0x8021d54)Ethernet capabilities supported: 10 Mbits 100 MbitsIPv6 unicast addresses (max 3): 2001:db8::1 manual preferred infinite fe80::280:e1ff:fe4f:acda autoconf preferred infiniteIPv6 multicast addresses (max 4): ff02::1 ff02::1:ff00:1 ff02::1:ff4f:acdaIPv6 prefixes (max 2): <none>IPv6 hop limit: 64IPv6 base reachable time: 30000IPv6 reachable time: 43513IPv6 retransmit timer : 0IPv4 unicast addresses (max 1): 192.0.2.1/255.255.255.0 manual preferred infiniteIPv4 multicast addresses (max 2): 224.0.0.1IPv4 gateway : 0.0.0.0
Interface net0 (0x2000131c) ( Virtual ) [2]=================================================Interface is down .
Interface net1 (0x2000142c) ( Virtual ) [3]=================================================Interface is down .
Interface net2 (0x2000153c) ( Virtual ) [4]=================================================Interface is down .
Interface net3 (0x2000164c) ( Virtual ) [5]=================================================Interface is down .
As a result of the test, a total of 5 interfaces are displayed, including the physical interface eth0, and virtual interfaces net0 to net3. Also, you can see that the virtual interfaces net0 to net3 are all down.
uart:~$ net iface up 2Interface 2 is up -> A downed interface can be brought up using this shell command.
uart:~$ net iface
Interface eth0 (0x2000120c) (Ethernet) [1]===================================================Virtual interfaces attached to this : 2 Link addr : 00:80:E1:4F:AC:DAMTU : 1500Flags: AUTO_START,IPv4,IPv6Device: ethernet@40028000 (0x8021d54)Ethernet capabilities supported: 10 Mbits 100 MbitsIPv6 unicast addresses (max 3): 2001:db8::1 manual preferred infinite fe80::280:e1ff:fe4f:acda autoconf preferred infiniteIPv6 multicast addresses (max 4): ff02::1 ff02::1:ff00:1 ff02::1:ff4f:acdaIPv6 prefixes (max 2): <none>IPv6 hop limit: 64IPv6 base reachable time: 30000IPv6 reachable time: 44989IPv6 retransmit timer : 0IPv4 unicast addresses (max 1): 192.0.2.1/255.255.255.0 manual preferred infiniteIPv4 multicast addresses (max 2): 224.0.0.1IPv4 gateway : 0.0.0.0
Interface net0 (0x2000131c) (Virtual) [2]=================================================Virtual interfaces attached to this : 3 Virtual name: <unknown>Attached: 1 (Ethernet / 0x2000120c)Link addr : BE:87:79:7C:01:E0MTU : 576Flags: POINTOPOINT,NO_AUTO_START,IPv4,IPv6Device: IP_TUNNEL0 (0x8021d40)IPv6 not enabled for this interface.IPv4 unicast addresses (max 1): <none>IPv4 multicast addresses (max 2): 224.0.0.1IPv4 gateway : 0.0.0.0
Interface net1 (0x2000142c) (Virtual) [3]=================================================Interface is down.
Interface net2 (0x2000153c) (Virtual) [4]=================================================Interface is down.
Interface net3 (0x2000164c) (Virtual) [5]=================================================Interface is down.
3.2 Virtual Interface Code AnalysisWe need to analyze the virtual interface example code, but the general outline is as follows.______________________________________________________________
struct ud { struct net_if *my_iface; // my interface (virtual interface) struct net_if *ethernet; //physical interface(eth0) struct net_if *ip_tunnel_1; //ip tunnel 1 interface (virtual interface) struct net_if *ip_tunnel_2; //ip tunnel 2 interface (virtual interface) struct net_if *ipip; //ipip tunnel interface (virtual interface)};📌 Only one of the four virtual interfaces is required.
static const struct virtual_interface_api virtual_test_iface_api = { .iface_api.init = virtual_test_iface_init , //virtual interface initialization
.get_capabilities = virtual_test_get_capabilities, .start = virtual_test_interface_start , //start virtual interface .stop = virtual_test_interface_stop , //Stop virtual interface .send = virtual_test_interface_send , //virtual interface packet send .recv = virtual_test_interface_recv , //virtual interface packet receive .attach = virtual_test_interface_attach , // Attach a virtual interface to another interface};
//Create a virtual interface. By calling the net_virtual_interface_attach() function, it is connected (bound) to another interface.NET_VIRTUAL_INTERFACE_INIT(virtual_test1, VIRTUAL_TEST, NULL, NULL, &virtual_test_context_data1, NULL, CONFIG_KERNEL_INIT_PRIORITY_DEFAULT; &virtual_test_iface_api, VIRTUAL_TEST_MTU);
main( ){ net_if_foreach(iface_cb, &ud) //Initialize iface by calling iface_cb() callback of 5 interfaces net_virtual_interface_attach() //Attach a specific virtual interface to another virtual or physical interface ... setup_iface() //interface setup i.e. setting ip/netmask, mtu, etc. ...}
<prj.conf>//The ip/netmask values of the virtual interface are set in prj.conf.CONFIG_NET_SAMPLE_IFACE2_MY_IPV6_ADDR="2001:db8:100::1"# TEST-NET-2 from RFC 5737CONFIG_NET_SAMPLE_IFACE2_MY_IPV4_ADDR="198.51.100.1"CONFIG_NET_SAMPLE_IFACE2_MY_IPV4_NETMASK="255.255.255.0"______________________________________________________________
...
chyi@earth:~/zephyrproject/zephyr$ west flash
...
$ sudo minicom -D /dev/ttyACM0
[Figure 3.3] Serial console execution
*** Booting Zephyr OS build v3.7.0-rc2-417-g1e20f58c17c1 ***
[00:00:01.551,000] <inf> net_virtual_interface_sample: Start application (dev VIRTUAL_TEST/0x8021d2c)
[00:00:01.551,000] <inf> net_virtual_interface_sample: My example tunnel interface 3 (VirtualTest-1 / 0x2000142c)
[00:00:01.551,000] <inf> net_virtual_interface_sample: Tunnel interface 2 (/ 0x2000131c)
[00:00:01.551,000] <inf> net_virtual_interface_sample: Tunnel interface 4 (VirtualTest-2 / 0x2000153c)
[00:00:01.551,000] <inf> net_virtual_interface_sample: IPIP interface -1 (0)
[00:00:01.551,000] <inf> net_virtual_interface_sample: Ethernet interface 1 (0x2000120c)
[00:00:01.552,000] <inf> net_virtual_interface_sample: This interface 3/0x2000142c attached to 2/0x2000131c
uart:~$ net iface
-> Prints the currently operating network interface.
Interface eth0 (0x2000120c) (Ethernet) [1]
===================================================
Virtual interfaces attached to this : 2
Link addr : 00:80:E1:4F:AC:DA
MTU : 1500
Flags: AUTO_START,IPv4,IPv6
Device: ethernet@40028000 (0x8021d54)
Ethernet capabilities supported:
10 Mbits
100 Mbits
IPv6 unicast addresses (max 3):
2001:db8::1 manual preferred infinite
fe80::280:e1ff:fe4f:acda autoconf preferred infinite
IPv6 multicast addresses (max 4):
ff02::1
ff02::1:ff00:1
ff02::1:ff4f:acda
IPv6 prefixes (max 2):
<none>
IPv6 hop limit: 64
IPv6 base reachable time: 30000
IPv6 reachable time: 43513
IPv6 retransmit timer : 0
IPv4 unicast addresses (max 1):
192.0.2.1/255.255.255.0 manual preferred infinite
IPv4 multicast addresses (max 2):
224.0.0.1
IPv4 gateway : 0.0.0.0
Interface net0 (0x2000131c) ( Virtual ) [2]
=================================================
Interface is down .
Interface net1 (0x2000142c) ( Virtual ) [3]
=================================================
Interface is down .
Interface net2 (0x2000153c) ( Virtual ) [4]
=================================================
Interface is down .
Interface net3 (0x2000164c) ( Virtual ) [5]
=================================================
Interface is down .
As a result of the test, a total of 5 interfaces are displayed, including the physical interface eth0, and virtual interfaces net0 to net3. Also, you can see that the virtual interfaces net0 to net3 are all down.
uart:~$ net iface up 2
Interface 2 is up
-> A downed interface can be brought up using this shell command.
uart:~$ net iface
Interface eth0 (0x2000120c) (Ethernet) [1]
===================================================
Virtual interfaces attached to this : 2
Link addr : 00:80:E1:4F:AC:DA
MTU : 1500
Flags: AUTO_START,IPv4,IPv6
Device: ethernet@40028000 (0x8021d54)
Ethernet capabilities supported:
10 Mbits
100 Mbits
IPv6 unicast addresses (max 3):
2001:db8::1 manual preferred infinite
fe80::280:e1ff:fe4f:acda autoconf preferred infinite
IPv6 multicast addresses (max 4):
ff02::1
ff02::1:ff00:1
ff02::1:ff4f:acda
IPv6 prefixes (max 2):
<none>
IPv6 hop limit: 64
IPv6 base reachable time: 30000
IPv6 reachable time: 44989
IPv6 retransmit timer : 0
IPv4 unicast addresses (max 1):
192.0.2.1/255.255.255.0 manual preferred infinite
IPv4 multicast addresses (max 2):
224.0.0.1
IPv4 gateway : 0.0.0.0
Interface net0 (0x2000131c) (Virtual) [2]
=================================================
Virtual interfaces attached to this : 3
Virtual name: <unknown>
Attached: 1 (Ethernet / 0x2000120c)
Link addr : BE:87:79:7C:01:E0
MTU : 576
Flags: POINTOPOINT,NO_AUTO_START,IPv4,IPv6
Device: IP_TUNNEL0 (0x8021d40)
IPv6 not enabled for this interface.
IPv4 unicast addresses (max 1):
<none>
IPv4 multicast addresses (max 2):
224.0.0.1
IPv4 gateway : 0.0.0.0
Interface net1 (0x2000142c) (Virtual) [3]
=================================================
Interface is down.
Interface net2 (0x2000153c) (Virtual) [4]
=================================================
Interface is down.
Interface net3 (0x2000164c) (Virtual) [5]
=================================================
Interface is down.
3.2 Virtual Interface Code Analysis
We need to analyze the virtual interface example code, but the general outline is as follows.
______________________________________________________________
struct ud {
struct net_if *my_iface; // my interface (virtual interface)
struct net_if *ethernet; //physical interface(eth0)
struct net_if *ip_tunnel_1; //ip tunnel 1 interface (virtual interface)
struct net_if *ip_tunnel_2; //ip tunnel 2 interface (virtual interface)
struct net_if *ipip; //ipip tunnel interface (virtual interface)
};
📌 Only one of the four virtual interfaces is required.
static const struct virtual_interface_api virtual_test_iface_api = {
.iface_api.init = virtual_test_iface_init , //virtual interface initialization
.get_capabilities = virtual_test_get_capabilities,
.start = virtual_test_interface_start , //start virtual interface
.stop = virtual_test_interface_stop , //Stop virtual interface
.send = virtual_test_interface_send , //virtual interface packet send
.recv = virtual_test_interface_recv , //virtual interface packet receive
.attach = virtual_test_interface_attach , // Attach a virtual interface to another interface
};
//Create a virtual interface. By calling the net_virtual_interface_attach() function, it is connected (bound) to another interface.
NET_VIRTUAL_INTERFACE_INIT(virtual_test1, VIRTUAL_TEST, NULL, NULL,
&virtual_test_context_data1,
NULL,
CONFIG_KERNEL_INIT_PRIORITY_DEFAULT;
&virtual_test_iface_api,
VIRTUAL_TEST_MTU);
main( )
{
net_if_foreach(iface_cb, &ud) //Initialize iface by calling iface_cb() callback of 5 interfaces
net_virtual_interface_attach() //Attach a specific virtual interface to another virtual or physical interface
...
setup_iface() //interface setup i.e. setting ip/netmask, mtu, etc.
...
}
<prj.conf>
//The ip/netmask values of the virtual interface are set in prj.conf.
CONFIG_NET_SAMPLE_IFACE2_MY_IPV6_ADDR="2001:db8:100::1"
# TEST-NET-2 from RFC 5737
CONFIG_NET_SAMPLE_IFACE2_MY_IPV4_ADDR="198.51.100.1"
CONFIG_NET_SAMPLE_IFACE2_MY_IPV4_NETMASK="255.255.255.0"
______________________________________________________________
______________________________________________________________static int virtual_test_interface_send (struct net_if *iface, struct net_pkt *pkt){ struct virtual_test_context *ctx = net_if_get_device(iface)->data;
if (ctx->attached_to == NULL) { return -ENOENT; }
#if 0 return net_send_data (pkt);#else //The code below will be added#endif}______________________________________________________________
static int virtual_test_interface_send (struct net_if *iface,
struct net_pkt *pkt)
{
struct virtual_test_context *ctx = net_if_get_device(iface)->data;
if (ctx->attached_to == NULL) {
return -ENOENT;
}
#if 0
return net_send_data (pkt);
#else
//The code below will be added
#endif
}
______________________________________________________________
We only need one virtual interface. So, after creating one virtual interface, let's remove all the code except the code that attaches it to the physical interface.
<virtual interface> => attach => <physical interface>
3.3 Integrate echo server and virtual interface example codeFrom now on, let's integrate the two programs introduced in Chapters 2 and 3 into one as shown below.
We only need one virtual interface. So, after creating one virtual interface, let's remove all the code except the code that attaches it to the physical interface.
<virtual interface> => attach => <physical interface>
chyi@earth:~/zephyrproject/zephyr/samples/net/sockets/wireguard$ ls -alTotal 44drwxrwxr-x 4 chyi chyi 4096 November 28 18:26 .drwxrwxr-x 25 chyi chyi 4096 November 28 18:26 ..-rw-rw-r-- 1 chyi chyi 596 Nov 28 17:50 CMakeLists.txt-rw-rw-r-- 1 chyi chyi 2384 Jul 12 12:40 Kconfig-rw-rw-r-- 1 chyi chyi 4359 Jul 12 12:40 README.rstdrwxrwxr-x 2 chyi chyi 4096 November 28 13:45 boards-rw-rw-r-- 1 chyi chyi 1618 Nov 28 18:10 prj.conf-rw-rw-r-- 1 chyi chyi 4307 Jul 12 12:40 sample.yamldrwxrwxr-x 2 chyi chyi 4096 Nov 28 18:26 srcchyi@earth:~/zephyrproject/zephyr/samples/net/sockets/wireguard$ cd srcchyi@earth:~/zephyrproject/zephyr/samples/net/sockets/wireguard/src$ ls -laTotal 28drwxrwxr-x 2 chyi chyi 4096 November 28 18:26 .drwxrwxr-x 4 chyi chyi 4096 November 28 18:26 ..-rw-rw-r-- 1 chyi chyi 1196 November 28 17:48 common.h //common header-rw-rw-r-- 1 chyi chyi 3911 November 28 17:53 pack_recv.c //udp packet reception codes-rw-rw-r-- 1 chyi chyi 7668 November 28 18:23 tunnel.c //virutal interface codes-rw-rw-r-- 1 chyi chyi 3481 November 28 17:33 wg_main.c //main routine
chyi@earth:~/zephyrproject/zephyr/samples/net/sockets/wireguard$ ls -al
Total 44
drwxrwxr-x 4 chyi chyi 4096 November 28 18:26 .
drwxrwxr-x 25 chyi chyi 4096 November 28 18:26 ..
-rw-rw-r-- 1 chyi chyi 596 Nov 28 17:50 CMakeLists.txt
-rw-rw-r-- 1 chyi chyi 2384 Jul 12 12:40 Kconfig
-rw-rw-r-- 1 chyi chyi 4359 Jul 12 12:40 README.rst
drwxrwxr-x 2 chyi chyi 4096 November 28 13:45 boards
-rw-rw-r-- 1 chyi chyi 1618 Nov 28 18:10 prj.conf
-rw-rw-r-- 1 chyi chyi 4307 Jul 12 12:40 sample.yaml
drwxrwxr-x 2 chyi chyi 4096 Nov 28 18:26 src
chyi@earth:~/zephyrproject/zephyr/samples/net/sockets/wireguard$ cd src
chyi@earth:~/zephyrproject/zephyr/samples/net/sockets/wireguard/src$ ls -la
Total 28
drwxrwxr-x 2 chyi chyi 4096 November 28 18:26 .
drwxrwxr-x 4 chyi chyi 4096 November 28 18:26 ..
-rw-rw-r-- 1 chyi chyi 1196 November 28 17:48 common.h //common header
-rw-rw-r-- 1 chyi chyi 3911 November 28 17:53 pack_recv.c //udp packet reception codes
-rw-rw-r-- 1 chyi chyi 7668 November 28 18:23 tunnel.c //virutal interface codes
-rw-rw-r-- 1 chyi chyi 3481 November 28 17:33 wg_main.c //main routine
4. Implementing WireGuard in Linux user space (C language version)As we all know, wireguard for linux kernel is implemented in C, but it is not easy to port it to other OS because it is specialized for linux kernel. In addition, there is also a problem with using wireguard-go code implemented in user space because the binary size compiled in Go is too large.📌 There is also a way to make the binary smaller using something like TinyGo, but it is not easy because there are some APIs that are not supported.
So, is there anything that is implemented in C and operates in user space? While I was thinking about this, I found a welcome open source. 😍
That's right, Daniel Hope wrote code in 2021 to make wireguard run on top of LwIP. The two links below are examples based on this code with some modifications (runs on ESP32, etc.).
4.1 wireguard-lwip code analysisTherefore, in this chapter , we will cover the second step of WireGuard porting for Zephyr RTOS, porting the wireguard-lwip code to Linux userspace . The reason for porting to Linux first is to verify in advance that there are no problems in the process of removing the LwIP code.
📌 As you will see later, if you port to Linux first, the process of porting to zephyr becomes much easier (reason: zephyr follows the POSIX standard).
[Figure 4.1] wireguard-lwip source code list
The above code appears to be a level below what was written by Jason A. Donenfeld, the original author of WireGuard, but I think it's still useful.
📌 Don't get me wrong~ The code written by Daniel Hope is also very good. 😂
[Figure 4.2] Wireguard initial setup flow (1)
[Figure 4.3] Wireguard initial setup flow (2)
[Figure 4.4] Packet receiving code main flow
[Figure 4.5] Packet transmission code main flow
Now that we have roughly looked at the wireguard-lwip code, the next step is to understand the operating principle of the Linux tun device and then reflect this in the code. The figure below shows the operating principle of the tun device.
[Figure 4.6] Linux tun device operation principle [Source - Reference 6]Based on this, the procedure for sending or receiving a VPN packet is as follows.
<Tun device based VPN packet transmission process>
First, assign (set) the VPN IP address 10.1.1.100/24 to the tun0 interface.
The app sends data to a peer VPN (e.g. 10.1.1.200). Since 10.1.1.200 belongs to the 10.1.1.0/24 network, the packet is forwarded to the tun0 interface.
Packets transmitted through the tun0 interface do not go out, but are accumulated in the (tun device driver) read buffer.
In the figure above, the VPN client, indicated as Process, receives a packet from the tun0 interface (calls the read function), then performs encryption and tunneling processing, and then sends out the VPN packet to the actual physical interface, eth0 (calls the sendto function).
[Figure 4.7] Userspace wireguard encryption (encapsulation) schematic diagram
<Tun device based VPN packet reception process>
Peer VPN transmits VPN packets to a specific VPN port.
The Process that received the packet, i.e. the VPN Client, writes the packet extracted through the decapsulation & decryption process to the tun0 interface. This is because the src/dst IP of the packet extracted through the decapsulation & decryption process is included in the IP range of the tun0 interface (this means that it is processed based on the routing operation principle).
Since the destination address of the packet transmitted to the tun0 interface is 10.1.1.100, i.e. itself, the packet is transmitted to the application (App).
[Figure 4.8] Userspace wireguard decryption (decapsulation) schematic diagram
4.2 Porting wireguard to run in Linux userlandNow that we have the code analysis and design direction set, let's try to port wireguard-lwip to Linux based on what we've explained above.
chyi@earth:/mnt/hdd/workspace/mygithub_prj/wireguard-c/src$ ls -la -> The actual work done is as follows:Total 200drwxrwxr-x 5 chyi chyi 4096 December 2 16:32 .drwxrwxr-x 6 chyi chyi 4096 December 2 15:41 ..-rw-rw-r-- 1 chyi chyi 957 Dec 1 16:15 Makefiledrwxrwxr-x 2 chyi chyi 4096 Dec 2 16:32 crypto-rw-rw-r-- 1 chyi chyi 2074 Nov 29 15:09 crypto.c-rw-rw-r-- 1 chyi chyi 3147 Dec 1 17:59 crypto.hdrwxrwxr-x 2 chyi chyi 4096 Dec 2 16:32 libdrwxrwxr-x 2 chyi chyi 4096 Nov 30 18:11 lwip_h-rw-rw-r-- 1 chyi chyi 6091 Dec 2 12:09 wg_comm.c-rw-rw-r-- 1 chyi chyi 845 Dec 2 15:27 wg_comm.h-rw-rw-r-- 1 chyi chyi 5153 Dec 2 16:20 wg_config.c-rw-rw-r-- 1 chyi chyi 2316 Dec 2 16:16 wg_config.h-rw-rw-r-- 1 chyi chyi 6486 Dec 2 14:24 wg_main.c-rw-rw-r-- 1 chyi chyi 733 Dec 2 15:28 wg_main.h-rw-rw-r-- 1 chyi chyi 1425 Dec 2 11:37 wg_timer.c-rw-rw-r-- 1 chyi chyi 383 Dec 2 15:28 wg_timer.h-rw-rw-r-- 1 chyi chyi 6172 Dec 2 13:30 wg_tun.c-rw-rw-r-- 1 chyi chyi 929 Dec 2 15:28 wg_tun.h-rw-rw-r-- 1 chyi chyi 2989 Dec 2 15:23 wireguard-platform.c-rw-rw-r-- 1 chyi chyi 3013 April 26 2024 wireguard-platform.h-rw-rw-r-- 1 chyi chyi 40247 April 26 2024 wireguard.c-rw-rw-r-- 1 chyi chyi 10883 Dec 2 15:29 wireguard.h-rw-rw-r-- 1 chyi chyi 3675 Dec 2 16:18 wireguard_vpn.c-rw-rw-r-- 1 chyi chyi 896 Dec 2 16:15 wireguard_vpn.h-rw-rw-r-- 1 chyi chyi 33667 Dec 2 15:24 wireguardif.c-rw-rw-r-- 1 chyi chyi 4795 Dec 2 15:26 wireguardif.h
chyi@earth:/mnt/hdd/workspace/mygithub_prj/wireguard-c/src$ make -> Compilation proceeds without any problems .gcc -W -Wall -D_GNU_SOURCE -g -c wg_main.c -o wg_main.ogcc -W -Wall -D_GNU_SOURCE -g -c wg_comm.c -o wg_comm.ogcc -W -Wall -D_GNU_SOURCE -g -c wg_config.c -o wg_config.ogcc -W -Wall -D_GNU_SOURCE -g -c wg_tun_device_common.c -o wg_tun_device_common.ogcc -W -Wall -D_GNU_SOURCE -g -c wg_tun_device_linux.c -o wg_tun_device_linux.ogcc -W -Wall -D_GNU_SOURCE -g -c wireguard_vpn.c -o wireguard_vpn.ogcc -W -Wall -D_GNU_SOURCE -g -c wireguardif.c -o wireguardif.ogcc -W -Wall -D_GNU_SOURCE -g -c wireguard.c -o wireguard.ogcc -W -Wall -D_GNU_SOURCE -g -c wireguard-platform.c -o wireguard-platform.ogcc -W -Wall -D_GNU_SOURCE -g -c wg_timer.c -o wg_timer.ogcc -W -Wall -D_GNU_SOURCE -g -c crypto.c -o crypto.ogcc -W -Wall -D_GNU_SOURCE -g -c crypto/blake2s.c -o crypto/blake2s.ogcc -W -Wall -D_GNU_SOURCE -g -c crypto/chacha20.c -o crypto/chacha20.ogcc -W -Wall -D_GNU_SOURCE -g -c crypto/chacha20poly1305.c -o crypto/chacha20poly1305.ogcc -W -Wall -D_GNU_SOURCE -g -c crypto/poly1305-donna.c -o crypto/poly1305-donna.ogcc -W -Wall -D_GNU_SOURCE -g -c crypto/x25519.c -o crypto/x25519.ogcc -W -Wall -D_GNU_SOURCE -g -c lib/log.c -o lib/log.ogcc -W -Wall -D_GNU_SOURCE -g -c lib/strlib.c -o lib/strlib.ogcc -W -Wall -D_GNU_SOURCE -g -o wireguard wg_main.o wg_comm.o wg_config.o wg_tun_device_common.o wg_tun_device_linux.o wireguard_vpn.o wireguardif.o wireguard.o wireguard-platform.o wg_timer.o crypto.o crypto/blake2s .o crypto/chacha20.o crypto/chacha20poly1305.o crypto/poly1305-donna.o crypto/x25519.o lib/log.o lib/strlib.o -lpthread
chyi@earth:/mnt/hdd/workspace/mygithub_prj/wireguard-c/etc$ vi wireguard.conf -> Before testing the VPN with a peer, open this file and modify it appropriately.
## wireguard-c configuration file#
#debug flagdebug=1
#Local information ===========================================#Local VPN IPv4 address & subnet maskmy_vpn_ip_address=10.1.1.100my_vpn_netmask=255.255.255.0my_vpn_netmask_CIDR=24
#local wireguard portlocal_wg_port=51820
#local private keylocal_wg_private_key="iHsZNqK/OW7ExUccUkLvAv6ihz787ZjQFXR9l0EbJkU="
#Peer information ===========================================================#Peer vpn ipv4 addresspeer_vpn_ip_address=10.1.1.200
#Endpoint addressendpoint_ip_address=192.168.8.139
#peer wireguard portpeer_wg_port=51820
#peer public keypeer_wg_public_key="isbaRdaRiSo5/WtqEdmpH+NrFeT1+QoLvnhVI1oFfhE="~
Let's run it as follows (for now, it runs without dying).
chyi@earth:/mnt/hdd/workspace/mygithub_prj/wireguard-c/src$ sudo ./wireguard -d ../etc/wireguard.conf[sudo] chyi password: Starting Wireguard VPN
The last thing left is to check the action.
<Test environment>
wireguard kernel (Ubuntu 18.04 10.1.1.200/24 ) <== Switch ==> wireguard-c daemon (Ubuntu 22.04 LTS, 10.1.1.100/24 )
[Figure 4.9] Wireguard daemon running
[Figure 4.10] Ping to VPN peer while wireguard daemon is running
OK, it works as expected. 😎
So, is there anything that is implemented in C and operates in user space? While I was thinking about this, I found a welcome open source. 😍
That's right, Daniel Hope wrote code in 2021 to make wireguard run on top of LwIP. The two links below are examples based on this code with some modifications (runs on ESP32, etc.).
4.1 wireguard-lwip code analysis
Therefore, in this chapter , we will cover the second step of WireGuard porting for Zephyr RTOS, porting the wireguard-lwip code to Linux userspace . The reason for porting to Linux first is to verify in advance that there are no problems in the process of removing the LwIP code.
📌 As you will see later, if you port to Linux first, the process of porting to zephyr becomes much easier (reason: zephyr follows the POSIX standard).
[Figure 4.1] wireguard-lwip source code list
The above code appears to be a level below what was written by Jason A. Donenfeld, the original author of WireGuard, but I think it's still useful.
📌 Don't get me wrong~ The code written by Daniel Hope is also very good. 😂
[Figure 4.2] Wireguard initial setup flow (1)
[Figure 4.3] Wireguard initial setup flow (2)
[Figure 4.4] Packet receiving code main flow
[Figure 4.5] Packet transmission code main flow
Now that we have roughly looked at the wireguard-lwip code, the next step is to understand the operating principle of the Linux tun device and then reflect this in the code. The figure below shows the operating principle of the tun device.
[Figure 4.6] Linux tun device operation principle [Source - Reference 6]
Based on this, the procedure for sending or receiving a VPN packet is as follows.
<Tun device based VPN packet transmission process>
First, assign (set) the VPN IP address 10.1.1.100/24 to the tun0 interface.
The app sends data to a peer VPN (e.g. 10.1.1.200). Since 10.1.1.200 belongs to the 10.1.1.0/24 network, the packet is forwarded to the tun0 interface.
Packets transmitted through the tun0 interface do not go out, but are accumulated in the (tun device driver) read buffer.
In the figure above, the VPN client, indicated as Process, receives a packet from the tun0 interface (calls the read function), then performs encryption and tunneling processing, and then sends out the VPN packet to the actual physical interface, eth0 (calls the sendto function).
[Figure 4.7] Userspace wireguard encryption (encapsulation) schematic diagram
<Tun device based VPN packet reception process>
Peer VPN transmits VPN packets to a specific VPN port.
The Process that received the packet, i.e. the VPN Client, writes the packet extracted through the decapsulation & decryption process to the tun0 interface. This is because the src/dst IP of the packet extracted through the decapsulation & decryption process is included in the IP range of the tun0 interface (this means that it is processed based on the routing operation principle).
Since the destination address of the packet transmitted to the tun0 interface is 10.1.1.100, i.e. itself, the packet is transmitted to the application (App).
[Figure 4.8] Userspace wireguard decryption (decapsulation) schematic diagram
4.2 Porting wireguard to run in Linux userland
Now that we have the code analysis and design direction set, let's try to port wireguard-lwip to Linux based on what we've explained above.
chyi@earth:/mnt/hdd/workspace/mygithub_prj/wireguard-c/src$ ls -la
-> The actual work done is as follows:
Total 200
drwxrwxr-x 5 chyi chyi 4096 December 2 16:32 .
drwxrwxr-x 6 chyi chyi 4096 December 2 15:41 ..
-rw-rw-r-- 1 chyi chyi 957 Dec 1 16:15 Makefile
drwxrwxr-x 2 chyi chyi 4096 Dec 2 16:32 crypto
-rw-rw-r-- 1 chyi chyi 2074 Nov 29 15:09 crypto.c
-rw-rw-r-- 1 chyi chyi 3147 Dec 1 17:59 crypto.h
drwxrwxr-x 2 chyi chyi 4096 Dec 2 16:32 lib
drwxrwxr-x 2 chyi chyi 4096 Nov 30 18:11 lwip_h
-rw-rw-r-- 1 chyi chyi 6091 Dec 2 12:09 wg_comm.c
-rw-rw-r-- 1 chyi chyi 845 Dec 2 15:27 wg_comm.h
-rw-rw-r-- 1 chyi chyi 5153 Dec 2 16:20 wg_config.c
-rw-rw-r-- 1 chyi chyi 2316 Dec 2 16:16 wg_config.h
-rw-rw-r-- 1 chyi chyi 6486 Dec 2 14:24 wg_main.c
-rw-rw-r-- 1 chyi chyi 733 Dec 2 15:28 wg_main.h
-rw-rw-r-- 1 chyi chyi 1425 Dec 2 11:37 wg_timer.c
-rw-rw-r-- 1 chyi chyi 383 Dec 2 15:28 wg_timer.h
-rw-rw-r-- 1 chyi chyi 6172 Dec 2 13:30 wg_tun.c
-rw-rw-r-- 1 chyi chyi 929 Dec 2 15:28 wg_tun.h
-rw-rw-r-- 1 chyi chyi 2989 Dec 2 15:23 wireguard-platform.c
-rw-rw-r-- 1 chyi chyi 3013 April 26 2024 wireguard-platform.h
-rw-rw-r-- 1 chyi chyi 40247 April 26 2024 wireguard.c
-rw-rw-r-- 1 chyi chyi 10883 Dec 2 15:29 wireguard.h
-rw-rw-r-- 1 chyi chyi 3675 Dec 2 16:18 wireguard_vpn.c
-rw-rw-r-- 1 chyi chyi 896 Dec 2 16:15 wireguard_vpn.h
-rw-rw-r-- 1 chyi chyi 33667 Dec 2 15:24 wireguardif.c
-rw-rw-r-- 1 chyi chyi 4795 Dec 2 15:26 wireguardif.h
chyi@earth:/mnt/hdd/workspace/mygithub_prj/wireguard-c/src$ make
-> Compilation proceeds without any problems .
gcc -W -Wall -D_GNU_SOURCE -g -c wg_main.c -o wg_main.o
gcc -W -Wall -D_GNU_SOURCE -g -c wg_comm.c -o wg_comm.o
gcc -W -Wall -D_GNU_SOURCE -g -c wg_config.c -o wg_config.o
gcc -W -Wall -D_GNU_SOURCE -g -c wg_tun_device_common.c -o wg_tun_device_common.o
gcc -W -Wall -D_GNU_SOURCE -g -c wg_tun_device_linux.c -o wg_tun_device_linux.o
gcc -W -Wall -D_GNU_SOURCE -g -c wireguard_vpn.c -o wireguard_vpn.o
gcc -W -Wall -D_GNU_SOURCE -g -c wireguardif.c -o wireguardif.o
gcc -W -Wall -D_GNU_SOURCE -g -c wireguard.c -o wireguard.o
gcc -W -Wall -D_GNU_SOURCE -g -c wireguard-platform.c -o wireguard-platform.o
gcc -W -Wall -D_GNU_SOURCE -g -c wg_timer.c -o wg_timer.o
gcc -W -Wall -D_GNU_SOURCE -g -c crypto.c -o crypto.o
gcc -W -Wall -D_GNU_SOURCE -g -c crypto/blake2s.c -o crypto/blake2s.o
gcc -W -Wall -D_GNU_SOURCE -g -c crypto/chacha20.c -o crypto/chacha20.o
gcc -W -Wall -D_GNU_SOURCE -g -c crypto/chacha20poly1305.c -o crypto/chacha20poly1305.o
gcc -W -Wall -D_GNU_SOURCE -g -c crypto/poly1305-donna.c -o crypto/poly1305-donna.o
gcc -W -Wall -D_GNU_SOURCE -g -c crypto/x25519.c -o crypto/x25519.o
gcc -W -Wall -D_GNU_SOURCE -g -c lib/log.c -o lib/log.o
gcc -W -Wall -D_GNU_SOURCE -g -c lib/strlib.c -o lib/strlib.o
gcc -W -Wall -D_GNU_SOURCE -g -o wireguard wg_main.o wg_comm.o wg_config.o wg_tun_device_common.o wg_tun_device_linux.o wireguard_vpn.o wireguardif.o wireguard.o wireguard-platform.o wg_timer.o crypto.o crypto/blake2s .o crypto/chacha20.o crypto/chacha20poly1305.o crypto/poly1305-donna.o crypto/x25519.o lib/log.o lib/strlib.o -lpthread
chyi@earth:/mnt/hdd/workspace/mygithub_prj/wireguard-c/etc$ vi wireguard.conf
-> Before testing the VPN with a peer, open this file and modify it appropriately.
#
# wireguard-c configuration file
#
#debug flag
debug=1
#Local information ===========================================
#Local VPN IPv4 address & subnet mask
my_vpn_ip_address=10.1.1.100
my_vpn_netmask=255.255.255.0
my_vpn_netmask_CIDR=24
#local wireguard port
local_wg_port=51820
#local private key
local_wg_private_key="iHsZNqK/OW7ExUccUkLvAv6ihz787ZjQFXR9l0EbJkU="
#Peer information ===========================================================
#Peer vpn ipv4 address
peer_vpn_ip_address=10.1.1.200
#Endpoint address
endpoint_ip_address=192.168.8.139
#peer wireguard port
peer_wg_port=51820
#peer public key
peer_wg_public_key="isbaRdaRiSo5/WtqEdmpH+NrFeT1+QoLvnhVI1oFfhE="
~
Let's run it as follows (for now, it runs without dying).
chyi@earth:/mnt/hdd/workspace/mygithub_prj/wireguard-c/src$ sudo ./wireguard -d ../etc/wireguard.conf
[sudo] chyi password:
Starting Wireguard VPN
The last thing left is to check the action.
<Test environment>
wireguard kernel (Ubuntu 18.04 10.1.1.200/24 ) <== Switch ==> wireguard-c daemon (Ubuntu 22.04 LTS, 10.1.1.100/24 )
[Figure 4.9] Wireguard daemon running
[Figure 4.10] Ping to VPN peer while wireguard daemon is running
Finally, you can check out what we've worked on so far on the github below.
5. Porting WireGuard in Zephyr Environment I - EthernetIn this chapter, we will introduce how to port the WireGuard protocol to Zephyr RTOS based on the contents of Chapters 2-4.
<Key Points>1) How to register a new network interface? Using the virtual interface example2) How to receive packets? In reality, after receiving a UDP packet, you can process it by using the wireguard packet recv processing routine.3) After encryption, how do you send the packet through the actual physical interface? It seems like you can send the packet to the ethernet driver after encryption processing in the send routine of the virtual interface.
[Figure 5.1] WireGuard schematic for Zephyr
<Things to note when porting>1) You need to check whether the packet send and recv parts are working properly.2) Check if IP forwarding is enabled.3) Since the wireguard protocol includes time-related fields in the header, the function for calculating time must work properly (if the time does not increase sequentially, the received packet is dropped).4) Pthread code must be appropriately replaced with zephyr thread code.5) The timer function should be replaced with the zephyr function.6) The log function needs to be replaced with zephyr code.7) Check if there are any problems with stack size, etc.
5.1 Porting wireguard for ZephyrThe initial addition of the code looks like this:
[Figure 5.2] Contents of CMakeLists.txt file for wireguard app
There is no problem with compilation either.
[Figure 5.1] WireGuard schematic for Zephyr
<Things to note when porting>
5.1 Porting wireguard for Zephyr
The initial addition of the code looks like this:
[Figure 5.2] Contents of CMakeLists.txt file for wireguard app
There is no problem with compilation either.
chyi@earth:~/zephyrproject/zephyr$ west build -b nucleo_f207zg samples/net/sockets/wireguard --pristine
chyi@earth:~/zephyrproject/zephyr$ west flash
<Note>
The exact reason is not yet known, but an Invalid checksum error (for ICMP packets) occurred during the ping test, so the test was conducted with the code below blocked on one line.
[Figure 5.3] Partial modification of zephyr/subsys/net/ip/icmpv4.c file
5.2 Testing wireguard operation for Zephyr
Let's check what we've worked on so far in the environment below.
<Test environment>
wireguard kernel (Ubuntu 18.04 10.1.1.200/24 ) <== Switch ==> zephyr wireguard app ( 10.1.1.50/24 )
<Test Results>1) Handshake works normally.2) There is nothing wrong with the send routine.3) After receiving the packet, there is no problem with the decryption process.4) The problem is that after decryption, when sending the response, it is sent to eth0 without using the virtual interface send function. 💣 -> But at this time, if you do net ping 10.1.1.200 in zephyr net shell, it goes out to the peer through the virtual interface send function.
<Troubleshooting>pkt = net_pkt_alloc_with_buffer ( eth_if , tot_len, AF_INET, IPPROTO_IP, K_NO_WAIT);->pkt = net_pkt_alloc_with_buffer ( tun_if , tot_len, AF_INET, IPPROTO_IP, K_NO_WAIT);
OK, I replaced the net_if part of the function that allocates net_pkt before calling the net_ipv4_input () function with virtual interface information, and it works fine(The above problem has been solved). 😎
[Figure 5.4] Normal operation of wireguard code for zephyr (1)
If you run the command below in this state, the virtual interface will come up and then wireguard handshaking will be started.
uart:~$ net iface up 2
[Figure 5.5] Normal operation of wireguard code for zephyr (2)
The two pictures below are pinging from a peer machine to zephyr wireguard and capturing packets with tcpdump. Of course, if you do net ping 10.1.1.200 on the zephyr console(net shell), the ping is normal.
$ ping 10.1.1.50
[Figure 5.6] Pinging from Peer (Ubuntu 18.04)
$ sudo tcpdump -i enp2s0 udp port 51820
[Figure 5.7] tcpdump on Peer (Ubuntu 18.04)
WireGuard source code for Zephyr is available on github below(Caution: It's still at the beginning level).
So far, we have looked at the process of implementing wireguard running on Zephyr OS in detail. I would like to express my gratitude to the readers who read to the end. 😎
6. Porting WireGuard in Zephyr Environment II - Wi-Fi
To be continued...
7. References[1] https://www.st.com/en/evaluation-tools/nucleo-f207zg.html
<Note>
The exact reason is not yet known, but an Invalid checksum error (for ICMP packets) occurred during the ping test, so the test was conducted with the code below blocked on one line.
[Figure 5.3] Partial modification of zephyr/subsys/net/ip/icmpv4.c file
5.2 Testing wireguard operation for Zephyr
<Test environment>
wireguard kernel (Ubuntu 18.04 10.1.1.200/24 ) <== Switch ==> zephyr wireguard app ( 10.1.1.50/24 )
[Figure 5.4] Normal operation of wireguard code for zephyr (1)
[Figure 5.5] Normal operation of wireguard code for zephyr (2)
[Figure 5.6] Pinging from Peer (Ubuntu 18.04)
댓글 없음:
댓글 쓰기