In this session, I would like to introduce the process of modifying the WireGuard kernel code to create L2 WireGuard that can operate in a Layer 2 environment and running it on NanoPi. 😎

Table of Contents

1. Understanding Layer2 VPN
2. How L2 WireGuard Works
3. L2 WireGuard Operation Test #1

4. L2 WireGuard Operation Test #2

5. Creating L2 VPN with GRETAP + L3 WireGuard

6. The story that hasn't been told yet

7. References

One winter, on the trail

Making WireGuard work on Layer 2 would be an interesting task. If we could change the timing of sending packets from the IP layer to the Ethernet layer, and improve the reception so that non-IP packets such as ARP packets can be received in addition to IP packets, wouldn't it be possible to create a Layer 2 WireGuard? 😹

1. Understanding Layer 2 VPN

I have already introduced in detail how SoftEther VPN, which is synonymous with Layer 2 VPN, works in the blog post below.

https://slowbootkernelhacks.blogspot.com/2020/05/glinet-mv1000-softether-vpn.html

SoftEther VPN has a very good feature set and is a good enough VPN for commercial use (stable and feature-rich), but it has one drawback: it is too complicated to set up (of course, this is my personal opinion 😋).

As you can see in the picture below, Layer 2 VPNs can handle several things that Layer 3 VPNs can't. Of course, these include protocols that aren't used much anymore, like IPX, AppleTalk, and PPPoE, but there are also quite a few things that can only be handled through Layer 2 VPNs, like ARP, 802.3x Pause frame, 802.1q VLAN, 802.1ad QinQ, STP, RIP, LACP, DHCP, and mDNS (printers, AppleTV, Google Chromecast, etc.).

[Figure 1.1] L3 VPN vs L2 VPN

📌 DHCP uses IP broadcast packets, and mDNS uses IP mulicast packets, but layer 3 VPNs cannot handle them.

The image below shows an example of remotely accessing a printer (printing directly from home to an office printer) using the mDNS protocol. This is absolutely impossible with a Layer 3 VPN, but it is not the case with a Layer 2 VPN (it is possible).

[Figure 1.2] Remote Printing Using L2 VPN

📌 If you want to print out a very important document in real time through a printer, wouldn't the above case make sense? Of course, you can encrypt the document and send it by email, and the recipient can open the file and print it, but...

A long time ago, I made a product with SoftEther VPN, and I'll introduce the drawing I made at that time here. I plan to examine whether a similar configuration is possible with L2 WireGuard, which will be explained in Chapters 2-4. 🔍

[Figure 1.3] How SoftEther L2 VPN works

📌 In fact, SoftEther VPN is a VPN that operates in user space, but it is very different from the monotonous way of using the existing tun interface, and it is a very cool VPN. 💯

Now, in the next chapter, let's modify the WireGuard code to create an easy-to-use L2 WireGuard that operates on Layer 2.

2. How L2 WireGuard works

(Personally) I've been using WireGuard for about 7-8 years now. During that time, I've created commercial products based on WireGuard and also worked on changing the protocol.

<Collection of previous posts related to WireGuard>

[1] https://slowbootkernelhacks.blogspot.com/2020/04/espressobin-wireguard-vpn.html

[2] https://slowbootkernelhacks.blogspot.com/2020/05/openwrt-gainstrong-minibox3-wireguard.html

[3] https://slowbootkernelhacks.blogspot.com/2020/09/wireguard-vpn.html

[4] https://slowbootkernelhacks.blogspot.com/2023/01/orangepi-r1-plus-lts-pqc-wireguard-vpn.html

[5] https://slowbootkernelhacks.blogspot.com/2023/02/esp32-wireguard-nat-router-pqc.html

[6] https://slowbootkernelhacks.blogspot.com/2023/02/nanopi-r4s-pq-wireguard-vpn-router.html

[7] https://slowbootkernelhacks.blogspot.com/2023/02/nanopi-r4s-pq-wireguard-vpn-router-part.html

[8] https://slowbootkernelhacks.blogspot.com/2024/01/dpdk-fdio-vpp-security-gateway.html

[9] https://slowbootkernelhacks.blogspot.com/2024/05/nanopi-wireguard-go-quantum-safe-vpn.html

[10] https://slowbootkernelhacks.blogspot.com/2024/12/wireguard-for-zephyr-rtos2.html

______________________________

As we all know, WireGuard was designed to operate basically at Layer 3, that is, IPv4/IPv6 layer. In fact, it can handle almost all protocols just by operating at Layer 3 (there is no problem at all in providing services). However, if WireGuard can be made to operate at Layer 2 like OpenVPN (L2 mode) or SoftEther VPN, that would also be quite exciting. 😍

Simple and fast WireGuard, now extended to Layer 2 !

The content introduced in this chapter is about L2 WireGuard, which modifies the WireGuard kernel code to operate on Layer 2, and is based on the content of the site below. Before going into the detailed explanation, I would like to express my gratitude to the original author, Fadis (NAOMASA MATSUBAYASHI) .

https://speakerdeck.com/fadis/l2-wireguard

📌 The document is in Japanese, so I had a hard time understanding it. But if you look closely, you can understand what it says. 😋

a) Basic design direction of L2 WireGuard

In order for WireGuard to operate at layer 2, the following three things must be considered first.

(1) In order to tunnel the entire Ethernet frame, how can we change the timing of sending the packet from the IP layer to the Ethernet layer?

When creating a wireguard interface (e.g. wg0), wouldn't it be okay to create it as an ethernet device?
To use an analogy, you can think of it as making a tun device into a tap device.

(2) In Ethernet packet communication, there are non-IP packets such as ARP in addition to IP packets. Therefore, what should be done to ensure that non-IP packets such as ARP packets are received in addition to IP packets when receiving packets?

After decoding and decapsulating the received packets, wouldn't it be possible to modify it to pass through non-IP packets?
What are some non-IP protocols? ARP, IPX, Appletalk, NETBEUI, VLAN, LACP...

(3) WireGuard basically performs table lookup(allowed_ips) based on src/dst IP . This is not a problem for IP packets, but it is a problem for non-IP packets. Also, even for IP packets, multicast and broadcast packets are problematic when processed. This part is directly related to the Layer 3 wireguard design idea, and modification is inevitable when switching to the Layer 2 method.

b) Create L2 wireguard interface and change packet transmission part

First, in order to tunnel the entire ethernet frame, we need to create an l2 interface. That means we need to write code to create an l2 wireguard-only interface using the ip link command. To do this, declare a new rtnl_link_ops structure as shown below and register it through the initialization function.

______________________________________________________

[Code 2.1] Declaring rtnl_link ops for l2

📌 For reference, most of the content introduced in this section is in the drivers/net/wireguard/device.c file.

______________________________________________________

[Code 2.2] wg_device_init() function - register l2link_ops

After doing this, you can create a wireguard l2 interface using a command like ip link add dev wg0 type l2wireguard.

For reference, the ether_setup(dev) function must be called in the l2wg_setup( ) function corresponding to the .setup field of l2link_ops so that the interface is recognized as an ethernet device.

______________________________________________________

[Code 2.3] Definition of l2wg_setup function

📌 For ethernet device driver implementation, it is also helpful to refer to the drivers/net/tun.c file (tun/tap driver).

Next, declare a netdevice dedicated to l2 and initialize it (register netdev_ops) as follows using the l2wg_setup( ) function described above.

dev->netdev_ops = &l2netdev_ops;

______________________________________________________

[Code 2.4] Declare netdevice ops for l2

📌 It should be noted that l2 wireguard has two additional fields, .ndo_set_mac_address and .ndo_validate_addr, unlike l3 wireguard netdevice.

Once netdevice is registered, packets (ethernet frames) can be transmitted via the wg_xmit( ) function. The problem is that the existing wg_xmit( ) function is designed to transmit only IP packets, so this function needs to be appropriately processed to ensure that non-IP packets (e.g. ARP packets) are normally transmitted to the peer.

The first part that needs to be modified is the function below. Modify it as shown below so that ARP packets and the like can pass through.

______________________________________________________

[Code 2.5] Modify the wg_check_packet_protocol() function in the queueing.h file.

WireGuard's basic principle is to allow only specific bands of IPv4 and IPv6 packets through the allowed_ips setting. Therefore, it is impossible to prevent non-IP packets such as ARP from being excluded from tunneling by using the existing allowed_ips lookup code as is. Therefore, this function must also be appropriately modified ( in a form where all packets are allowed, i.e. allowed-ips 0.0.0.0/0 ) .

______________________________________________________

[Code 2.6]  wg_allowedips_lookup_src/dst function in allowedips.c file (before modification)

______________________________________________________

[Code 2.7] wg_allowedips_lookup_src/dst function for l2wg (after modification)

Therefore, from now on, in order to use L2 WireGuard, you must set it as " wg set ... allowed-ips 0.0.0.0/0 ". This also means that in order to switch to L2 WireGuard, you will no longer be able to use the concept of allowed-ips.

📌 In some ways, it seems that the basic concept is a bit shaky because L2 Wireguard was implemented while maintaining the basic Wireguard code framework.

c) Change the packet reception part

The next thing to look at is the packet receiving part. The packet receiving part is mostly in the receive.c file, and we need to look at the contents of this file, focusing on the wg_packet_rx_poll( ) function. For reference, please refer to the contents of Chapter 4 of the previous blog post regarding the WireGuard packet receiving processing flow.

https://slowbootkernelhacks.blogspot.com/2020/09/wireguard-vpn.html

Receive encrypted IP tunnel packet (wg_receive) -> Remove tunnel and decrypt ( decrypt_packet) -> Forward to upper level ( wg_packet_rx_poll )

The wg_packet_rx_poll( ) function takes packets that have already been decrypted (decapsulated) from the queue, processes them appropriately, and then sends them up to the tcp/ip stack using the napi_schedule function. In this process, since the existing code is implemented only for IP packets, non-IP packets such as ARP are excluded from the processing target. Therefore, this needs to be appropriately modified, and the function that needs to be modified is wg_packet_consume_data_done( ) .

______________________________________________________

[Code 2.8] Packet receiving function

There are two main parts that need to be modified in the wg_packet_consume_data_done( ) function.

Part that filters IPv4 and IPv6 packets. In other words, part that filters only IPv4 and IPv6 packets.

The part that attempts an allowedips lookup using the source ip address.

As for the part where allowedips lookup (lookup targeting the source address) is attempted when receiving a packet, since this has already been explained (same concept) in the transmission processing (lookup targeting the destination address), we will focus here on implementing the code to prevent non-IP packets from being filtered. To this end, L2 WireGuard additionally implemented a function to obtain the length of a packet as shown below , and modified the code to prevent non-IP packets from being filtered using this function.

______________________________________________________

[Code 2.9] l2wg_get_packet_length( ) function

📌 In this blog post, only ARP was mainly mentioned, but many other aspects such as IPX, Appletalk, 802.1q, QinQ, 802.1ah, pause frame, PPPoE, etc. should be considered when processing L2 VPN.

The code below is an excerpt from the part where the l2wg_get_packet_length( ) function is applied within the wg_packet_consume_data_done( ) function.

...

______________________________________________________

[Code 2.10] The part where the l2wg_get_packet_length( ) function is applied in the wg_packet_consume_data_done() function content

I tried to express the design principles/background of L2 WireGuard in my own way as much as possible, but there are still some awkward parts. 😂 Please read the pdf document written by Fadis to find the missing parts.

--------------------------------------------

So far, we have looked into the basic design idea of L2 WireGuard, the creation of L2 WireGuard interface, and the modifications to the packet transmitter and packet receiver. It is easier said than done, but in reality, the task of converting from L3 to L2 is something that can only be achieved through long experience and trial and error. 😔

In the next chapter, we will verify that the L2 WireGuard implemented in this way really works without any problems using two NanoPis.

3. L2 WireGuard Operation Test #1
In this chapter, we will use one NanoPi R2S Plus and one NanoPi R5C to verify whether L2 WireGuard is operating normally.

I bought a NanoPi R2S Plus and a R5C board from FriendlyElec. The reason is to make L2 WireGuard VPN Routers.

<NanoPi R2S Plus>

https://www.friendlyelec.com/index.php?route=product/product&product_id=296

<NanoPi R5C>

https://www.friendlyelec.com/index.php?route=product/product&product_id=290

The price is very cheap. $36 including case (but excluding power adapter) => $40 is enough even if you get everything!
It has two Ethernet ports (the WAN port is USB 3.0 based) so it can be used as a Security Gateway.
It is equipped with eMMC (32GB). If the structure boots only with microSD, it may be a security issue.
It has a Wi-Fi chipset (this is optional and costs $9.99 extra).
There is a debug interface (USB 3.0) on the outside of the case.
NanoPi is considerably more stable than other Pi types (there is no problem in commercializing it).

[Figure 3.1] NanoPi R2S Plus(1) - Key Features [Source - Reference 12]

[Figure 3.2] NanoPi R2S Plus(2) - Board top/bottom [Source - Reference 12]

[Figure 3.3] NanoPi R4S Plus(3) - Metal Case [Source - Reference 12]

[Figure 3.4] NanoPi R4S Plus(4) - Network Speed [Source - Reference 12]

___________________________________________________________

I have already introduced NanoPi in a previous blog post (Chapter 3). Therefore, please refer to the information below for detailed development environment.

https://slowbootkernelhacks.blogspot.com/2024/05/nanopi-wireguard-go-quantum-safe-vpn.html

📌 The development environment of NanoPi R2S Plus and R5C is not much different.

<The 1st test environment>

[Figure 3.5] L2 WireGuard test environment (left: NanoPi R5C, right: NanoPi R2S Plus)

📌 The product in the middle is my personal favorite, the MT1300 from Gl.iNet.

NanoPi R5C WAN(eth0) <==== Gl.iNet MT1300 (LAN1, LAN2) ====> NanoPi R2S Plus WAN(eth0)

a) Build L2 WireGuard

Based on the changes made so far, let's build the kernel again and create the wireguard.ko file.

$ cd friendlywrt23-rk3328

$ cd kernel/drivers/net/wireguard

$ grep -rl L2_WIREGUARD
=> Based on Fadis' code, I modified the file below and processed the L2_WIREGUARD feature to make it easier to distinguish the code.

Makefile
allowedips.c

allowedips.h

device.c

device.h

netlink.c

queueing.h

receive.c

📌 Note) Since the IPX-related code is not visible in kernel 6.1.63 used in NanoPi, the IPX-related code in the l2 wireguard code was removed for convenience. It seems that they decided that it is no longer used because it is such an old protocol.

$ cd friendlywrt23-rk3328

$ ./build.sh kernel
For NanoPi, the kernel build is performed like this.

...

CC [M] /mnt/hdd/workspace/mini_devices/friendlywrt23-rk3328/scripts/sd-fuse/out/rtl8812au/hal/phydm/rtl8821a/phy
dm_rtl8821a.o
CC [M] /mnt/hdd/workspace/mini_devices/friendlywrt23-rk3328/scripts/sd-fuse/out/rtl8812au/hal/phydm/halrf/rtl882
1a/halrf_iqk_8821a_ce.o
CC [M] /mnt/hdd/workspace/mini_devices/friendlywrt23-rk3328/scripts/sd-fuse/out/rtl8812au/platform/platform_ops.
o
LD [M] /mnt/hdd/workspace/mini_devices/friendlywrt23-rk3328/scripts/sd-fuse/out/rtl8812au/rtl8812au.o
MODPOST /mnt/hdd/workspace/mini_devices/friendlywrt23-rk3328/scripts/sd-fuse /out/rtl8812au/Module.symvers
CC [M] /mnt/hdd/workspace/mini_devices/friendlywrt23-rk3328/scripts/sd-fuse/out/rtl8812au/rtl8812au.mod.o
LD [M] /mnt/hdd/workspace/mini_devices/friendlywrt23-rk3328/scripts/sd-fuse/out/rtl8812au/rtl8812au.ko
make[1]: Exit directory '/mnt/hdd/workspace/mini_devices/friendlywrt23-rk3328/kernel'
'rtl8812au.ko' -> '/mnt/hdd/workspace/mini_devices/friendlywrt23-rk3328/scripts/sd-fuse/out/output_rk3328_kmodules/
lib/modules/6.1.63/rtl8812au.ko'
depmod /mnt/hdd/workspace/mini_devices/friendlywrt23-rk3328/scripts/sd-fuse/out/output_rk3328_kmodules 6.1.63 ...
building kernel ok.
====Building kernel ok!====

$ cd kernel/drivers/net/wireguard

$ ls -l wireguard.ko
-rw-rw-r-- 1 chyi chyi 863608 June 12 14:55 wireguard.ko

$ scp ./wireguard.ko root@192.168.2.1:~/workspace/ l2_wireguard
=> Copy to target board.
root@192.168.2.1's password:
wireguard.ko 100% 843KB 14.4MB/s 00:00

b) Running it on NanoPi (1st test)

First, let's log in to NanoPi R2S Plus and proceed with the Wireguard setup.

$ ssh root@192.168.2.1

root@192.168.2.1's password:
___ _ _ _ __ __ _
| __| _(_)___ _ _ __| | |_ \ \ / / _| |_
| _| '_| / -_) ' \/ _` | | || \ \/\/ / '_| _|
|_||_| |_\___|_||_\__,_|_|\_, |\_/\_/|_| \__|
|__/
----------------------------------------------- ------
FriendlyWrt 23.05.3, r23809-234f1a2efa
------------------------------------ -----------------

📌 NanoPi R2S Plus has a console port (/dev/ttyUSB0, 1500000, 8N1) externally, so you can log in using that.

$ cd ~/workspace/l2_wireguard

root@nanobox-r2s-plus:~/workspace/l2_wireguard# uname -a
Linux nanobox-r2s-plus 6.1.63 #1 SMP Sun Apr 14 15:07:49 KST 2024 aarch64 GNU/Linux

root@nanobox-r2s-plus:~/workspace/l2_wireguard# cp wireguard.ko /lib/modules/6.1.63/wireguard.ko
=> Replace the currently running wireguard kernel module with l2 wireguard.

root@nanobox-r2s-plus:~/workspace/l2_wireguard# sync

root@nanobox-r2s-plus:~/workspace/l2_wireguard# reboot

After rebooting, log in again and configure l2 wireguard settings as follows.

# wg genkey | tee ./privatekey | wg pubkey > ./publickey
=> Generate curve25519 keypair.

# ip link add dev wg1 type l2wireguard
=> You must specify l2wireguard to tunnel the entire ethernet frame.

# ip address add dev wg1 192.168.157.1/24
=> Specify an IP for the wg1 interface. In the case of l3 wireguard, the IP for wgX means the vpn IP, but there is no need to specify it for l2 vpn. The reason for specifying the IP here is to do a ping test.

# ip link set dev wg1 address 56:97:4A:00:00:01
=> In the case of l2 wireguard, since it is an ethernet device, the MAC address must be specified.

# ip link set wg1 up
=> Brings up the l2 wireguard interface.

# wg set wg1 listen-port 51820 private-key ./keys/privatekey peer
FppVj8pZvJupnRI9admT8LPiXcwZ4eILHNhfUyU+XWk= allowed-ips 0.0.0.0/0 endpoint 192.168.8.125:51820

=> Set up l2 wireguard peer.

📌 Note 1) Of course, wireguard interface can be used from wg0, and wg1 is used here (in homage to the original author).
📌 Note 2) As mentioned above, allowed-ips must be set to 0.0.0.0/0 for normal operation.
📌 Note 3) Again, l2 wireguard must have a MAC address set.

Next, let's go through the same setup process on NanoPi R5C. Here, we will only cover the l2 wireguard setup part.

Okay, now that all the basic settings are done, let's check it out. As expected, pinging to the peer vpn ip works fine. 😃

[Figure 3.6] Confirming normal operation of L2 WireGuard(1)

📌 You can see that l2 wireguard has the MTU value set to 1420 - 14 (ethernet header size) = 1406. This part can also be confirmed in the device.c code.

If you check the tcpdump result in this state, you can see that l2 wireguard is operating normally.

[Figure 3.7] L2 WireGuard normal operation verification(2) - wireguard packet capture

📌 If you do a ping test with l3 wireguard, the length comes out as 128 under the same conditions. 144-128 = 16, but it has increased by ethernet header size (14) + 2.

If you tcpdump against the wireguard interface, you will see raw packets (this is also normal).

[Figure 3.8] L2 WireGuard normal operation verification(3) - raw packet capture

So far, we have installed the l2 wireguard kernel module on two NanoPis and verified that l2 wireguard is applied during 1:1 communication between NanoPis.

4. L2 WireGuard Operation Test #2

Well, but, the previous test is lacking by 2%. It is also difficult to tell what has changed compared to l3 wireguard. From now on, I will build the same environment as the one used to test SoftEther VPN mentioned in Chapter 1, and check if l2 wireguard operates normally in this state.

[Figure 4.1] L2 WireGuard tunnel packet

When using L2 VPN, since the Ethernet frame is tunneled as a whole and transmitted to the peer as shown in the figure above, the machine sending the packet (e.g. Notebook on the left of the figure below) can enjoy the effect of being on the same network as the machine receiving it (e.g. server on the right of the figure below). It goes without saying that the IP bands must be the same in order to communicate on the same network. Therefore, if you look at the test content below, after connecting the tunnel, there is a part where the IP of the sending side is changed (manually or DHCP) to the NanoPi R5C internal network band (192.168.3.0/24). This is the reason for this.

📌 This part is conceptually very different from the case of L3 VPN.

<The 2nd test environment>

In reality, we need to test it in an Internet environment, but let's first conduct a test operation in a test environment like the one below.

[Figure 4.2] Second test configuration using L2 WireGuard

The above configuration being possible means that when using L2 VPN, remote DHCP is also possible, as shown in the figure below.

[Figure 4.3] Remote DHCP (including DNS settings) using L2 WireGuard

First, let's set up the NanoPi R2S Plus.

[Figure 4.4] NanoPi R2S Plus network configuration

<NanoPi R2S Plus>
# wg genkey | tee ./privatekey | wg pubkey > ./publickey
=> Generate curve25519 keypair.

# ip link add dev wg1 type l2wireguard
=> You must specify l2wireguard to tunnel the entire ethernet frame.

# wg set wg1 listen-port 51820 private-key ./keys/privatekey peer
FppVj8pZvJupnRI9admT8LPiXcwZ4eILHNhfUyU+XWk= allowed-ips 0.0.0.0/0 endpoint 192.168.8.125 :51820
=> Set device and peer settings. allowed-ips must be set to 0.0.0.0/0.

# ip link set dev wg1 address 02:ca:fe:f0:0e:04
=> Set the MAC address for the wg1 interface. Each company has its own MAC address range, so let's utilize that.

# ip link set wg1 up
=> Link up the wg1 interface.

# ifconfig br-lan down
# brctl delbr br-lan
=> Take down the previously configured br-lan bridge interface.

# ifconfig eth1 0.0.0.0
# ifconfig wg1 0.0.0.0
=> Remove the IP address of the interface to be included in the bridge. In the case of openwrt, eth1 is a LAN interface.

# brctl addbr br-lan
# brctl addif br-lan eth1
# brctl addif br-lan wg1
=> Create a new br-lan interface and include the eth1 and wg1 interfaces in the bridge.

# ip link set dev br-lan address 56:97:4A:F7:7B:2E
=> Sets the address of the br-lan interface. Normally, the MAC address of eth1 is used as is.

# ~~ip address add dev br-lan 192.168.157.1/24~~
=> This setting is actually unnecessary, so remove it.

# ip link set br-lan up
=> Turns the bridge interface up.

_________________________________________________________________________

#!/bin/sh

#wg genkey | tee ./privatekey | wg pubkey > ./publickey

ip link add dev wg1 type l2wireguard
wg set wg1 listen-port 51820 private-key ./keys/privatekey peer FppVj8pZvJupnRI9admT8LPiXcwZ4eILHNhfUyU+XWk=
allowed-ips 0.0.0.0/0 endpoint 192.168.8.125:51820

ip link set dev wg1 address 02:ca:fe:f0:0e:04
ip link set wg1 up

ifconfig br-lan down
brctl delbr br-lan

ifconfig eth1 0.0.0.0
ifconfig wg1 0.0.0.0

brctl addbr br-lan
brctl addif br -lan eth1
brctl addif br-lan wg1
brctl setfd br-lan 1
brctl stp br-lan 1
ip link set dev br-lan address 56:97:4A:F7:7B:2E
ip link set br-lan up

_________________________________________________________________________

[Figure 4.5] Settings on the NanoPi R2S Plus Console

Next, let's proceed with setting up the NanoPi R5C side.

[Figure 4.6] NanoPi R5C network configuration

<NanoPi R5C>
# wg genkey | tee ./privatekey | wg pubkey > ./publickey
=> Generate curve25519 keypair.

# ip link add dev wg1 type l2wireguard
=> You must specify l2wireguard to tunnel the entire ethernet frame.

# wg set wg1 listen-port 51820 private-key ./keys/privatekey peer jkOW74X1CRS8fobzmYYPGDkyVhR5rd Ea9uh8QA2O/0k= allowed-ips 0.0.0.0/0 endpoint 192.168.8.182:51820

=> Set device and peer settings. Allowed-ips must be set to 0.0.0.0/0.

# ip link set dev wg1 address 02:ca:fe:f0:0e:05
=> Set the MAC address for the wg1 interface. Each company has its own MAC address range, so let's utilize that.

# ip link set wg1 up
=> Link up the wg1 interface.

# ifconfig br-lan down
# brctl delbr br-lan
=> Take down the previously configured br-lan bridge interface.

# ifconfig eth1 0.0.0.0
# ifconfig wg1 0.0.0.0
=> Remove the IP address of the interface to be included in the bridge. In the case of openwrt, eth1 is a LAN interface.

# brctl addbr br-lan
# brctl addif br-lan eth1
# brctl addif br-lan wg1
=> Create a new br-lan interface and include the eth1 and wg1 interfaces in the bridge.

# ip link set dev br-lan address 1E:45:77:76:DE:26
=> Sets the address of the br-lan interface. Normally, the MAC address of eth1 is used as is.

# ip address add dev br-lan 192.168.3.1/24
=> You need to set it like this for remote dhcp to work properly (maintain existing settings).

# ip link set br-lan up
=> Turns the bridge interface up.

__________________________________________________________

#!/bin/sh

#wg genkey | tee ./privatekey | wg pubkey > ./publickey

ip link add dev wg1 type l2wireguard

wg set wg1 listen-port 51820 private-key ./keys/privatekey peer jkOW74X1CRS8fobzmYYPGDkyVhR5rd Ea9uh8QA2O/0k= allowed-ips 0.0.0.0/0 endpoint 192.168.8.182:51820

ip link set dev wg1 address 02:ca:fe:f0:0e:05
ip link set wg1 up

ifconfig br-lan down
brctl delbr br-lan

ifconfig eth1 0.0.0.0
ifconfig wg1 0.0.0.0

brctl addbr br -lan
brctl addif br-lan eth1
brctl addif br-lan wg1
brctl setfd br-lan 1
brctl stp br-lan 1
ip link set dev br-lan address 1E:45:77:76:DE:26

ip address add dev br-lan 192.168.3.1/24
ip link set br-lan up

__________________________________________________________

<Linux Notebook>
With the L2 wireguard tunnel created, manually change the IP settings of the Linux notebook (on the left) as shown below.

(Old) 192.168.2.200/24 => (Modified) 192.168.3.200/24

Alternatively, DHCP settings are also possible, as in Figure 4.3.

[Figure 4.7] Setting up DHCP on a Linux notebook

📌 When DHCP is set, the entire DHCP packet (including broadcast packets) is encrypted and then transmitted to the NanoPi R5C's LAN. After going through the decryption and decapsulation process, it is transmitted to the dnsmasq server to receive the final DHCP response.

[Figure 4.8] DHCP Protocol Operation Principle [Figure Source - Reference 10]

After that, try pinging server 192.168.3.139. OK, it pings (meaning l2 wireguard tunneling is working).

[Figure 4.9] Ping between Linux notebook => Server

Let's try the opposite, server -> Notebook. It also pings. It feels as if the two machines are on the same network 192.168.3.0/24.

[Figure 4.10] Ping between Linux notebook and server

Looking at the tcpdump results, we can see that l2 wireguard is operating normally.

[Figure 4.11] Linux notebook => tcpdump between Server (captured on R2S Plus)

If you drop the tcpdump contents as a pcap file as shown below and check it with wireshark, you can more easily check whether it is a wireguard packet.

# tcpdump -i eth0 -w lgwg.pcap

[Figure 4.12] Wireshark view between Linux notebook => Server

Finally, the wireguard packet transfer status was checked and all was normal.

# wg show wg1

[Figure 4.13] wg show wg1 execution (R2S Plus)

br-lan is a bridge interface, and since l2 wireguard (wg1) is attached to it, the fact that ping is possible implicitly means that the ethernet frame is tunneled properly.

Oh, but up to this point, I thought everything was working perfectly in the test, but there was one big problem. That is, ping to the internet, dns, etc. are working fine, but web, ssh, ftp, etc. are not working well. Why is that?

While trying to figure out the cause of the problem, I tried to connect to the web from the Linux notebook and ran tcpdump on the NanoPi R5C side, and I saw that there was a problem where no response was received to the ARP request , as shown below .

<NanoPi R5C>

# tcpdump -i eth0

21:45:56.547525 ARP, Request who-has 112.175.235.196 tell 192.168.8.182 , length 46

__________________________________________________________________________

Looking at the content, it is asking 192.168.8.182, i.e. NanoPi R2S Plus (wan IP), to tell the MAC address of 112.175.235.196 (web server). If it were not an L2 VPN situation, Gl.iNet MT1300 would have immediately sent an ARP reply.

(No L2 VPN situation) Linux Notebook -> NanoPi R2S Plus(wan: 192.168.8.182) -> Gl-iNet MT1300

However, in the L2 VPN situation, the ARP Reply does not go to the NanoPi R2S Plus via the L2 Wireguard as shown in the figure below, but rather the response goes directly ( B - ARP Reply ). Therefore, in this case, if the NanoPi R5C in the middle acts as an ARP proxy (instead, sends an ARP reply with its own MAC address), the problem can be solved ( C - ARP Reply ).

[Figure 4.14] Situations where Proxy ARP settings are required

📌 ARP packets cannot leave the same broadcast domain.

# echo 1 > /proc/sys/net/ipv4/conf/br-lan/proxy_arp

=> You need to do this to receive an ARP reply to the ARP request requested from R2S Plus.

OK, I tried to connect to the web again in this state and it works fine. Cool~ 🍺

[Figure 4.15] Internet connection using L2 WireGuard

📌 As shown in the picture, the Notebook on the left has changed its location to the right, and operates as if it is on the same network as the Server.

_______________________________________________________________

For reference, if you use L2 WireGuard, the following basic network configuration is possible. You can enjoy the same effect as if you were connecting a very long LAN cable from your home to your office.

[Figure 4.16] L2 WireGuard network configuration example 1 - Gateway configuration

If you use L2 WireGuard, you can also create a VPN configuration that operates (standalone server) without changing the office network configuration (right side of the figure below).

[Figure 4.17] Network configuration example 2 for L2 WireGuard VPN - Standalone server configuration

Above, I analyzed the source code of L2 WireGuard implemented by Fadis (NAOMASA MATSUBAYASHI) and introduced the actual operation process in the NanoPi environment. L2 WireGuard is pretty great~ 😍

5. Creating L2 VPN with GRETAP + L3 WireGuard

In this chapter, I will briefly introduce how to create an L2 VPN by combining GRE Tunnel and WireGuard. It would also be interesting to compare the differences with L2 WireGuard, which was explained in the previous chapter. 👌

Implementing Layer 2 VPN using GRETAP over L3 WireGuard
See the site below

https://gist.github.com/zOrg1331/a2a7ffb3cfe3b3b821d45d6af00cb8f6

https://notes.superlogical.ch/pages/note_wg/nolayer2/

[Figure 5.1] GRE header [Image source - Reference 11]

📌 The GRE header is usually 4 bytes, and optional fields can be added.

[Figure 5.2] Tunneling configuration using GRE [Image source - Reference 11]

Since the background explanation has already been sufficiently provided in the previous chapter, only the setup method and test results will be mentioned here.
(Of course) before the test, wireguard.ko must be replaced with the original version (l3 wireguard) and then the test must be conducted.

<Test environment>

[Figure 5.3] GRETAP + WireGuard operation principle (1)

[Figure 5.4] GRETAP + WireGuard operation principle (2)

First, let's proceed with setting up the NanoPi R2S Plus (Client side).

<NanoPi R2S Plus>

# wg genkey | tee ./privatekey | wg pubkey > ./publickey

# ip link add dev wg0 type wireguard
# ip address add dev wg0 10.1.1.1/24
# ip link set wg0 mtu 1402 up

# wg set wg0 listen-port 51820 private-key ./keys/privatekey peer FppVj8pZvJupnRI9admT8LPiXcwZ4eILHNhfUyU+XWk= allowed-ips 0.0.0.0/0 endpoint 192.168.8.125:51820

# ip link add name gretap1 type gretap local 10.1.1.1 remote 10.1.1.2
# ip link set gretap1 up

# ifconfig br-lan down
# brctl delbr br-lan
# ifconfig eth1 0.0.0.0

# brctl addbr br-lan
# brctl addif br -lan eth1
# brctl addif br-lan gretap1

# brctl setfd br-lan 1
# brctl stp br-lan 1
# ip link set br-lan up

Next, let's proceed with setting up the NanoPi R5C side (server side).

<NanoPi R5C>

# wg genkey | tee ./privatekey | wg pubkey > ./publickey
# ip link add dev wg0 type wireguard
# ip address add dev wg0 10.1.1.2/24
# ip link set wg0 mtu 1402 up
# wg set wg0 listen-port 51820 private-key ./keys/privatekey peer jkOW74X1CRS8fobzmYYPGDkyVhR5rdEa9uh8QA2O/0k= allowed-ips 0.0.0.0/0 endpoint 192.168.8.182:51820

# ip link add name gretap1 type gretap local 10.1.1.2 remote 10.1.1.1
# ip link set gretap1 up

# ifconfig br-lan down
# brctl delbr br-lan
# ifconfig eth1 0.0.0.0

# brctl addbr br-lan
# brctl addif br-lan eth1
# brctl addif br-lan gretap1
# brctl setfd br-lan 1
# brctl stp br-lan 1

# ip address add dev br-lan 192.168.3.1/24
# ip link set br-lan up

# echo 1 > /proc/sys/net/ipv4/conf/br-lan/proxy_arp

📌 Although it was not specifically mentioned when explaining L2 WireGuard, both L2 WireGuard and GRETAP + WireGuard may require TCPMSS settings if necessary.

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

<Linux Notebook>

(Client side) Change the IP setting of the Linux notebook to DHCP.

[Figure 5.5] Checking the routing table in Linux Notebook

In this state, try pinging the default gateway, i.e. 192.168.3.1. OK, it pings.

Try connecting to the Internet with a web browser. OK, it's connected properly. Cool~ ☕

<NanoPi R2S Plus>

# tcpdump -i eth0

[Figure 5.6] tcpdump on NanoPi R2S Plus

Oh, it's better than I thought. The speed doesn't seem that slow either... Hmm, this method is quite useful, too. 😁

6. The story that hasn't been told yet

1) Apply L2 WireGuard between Client and Server located in the same LAN

A situation where you want to communicate encrypted between two machines on the same LAN without changing IP settings.
How to solve loop problem? Is it possible?

[Figure 6.1] L2 WireGuard between two machines on the same LAN

2) Applying PQC Algorithm(CRYSTALS Kyber KEM) to L2 WireGuard

As briefly introduced in a previous blog post, let's try applying CRYSTALS Kyber KEM to L2 WireGuard.

https://slowbootkernelhacks.blogspot.com/2023/02/nanopi-r4s-pq-wireguard-vpn-router-part.html

3) Applying L2 WireGuard to wireguard-go code

Let's apply l2 wireguard to wireguard-go code for userspace.

4) Connecting L2 WireGuard and FD.io VPP

Let's connect VPP and L2 WireGuard using the tap interface.
Wouldn't that be possible?

https://slowbootkernelhacks.blogspot.com/2024/06/dpdk-fdio-vpp-security-gateway.html

To be continued...

________________________________________________________

WireGuard® is a registered trademark of Jason A. Donenfeld.

May The Source Be With You

7. References

[1] https://speakerdeck.com/fadis/l2-wireguard?slide=2

[7.10] https://slowbootkernelhacks.blogspot.com/2024/12/wireguard-for-zephyr-rtos.html
[8] https://gist.github.com/zOrg1331/a2a7ffb3cfe3b3b821d45d6af00cb8f6

[9]  https://notes.superlogical.ch/pages/note_wg/nolayer2/
[10]  https://www.netmanias.com/ko/post/blog/5348/dhcp-ip-allocation-network-protocol/understanding-the-basic-operations-of-dhcp
[11]  https://info.support. huawei.com/info-finder/encyclopedia/en/GRE.html

[12] https://www.friendlyelec.com/index.php?route=product/product&product_id=296
[13] And Google~

Slowboot