pfSense and PPPoE

pfSense is perhaps not a good idea for most home networks

Being in IoT industry as a tech guy means that you'll definitely get exposed to networking a lot. And one of the debates that essentially comes up is using OpenWRT vs pfSense. Following is a summary of what most feedback I'd uptil now.

pfSense - Runs on x86 / x86_64. Security focused. Based on FreeBSD. Has intrusion detection and prevention. Advanced Enterprise level features.

OpenWRT - Runs on small to large any hardware footprint. Oriented for performance and light footprint. Based on Linux. Can be configured to be secure etc, but hard to configure out of the box.

I'd been running an OpenWRT on Pi4 as my main router for over an year, and everything used to run fine except for the daily restart to renew the DHCPv6 leases (maybe it's my bad configuration of BSNL to blame here, but IDK yet about this). It has support for multi-WAN and failover, built-in wireguard, port-forwards, routing etc and basic zones and firewall configuration.

But one annoying thing that is an issue with OpenWRT is that base images have a very small partition table, and if you change it, in-place upgrades become very difficult. This issue is further compunded by the face that I use USB-Gigabit dongles to provide as additional interface for my device. And these require additional packages to be installed, which are lost during the whole upgrade. And this was an unacceptable security issue, since this router also faces the internet and hence it is not advisable to leave it as is.

Thus after much thought, I started with testing pfSense inside a Proxmox VM. Till here, everything was fine. In fact, I was impressed with ease and the better UI. So I tried to pass my WAN connection interface directly to the VM, to let it handle the internet connection for me. And thus began the long tale.

At first the speeds were horrible. I read around a lot, and it turned out the PPPoE connections required good single core speed, and apparently running it inside a VM hampered the performance. I tried passing more and more features and enabling various flags recommended on Reddit and support forums, but to no help. On a 300 Mbps symmetric connection, I was getting 70Mbps download! I blamed it on my tiny NUC running the Proxmox VM perhaps ineffectively.

Hence, I flashed the Proxmox directly on an old NUC, configured as 4C / 8 GB RAM / 256 GB SSD. However the results were slightly better in upload, but almost identical in download.  And this was after debugging strange issues like FreeBSD not liking my PenDrive to boot from it! This again I identified after a lot of searching the forums etc.

After spending more than 48 hours on this project. I just gave up. I just turned on my RPi4, and it performed perfectly, with less resources. I did an in-place upgrade, wasted a few hours to manage the above issues. And everything works fine now. I'd have liked the ability to expoit better features for security, but frankly a 5X drop in performance or increase in cost for a personal network is not worth it. And PPPoE is a very common protocol for home networks even today. I assumed that such a basic industry standard would be a focus area for performance improvement, but it simply does not exists. Also I noticed how people who were posting high benchmarks, were running instances on Xeons, sometimes with hardware accelerated interfaces. Thus, pfSense is indeed aimed and tuned for enterprise only. It won't be cost effective for home lab running a PPPoE WAN connection.

This experience made me realize, that once I used to romanticise the idea of using a FreeBSD native node, tuned for reliability and performance. How the nodes could be effectively tuned to run native machine code. And indeed in my college days, I did that and gained performance as well over linux. But now, Linux is indeed has come so far ahead, that having opinions from even 5 years ago can hurt you. Even with less powerful CPU, my OpenWRT barely hits the CPU at all, while Proxmox was using 100% of a single core as soon as I ran benchmarks.

In future, I'll try to blog about the router configuration and certain enhancements that I've configured for OpenWRT.