Happy Tomato

Hello :wave:

Recently, I’ve upgraded the firmware on one of my routers which has the Tomato firmware installed (FreshTomato mod).

And after setting it up, I’ve returned to the idea of using the custom SSL certificate for its admin web UI. Why? Because I can create a custom (self-signed and trusted) certificate authority, make it trusted by my machines, and then install the certificate issued by that CA on the router ultimately having its web UI better trusted (because once in a while it might so happen that malicious 3rd party may hack the router, swap the certificate and reduce the security of the system).

Another reason for me thinking about this is the following: previously (when I was using the previous version of the firmware) I’ve been using the provided SSL certs by Tomato and restricted web UI to support HTTPS only. But some time later it turned out that the certificate expired, and all the browsers were rejecting communication with the website because of that. Effectively, I became blocked and couldn’t access the admin web UI anymore at that time :cold_sweat:. And I had to re-flash the firmware to get the access back :man_facepalming:.

Is it possible at all to use a custom cert? Maybe, the certificate is built into the firmware? Let’s check!

:warning: Note: In this post, my journey describes the behavior of FreshTomato Version 2024.2 MIPS (the most recent at the moment of writing)

If we enable HTTPS for admin UI at say 1 pm and check the HTTPS endpoint, we’ll get back the expected browser warning about the cert, which after bypassing lets us in into the web UI. And the certificate info shows that it was auto-generated by the router at 1 pm. If we reboot the router at 2 pm, we’ll observe that this time the certificate has been issued at 2 pm. Alright: so it has been regenerated on each boot.

Later I found out that it’s possible to not regenerate the certificate on each boot and instead re-use the previously auto-generated certificate.

I don’t remember exactly the settings that I used with the previous version of the firmware but it looks like previously that was the root cause of my issue: I enabled HTTPS, the router generated a certificate, and after some time (without proper understanding the possible implications) I enabled re-usage of the certificate, and since it wasn’t regenerated and had some expiration threshold it was only a matter of time when it would expire (and lock me out :smile::disappointed::man_facepalming:)

It took several days for me to try out the possible next steps that might lead to success, and fail on them until I finally stopped at the following ones.

Note: I’ve investigated the auto-generated certificates and did my best to replicate them by using the same encryption type, hash size, and additional info like DNS names in order to have better confidence that the router’s software would not fail on my cert (let’s not forget that the firmware size is limited and much of the software present on the router is custom built and do not support all the possible options/formats as you may expect on your Linux distro).

0) :man_student: Out of Curiosity Investigation (Optional)

Let’s take a look at the currently mounted file systems:

mount

Output:

rootfs on / type rootfs (rw)
/dev/root on / type squashfs (ro)
proc on /proc type proc (rw)
tmpfs on /tmp type tmpfs (rw)
devfs on /dev type tmpfs (rw,noatime)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw)

Let’s try to locate the auto-generated certificate:

find / -iname *.crt
find / -iname *.pem
find / -iname *.key
find / -iname *cert*
find / -iname *ssl*
ls -la /etc/*.pem

Output:

find / -iname *.crt
/rom/etc/ssl/certs/ca-certificates.crt

find / -iname *.pem
/rom/etc/ssl/cert.pem
/tmp/etc/cert.pem
/tmp/etc/key.pem
/tmp/etc/server.pem

find / -iname *.key

find / -iname *cert*
/rom/etc/l7-protocols/validcertssl.pat
/rom/etc/ssl/cert.pem
/rom/etc/ssl/certs
/rom/etc/ssl/certs/ca-certificates.crt
/tmp/etc/cert.pem
/usr/sbin/gencert.sh

find / -iname *ssl*
/rom/etc/l7-protocols/ssl.pat
/rom/etc/l7-protocols/validcertssl.pat
/rom/etc/ssl
/rom/etc/ssl/openssl.cnf
/tmp/etc/ssl
/usr/lib/libmssl.so
/usr/lib/libssl.so
/usr/lib/libssl.so.1.1
/usr/sbin/openssl
/usr/sbin/openssl11

ls -la /etc/*.pem
-rw-r--r--    1 root     root          1391 Jul 22 10:46 /etc/cert.pem
-rw-r-----    1 root     root           302 Jul 22 10:46 /etc/key.pem
-rw-r--r--    1 root     root          1693 Jan  1  1970 /etc/server.pem

Let’s see the properties of the auto-generated certificate:

openssl x509 -noout -text -in /etc/cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            6c:9c:0d:f8:bb:18:2a:11
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: emailAddress = root@localhost, C = US, ST = Ohio, L = Columbus, CN = 192.168.1.1, O = FreshTomato, OU = FreshTomato Team, emailAddress = root@localhost
        Validity
            Not Before: Jul 22 08:03:06 2024 GMT
            Not After : Jul 23 08:03:06 2034 GMT
        Subject: emailAddress = root@localhost, C = US, ST = Ohio, L = Columbus, CN = 192.168.1.1, O = FreshTomato, OU = FreshTomato Team, emailAddress = root@localhost
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:f6:4f:45:82:e5:2f:db:58:13:9c:36:85:8e:26:
                    af:03:83:fb:43:3b:d1:73:fc:63:8b:31:24:ff:83:
                    a4:ef:4b:85:19:1a:6c:3b:fb:b8:78:c3:d6:20:61:
                    6e:84:f7:78:99:1c:5d:b0:d0:dd:18:b9:93:bd:af:
                    c1:9c:1a:54:f9
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Key Usage:
                Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
            X509v3 Subject Alternative Name:
                IP Address:192.168.1.1, DNS:192.168.1.1, DNS:HelloWorld, DNS:helloworld
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
            X509v3 Subject Key Identifier:
                A0:94:63:C5:6B:A9:44:FB:DC:6F:95:CD:4C:68:37:EB:7A:D8:AE:68
            X509v3 Authority Key Identifier:
                keyid:A0:94:63:C5:6B:A9:44:FB:DC:6F:95:CD:4C:68:37:EB:7A:D8:AE:68
                DirName:/emailAddress=root@localhost/C=US/ST=Ohio/L=Columbus/CN=192.168.1.1/O=FreshTomato/OU=FreshTomato Team/emailAddress=root@localhost
                serial:6C:9C:0D:F8:BB:18:2A:11

            X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:21:00:f8:db:ee:4e:43:ff:09:c0:fc:60:94:1c:9c:
         49:e9:bd:2b:c7:2e:fe:c4:ea:d1:ff:b4:36:f0:ee:f1:25:f9:
         3b:02:20:39:cf:84:df:db:0c:83:25:16:18:c8:bb:ad:6a:f2:
         12:4d:9d:6e:8d:06:3c:44:bd:92:11:a2:ac:80:b0:74:df

It matches the certificate information we can see in the browser when accessing the HTTPS web UI, so:

  • /etc/cert.pem is the certificate for web UI
  • /etc/key.pem is the private key for this certificate
  • /etc/server.pem is the CA used to sign this certificate

Ideally, as I try to create a certificate with similar properties I’ll also generate the one with SHA256 hashing for signature (Signature Algorithm: ecdsa-with-SHA256). The smaller the signature the smaller the size, and in this case size matters since we’ll be storing data in NVRAM.


The built-in router’s web server is some custom build of the httpd, and it uses /etc/cert.pem and /etc/key.pem files for its SSL endpoint. Obviously, the first and most basic thing that came to my mind for testing the idea of a custom certificate was the following: generate a custom cert, replace the above-mentioned files in the router, and check in the browser (ensuring that the permissions for these new files would match the ones for auto-generated ones via the chmod and chown commands).

The fresh instance of the browser was able to open the HTTPS endpoint after this change, however, the provided certificate was the old one (auto-generated).

Look like the web server caches this info. Alright, then I got the PID and command line of the httpd via the top command, killed it kill -9 <HTTPD_PID>, and restarted the web server with the same command line (which happens to not have any args: just httpd).

Output of the top command on the router

Kill httpd via its PID on the router

After the server restart, the browser was given the correct certificate (created by me):

Browser view of the certificate after httpd restart

But after I accepted in the browser this untrusted cert, I got back an error:

Web error after httpd restart

And here is the error in the router’s command line:

httpd CLI error

I’ve also looked through system logs, but there was nothing useful. And googling this error didn’t help much (at least in the case of this tiny custom build of the httpd).

So, it was not that easy. And I was thinking that it would be a (very) long journey to somehow use my own certificate (perhaps, requiring making a custom build of the firmware).

Luckily, I found a way :wink:


If we have HTTPS web UI enabled and also enabled the option for Save In NVRAM then the auto-generated certificate persists across reboots (instead of being re-generated upon each boot/start of the web server). This is a clue that it might be useful to take a look at NVRAM.

nvram show | less

We can see the HTTPS-related settings:

NVRAM settings related to HTTPS

The value of the https_crt_file looks like a base64 encoded value. Based on the name of the variable, it’s some file. What’s inside?

Let’s decode this value (it’s not that easy since the router does not have the base64 command):

mkdir /tmp/test
cd /tmp/test

nvram get https_crt_file > nvram_file.base64
openssl enc -base64 -v -d -in nvram_file.base64 -out nvram_file -A

But what does it represent: a certificate, a key, or some combination of both?

cat nvram_file

Output:

�VKs�8�ٿ"���c� ɲ,@ccn~�W��a���+`2�$S;{�٩��wi������Y�O�r��e���E�M�b�Z���޵�G�k␦F�k��-Co����_u�[��x�/��ߝ����)�8bB�{�=��_�␦��j��ZI!�������.��HDtX,h�ֱz/E�+� �/�fE���!�8��e��@B��(�����$�pN���
                            h�0�
��r6ߜk��5�\�����<�t4��Ih�*����K���`�␦�,1ZyFz" A��8c�@44�B�4i[���]��0<$��'$/�6�N��lͥ�КY��5��}
            �k����ϸ.���b�Mb�-���u�9X�VbT�$����D;�       *6b����1Ff��M
                                                                     �z1w�����SA7Ҿi������Eh��K"�����.��N�}����
                                                                                                              ��~�eTg�*gf������%v�N�/%��}���no���V��*���c&r�)߬�0��ӕ���4�!��p�g�dԻ�Av�����N;���~�2L-0:���V�Ap�
                                                 T���A�ˡ�b9���M��&�&*����       �"��>g�g�6y     ��@^u!��Q�I�����#lO�`9\���N��嶻��x�/U�[�:��S��H������/y|���(���G
    l���
VY��6J�>��"{����K}��RK��=ͫ�9,,�ا'�G�*��W�]ĕ�ѭ�4����`~�z�1��u���Q�7Dng��v!�%��v4��`��y
�0<C�[�çl�=Y�~Vv�                                                                    CX�;�(���r
                 �����*��H��Ű|2ź�����6i�H+��e�KrX��=��e�E�΍e�� g�?k��]��H�����j�o���\�_�����ju��x��[j���?��������xJ>��W�B!��    g��7����7�<␦(��q��߄�"�1���7�ڡ��g�|�5I�W���A�q��I���j�MQ̸���x��,�a   ��SYu���ڂɤ���`�b g�@��|q:f��W�6�V�l`-��q
?%���9%|�{���~w�>�|���_a�)

After searching and walking inside the try-and-fail loop, I figured it out: that’s a gzipped archive with the certificate and key (both placed inside the etc directory in the archive):

tar -tvzf nvram_file

Output:

-rw-r--r-- 0/0      1391 2024-07-22 12:23:06 etc/cert.pem
-rw-r----- 0/0       302 2024-07-22 12:23:05 etc/key.pem

If we extract and check the certificate, we’ll see that it’s the one being used by the web server.

So, here’s the idea:

  • place our custom certificate with its key in the etc folder in some temporary location
  • gzip this etc folder
  • store base64 encoded value for this archive under the https_crt_file key in NVRAM
  • reboot the router

1) Create a Private Certificate Authority (CA)

This would give us a CA certificate valid for 11000 days:

openssl ecparam -genkey -name prime256v1 -out ca.key

cat > ca.config <<EOF
[req]
distinguished_name = req_distinguished_name
prompt = no

[ ext ]
basicConstraints=CA:TRUE,pathlen:0

[req_distinguished_name]
countryName = UA
stateOrProvinceName = Kyiv
localityName = Kyiv
organizationName = TrickyCat
commonName = TrickyCat Router CA for Blog Post
EOF

openssl req -x509 -new -SHA256 -nodes -key ca.key -days 11000 -out ca.crt -config ca.config -extensions ext

Let’s check it:

openssl x509 -noout -text -in ca.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            58:14:bc:95:be:b9:3a:bc:cb:f5:40:54:ae:8e:f0:87:3c:02:56:62
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C = UA, ST = Kyiv, L = Kyiv, O = TrickyCat, CN = TrickyCat Router CA for Blog Post
        Validity
            Not Before: Jul 22 11:01:11 2024 GMT
            Not After : Sep  3 11:01:11 2054 GMT
        Subject: C = UA, ST = Kyiv, L = Kyiv, O = TrickyCat, CN = TrickyCat Router CA for Blog Post
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:b6:af:0f:09:9e:ec:98:6a:cc:7d:0c:f6:1c:b1:
                    de:8d:e8:ad:c6:32:41:26:fa:90:0f:90:43:4f:04:
                    5b:e6:8c:52:89:e1:68:ad:84:24:08:70:cf:34:fe:
                    27:7f:fe:78:9f:2f:77:0c:93:bb:2c:1e:af:17:02:
                    c4:43:49:8d:0c
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:TRUE, pathlen:0
    Signature Algorithm: ecdsa-with-SHA256
         30:44:02:20:60:4b:27:bf:54:a4:30:18:46:8f:6d:38:40:fc:
         ae:7d:18:f0:5f:9c:d6:a9:53:f8:5f:19:41:00:b2:66:a1:70:
         02:20:61:04:70:e5:a6:fc:1b:c7:69:68:e7:09:6b:c7:2c:dd:
         82:8c:be:32:03:91:5d:2a:2f:fb:c5:16:28:c3:ea:40

2) Create Certificate

openssl ecparam -genkey -name prime256v1 -out server.key

cat > server.config <<EOF
[req]
distinguished_name = req_distinguished_name
prompt = no

[req_distinguished_name]
commonName = TrickyCat HelloWorld Router for Blog Post
countryName = UA
stateOrProvinceName = Kyiv
localityName = Kyiv
postalCode = 666
streetAddress = Elm Street 13
organizationName = TrickyCat
organizationalUnitName = TrickyCat Routers
emailAddress = router-admin@routers.blog.tricky.cat.example.com
EOF

openssl req -new -SHA256 -key server.key -config server.config -nodes -out server.csr -verbose

cat > server.v3_ext <<EOF
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
subjectAltName = @alt_names
extendedKeyUsage = serverAuth

[alt_names]
IP.1 = 192.168.1.1
DNS.1 = 192.168.1.1
DNS.2 = helloworld
DNS.3 = HelloWorld
EOF

openssl x509 -req -SHA256 -extfile server.v3_ext -days 11000 -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt

View the csr & cert:

# openssl req -text -noout -verify -in server.csr
openssl x509 -noout -text -in server.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            2b:a6:a1:e4:5b:a7:b7:47:17:89:f1:44:96:27:e7:df:ab:88:7b:c8
        Signature Algorithm: ecdsa-with-SHA256
        Issuer: C = UA, ST = Kyiv, L = Kyiv, O = TrickyCat, CN = TrickyCat Router CA for Blog Post
        Validity
            Not Before: Jul 22 11:02:07 2024 GMT
            Not After : Sep  3 11:02:07 2054 GMT
        Subject: CN = TrickyCat HelloWorld Router for Blog Post, C = UA, ST = Kyiv, L = Kyiv, postalCode = 666, street = Elm Street 13, O = TrickyCat, OU = TrickyCat Routers, emailAddress = router-admin@routers.blog.tricky.cat.example.com
        Subject Public Key Info:
            Public Key Algorithm: id-ecPublicKey
                Public-Key: (256 bit)
                pub:
                    04:ee:d1:0f:03:32:9f:cb:0a:4c:cd:ba:c6:f8:b7:
                    cd:fd:bf:05:70:a8:cd:fe:f7:ce:a3:5d:0a:bf:60:
                    5b:ec:46:34:01:32:0f:bd:4f:00:c2:e3:dc:f5:eb:
                    32:cf:40:ea:c9:89:d5:e6:48:11:57:16:19:14:87:
                    d3:91:96:2a:de
                ASN1 OID: prime256v1
                NIST CURVE: P-256
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
            X509v3 Subject Alternative Name:
                IP Address:192.168.1.1, DNS:192.168.1.1, DNS:helloworld, DNS:HelloWorld
            X509v3 Extended Key Usage:
                TLS Web Server Authentication
    Signature Algorithm: ecdsa-with-SHA256
         30:45:02:21:00:94:4c:8c:a5:a3:4d:d2:e0:59:19:26:2a:40:
         61:28:35:5b:ee:e4:0c:3c:2f:3e:0a:7b:53:04:9e:d2:4b:a8:
         99:02:20:0a:5a:d7:cc:f1:f5:ee:94:2e:87:3d:33:1a:56:3b:
         77:81:d8:1c:3e:7e:9c:04:2d:15:66:dd:2f:54:97:94:90

3) Configure Admin Access on the Router

3.1) Enable HTTPS Access

Ensure that HTTPS admin access is enabled (for now leave HTTP as a fallback too):

Enable HTTPS Web Admin Access

:white_check_mark: And be sure to select the Save In NVRAM option.

:x: Note: The Regenerate option must NOT be enabled or the router will regenerate the certificate.

3.2) Enable SSH Connections

Enable SSH on the router’s web UI and connect to it via the key (as I did) or password.

Enable SSH Access

4) Log In via SSH and Do Preparations

  • log into the router

    ssh -i "my_router_ssh_key_file" root@192.168.1.1
    
  • create some temporary folder and inside that folder create a folder named etc

    e.g.: create a folder /tmp/trickycat/etc

    mkdir -p /tmp/trickycat/etc
    

5) Upload the Certificate

Upload the generated certificate and private key (e.g. server.crt and server.key from above) from your host machine to the router’s temporary etc folder created above.

The names of target files matter here! They must be cert.pem and key.pem

E.g.:

  • server.crt => cert.pem

  • server.key => key.pem

scp -i "my_router_ssh_key_file" server.crt root@192.168.1.1:/tmp/trickycat/etc/cert.pem
scp -i "my_router_ssh_key_file" server.key root@192.168.1.1:/tmp/trickycat/etc/key.pem

P.S. Or we could generate the CA and certificate on the router, and then copy the CA/cert from the router back to our host machine.

6) Switch Back to Router SSH Session

Configure The Cert Payload

Go back to our temporary folder:

cd /tmp/trickycat

Verify the structure and permissions:

ls -laR
.:
drwxr-xr-x    3 root     root            60 Jul 22 09:19 .
drwxrwxrwx    9 root     root           280 Jul 22 09:19 ..
drwxr-xr-x    2 root     root            80 Jul 22 09:20 etc

./etc:
drwxr-xr-x    2 root     root            80 Jul 22 09:20 .
drwxr-xr-x    3 root     root            60 Jul 22 09:19 ..
-rw-r--r--    1 root     root           964 Jul 22 09:20 cert.pem
-rw-r--r--    1 root     root           302 Jul 22 09:20 key.pem

Create a GZip-ed archive from the etc directory:

tar czf cert.tar.gz etc/

“Upload” the created archive to NVRAM (to a variable named https_crt_file) as a base64-encoded value:

nvram setfb64 https_crt_file cert.tar.gz

Commit the changes in NVRAM to make them persistent:

nvram commit

Reboot the router:

reboot

After the boot, reconnect back to the router via SSH and validate:

# Check the contents of NVRAM:

mkdir /tmp/check
cd /tmp/check

# Save the contents of the NVRAM variable into a file (after base64 decoding)
nvram getfb64 https_crt_file mytest.tgz

# Test the archive
tar tvzf mytest.tgz

# Extract the archive content
# This would create (in the current folder) a directory named `etc`
# with two files:
# - etc/cert.pem
# - etc/key.pem
tar xvzf mytest.tgz

# Check the certificate extracted from the archive
openssl x509 -noout -text -in ./etc/cert.pem

#########################################################

# Check the certificate being used by the web server on the router:
openssl x509 -noout -text -in /etc/cert.pem

7) Earn Trust

Let’s go to web UI:

Our custom certificate in the browser (without machine trust to custom CA)

So, the web server served our custom certificate without a problem. But the cert is still not trusted.

To overcome this, let’s add our custom CA certificate to the trusted store on a machine.

For Windows, this would be the following:

  • open the certificate manager for the current machine

    • open the Windows menu

    • type/search for certificate

    • open the “Manage computer certificates” search result

      "Manage computer certificates" search result

  • go to the Trusted Root Certification Authorities/Certificates folder

    `Trusted Root Certification Authorities/Certificates` folder

  • right click on the free area in the list of CAs and open All Tasks -> Import... or follow the main menu Action -> All Tasks -> Import...

  • this opens the Certificate Import Wizard

  • follow the wizard and import our custom-generated CA certificate (that we used to sign the router’s cert)

    Imported custom CA certificate into the machine store

  • after the import, better reopen the browser (since it may cache the certificate info and this caching survives even the hard reload of pages, at least that’s the experience I have with the Chrome browser) or reboot the machine

  • next, go to the router’s HTTPS admin page and proudly see that it’s now recognized as secure and trusted

    View of the browser trusting our custom certificate after installing custom CA cert

Voila!

:v: