LG WEBOS 4.0 and below. Letsencrypt root certificated expired.

[shdwlynx]

I decided to try downloading and running the update-ca-certs.sh script again and I got the following message:

----------
FIX FAILED
----------

Error: Homebrew Channel init.d directory does not exist

       /var/lib/webosbrew/init.d

Before running this script, ensure you have rooted your TV.

To root your TV, visit https://rootmy.tv/ in your TV's browser.

To learn more about this script, visit https://github.com/tf318/lg

Well, that's interesting. My Homebrew Channel comes up just fine on the TV and shows "Root status" as "OK".

Oddly, while the /var/lib/webosbrew directory did exist, init.d was definitely gone, and I have no idea why/how that happened... but my guess would be that my kids turned the TV on/off quickly a few times and kicked it into Failsafe Mode and maybe that screwed things up.  I recommend following the advice provided by rootmy.tv: 

Quote

It is recommended to have "Quick Start+" functionality enabled. This will make shutdown button on a remote not do a full system shutdown. If you quickly turn the TV on and off without Quick Start+, our "Failsafe Mode" may get triggered (which is there to prevent startup scripts bricking the TV) which will go away after switching relevant switch in Homebrew Channel Settings.

 So, I turned on Quick Start+ and then went back to my very first step, going to rootmy.tv to reinstall Homebrew.  After that process completed (two reboots later), I was able to still ssh into the TV.  I downloaded and ran the update-ca-certs.sh script again (things in /tmp often get nuked upon reboot) without any issues.

But Plex is still not working.

Which is weird because /var/lib/webosbrew/init.d/overlay-letsencrypt-ca-certs-fix exists, and its contents look fine:

#!/bin/bash
# Overlay our custom CA certificate configuration (to replace outdated CA certs) and update trust store
mount --bind /home/certfix-overlay/fixed-ca-certificates.conf /etc/ca-certificates.conf
mount -t overlay overlay -o lowerdir=/etc/ssl,upperdir=/home/certfix-overlay/etc_ssl,workdir=/home/certfix-overlay/work-etc_ssl /etc/ssl
mount -t overlay overlay -o lowerdir=/usr/share/ca-certificates,upperdir=/home/certfix-overlay/usr_share_ca-certificates,workdir=/home/certfix-overlay/work-usr_share_ca-certificates /usr/share/ca-certificates
update-ca-certificates

So, I checked and all those directories and expected files are in the right place.  I tried accessing

link hidden, please login to view

 since it (obviously) must use a Let's Encrypt SSL certificate itself, and got an interesting response:

root@LGwebOSTV:~# curl https://letsencrypt.com
curl: (77) error setting certificate verify locations:
  CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none

The ca-certificates.crt file does not exist in /etc/ssl/certs so I went hunting.

find / -name "ca-certificates.crt"

The file does exist in /home/certfix-overlay/etc_ssl and the mount commands above should be mapping that directory to /etc/ssl/certs.  So, I ran a mount command to see if the mounts are actually working:

root@LGwebOSTV:~# mount | grep cert
/dev/mmcblk0p51 on /etc/ca-certificates.conf type ext4 (rw,nosuid,relatime,data=ordered)
overlay on /etc/ssl type overlay (rw,relatime,lowerdir=/etc/ssl,upperdir=/home/certfix-overlay/etc_ssl,workdir=/home/certfix-overlay/work-etc_ssl)
overlay on /usr/share/ca-certificates type overlay (rw,relatime,lowerdir=/usr/share/ca-certificates,upperdir=/home/certfix-overlay/usr_share_ca-certificates,workdir=/home/certfix-overlay/work-usr_share_ca-certificates)
overlay on /var/palm/jail/netflix/etc/ssl type overlay (ro,nosuid,relatime,lowerdir=/etc/ssl,upperdir=/home/certfix-overlay/etc_ssl,workdir=/home/certfix-overlay/work-etc_ssl)
overlay on /var/palm/jail/com.webos.app.browser/etc/ssl type overlay (ro,nosuid,relatime,lowerdir=/etc/ssl,upperdir=/home/certfix-overlay/etc_ssl,workdir=/home/certfix-overlay/work-etc_ssl)
overlay on /var/palm/jail/com.webos.app.screensaver/etc/ssl type overlay (ro,nosuid,relatime,lowerdir=/etc/ssl,upperdir=/home/certfix-overlay/etc_ssl,workdir=/home/certfix-overlay/work-etc_ssl)

And they are there, as expected.  The mount command meant to affect /etc/ssl is meant to pile the contents of /home/certfix-overlay/etc_ssl on top of /etc/ssl, effectively "temporarily adding" four files (one of which is ca-certifiactes.crt) to /etc/ssl:

root@LGwebOSTV:/home/certfix-overlay/etc_ssl/certs/# ls
DST_Root_CA_X3.pem   ca-certificates.crt  isrgrootx1.pem       lets-encrypt-r3.pem

And yet, it clearly wasn't working, because both DST_Root_CA_X3.pem  and ca-certificates.crt were not present in /etc/ssl.

I kept digging.  I discovered that two of those files were not actually in /home/certfix-overlay/etc_ssl but instead were coming from /usr/share/ca-certificates instead:

root@LGwebOSTV:/etc/ssl/certs# stat lets-encrypt-r3.pem
  File: 'lets-encrypt-r3.pem' -> '/usr/share/ca-certificates/lets-encrypt-r3.crt'
root@LGwebOSTV:/etc/ssl/certs# stat isrgrootx1.pem
  File: 'isrgrootx1.pem' -> '/usr/share/ca-certificates/isrgrootx1.crt'

Yeah, I'm confused.  I think I'm on the right track, but I am going to have to sleep on this and look at it with a fresh brain tomorrow.

[shdwlynx]

Continuing my quest, I checked out the filesystem.  Running the Find Mount command to search for overlays yielded the following:

root@LGwebOSTV:~# findmnt -t overlay
...
|-/etc/ssl                       overlay  overlay rw,relatime,lowerdir=/etc/ssl,upperdir=/home/certfix-overlay/etc_ssl,workdir=/h...
`-/usr/share/ca-certificates     overlay  overlay rw,relatime,lowerdir=/usr/share/ca-certificates,upperdir=/home/certfix-overlay/...

This states that the /usr/share/ca-certificates directory is overlayed by /home/certfix-overlay/usr_share_ca-certificates, and looking at the contents of the two directories confirms that the overlay is working as expected (/usr/share/ca-certificates includes the two certificate files):

root@LGwebOSTV:~# ls /home/certfix-overlay/usr_share_ca-certificates/
isrgrootx1.crt       lets-encrypt-r3.crt

root@LGwebOSTV:~# ls /usr/share/ca-certificates/
isrgrootx1.crt       lets-encrypt-r3.crt  mozilla              sdp

More importantly, it states that /etc/ssl is supposed to be overlayed by /home/certfix-overlay/etc_ssl, but looking at the two directories confirms that it is not working as expected:

root@LGwebOSTV:~# ls /home/certfix-overlay/etc_ssl/certs/
DST_Root_CA_X3.pem   ca-certificates.crt  isrgrootx1.pem       lets-encrypt-r3.pem

root@LGwebOSTV:~# cd /etc/ssl/certs
root@LGwebOSTV:/etc/ssl/certs# ls DST_Root_CA_X3.pem ca-certificates.crt isrgrootx1.pem lets-encrypt-r3.pem
ls: DST_Root_CA_X3.pem: No such file or directory
ls: ca-certificates.crt: No such file or directory
isrgrootx1.pem       lets-encrypt-r3.pem

I went back to /var/lib/webosbrew/init.d/overlay-letsencrypt-ca-certs-fix and disabled it execution with chmod -x then rebooted the TV with reboot now.  Once it turned back on, of course now the overlay mounts didn't automatically run.  My plan is to run each command individually in the file to see how each one works.

Ultimately, I changed the mounting in the fix script above so that it's a bit more limited, mounting to /etc/ssl/certs instead of /etc/ssl: 

mount -t overlay overlay -o lowerdir=/etc/ssl/certs,upperdir=/home/certfix-overlay/etc/ssl/certs,workdir=/home/certfix-overlay/work-etc_ssl_certs /etc/ssl/certs

I had to create that workdir, as it did not already exist:

root@LGwebOSTV:~# mkdir /home/certfix-overlay/work-etc_ssl_certs

I have no idea why this works/made any difference, but now curl now happily accessed letsencrypt.com:

root@LGwebOSTV:~# curl https://letsencrypt.org
<!DOCTYPE html>
<html
        dir="ltr"
        lang="en-US">
(whole bunch of web page goodness)
</html>

And yet, Plex is still not working correctly.  So, I am currently grasping at straws.  Anyone have any?

[C Smith]

22 hours ago, shdwlynx said:

 

On 1/28/2022 at 1:45 AM, C Smith said:

I just had a pop up letting me know about firmware 05.65.03 for my OLED55B6.

Someone has to be the guinea pig to see if this updates the cert issue. Might as well be me.

Did it work?  Or break things for you?  I suspect it will break things since...

Please keep in mind - I don't use Plex, so I don't know if this will be useful to you.

I ultimately found that a combination of the most recent firmware AND a full factory reset got me back to the point where I would be at least allowed to override the cert errors. This is still inconvenient, but manageable. The full factory reset is much less convenient - you better document all apps installed, and be prepared for a couple of hours to put everything back together. I have zero idea as to why this should fix anything at all.

Without a browser update past Chrome 38, the value in the browser is increasingly less as more sites simply refuse to cooperate. The difference between rooting and what I have now would gain me a couple less mouse clicks, nothing more.

I really wish LG saw fit to open source this, as I think the community would be capable of bringing this more up to date. But I also suspect that features such as HDMI DRM make that unworkable.

[shdwlynx]

10 hours ago, C Smith said:

Please keep in mind - I don't use Plex, so I don't know if this will be useful to you.

I ultimately found that a combination of the most recent firmware AND a full factory reset got me back to the point where I would be at least allowed to override the cert errors. This is still inconvenient, but manageable. The full factory reset is much less convenient - you better document all apps installed, and be prepared for a couple of hours to put everything back together. I have zero idea as to why this should fix anything at all.

Without a browser update past Chrome 38, the value in the browser is increasingly less as more sites simply refuse to cooperate. The difference between rooting and what I have now would gain me a couple less mouse clicks, nothing more.

I really wish LG saw fit to open source this, as I think the community would be capable of bringing this more up to date. But I also suspect that features such as HDMI DRM make that unworkable.

Thanks for the update.  At least we know that we can install the update (and then do the factory reset with all the pain that brings) and still get back to a rooted state where we can still install the cert override.  I may still do this just to be as up to date as possible, and since my Plex doesn't work anyhow... 

[C Smith]

9 hours ago, shdwlynx said:

Thanks for the update.  At least we know that we can install the update (and then do the factory reset with all the pain that brings) and still get back to a rooted state where we can still install the cert override.  I may still do this just to be as up to date as possible, and since my Plex doesn't work anyhow... 

I did not try the rooting - I don't think I left open the interpretation that I did, but to be clear - never did it.

I was seriously hoping that LG might actually fix this. Even leaving Chrome 38 in place, it should be possible - really just applying the needed cert updates. I did actually report this to LG, but I am increasingly now expecting them to ignore this.

I did eventually do the factory reset, and re-installed all the apps I wanted to keep, and discovered that the cert errors in the browser could now all be overridden. The TV is never going to be used for banking or email, but it does get used to browse news sites and Wikipedia. However, the other (not connected!) issue is that modern web pages increasingly push several megabytes per page, and the tv browser instance will eventually just give up and say 'too big'. If it is feeling gracious, it just restarts with only the one page - this only usually happens when multiple tabs are open or a 'heavy trail' is being left - things like a mapping site.

[tam]

On 5/1/2022 at 8:29 PM, shdwlynx said:

More importantly, it states that /etc/ssl is supposed to be overlayed by /home/certfix-overlay/etc_ssl, but looking at the two directories confirms that it is not working as expected:

root@LGwebOSTV:~# ls /home/certfix-overlay/etc_ssl/certs/
DST_Root_CA_X3.pem   ca-certificates.crt  isrgrootx1.pem       lets-encrypt-r3.pem

root@LGwebOSTV:~# cd /etc/ssl/certs
root@LGwebOSTV:/etc/ssl/certs# ls DST_Root_CA_X3.pem ca-certificates.crt isrgrootx1.pem lets-encrypt-r3.pem
ls: DST_Root_CA_X3.pem: No such file or directory
ls: ca-certificates.crt: No such file or directory
isrgrootx1.pem       lets-encrypt-r3.pem

When things are working (i.e. you have manually run the update-ca-certs.sh script, the TV has rebooted, and it has automatically run the post-boot /var/lib/webosbrew/init.d/overlay-letsencrypt-ca-certs-fix script), then:

*  DST_Root_CA_X3.pem should not exist in /etc/ssl/certs - the post-boot script explicitly removes its entry from the (overlaid) /etc/ca-certificates.conf, and the content of /etc/ssl/certs is dynamically updated post-boot, based on this configuration file, by the update-ca-certificates command run in the post-boot script. DST_Root_CA_X3.pem is explicitly excluded because it has expired.

* ca-certificates.crt should exist in /etc/ssl/certs - it is created dynamically post-boot, by the update-ca-certificates command run in the post-boot script.

So clearly, as you say, the post-update script was not running correctly for you. I assume this is because the TV must have been in failsafe mode at the time.

I am not entirely sure that the TV's default GUI web browser uses this same system-wide certificate truststore, but may instead have its own (which LG may take more care about updating). So, if one has updated LG firmware, it may be entirely possible that the GUI browser will talk to sites using the new LetsEncrypt certs without any "workarounds" like my script (if LG have updated the browser's proprietary certificate truststore as part of that firmware update). 

I unfortunately do not have access to a Plex server using a new LetsEncrypt cert, so can't try to replicate the issue. I am using it with an Emby server using a new LetsEncrypt cert, and the latest incarnation of the script works well for me (I too enabled the Quite Start+ functionality on the TV as I got fed up of failsafe mode kicking in all the time - the experience has been much smoother since).

[shdwlynx]

3 hours ago, tam said:

When things are working (i.e. you have manually run the update-ca-certs.sh script, the TV has rebooted, and it has automatically run the post-boot /var/lib/webosbrew/init.d/overlay-letsencrypt-ca-certs-fix script), then:

*  DST_Root_CA_X3.pem should not exist in /etc/ssl/certs - the post-boot script explicitly removes its entry from the (overlaid) /etc/ca-certificates.conf, and the content of /etc/ssl/certs is dynamically updated post-boot, based on this configuration file, by the update-ca-certificates command run in the post-boot script. DST_Root_CA_X3.pem is explicitly excluded because it has expired.

* ca-certificates.crt should exist in /etc/ssl/certs - it is created dynamically post-boot, by the update-ca-certificates command run in the post-boot script.

...

I unfortunately do not have access to a Plex server using a new LetsEncrypt cert, so can't try to replicate the issue. I am using it with an Emby server using a new LetsEncrypt cert, and the latest incarnation of the script works well for me (I too enabled the Quite Start+ functionality on the TV as I got fed up of failsafe mode kicking in all the time - the experience has been much smoother since).

First, thanks for replying, Tam, and thanks for all the work you did!  I'm really not sure why the DST_Root_CA_X3.pem file was present at all as it was zero bytes in size, so I just renamed it to *.pem.garbage (I'm sure I could have just deleted it, but I have a hard time letting go sometimes, heh).  I'm just confused as to why I can now use curl to access LetsEncrypt websites without an issue (which makes me think your scripts did indeed work) but Plex is still angry about it.  I need to dig into Plex logs (if any actually exist within the app or on the file system) to see what it's unhappy about.  If you have advice about WebOS app logging in general, I would appreciate hearing any.

[m33ts4k0z]

On 5/3/2022 at 3:54 PM, shdwlynx said:

First, thanks for replying, Tam, and thanks for all the work you did!  I'm really not sure why the DST_Root_CA_X3.pem file was present at all as it was zero bytes in size, so I just renamed it to *.pem.garbage (I'm sure I could have just deleted it, but I have a hard time letting go sometimes, heh).  I'm just confused as to why I can now use curl to access LetsEncrypt websites without an issue (which makes me think your scripts did indeed work) but Plex is still angry about it.  I need to dig into Plex logs (if any actually exist within the app or on the file system) to see what it's unhappy about.  If you have advice about WebOS app logging in general, I would appreciate hearing any.

Hello,

I experience the same  issue with my TV. In my case though, I can see all the certs in their correct place, the script executed without errors but still the Plex app doesnt allow connections. I see that in order for the TV app to create logs, it needs to connect to the server first so since there is no connection, we cant receive any logs. Really silly. Did you find any solution to this?

 

Thanks

[shdwlynx]

On 6/26/2022 at 4:27 AM, m33ts4k0z said:

Hello,

I experience the same  issue with my TV. In my case though, I can see all the certs in their correct place, the script executed without errors but still the Plex app doesnt allow connections. I see that in order for the TV app to create logs, it needs to connect to the server first so since there is no connection, we cant receive any logs. Really silly. Did you find any solution to this?

 

Thanks

Unfortunately, I haven't had time to dig in and have just fallen back to using my old Xbox One to watch Plex.  I was hoping someone smarter than me would have figured it out by now.

[m33ts4k0z]

2 minutes ago, shdwlynx said:

Unfortunately, I haven't had time to dig in and have just fallen back to using my old Xbox One to watch Plex.  I was hoping someone smarter than me would have figured it out by now.

Actually I did find a solution. I used a ZeroSSL certificate that is also free and has to be renewed every 3 months as the lets encrypt one. My use case is only for the Plex app that refuses to connect with the encryption turned on. I just instructed the plex server to use that certificate and it looks like even my 2016 LG works fine with it. I so far tried it with 2 LG TVs with the issue and they both worked at once.

I dont seem to have issues with the built in browser anywho.

[bdjohnso10]

On 12/6/2021 at 7:20 AM, shdwlynx said:

Nobody knows if LG is going to fix it, but honestly, the community already has.  I'll summarize what I did which was based on everyone's helpful instructions above, meant for someone who knows little to nothing about telnet, ssh, etc.  (If you are familiar with those applications, this will be a breeze.  If not, just follow the instructions and search the web if you'd like to learn more about the commands used.)

Note that most of these instructions are taken from 

link hidden, please login to view

 and may be slightly different depending on your TV.

On your TV...

1. Turn on your TV
2. Open the web browser app
3. Browse to rootmy.tv
4. Slide the little "Slide to root" bar to the right if you have the kind of remote that lets you use an on-screen pointer or just press the number 5 on your remote
5. Accept the security prompt
6. The exploit will proceed automatically and the TV will reboot itself; choose to reboot a second time to finalize the installation of the Homebrew Channel (the TV may turn off instead of rebooting - if this happens, just turn the TV back on again)
7. Your TV should now have Homebrew Channel app installed, and an unauthenticated(!) root telnet service exposed
8. Go to the TV's network settings and write down its IP address

Note that your TV must be turned on in order to connect to it remotely; when it's off, it's all the way off (although there may be a non-power-saving setting you can enable to keep the TV's operating system running when it's off so that it will turn on faster, in which case you might be able to connect to it even when it's off).

On your computer...

1. Enable telnet (sorry Mac users, you'll need to figure this out on your own)
   • Click on the Windows Start Menu
   • Type "Windows Features" and click on "Turn Windows features on and off"
   • Scroll down to Telnet client and check the box
   • Click OK
2. Open a command prompt (Win+R then type "cmd")
3. Generate an encryption key pair that will be used to connect securely to the TV
   • Type "mkdir .ssh" to create the secure shell directory if it's not there already
   • Type "ssh-keygen -t rsa -f .\.ssh\lgtv_rsa" (the file name can be anything you want, but that's what I used)
   • Type "notepad .ssh\config"
   • Paste the following contents into the newly created file:

     Host 1.2.3.4
         User root
         IdentityFile C:\Users\yourusername\.ssh\lgtv_rsa

     Replace "1.2.3.4" with your TV's IP address you wrote down in step 8 above
     Replace "yourusername" with the directory name where your user data is stored

   • Save the file and quit Notepad

   • Type "type .ssh\lgtv_rsa.pub" (it will spit out some ugly text - this is your public encryption key)

   • Select all the text and copy it to your clipboard - you'll need this shortly

4. Type "telnet 1.2.3.4", where 1.2.3.4 is your TV's IP address you wrote down in step 8 above, to connect to your TV
   (No username or password is required to connect as the "root" user - we'll fix this security hole in a moment)
5. Type "vi /home/root/.ssh/authorized_keys"
6. Once the new empty file opens, type the letter "i" to enter insert mode
7. Press Ctrl-V to paste in the clipboard contents, which is the public key generated in step 3
8. Press the "Esc" (Escape) key to exit insert mode
9. Type ":wq!" and press Enter to write the file and then quit the editor immediately
10. Type "exit" to exit the telnet application
11. Type "exit" again to exit the command prompt

Back on your TV...

1. Open the Homebrew Channel
2. Go to Settings (the Gear icon)
3. Disable Telnet
4. Enable SSH Server
5. Enable Block system updates
6. Reboot the TV

Back on your computer, apply Tam's SSL fix...

1. Disable telnet (follow the instructions in "On your computer" above but uncheck the box next to Telnet client instead of checking it this time)
2. Open a command prompt (Win+R then type "cmd")
3. Type "ssh 1.2.3.4" (not telnet this time!), where 1.2.3.4 is your TV's IP address, to connect to your TV
   (again, no username or password is required, but it's still secure since the encryption key you generated above is used)
4. Follow Tam's instructions above, repeated here (I suggest copy/pasting the "wget" line from the browser into your command prompt window):
   (Now, if you're like me, the "wget" command will fail because the version running on the TV doesn't have the capability of connecting to secure HTTPS web sites.  Skip to the Alternate Instructions if you have the same issue with the "wget" command.)

   cd /tmp
   wget  https://raw.githubusercontent.com/tf318/lg/main/update-ca-certs.sh
   chmod +x update-ca-certs.sh
   ./update-ca-certs.sh

Alternate instructions

If the "wget" command failed, follow these steps instead:

1. Open Tam's script file in this browser in another tab: 
2. Copy the text and then close that tab
3. Return to the command prompt...
4. Type "vi /tmp/update-ca-certs.sh"
5. Once the new empty file opens, type the letter "i" to enter insert mode
6. Press Ctrl-V to paste in the clipboard contents, which is the Tam's script
7. Press the "Esc" (Escape) key to exit insert mode
8. Type ":wq!" and press Enter to write the file and then quit the editor immediately
9. Type the following two commands:

   chmod +x update-ca-certs.sh
   ./update-ca-certs.sh

 

Whether using Tam's original instructions or my Alternate version, the TV will reboot (twice, maybe) and Plex should now open and be happy.  At least mine was.  Thanks again, RootMy.TV team and Tam, for all your research and hard work!

This works perfectly, thank you so much!

[Pavel Zelenka]

Today I received firmware update for my OLED65B7 and the issue releated to the root certificate was resolved. Now I can use the browser again.

[C Smith]

On 10/11/2022 at 5:46 AM, Pavel Zelenka said:

Today I received firmware update for my OLED65B7 and the issue releated to the root certificate was resolved. Now I can use the browser again.

My 55B6 received version 05.65.15 a couple days ago, and I agree - certificate warnings are no longer a problem when browsing.

[m33ts4k0z]

This is sick but even my 2016 OLED received the update last week and now, no workarounds are needed. Impressed by LG although it took longer than a year to do a so simple update.

[shdwlynx]

TL;DR: The new update from LG works!

I have a OLED65E6P (the most beautiful TV I've ever owned in every way) and while the HomeBrew fix worked when I did it last year, it stopped working for some reason several months ago.  I never tried repeating the steps, so it could have worked if I ran through it again.

I just went in and turned off update-blocking and updated to LG's WebOS 3 05.65.10 (.15 is not available for my TV, apparently) and now SSL works properly in my web browser and Plex is now fully functional again.  HomeBrew puked (so be ready for that), but I could probably put that back on if I wanted to.

[shdwlynx]

Thanks to everyone on this forum for alerting us to this new version!