Oh, I backup religiously since Blue failed right after I moved and backup my backups on my laptop as well. (literally failed; I lost everything and had to run photorec and three other tools to pick out everything I'd done for the previous six months, since that I hadn't copied to a backup on my server because I was prepping to move at the time).

So far, OTBR is the biggest stopping issue since HA runs it but nothing sticks. I admit, moving zwave is my actual biggest dread; zigbees I can do probably in a weekend, but zwave is such hell to unpair and re-pair (thought it makes up for it by sticking forever). That's part of the reason I love Thread and Matter; they're almost as sticky as zwave once they pair, and while pairing them is variable (sometimes fast, sometimes not so much) they repair themselves pretty consistently if the outage is under 24 hours and you can deliberately unpair them fairly easily.

I've been running Home Assistant for roughly five-six years (Pi, then Blue, now Amber and a second instance on my server for network integrations like nmap and netgear), but since my SmartThings hub was taking care of zigbee/zwave, until now I used HA as a coordinator for every smart device ecosystem I was using (Hue, Wyze, Ring, Blink, Alexa, August, Arlo, et al). Sorry that wasn't clear.

While Ive started slowly adding zigbee devices directly, I haven't started with zwave and thread isn't working for me yet (OTBR is running but nothing sticks). And I really don't want to have my hub fail and all my thread/matter devices useless when I don't have anything that can access them.

So far, the OTBR on HA isn't working, but...if it's an age and device issue, it may be migrating the zigbee and zwave over to HA and leave my SmartThings for my OTBR devices will work for now. That may at least buy some time to work out how to make the HA's OTBR work.

It's not reliable on thread/matter.

I use zigbee, zwave, and thread. I've migrated half my contact sensors, a few motion and presence, and two rooms of light bulbs to thread, which HA is...questionable on, hence beginning my search.

The shell integration is why this happened.; I wanted to run the update script as a service so it could be triggered when the Supervisor or Core versions changed so it would automatically symlink my scripts in /usr/local/bin in the ssh_addon container. The shell integration runs in the homeassistant container, so that's when it became complicated.

So it can be done, it just--required a lot of steps and me making a mapping spreadsheet of all the containers. But! Automations and scripts run in the homeassistant container, while when you ssh, you're going into the ssh addon container which should have been obvious and really was once I finished mapping all the containers.

Goal: I need /usr/local/bin in the ssh container so I can run scripts over ssh and access my function library script easily without ./path/to/script.

Summary: ssh into HAOS from the homeassistant container with an HAOS root user (port 22222), run docker exec to get into the ssh addon container, then make your symlinks for /usr/local/bin.

(Note: this is ridiculously complicated and I know there has to be a better way. But this works so I win.)

1. Get access to HAOS itself as root: https://developers.home-assistant.io/docs/operating-system/debugging. Verify you can login successfully.
2. In homeassistant container:

• a. create an .ssh folder (/config/.ssh)
• b. add the authorized_keys file you made for step one.
• c. add the public and private keys you made for step one (should be in the ssh addon container).
• d. set permissions;

chmod 600 /config/.ssh/authorized_keys
chmod 600 /config/.ssh/PRIVATE_KEY
chmod 644 /config/.ssh/PUBLIC_KEY
chmod 700 /config/.ssh

• e. In /config/shell_scripts.yaml or wherever you put your shell scripts, add the script you want to use to update /usr/local/bin: UPDATE_BIN_SCRIPT: /config/shell_scripts/UPDATE_BIN_SCRIPT
• f. Restart HA.
• g. Check it in Developer Tools->Services

I have no idea how consistent the ssh addon container name is usually but it's different on all three of my installs, so insert your container name for SSH_ADDON_CONTAINER_NAME

Steps: login to HAOS, go into the SSH Container, and do the update. This is horribly messy but hey, it works.

UPDATE_BIN_SCRIPT

#!/bin/bash

# OPTIONAL: Update some of the very outdated alpine packages in both homeassistant and the ssh addon (figlet makes cool ascii art of my server
# name).   You'll need to run it twice; once for the homeassistant container, then again in the ssh container.  Assuming you want to update packages,
# anyway
# update homeassistant container packages
apk add coreutils figlet iproute2 iw jq ncurses procps-ng sed util-linux wireless-tools

# ssh into HAOS and access docker container
ssh -i /config/.ssh/PRIVATE_KEY -p 22222 root@HA_IP_ADDRESS << EOF
	docker exec SSH_ADDON_CONTAINER_NAME \
	bash -c \
       'apk add coreutils figlet iproute2 iw jq ncurses procps-ng sed util-linux wireless-tools; \
	if [ ! -h /usr/local/bin/SCRIPT1 ]; then echo "SCRIPT1 does not exist"; \
	ln -s /homeassistant/shell_scripts/SCRIPT1 /usr/local/bin/SCRIPT1; echo "Link created"; \
	else echo "Link exists";fi; \
	if [ ! -h /usr/local/bin/SCRIPT2 ]; then echo "SCRIPT2 does not exist"; \
	ln -s /homeassistant/shell_scripts/SCRIPT2 /usr/local/bin/SCRIPT2; echo "Link created"; \
	else echo "Link exists";fi'
EOF

echo "Done"

I am going to feel really stupid when I find out there's a much easier way.

Docker containers are designed to be immutable. The moment they’re stopped and recreated, any changes to them ads thrown out. You’re supposed to add a layer to your Docker image if you want to add command lines and such. That’s why it’ll keep deleting your stuff every time you update.

It took me until I put Home Assistant on my server in a docker container to realize what was going on there. I use docker more now, but it's really, really nothing like this.

Running the script inside Docker should put it in the right place, but I wouldn’t advice doing it that way.

That's what I've been doing manually over regular ssh (not the 22222 port one).

To work around the path issue, maybe consider using hard links rather than soft links?

That's what I think I need to do, but the only 'hard' links--at least according to multiple find -name/find -iname searches on the ssh 22222 port--are all in /mnt/data/docker/overlay2 and /var/lib/docker/overlay2. I get there's a working pattern with the overlays but dear God why.

Alternatively, you could figure out where HAOS stores the Docker config and add a volume definition of your own. You’ll probably be able to put all of your files in /usr/local/bin by adding a line like “- /path/home/host:/usr/local/bin” in the right place. I don’t know where this config is stored, though.

Okay that makes sense. I guess the first step is to get the container structure and volume.

Thanks so much! I'll update if I find the solution or die trying.

You know, I didn't think of that. I've never run an OS in docker; all I tested my data collection scripts on were my regular VM's a few times just for fun. And for that matter, most LXC containers I run in Proxmox are privileged to get around restrictions (still haven't found a way for LXC's to let me compile different architectures, though. HA may have updated their docker to current, which would explain why it happened so suddenly.

And yes, for now, I'll just do root login to grab the information; it's technically more accurate, I am just knee-jerk distrustful of using root to the point until Proxmox and this last year, I almost forgot it existed unless there's a very weird linux problem I need it for. Thanks for this information, though; I've only just started seriously working with LXC and docker containers, so that's not an approach I woudl have considered.

Full disclosure: I just--and I mean just--got my head wrapped around docker and containers due to installing Proxmox on my server. Right now, my Proxmox server runs a LXC container for docker, and in docker I run Handbrake and MakeMKV images that run the GUIs in a browser or run with command line. They connect to each other through mounting the LXC's /home/user into both., then added a connection to the remote shares on my other server so I can send them to my media server. Yes, I did have to map all the mountings out first before I started but hey, that's how I learn.

Long way of saying: I am just now able to start understanding how Home Assistant works--someone said Home Assistant OS was basically really a hypervisor overseeing a lot of containers and now that I use Proxmox, that really helped--but I'm still really unfamiliar with the details.

I installed the full Home Assistant on a dedicated Pi4, so it's the only thing it does. Until yesterday, the only part I actually interacted with was the data portion, which is where all my files are, where I configure my GUI and script, store addons, etc. The container for this portion runs on Alpine Linux; I can and have and do install/update/change/build packages I need or like to use. in there It's ephemeral; anything I do outside the data directory (it holds /config, /addons, etc) gets wiped clean on update, so I reinstall them whenever HA does an update .

When I run my data collection scripts on my Home Assistant SBC, they take their information from the container aka Alpine Linux., including saying my OS was Alpine. All of this worked correctly up until--according to the directory dates, December 10th at 2:40 AM when the /sys/firmware was last updated and everything in it vanished, breaking the symlink to /proc/device-tree/model. This also updated the container OS to Alpine 3.19.0. Data collection runs hourly; one of my Pis ssh's into each computer to run four data collection scripts and updates a browser page I run off apache, so I can check current presence and network status and also check the OS/hardware/running services of all my computers from the browser (the services script doesn't work on Alpine yet; different structure). I didn't notice until recently because work got super busy, so I only verified availability and network status regularly.

These are the packages I install or switch to an updated/different version the Alpine container to help with this or just have fun: -figlet (it's just cute ASCII art for an ssh banner), -iproute2 (network info, when updated has option to store network info in a variable as a json),

• iw (wireless adapter info),
• jq (reads and processes json files),
• procps-ng (updated uptime package for more options),
• sed (updated can do more than the installed one),
• util-linux (for column command in bash),
• wireless-tools (iwconfig, more wireless data if iw doesn't have it) (Note: I think tr may also be updated by one of these.)

These are the ones I use for data collection that are already installed:

• lscpu ("Model name" "Vendor ID" "Architecture" "CPU(s)" "CPU min MHz" "CPU max MHz")
• uname (kernel)

These are the files I access for data collection:

• /proc/device-tree/model (Computer model)
• /proc/meminfo (RAM)
• /proc/uptime (Uptime)
• /etc/os-release (Current OS data)
• /sys/class/thermal/thermal_zone0/temp (CPU temperature for all my SBCs except BeagleBone Black)

Until this month, all of those files were accessible both before I do the package updates and after. The only one affected was maybe /proc/uptime by the uptime update to get more options. Again: I've been running these scripts or versions of them for well over a year and I test individually on each SBC before adding them to my data collection scripts to run remotely; all of these worked on every computer, including whatever SBC was running Home Assistant. (Odroid N2+ until it died a few months ago) And all of them work right now--except /proc/device-tree/model on my Home Assistant SBC. The only way I can get model info is to add an extra ssh to Home Assistant itself as root and grab the data off that file (and while I"m there, get the OS data for Home Assistant instead of Alpine), save it to my shell script directory in my data container, and have the my script process that file for my data after it gets the rest from the container.

That's why I'm weirded out; this is one of the things that is the same on every single Linux OS I've used and on Alpine, so why on earth would this one thing change?

This could conceivably be an Alpine issue; I downloaded Alpine 3.19.0 to run in Proxmox when I get a chance, and I kind of hope that it's a deliberate change in Alpine, because otherwise, I can't imagine why on earth the HA team would alter Alpine to break that symlink. Or they could be templating Alpine for the container each time and this time it accidentally broke. The entire thing is just so weird. Or maybe--though not likely--a bug in Alpine 3.19.0, but I doubt it; I can't possibly be the first to notice, it was released at least three weeks ago and I googled a lot.

I'm honestly not sure it affects anything at all, but it bothers me so here we are. Though granted, it did make me finally get off my ass and figure out how to login as root into HA as well as do a badly needed refactor of my main data collection script (the one that does the ssh'ing) as well as clean and refactor my computer information scripts, so maybe it was destiny.

Which is why I"m not sure I need a bug report. The part I have non-root access to is inside a docker container and that's all I needed to collect data. But it's such a random thing to go missing.since that core update.

Since probably October, I've noticed some really really random problems show up that never used to. And for once, I know it wasn't me messing with the code; I took a sabbatical from HA to learn how to use Proxmox a couple of months ago. and everything worked fine. It was actually a clean install to a new Raspberry Pi as my Odroid decided to stop working and I haven't had time to learn to solder (hopefully this week, tho). I was kind of wondering if it was the Pi that was the problem.