Manually upgrading

Hello. This is my first post here. And even partly a rant as I am pretty annoyed at an update procedure that I do not fully understand how it works and that does not seem to be documented. Or at least I didn´t find any documentation.

Call me old-fashioned, but as with all my Debian servers I want to be the one who decides when I upgrade my system. I also want to remove packages at will and have the updater only upgrade packages I have installed. I do not want the upgrade to insist of having a specific set of packages installed and always re-installing for example czech translations for luci modules and so on. I do not want to be bothered to tell the updater that it should not install a package I just removed with opkg before – actually I removed it and yes I am pretty damn sure about it. In short I want: apt update ; apt upgrade or well opkg update ; opkg upgrade – which by the way does not ask anything – or something similar to that. I also do not want to be bothered with any terms of service and things like that.

So far I did this:

  1. opkg update
  2. schnapps create -p pre “before upgrade”
  3. opkg upgrade
  4. And if I remembered schnapps create -p post “after upgrade”

and it worked just nice, I was just wondering about whether I will get any notifications in foris web gui about what is new or what else this way.

Now I get this:

root@[…]~# opkg upgrade
The opkg upgrade is *DANGEROUS* on TurrisOS releases.
It *WILL* break your system.
Do you really want to continue? Write upper-case yes if do.
^C

Okay, now you scared me.

So I tried pkgupdate

This ran. But it hung on restarting lighttpd which I had to restart manually for the web interface to appear before. After some time I hit Ctrl-C cause I had the impression nothing is happening on my system anymore. Now I get this:

[…]
INFO:Queue install of fswebcam/turris/20140113-1
INFO:Queue install of motion/turris/3.4.0-20141018-9479d910f2149b5558788bb86f97f26522794212-3
INFO:Queue install of ffmpeg/turris/2.7.6-1
INFO:Queue removal of ddns-scripts_nsupdate
INFO:Queue removal of luci-i18n-ntpc-en
INFO:Queue removal of lxc-autostart
INFO:Queue removal of ddns-scripts_cloudflare
INFO:Queue removal of ddns-scripts_no-ip_com
INFO:Queue removal of kmod-crypto-rng-jitterentropy
Press return to continue, CTRL+C to abort

INFO:Executing preupdate hooks...
INFO:Subprogram output: /etc/updater/hook_preupdate/05_schnapps.sh:
Snapshot number 14 created

INFO:End of subprogram output
line not found
line not found
line not found
line not found
DIE:
[string "transaction"]:344: Unfinished journal exists
Aborted

So pkgupdate has some unfinished transaction, yet… and a bug it seems as “line not found” indicates – also the error message is not partically helpful, as it doesn´t tell the file in which the error occured on line 344. Also neither “updater-unstuck.sh” nor does pkgupdater seem to offer an option to deal with that situation. updater.sh also does not work, as I didn´t agree to the terms of service.

Seriously: Is there any sane way to update Omnia Turris manually at exactly the time I want it to and actually looking at what the updater will do before I confirm the changes it is about to make? Preferably with a apticron or cron-apt like thing which sends me a mail once package upgrades are available.

Thing is: I want to be root on my router. I would accept not using opkg, but something else like pkgupdate, if you really insist, but really: Can I have a manual way to manually trigger upgrading my router exactly when I wish and by default also have it only upgrade packages I have installed and not insisting on installed a specific set of packages?

Pretty please.

Or do I have to replace Omnia Turris OS with something else in order to regain that freedom? I bought the router in order to have updates and freedom.

Also please why do to reinvent the wheel? Why is opkg not sufficient for you?

Okay I moved /usr/share/updater/journal out of the way (not a good place to put a transaction log file if I go by the FHS). Now pkgupdate did this:

INFO:Queue install of libgd/turris/2.0.35-2
INFO:Queue install of fswebcam/turris/20140113-1
INFO:Queue install of motion/turris/3.4.0-20141018-9479d910f2149b5558788bb86f97f26522794212-3
INFO:Queue install of ffmpeg/turris/2.7.6-1
INFO:Queue removal of ddns-scripts_nsupdate
INFO:Queue removal of luci-i18n-ntpc-en
INFO:Queue removal of lxc-autostart
INFO:Queue removal of ddns-scripts_cloudflare
INFO:Queue removal of ddns-scripts_no-ip_com
INFO:Queue removal of kmod-crypto-rng-jitterentropy
Press return to continue, CTRL+C to abort

INFO:Executing preupdate hooks...
INFO:Subprogram output: /etc/updater/hook_preupdate/05_schnapps.sh:
Snapshot number 15 created

INFO:End of subprogram output
Output from foris-diagnostics-plugin.postinst:
+ [ -n  ]
+ /etc/init.d/lighttpd restart

and lighthttpd restart worked okay this time.

Okay, pkgupdate seemed to have recognized that the packages have been upgrade at the first run already.

root@[…]~# pkgupdate
WARN:Script revision-specific not found, but ignoring its absence as requested
WARN:Script serial-specific not found, but ignoring its absence as requested
WARN:Multiple candidates from same repository with same version for package shairport-sync-openssl
Press return to continue, CTRL+C to abort

WARN:Lock on //var/lock/opkg.lock released by garbage collector

Okay, I consider this solved for now, but I really would like to hear why you complicate the update process in that way – so I am not yet setting solved tag. Also I think I can remove some of the accumulated BTRFS snapshots now.

Okay, this is not yet solved. pkgupdate also insisted of a set of installed packages and reinstall all czech translations as well as the transmission packages I don´t use. I want an updater that does not impose the will of its developers onto me. If you want to mark some packages as essential that are really needed… like Debian does, I am fine with that. But please do not try to reinvent proper package management (in a bad way). If I tell opkg to remove a package, I mean it.

Okay, I found the thread Removed packages reinstall themselves. So it seems in the future updater will recognize what packages will be removed. So if I would answer my question myself, I would come to this conclusion:

  1. For now uncheck all packages sets in Foris GUI. Which I can´t even do unless I agree to automatic updates. But well, I edit /etc/config/updater then.

  2. Use pkgupdate for any updates, instead of opkg upgrade.

In the future it would be possible to use package lists and the update still recognizes what I uninstalled manually. Even though I don´t even know why at all I should use those package lists.

You have freedom to implement your own update mechanism if you don’t like the standard one :wink:

Nope removing all pkglists from /etc/config/updater is VERY dangerous it seems:

root@amrita:~# pkgupdate
WARN:Script revision-specific not found, but ignoring its absence as requested
WARN:Script serial-specific not found, but ignoring its absence as requested
INFO:Queue removal of kmod-crypto-cmac
INFO:Queue removal of kmod-nf-nathelper-extra
INFO:Queue removal of luci-proto-vpnc
INFO:Queue removal of luci-i18n-watchcat-en
INFO:Queue removal of kmod-fs-exfat
INFO:Queue removal of fstrim
INFO:Queue removal of wol
INFO:Queue removal of kmod-nls-cp862
INFO:Queue removal of kmod-fs-minix
INFO:Queue removal of kmod-nls-cp1251
INFO:Queue removal of kmod-fs-cramfs
INFO:Queue removal of rsyncd
INFO:Queue removal of pptpd
INFO:Queue removal of kmod-usb-net-qmi-wwan
INFO:Queue removal of kmod-fs-f2fs
INFO:Queue removal of luci-app-sqm
INFO:Queue removal of mkdosfs
INFO:Queue removal of kmod-fs-xfs
INFO:Queue removal of resize2fs
INFO:Queue removal of kmod-fs-jfs
INFO:Queue removal of kmod-md-linear
INFO:Queue removal of openvpn-openssl
INFO:Queue removal of lxc-console
INFO:Queue removal of kmod-crypto-user
INFO:Queue removal of luci-i18n-samba-en
INFO:Queue removal of mount-utils
INFO:Queue removal of ppp-mod-pppoa
INFO:Queue removal of kmod-veth
INFO:Queue removal of nfs-kernel-server-utils
INFO:Queue removal of acl
INFO:Queue removal of libacl
INFO:Queue removal of luci-i18n-ahcp-en
INFO:Queue removal of kmod-fs-udf
INFO:Queue removal of kmod-lib-crc-itu-t
INFO:Queue removal of kmod-cryptodev
INFO:Queue removal of luci-i18n-statistics-en
INFO:Queue removal of lxc-info
INFO:Queue removal of kmod-usb-serial-option
INFO:Queue removal of kmod-nls-iso8859-15
INFO:Queue removal of lsblk
INFO:Queue removal of kmod-nls-cp852
INFO:Queue removal of hfsfsck
INFO:Queue removal of kmod-crypto-cbc
INFO:Queue removal of kmod-usb-uhci
INFO:Queue removal of kmod-usb-wdm
INFO:Queue removal of kmod-fs-ext4
INFO:Queue removal of kmod-lib-crc16
INFO:Queue removal of bind-dig
INFO:Queue removal of kmod-usb2
INFO:Queue removal of luci-app-samba
INFO:Queue removal of blkid
INFO:Queue removal of kmod-mppe
INFO:Queue removal of kmod-crypto-sha1
INFO:Queue removal of luci-app-mjpg-streamer
INFO:Queue removal of kmod-md-raid10
INFO:Queue removal of mdadm
INFO:Queue removal of lxc-snapshot
INFO:Queue removal of kmod-usb-storage-extras
INFO:Queue removal of file
INFO:Queue removal of ethtool
INFO:Queue removal of kmod-crypto-marvell-cesa
INFO:Queue removal of mountd
INFO:Queue removal of fswebcam
INFO:Queue removal of usb-modeswitch
INFO:Queue removal of motion
INFO:Queue removal of luci-i18n-firewall-en
INFO:Queue removal of kmod-nls-cp1250
INFO:Queue removal of vsftpd
INFO:Queue removal of samba36-server
INFO:Queue removal of kmod-md-raid456
INFO:Queue removal of usbutils
INFO:Queue removal of bind-client
INFO:Queue removal of kmod-fs-autofs4
INFO:Queue removal of xfs-growfs
INFO:Queue removal of cryptsetup-openssl
INFO:Queue removal of luci-proto-relay
INFO:Queue removal of relayd
INFO:Queue removal of partx-utils
INFO:Queue removal of kmod-nls-iso8859-6
INFO:Queue removal of uqmi
INFO:Queue removal of kmod-nls-iso8859-1
INFO:Queue removal of sqm-scripts
INFO:Queue removal of kmod-ifb
INFO:Queue removal of swap-utils
INFO:Queue removal of kmod-fs-isofs
INFO:Queue removal of kmod-fs-msdos
INFO:Queue removal of kmod-fs-vfat
INFO:Queue removal of luci-i18n-upnp-en
INFO:Queue removal of luci-app-rainbow
INFO:Queue removal of kmod-ata-ahci
INFO:Queue removal of badblocks
INFO:Queue removal of kmod-nls-iso8859-2
INFO:Queue removal of iptables-mod-ipopt
INFO:Queue removal of kmod-crypto-xts
INFO:Queue removal of luci-i18n-hd-idle-en
INFO:Queue removal of kmod-fs-hfsplus
INFO:Queue removal of kmod-nls-utf8
INFO:Queue removal of kmod-gre
INFO:Queue removal of ffmpeg
INFO:Queue removal of losetup
INFO:Queue removal of 6rd
INFO:Queue removal of br2684ctl
INFO:Queue removal of linux-atm
INFO:Queue removal of iptables-mod-conntrack-extra
INFO:Queue removal of attr
INFO:Queue removal of libusb-1.0
INFO:Queue removal of bind-libs
INFO:Queue removal of lxc-auto
INFO:Queue removal of lxc-stop
INFO:Queue removal of luci-i18n-tinyproxy-en
INFO:Queue removal of wget
INFO:Queue removal of ntfs-3g-utils
INFO:Queue removal of ntfs-3g
INFO:Queue removal of kmod-fs-configfs
INFO:Queue removal of kmod-ipt-ipopt
INFO:Queue removal of luci-proto-openconnect
INFO:Queue removal of tar
INFO:Queue removal of kmod-md-multipath
INFO:Queue removal of luci-app-lxc
INFO:Queue removal of getopt
INFO:Queue removal of xz
INFO:Queue removal of liblzma
INFO:Queue removal of cifsmount
INFO:Queue removal of kmod-nls-cp932
INFO:Queue removal of kmod-nls-cp866
INFO:Queue removal of hdparm
INFO:Queue removal of kmod-crypto-sha512
INFO:Queue removal of 6in4
INFO:Queue removal of nfs-kernel-server
INFO:Queue removal of kmod-fs-nfsd
INFO:Queue removal of portmap
INFO:Queue removal of librpc
INFO:Queue removal of kmod-fs-exportfs
INFO:Queue removal of kmod-fs-nfs
INFO:Queue removal of xfs-fsck
INFO:Queue removal of tcpdump
INFO:Queue removal of libpcap
INFO:Queue removal of kmod-ipt-conntrack-extra
INFO:Queue removal of luci-proto-3g
INFO:Queue removal of comgt
INFO:Queue removal of lxc-ls
INFO:Queue removal of lxc-config
INFO:Queue removal of kmod-nls-koi8r
INFO:Queue removal of xz-utils
INFO:Queue removal of libmount
INFO:Queue removal of lxc-monitord
INFO:Queue removal of mkhfs
INFO:Queue removal of kmod-fs-hfs
INFO:Queue removal of smartd
INFO:Queue removal of lxc-start
INFO:Queue removal of openconnect
INFO:Queue removal of kmod-nls-cp437
INFO:Queue removal of kmod-crypto-ccm
INFO:Queue removal of kmod-crypto-ctr
INFO:Queue removal of kmod-crypto-seqiv
INFO:Queue removal of kmod-crypto-iv
INFO:Queue removal of kmod-crypto-wq
INFO:Queue removal of luci-i18n-base-en
INFO:Queue removal of libmagic
INFO:Queue removal of kmod-nls-cp850
INFO:Queue removal of libgd
INFO:Queue removal of 6to4
INFO:Queue removal of kmod-sit
INFO:Queue removal of kmod-iptunnel
INFO:Queue removal of kmod-iptunnel4
INFO:Queue removal of libpng
INFO:Queue removal of lxc-attach
INFO:Queue removal of kmod-nls-cp864
INFO:Queue removal of kmod-fs-nfs-common
INFO:Queue removal of kmod-md-raid1
INFO:Queue removal of lvm2
INFO:Queue removal of libdevmapper
INFO:Queue removal of kmod-dm
INFO:Queue removal of samba36-client
INFO:Queue removal of kmod-nls-iso8859-8
INFO:Queue removal of kmod-crypto-rng
INFO:Queue removal of kmod-fs-afs
INFO:Queue removal of kmod-dnsresolver
INFO:Queue removal of kmod-fs-fscache
INFO:Queue removal of kmod-rxrpc
INFO:Queue removal of kmod-crypto-pcbc
INFO:Queue removal of kmod-crypto-fcrypt
INFO:Queue removal of rpcd-mod-lxc
INFO:Queue removal of kmod-ata-mvebu-ahci
INFO:Queue removal of kmod-ata-ahci-platform
INFO:Queue removal of kmod-ata-core
INFO:Queue removal of block-mount
INFO:Queue removal of kmod-video-uvc
INFO:Queue removal of kmod-video-videobuf2
INFO:Queue removal of kmod-video-core
INFO:Queue removal of kmod-nls-cp775
INFO:Queue removal of kmod-dma-buf
INFO:Queue removal of sfdisk
INFO:Queue removal of kmod-fs-btrfs
INFO:Queue removal of kmod-lib-crc32c
INFO:Queue removal of kmod-crypto-crc32c
INFO:Queue removal of kmod-lib-lzo
INFO:Queue removal of kmod-lib-xor
INFO:Queue removal of blkdiscard
INFO:Queue removal of cfdisk
INFO:Queue removal of kmod-usb-net-rndis
INFO:Queue removal of kmod-usb-net-cdc-ether
INFO:Queue removal of kmod-usb-net
INFO:Queue removal of kmod-mii
INFO:Queue removal of lxc-create
INFO:Queue removal of lxc-hooks
INFO:Queue removal of lxc-templates
INFO:Queue removal of tune2fs
INFO:Queue removal of e2fsprogs
INFO:Queue removal of libext2fs
INFO:Queue removal of kmod-crypto-gf128
INFO:Queue removal of kmod-fs-cifs
INFO:Queue removal of kmod-crypto-hmac
INFO:Queue removal of kmod-crypto-md5
INFO:Queue removal of kmod-crypto-md4
INFO:Queue removal of kmod-fs-ntfs
INFO:Queue removal of kmod-crypto-authenc
INFO:Queue removal of kmod-lib-textsearch
INFO:Queue removal of lxc-monitor
INFO:Queue removal of lxc-common
INFO:Queue removal of liblxc
INFO:Queue removal of ds-lite
INFO:Queue removal of kmod-ip6-tunnel
INFO:Queue removal of flock
INFO:Queue removal of sshfs
INFO:Queue removal of fuse-utils
INFO:Queue removal of kmod-nls-iso8859-13
INFO:Queue removal of kmod-iptunnel6
INFO:Queue removal of kmod-lib-raid6
INFO:Queue removal of smartmontools
INFO:Queue removal of uclibcxx
INFO:Queue removal of fdisk
INFO:Queue removal of libsmartcols
INFO:Queue removal of luci-i18n-wol-en
INFO:Queue removal of luci-i18n-minidlna-en
INFO:Queue removal of kmod-crypto-des
INFO:Queue removal of udev
INFO:Queue removal of mjpg-streamer
INFO:Queue removal of kmod-crypto-sha256
INFO:Queue removal of vpnc
INFO:Queue removal of vpnc-scripts
INFO:Queue removal of resolveip
INFO:Queue removal of luci-i18n-commands-en
INFO:Queue removal of dosfsck
INFO:Queue removal of gnupg-utils
INFO:Queue removal of gnupg
INFO:Queue removal of libreadline
INFO:Queue removal of davfs2
INFO:Queue removal of libneon
INFO:Queue removal of chat
INFO:Queue removal of kmod-crypto-ecb
INFO:Queue removal of kmod-crypto-manager
INFO:Queue removal of kmod-crypto-aead
INFO:Queue removal of kmod-crypto-null
INFO:Queue removal of kmod-crypto-hash
INFO:Queue removal of kmod-crypto-pcompress
INFO:Queue removal of kmod-crypto-deflate
INFO:Queue removal of kmod-lib-zlib
INFO:Queue removal of libfuse
INFO:Queue removal of kmod-fuse
INFO:Queue removal of libgcrypt
INFO:Queue removal of libgpg-error
INFO:Queue removal of xfs-mkfs
INFO:Queue removal of lxc-configs
INFO:Queue removal of lxc
INFO:Queue removal of kmod-md-raid0
INFO:Queue removal of kmod-md-mod
INFO:Queue removal of kmod-pppoa
INFO:Queue removal of kmod-atm
Press return to continue, CTRL+C to abort

You aren´t really serious about that, or are you? Adding the pkglists back in. Luckily I have the configuration file in a git repo.

I really prefer to have something sane as update mechanism out of the box. Like in Debian for example. So please let me first appeal for some sanity here.

1 Like

Hello

I really don’t want to read all text you posted, really not. It would took me time that can be used to updater improvements. Just one point, updater is just different than apt. At first, updater configuration describes what should be in system. That’s why if you tell it that you want no packages (no user lists) it removes whole system. Also tell me your experience with Debian version upgrade. It almost never goes smooth, it is more less hands on update. Updater has ambitions to have this smooth same as installation of additional software and specially it is hands off. Don’t expect that it is same. Comparing updater with apt is same as comparing OSX with GNU/Linux in my eyes. Both does same thing, but both does it differently.

You are root. You can update when ever you want it is just executed using cron, so where is problem? See /etc/updater configuration files. Updater isn’t interactive software, you don’t expect apache to ask you if you want start specific servers on its start nor you should expect updater to ask you what packages you really want update.

Thing is: I don´t want to tell updater what packages to upgrade. I just want it to upgrade all the packages that are installed, without altering the set of installed packages to whatever it sees fit, without completely changing the set of installed packages basically undoing all my work to declutter the OS.

Even

root@[…]/etc/updater# cat user.lua 
Uninstall "luci-app-transmission" {priority=60}
Uninstall "luci-i18n-ahcp-cs" {priority=60}
Uninstall "luci-i18n-base-cs" {priority=60}
Uninstall "luci-i18n-commands-cs" {priority=60}
Uninstall "luci-i18n-ddns-cs" {priority=60}
Uninstall "luci-i18n-firewall-cs" {priority=60}
Uninstall "luci-i18n-minidlna-cs" {priority=60}
Uninstall "luci-i18n-statistics-cs" {priority=60}
Uninstall "luci-i18n-tinyproxy-cs" {priority=60}
Uninstall "luci-i18n-transmission-cs" {priority=60}
Uninstall "luci-i18n-transmission-en" {priority=60}
Uninstall "luci-i18n-upnp-cs" {priority=60}
Uninstall "luci-i18n-watchcat-cs" {priority=60}
Uninstall "luci-i18n-wol-cs" {priority=60}
Uninstall "luci-i18n-hd-idle-cs" {priority=60}
Uninstall "luci-i18n-samba-cs" {priority=60}
Uninstall "transmission-daemon-openssl" {priority=60}

gives me a lot of warnings like:

root@[…]# pkgupdate
WARN:Script revision-specific not found, but ignoring its absence as requested
WARN:Script serial-specific not found, but ignoring its absence as requested
WARN:Multiple candidates from same repository with same version for package shairport-sync-openssl
WARN:Request not satisfied to uninstall package: luci-app-transmission
WARN:Request not satisfied to uninstall package: luci-i18n-ahcp-cs
WARN:Request not satisfied to uninstall package: luci-i18n-base-cs
[…]
WARN:Request not satisfied to install package: luci-app-transmission
WARN:Request not satisfied to install package: luci-i18n-ahcp-cs
WARN:Request not satisfied to install package: luci-i18n-base-cs
[…]

At least it basically appears to work:

Press return to continue, CTRL+C to abort

WARN:Lock on //var/lock/opkg.lock released by garbage collector

So to summarize:

  1. I should not use schnapps, opkg update and opkg upgrade, schnapps which would just do what I want.

  2. I cannot just remove package lists as pkgupdate basically would deinstall critical packages then, probably making the system unusable.

  3. And even if I use user.lua I have to filter out a ton of warnings each time I run pkgupdate in order to find out whether it has any real issues.

Which means I need to wait for a fix that updater-ng recognized packages I removed on my own.

Seriously I do not feel like being root on that box. I know I am cause I logged in via SSH as root, but the update software does not treat me in that way. Instead it treats me as if I would be total noob and not someone with 10+ years of deep Linux practice. That is lacking quite a bit of respect for me as the one who owns and administrates that Omnia.

Let me compare that to Debian:

  1. It only upgrades installed packages. Only with dist-upgrade it may install additional packages and I can even influence that with --no-install-recommends.

  2. It still protects me from doing very silly stuff like apt remove bash.

Seriously: What actually is wrong with opkg? Why shouldn´t I use it? Why would it break my system? Why did you invent your own updater instead of enhance opkg in case you see a need to? I really would like some transparancy here. I just find quite a bit of mostly undocumented updater magic on that box and when I want to use the standard way to upgrade via opkg that magic tells me not to.

1 Like

Why you shouldn’t be using schnapps? You can. Also opkg update can be used, it is even essential for opkg usage. Only one not advised is opkg upgrade.

You can just remove package list. You just have to create your own package list instead containing all packages you need. Only essential list is base. And that one shouldn’t be in /etc/config/updater and is always sourced.

This is really issue. But in reality it is just about face of updater. And for this updater isn’t yet polished enough. Yes there are some warnings that should been info more probably, some output lines that slip at some point and so on. But that will get better. I will work on that. And if you are really annoyed by it, you can look to source codes and send patches, they will be welcomed.

If you installed them using opkg than you can remove them using opkg. If they are from user list, then opkg has nothing to do with it. See Sign in · GitLab

Even trough those years of Linux practice you are not too much open to new things as it seems.
Also you seem to give no respect to software we implemented expecting that it is just another package manager without looking into what it really is.[quote=“helios21, post:10, topic:3067”]
It still protects me from doing very silly stuff like apt remove bash.
[/quote]
Did you try to remove busybox or kernel on Omnia?

Updater only updates configured packages. And can be influenced trough its configuration.

Advantage of it is that config describes system, not some state database. This should be very appealing to you as system administrator. Config can be same for multiple hosts. I have never seen shared package state database.

See my colleague’s presentation. https://www.youtube.com/watch?v=SV50D8UTbeg

Updater has documentation in its repository (see github or gitlab). With your deep Linux practice you should parse some knowledge from it. More user friendly documentation is on the way and will be on turris.cz/doc in some time in future.

Yes, but thats exactly the one that would give me the desired functionality of upgrading all installed packages without doing any other changes like installing other packages.

Warnings when using user.lua:

Okay thanks for seeing this.

Well thats exactly the thing I like to see.

Cause for me…

what you call state database is actually the configuration. I configured my system to have a specific set of packages using opkg… being required to repeat that step using some lua based configuration files just does not make sense to me. I expect it to keep that state. If I want to have something that enforces a specific state on mutiple hosts I´d use a configuration management software like Puppet or so.

I will look into that.

Okay, thanks for that. I may look into the gitlab documentation.

I close with:

I am open to new things when I actually see an improvement over what I know. There have been and are numerous attempts to reinvent or improve upon a traditional package manager… including putting apps into containers, yet none of them replaced traditional Linux distro package managers and I think thats for a reason and many of them have completely new issues.

But okay, I will watch the video… and I am willing to learn about updater… just so far don´t see the improvement. And I also still do not understand why using opkg upgrade could be harmful or even leave the system in a broken state. Maybe I find an explanation in the video you linked to, but I prefer to see this kind of information in the documentation. Maybe it is in the docs in the updater repo you mentioned.

Then why don’t you dump the state database and update the configuration file based on that dump every time you want to upgrade? You can automate that as a wrapper.

Good point, but this would be useless, because packages currently installed in system using opkg by user are because of opkg wrapper pretty much well tracked by updater (file /etc/updater/auto.lua). Problem is in miss understanding of that package lists are not part of that charade and removing package that comes from those lists is problematic. My advice would be disable all user lists and install only packages needed. Only exception is little bit extensive base list, that is also marked as critical, so it is not possible to uninstall such packages at all. We know about this issue and will solve it hopefully, but it is not most pressing issue now.

about state database as configuration:

Well… I´d expect that to be in place already: Right now you have pkgupdate and opkg using /usr/bin/opkg-pkgupdate-wrapper.sh which then calls in to opkg-cl which seems to be the original opkg, but also records of the state information into updater lua configuration files, like packages I installed additional into auto.lua. Yet it does not yet record when I delete packages with opkg remove to the configuration files. In any way the wrapper would just duplicate information that is already there. opkg has the information that the package is removed… yet pkgupdate would install it again. So it actually keeps state in two location. In one location you call it state and in the other location you call it configuration… yet it is state nonetheless at least when it has been changed by the wrapper. (Also in a sense all configuration is state.)

I now watched both the Youtube video and also read through the design docs of the updater repo. I got it that opkg has some limitations that dpkg would not have and that updater-ng doesn´t have. And also that it is missing elaborated dependency handling. I bet it also cannot upgrade to next major OpenWrt version. I got it that you eventually plan to replace opkg completely, yet this does not yet seem to be the case, as I don´t see that pkgupdate would recognize “install” or “remove” commands – also the command name pkgupdate would not really be fitting for a general purpose package manager. I also got it from the video talk that there is a provision for asking the user whether he wants an update.

If both of these things would be in place and thus updater-ng would be a complete package manager and there would be only one place where I tell it that I want a package to be removed and to stay removed, and it would ask me whether I want an update… then I would even be willing to agree to some kind of semi-automatic updating. Like when it would send me a mail with proposed updates giving me a link to click in order to confirm them to be done.

Right now I have to states to keep in sync: opkg remove + user.lua. I would like to have one state for this. And if that would mean I tell updater directly that I don´t want this package to be installed that would be okay as well. But having two states for the same information “package removed” just asks for all kinds of trouble – as its clearly visible as I am not the only one who stumbled upon this.

So maybe most of my concerns would be fixed once updater-ng actually really replaces opkg.

To clarify. Difference between state and configuration is that state says what is in system and in what state. Configuration is more general. It is in fact lua script, so you can do basically anything in it. It’s just more powerful. I understand that it is not too much of interest for normal user.

Updater-ng really will one day replace opkg. But as you see it has long time to go before it is viable.

If you have idea how automate it in opkg wrapper script, than I am open to discussions. But simple and strait forward solution that we place for every opkg remove command line Uninstall "pkg" {priority=60} wont work. That’s why it is shelved for now, because it requires some non-trivial time to figure out good solution and there are more pressing issues (for example migration of old turris routers from old updater to updater-ng).

Okay, I now understand that this is not the finally intended state. And I see the more of the intentions and goals behind updater-ng. Thats exactly what I asked for, some transparency on why you did updater-ng. Granted the issues I faced are not yet solved, but now I understand more about why they are there.

Can you elaborate why you think the simple and straight forward solution won´t work? Then I have an actual idea about the challenge to master. Of cause an “uninstall” line would need to be removed again when… someone does “opkg install” same package at a later time. Of cause an “uninstall” line should not be added when opkg has been unable to remove the package due to dependencies. If one removes several package with one call to opkg remove… I am not sure whether its easy to find on which packages it may have failed.

I will give you example case. We have two packages A and B. B is requested by user list, so in default B is installed. If user tries to uninstall it using opkg, with strait forward way we would have line Uninstall "B" {priority=60} in configuration of updater. This would really ensured that it isn’t in system. But if user tries install package A, what happens is that opkg installs both A and B and wrapper adds Install "A" to updater configuration. Now if updater is ran, it sees as more prioritized that B isn’t in system and so it decides that A can’t be in system either. So updater removes both A and B.

cynerd, I think the current approach to package lists is broken then. Debian uses meta packages – packages that just contain dependencies – that depend on several other packages for similar purposes as you use package lists.

Yet, if at one time you remove on of the packages the meta package depends on, it will also say to the user it will have to remove the metapackage. Then I can decide whether I want this. Also if I install a metapackage and I set a pinning not to install one of the the packages it depends on, it will clearly tell me so.

So this all works, cause its all tightly integrated within one tool with one state.

So… I think there is a quite obvious way how to handle this:

  • User told updater not to install package B
  • User tries to install package A, which depends on package B
  • opkg tells user that installing package A is not possible, unless he agrees also to install package B which is not to be installed due to her previous preference
  • User decides whether he wants this.

Yes, that requires integration both ways between pkgupdate and opkg. Or getting rid of opkg completely and replacing it by opkg. Or extending opkg instead so that it provides the needed functionality.

But right now, you have two tools whose configurations can conflict in a way that render puts the package management system in an inconsistent state and have both tools work against each other.

I stated that we are going to replace opkg by updater. So extending opkg makes no sense. For now we have opkg wrapper that handles packages outside of user lists pretty well. Yes problem is with user lists. We know that those are essentially broken now. Idea isn’t incorrect, it’s like portage profiles or debian meta packages. Problem is that currently we can’t suggest user that he should add specific uninstall rule to its configuration file, because we even don’t know that. Yes final usage point would probably be something like emerge does, if we encountered some problem we suggest automatic fix or user can fix it in its configuration on its own. But we are far from that and simple hack would be currently almost impossible. Not only we would need to implement application that would do such check, but also this new interactiveness would have to be propagated to luci.

As I see it, only way out of this is to drop opkg and replace it completely with updater. That’s because if we have features required for such interactiveness for opkg wrapper then it’s one small step from full opkg replacement (ok not for searching and such but as tool for installing/removing software). But there have to be I would say about a year of development before we can do that.

In reality, we are currently in state where updater just reports that he decided not to fulfill specific request. But to found out why, you have to enable debug output and see boolean proof why he decided to do so. We have way too much work ahead of us.

Problem with opkg is that it works even against it self. It’s so minimalistic that it for example allows you to install packages that provides same files. So in case of opkg even order of installation matters! This is nightmare when you try to do updates. As we see it, opkg is broken by design and have to die as soon as possible. Don’t even dare to ask how many hacks we have in updater to ensure that we don’t fail if opkg does something stupid. But users expects interactive tool for software installations and because updater isn’t ready for that, we had to ship with opkg too. This is necessity. Updates with opkg are catastrophic (users of Turris 1.x would tell you more) and updater can interact with user only trough configuration files. Which is not acceptable by most users.

I hope that this gives you idea where we stand. We just need more time. And of course some help from community in form of patches would be awesome. O:-)