[SOLVED] New feature: localrepo

Jirka · January 3, 2018, 12:18pm

Hi,

As I dont want to hijack more adblock thread - Adblock not creating/updating adb_list.overall (Omnia with adblock1.3.3-1), and due to lack of official documentation regarding new localrepo feature in updater, I would like to ask some more things here. I really need to figure out this before update, since I need to correctly setup new way to treat packages (and I didnt find something like dry run option of updater https://doc.turris.cz/doc/cs/howto/updater#prikaz_package to test syntax).

@cynerd mentioned: That wasn’t possible before with content as that just simply added given package version (I might be wrong. I don’t remember everything and I am using localrepo for about a half year for now). Now with localrepo you can specify source repository with ease in install command (Install(“pkg”, {repository = “localrepo-NAME”})).

So, this localrepo isnt hosted on www server/port, but its kinda virtual www, using just routers directory?

Where should we copy custom packages, to be able using them within user.lua Install directive?

repository = "localrepo-NAME confuses me a little. What is the correct steps to create custom local repository, copy packages into it and use in user.lua?

Thanks in advance

Solution for custom packages since version 3.9

user.lua Turris OS < 3.9

Package("adblock", { content = "file:///root/upravy/packages/adblock_3.1.1-1_all.ipk" })
Install("adblock")

–> MIGRATION --> (updated)

localrepo add /root/upravy/packages/adblock_3.1.1-1_all.ipk

user.lua Turris OS >= 3.9

Install("adblock", {repository = "localrepo-user"})

P.S.:
localrepo add command creates local repository named user:

create /usr/share/updater/localrepo/user
add it into configuration /usr/share/updater/localrepo/localrepo.lua)
copy specified packages there

cynerd · January 3, 2018, 7:15pm

No it’s real repository hosted on router it self in path /usr/share/updater/localrepo/NAME. Where NAME is just a repository name user can choose. In default user is used and auto is also used by opkg wrapper and to that repository are migrated all content packages from /etc/updater/auto.lua.

Updater is notified about these repositories trough file /etc/updater/conf.d/localrepo.lua (which points to /usr/share/updater/localrepo/localrepo.lua file). So you can see that it’s no magic, no hidden repository. It’s real repository created on router it self using localrepo script.

Edit: And oh yes repository is not just thing that is hosted on web. It’s just some set of packages and index of them. In reality it can be ftp, local directory or anything you like (well curl can resolve). So no there is no virtual www server, that would be stupid.

No migration would be something like this:

localrepo add /root/updavy/packages/*.ipk

And then you can edit your user.lua to contain just Install commands in form Install("pkg", {repository = "localrepo-user"}).

--init wasn’t used (read: it does nothing in all versions) at all and in future version it will be dropped. When localrepo needs it always initializes new repository.
--path is just if you want to specify different repositories root. You don’t want that as otherwise updater won’t found those repositories automatically.

On top of that you can of course add --repo custom to localrepo call and that way work with repository named custom instead of default user. Then of course repository name is not localrepo-user but localrepo-custom (to see that just cat file /usr/share/updater/localrepo/localrepo.lua.

Jirka · January 3, 2018, 7:53pm

Thanks a lot for explanation…Appretiate, you spent free time at this. As “You don’t want that as otherwise updater won’t found those repositories automatically”, there is really no point for fiddling with different repos I think - the whole point is have clean / differentiated custom packages, in symbiosis with future updates.

So, if I understand it finally, as the default (if not specified otherwise) for add command is used user repository. We should use it, and we should be fine. I test it, and works nice, .gz was created automatically. I will update my user.lua and hope I will be fine after planned cleanup

Couple assurements:
For backup purposes, it is suitable to backup just /usr/share/updater/localrepo; in rollback/clean install it should be enough to copy contents there, and updater should load repository without problem, correct?

User repository will be treated as before user.lua, so no automatic changes/updates during updates, correct? I noticed that auto was always dynamic, but user following his name, and I never get conf changed…

cynerd · January 3, 2018, 8:01pm

Yes.

user.lua is never touched. Ok not exactly. It was moved as part of migration to conf.d directory in Turris OS 3.9. But it’s content is strictly user’s. From 3.9 it’s not even shipped by updater-ng package. In fact because it’s now in conf.d directory you can name it how ever you want and it will be loaded (it at least has to have .lua extension).

Jirka · January 3, 2018, 8:07pm

Great! No further questions , I am happy to finally understand logic (hopefully). I tried to summarize and complete previous post, in case others will look into the same problem (or even worse, they could overlook Obsolete updater message). I will try to summarize this also for community documentation wiki, as this is quite important to be searchable only on forum.

Once again, many thanks and good night!

EDIT: https://doc.turris.cz/doc/en/public/custom_packages

jork · January 4, 2018, 8:48pm

I followed the instruction for Turis OS >= 3.9 and when I finished I got email with

Error notifications

Updater failed: Unknown error

can someone explain me that Unknown error?

                    localrepo add /root/adblock.ipk

added def: Install(“adblock”, {repository = “localrepo-user”}) to the /etc/updater/conf.d/user.lua

cynerd · January 4, 2018, 8:59pm

Unknown error is an unknown error.

Basically what it commonly means is that updater.sh (script handling updater’s execution) detected that pkgupdate exited with error but no error was reported by it. This can be because pkgupdate failed big time or just because there is something that we forgot to catch in updater it self.

What you have to do is to run updater in terminal to see what is really happening. updater.sh (or updater.sh -e DBG for debug output).

dibdot · January 16, 2018, 8:11am

I’ve followed the steps described here

for release level 3.9.1 and ended up with an updater error:

user configuration:

root@turris:/usr/share/updater/localrepo$ cat /etc/updater/conf.d/user.lua 
--[[
A place for user definitions.

Repository "name" "URI" { ca = "file:///etc/ssl/ca.pem", pubkey = "file:///etc/repo.pubkey" }
Install "pkgname" "other"
]]
Install("custom", {repository = "localrepo-user"})

Both ‘local add’ commands working fine (at least without any error message)

local repo root directory after that:

root@turris:/usr/share/updater/localrepo$ ls -la /usr/share/updater/localrepo/
drwxr-xr-x    1 root     root            34 Jan 16 08:51 .
drwxr-xr-x    1 root     root           156 Jan 16 08:51 ..
-rw-r--r--    1 root     root           116 Jan 16 08:51 localrepo.lua
drwxr-xr-x    1 root     root           154 Jan 16 08:51 user

local user repo:

root@turris:/usr/share/updater/localrepo$ ls -la /usr/share/updater/localrepo/user/
drwxr-xr-x    1 root     root           154 Jan 16 08:51 .
drwxr-xr-x    1 root     root            34 Jan 16 08:51 ..
-rw-r--r--    1 root     root           655 Jan 16 08:51 Packages.gz
-rw-r--r--    1 root     root         10691 Jan 16 08:51 adblock_3.4.3-1.ipk
-rw-r--r--    1 root     root          7912 Jan 16 08:51 luci-app-adblock_git-18.011.23874-43867a8-1.ipk

Whenever I start updater.sh -e DBG, I receive the following error output:

[...]
DEBUG:requests.lua:440 (func):Running script file:////etc/updater/conf.d/opkg-auto.lua
DEBUG:requests.lua:366 (submit):Request install of mc
DEBUG:requests.lua:366 (submit):Request install of nano
DEBUG:requests.lua:440 (func):Running script file:////etc/updater/conf.d/opkg.lua
DEBUG:requests.lua:440 (func):Running script file:////etc/updater/conf.d/localrepo.lua
DEBUG:requests.lua:440 (func):Running script file:///usr/share/updater/localrepo/localrepo.lua
DEBUG:requests.lua:440 (func):Running script file:////etc/updater/conf.d/example.lua
DEBUG:requests.lua:440 (func):Running script file:////etc/updater/conf.d/user.lua
DEBUG:requests.lua:366 (submit):Request install of custom
DEBUG:postprocess.lua:47 (get_repos):Getting repos
DEBUG:postprocess.lua:111 (cback):Received repository index turris/https://repo.turris.cz/omnia/packages/base/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index turris/https://repo.turris.cz/omnia/packages/base/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:111 (cback):Received repository index turris/https://repo.turris.cz/omnia/packages/routing/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index turris/https://repo.turris.cz/omnia/packages/routing/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:111 (cback):Received repository index turris/https://repo.turris.cz/omnia/packages/php/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index turris/https://repo.turris.cz/omnia/packages/php/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:111 (cback):Received repository index turris/https://repo.turris.cz/omnia/packages/turrispackages/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index turris/https://repo.turris.cz/omnia/packages/turrispackages/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:111 (cback):Received repository index turris/https://repo.turris.cz/omnia/packages/lucics/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index turris/https://repo.turris.cz/omnia/packages/lucics/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:111 (cback):Received repository index turris/https://repo.turris.cz/omnia/packages/hardware/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index turris/https://repo.turris.cz/omnia/packages/hardware/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:111 (cback):Received repository index turris/https://repo.turris.cz/omnia/packages/packages/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index turris/https://repo.turris.cz/omnia/packages/packages/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:111 (cback):Received repository index turris/https://repo.turris.cz/omnia/packages/telephony/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index turris/https://repo.turris.cz/omnia/packages/telephony/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:111 (cback):Received repository index turris/https://repo.turris.cz/omnia/packages/printing/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index turris/https://repo.turris.cz/omnia/packages/printing/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:111 (cback):Received repository index turris/https://repo.turris.cz/omnia/packages/management/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index turris/https://repo.turris.cz/omnia/packages/management/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:111 (cback):Received repository index localrepo-user/file:///usr/share/updater/localrepo/user/Packages.gz
DEBUG:postprocess.lua:118 (cback):Index localrepo-user/file:///usr/share/updater/localrepo/user/Packages.gz is compressed, decompressing
DEBUG:src/lib/interpreter.c:317 (lua_run_generic):Util command: gzip -dc 
DEBUG:postprocess.lua:103 (Globals):Decompression of localrepo-user/file:///usr/share/updater/localrepo/user/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index localrepo-user/file:///usr/share/updater/localrepo/user/Packages.gz
DEBUG:postprocess.lua:103 (Globals):Decompression of turris/https://repo.turris.cz/omnia/packages/printing/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index turris/https://repo.turris.cz/omnia/packages/printing/Packages.gz
DEBUG:postprocess.lua:103 (Globals):Decompression of turris/https://repo.turris.cz/omnia/packages/hardware/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index turris/https://repo.turris.cz/omnia/packages/hardware/Packages.gz
DEBUG:postprocess.lua:103 (Globals):Decompression of turris/https://repo.turris.cz/omnia/packages/management/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index turris/https://repo.turris.cz/omnia/packages/management/Packages.gz
DEBUG:postprocess.lua:103 (Globals):Decompression of turris/https://repo.turris.cz/omnia/packages/php/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index turris/https://repo.turris.cz/omnia/packages/php/Packages.gz
DEBUG:postprocess.lua:103 (Globals):Decompression of turris/https://repo.turris.cz/omnia/packages/routing/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index turris/https://repo.turris.cz/omnia/packages/routing/Packages.gz
DEBUG:postprocess.lua:103 (Globals):Decompression of turris/https://repo.turris.cz/omnia/packages/telephony/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index turris/https://repo.turris.cz/omnia/packages/telephony/Packages.gz
DEBUG:postprocess.lua:103 (Globals):Decompression of turris/https://repo.turris.cz/omnia/packages/turrispackages/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index turris/https://repo.turris.cz/omnia/packages/turrispackages/Packages.gz
DEBUG:postprocess.lua:103 (Globals):Decompression of turris/https://repo.turris.cz/omnia/packages/lucics/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index turris/https://repo.turris.cz/omnia/packages/lucics/Packages.gz
DEBUG:postprocess.lua:103 (Globals):Decompression of turris/https://repo.turris.cz/omnia/packages/base/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index turris/https://repo.turris.cz/omnia/packages/base/Packages.gz
DEBUG:postprocess.lua:103 (Globals):Decompression of turris/https://repo.turris.cz/omnia/packages/packages/Packages.gz done
DEBUG:postprocess.lua:86 (parse):Parsing index turris/https://repo.turris.cz/omnia/packages/packages/Packages.gz
DEBUG:postprocess.lua:386 (pkg_aggregate):Aggregating packages together
line not found
line not found
line not found
line not found
ERROR:src/pkgupdate/main.c:249 (main):
[string "planner"]:48: bad argument #1 to 'pairs' (table expected, got string)
DEBUG:src/lib/locks.c:81 (lua_lock_release):Released lock at //var/lock/opkg.lock
Working on message: 1516089286-23738
Working on message: 1516089956-24590
There is no message to send.

@cynerd is this a known error?

Edit: Trace output of the error:

TRACE:src/lib/interpreter.c:105 (interpreter_error_result):
Stack Traceback
===============
(1) Lua function '?' at line 64 of chunk '"logging"]'
	Local variables:
	 err = string: "[string \"planner\"]:48: bad argument #1 to 'pairs' (table expected, got string)"
	 err2string = Lua function '?' (defined at line 45 of chunk "logging"])
	 msg = string: "\
[string \"planner\"]:48: bad argument #1 to 'pairs' (table expected, got string)"
	 (*temporary) = table: 0x2009760  {msg:
[string "planner"]:48: bad argument #1 to 'pairs' (table expected, got string) (more...)}
(2)  C function 'function: 0x671fa0'
(3) upvalue C function 'pairs'
(4) Lua global 'candidates_choose' at line 48 of chunk '"planner"]'
	Local variables:
	 candidates = table: 0x12b8f00  {}
	 pkg_name = string: "custom"
	 version = nil
	 repository = string: "localrepo-user"
	 repos = table: 0x20094c0  {}
(5) Lua global 'sat_dep' at line 178 of chunk '"planner"]'
	Local variables:
	 state = table: 0x1ec90c0  {pkgs:table: 0x6462f0, pkg2sat:table: 0x1ec9200, sat:userdata: 0x1ec8b88 (more...)}
	 pkg = table: 0xd9a2c0  {name:custom, tp:package, final:table: 0x17376e0}
	 version = nil
	 repository = string: "localrepo-user"
	 name = string: "custom"
	 group_var = number: 2263
	 var = number: 2264
(6) Lua upvalue 'sat_build' at line 269 of chunk '"planner"]'
	Local variables:
	 sat = userdata: 0x1ec8b88
	 pkgs = table: 0x6462f0  {kmod-video-gspca-vc032x:table: 0x12b9980, iptables-mod-clusterip:table: 0xf84c90 (more...)}
	 requests = table: 0x654fc0  {1:table: 0xd3afa0, 2:table: 0xd3b230, 3:table: 0xd12000, 4:table: 0xd121d0 (more...)}
	 state = table: 0x1ec90c0  {pkgs:table: 0x6462f0, pkg2sat:table: 0x1ec9200, sat:userdata: 0x1ec8b88 (more...)}
	 (for generator) = C function: 0xb6c036d0
	 (for state) = table: 0x654fc0  {1:table: 0xd3afa0, 2:table: 0xd3b230, 3:table: 0xd12000, 4:table: 0xd121d0 (more...)}
	 (for control) = number: 339
	 _ = number: 339
	 req = table: 0xd9a320  {package:table: 0xd9a2c0, priority:50, tp:install, repository:localrepo-user (more...)}
	 req_var = number: 2262
(7) Lua function '?' at line 493 of chunk '"planner"]'
	Local variables:
	 pkgs = table: 0x6462f0  {kmod-video-gspca-vc032x:table: 0x12b9980, iptables-mod-clusterip:table: 0xf84c90 (more...)}
	 requests = table: 0x654fc0  {1:table: 0xd3afa0, 2:table: 0xd3b230, 3:table: 0xd12000, 4:table: 0xd121d0 (more...)}
	 sat = userdata: 0x1ec8b88
(8) tail call
	Local variables:
	 (*temporary) = C function: 0x671fa0
(9) Lua function '?' at line 67 of chunk '"updater"]'
	Local variables:
	 entrypoint = string: "internal:entry_lua"

ERROR:src/pkgupdate/main.c:249 (main):
[string "planner"]:48: bad argument #1 to 'pairs' (table expected, got string)

cynerd · January 16, 2018, 9:24am

Yes it’s known “bug”. Documentation specifies that repository option can be only a table (so you can specify multiple repositories). But I kind of forgot about that when I was implementing verifier so it allows table or string but later in the older code it fails. I know about it for some time but I had no time to fix it. From user’s point fix is just to add brackets around repository specification. So it should look like something like this:

Install("custom", {repository = {"localrepo-user"}})

Also, you really want to install package ‘custom’? You might be missing point of that command. Custom in that sense means name of custom package. So what you probably want is Install("adblock") or something like that.

Just for reference https://gitlab.labs.nic.cz/turris/updater/issues/212

dibdot · January 16, 2018, 9:30am

Thanks, works fine! The “custom” naming was only for this test …

Thanks again!

Jirka · February 3, 2018, 4:17pm

One more questions… How about updating already installed packages in localrepo, what is correct approach?

Must we use updater.sh to install/update packages from localrepo to system?

I noticed, that when I

Update package in localrepo manually / or use localrepo rm + add (for possible future installs by updater)
Use opkg install to update system version of package

the package is added to the auto repository automatically. I can delete it from /usr/share/updater/localrepo/auto directory, and then run localrepo check --fix, but I think its not the best approach.