Cloud backups outage

Dear Turris OS 3.x users,

there are some issues happening with cloud backup servers since yesterday. We are aware of these issues and we are trying to solve it as soon as possible.

We will post an update here once resolved.
Thanks for patience.

6 Likes

Everything is up and running. Please let us now (here, preferably) if you encounter any Cloud backups related problem from now on.

2 Likes

Still failing for me. Cloud backup section immediately says something like “Unable to download backup list”.

Turris OS 3.11.21, Forris 100.5.1

XHR endpoint https://192.168.9.1/foris/config/main/ssbackups/ajax?action=list returns 500 Internal server error with:

KeyError('backups',)
Stack trace
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/foris/middleware/reporting.py", line 93, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python3.6/site-packages/foris/middleware/backend_data.py", line 86, in __call__
    return self.app(environ, start_response)
  File "/usr/lib/python3.6/site-packages/bottle_i18n.py", line 68, in __call__
    return self.app(e,h)
  File "/usr/lib/python3.6/site-packages/bottle.py", line 979, in __call__
    return self.wsgi(environ, start_response)
  File "/usr/lib/python3.6/site-packages/bottle.py", line 954, in wsgi
    out = self._cast(self._handle(environ))
  File "/usr/lib/python3.6/site-packages/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/usr/lib/python3.6/site-packages/bottle.py", line 682, in mountpoint_wrapper
    body = app(request.environ, start_response)
  File "/usr/lib/python3.6/site-packages/bottle.py", line 979, in __call__
    return self.wsgi(environ, start_response)
  File "/usr/lib/python3.6/site-packages/bottle.py", line 954, in wsgi
    out = self._cast(self._handle(environ))
  File "/usr/lib/python3.6/site-packages/bottle.py", line 862, in _handle
    return route.call(**args)
  File "/usr/lib/python3.6/site-packages/bottle.py", line 1740, in wrapper
    rv = callback(*a, **ka)
  File "/usr/lib/python3.6/site-packages/foris/utils/__init__.py", line 79, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/foris/config/__init__.py", line 225, in config_ajax
    result = config_page.call_ajax_action(action)
  File "/usr/lib/python3.6/site-packages/foris_plugins/ssbackups/__init__.py", line 142, in call_ajax_action
    return self._action_list()
  File "/usr/lib/python3.6/site-packages/foris_plugins/ssbackups/__init__.py", line 97, in _action_list
    backups=self._prepare_list_data(data["backups"]),
KeyError: 'backups'

Do you have data collection enabled, registered and running on your router? Does this error appear another day as well?

I think the best way to debug cloud backups is from command-line:

# /usr/share/server-uplink/registration_code.sh
# ssbackups list; echo

I believe that data collection is enabled

Data collection status
uCollect data:	 Online status updated 78 seconds ago
firewall logs:	 Online status updated 413 seconds ago

Yes, this error appears in some last days.

# /usr/share/server-uplink/registration_code.sh
/usr/share/server-uplink/registration_code.sh: line 43: can't create /usr/share/server-uplink/registration_code: Read-only file system

# ssbackup list
{"detail": "Command << ['/usr/bin/curl', '--fail', '-X', 'GET', '-m', '10', '-H', 'Accept:application/json', '-H', 'Authorization:Token C6***', 'https://rb.turris.cz/backups/'] >> ended with << return code = 22 >> and with output << b'' >>.", "exit_code": -1

# echo $?
255

# mount
/dev/mmcblk0p1 on / type btrfs (ro,noatime,ssd,space_cache,commit=5,subvolid=260,subvol=/@)
proc on /proc type proc (rw,noatime)
sysfs on /sys type sysfs (rw,noatime)
none on /sys/fs/cgroup type cgroup (rw,relatime,cpuset,cpu,cpuacct,blkio,memory,devices,freezer,net_cls,pids)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noatime)
tmpfs on /dev type tmpfs (rw,relatime,size=512k,mode=755)
devpts on /dev/pts type devpts (rw,relatime,mode=600,ptmxmode=000)
debugfs on /sys/kernel/debug type debugfs (rw,noatime)
mountd(pid3501) on /tmp/run/mountd type autofs (rw,relatime,fd=5,pgrp=3494,timeout=60,minproto=5,maxproto=5,indirect)

# ls -la /usr/share/server-uplink
drwxr-xr-x    1 root     root           162 Oct 14  2019 .
drwxr-xr-x    1 root     root           510 Jul 14 21:44 ..
-rw-r--r--    1 root     root            11 Nov 14 02:00 contract_valid
-rwxr-xr-x    1 root     root          1547 Oct  8  2019 contract_valid.sh
-rwxr-xr-x    1 root     root          1270 Oct  8  2019 registered.sh
-rw-r--r--    1 root     root            16 Nov 17 00:00 registration_code
-rwxr-xr-x    1 root     root          1379 Oct  8  2019 registration_code.sh

Is it ok that /dev/mmcblk0p1is mounted read-only? I also tried to reboot router.

# btrfs check --force --readonly /dev/mmcblk0p1
...
found 472510464 bytes used, no error found

no, it’s not ok :disappointed:

Your issue is not related to Cloud backups outage several days before. There is some problem with your eMMC flash storage or btrfs filesystem. dmesg should provide some more info.

You better reach our support for further assistance.

This topic was automatically closed after 8 days. New replies are no longer allowed.