[GUIDE] Cross-compiling packages for Turris Omnia with TOS 3.xx (using Nginx as example)

sik0vny · May 9, 2020, 6:59pm

So my aim was to compile Nginx package, with possibility to add custom modules to it later.
But Nginx is just an example here, with this method you should be able to compile other packages too.

This tutorial is a result of two weeks of trials and errors. I never had any experiences with Linux compiling.
So idea of cross-compiling (especially for such a non-standard platform as Turris Omnia) was for me just a big brainf*ck.
Nevertheless I had a big urge to learn it, because I wanted to have bigger freedom in what I can run on my Omnia.

The biggest problem was that I didn’t even know how to search on this subject, I did’t know how to formulate my questions in Google.
It is also possible that there is somewhere a comprehensive tutorial how accomplish my task. Well, I didn’t find it.
Now, with help of a few friends ( thanks to: glaeken, kixorz, Pinky, brozkeff) I have some small successes, so I decided to publish my findings.

Disclaimer: I’m still a noob in this subject, some of the stages of this process are still a mystery for me - I don’t fully understand them.
So parts of this tutorial are probably not optimal and can be written better - feel free to comment on them, your feedback is welcomed.
The most important is: it worked for me, so I hope it will work for people who will find this post in the future.

So this is how to do it:

You need Ubuntu 14.04.

My first try was on Debian10, and that was a failure. Later on I discovered that I need Ubuntu 14.04:
Unable to build turris openwrt on debian - compile error in freadahead.c in m4 - #4 by mkyral - SW bugs discussion - Turris forum
I’m assuming you are starting in /root folder. Create a folder in which you will have your cross compilation stuff
I will call my “cc”, go to that dir
```
mkdir cc
cd cc
```

We are now in /root/cc/

Download an official OpenWRT SDK for Turris
Based on technical info about Turris Omnia and Downloads section of using_the_sdk page, we can download the SDK. This link might change in the future.

wget https://downloads.openwrt.org/releases/18.06.2/targets/mvebu/cortexa9/openwrt-sdk-18.06.2-mvebu-cortexa9_gcc-7.3.0_musl_eabi.Linux-x86_64.tar.xz -O openwrt-sdk.tar.xz

This will download the SDK as a file openwrt-sdk.tar.xz

Unpack archive, remove archive, change extracted dir name

tar xvf openwrt-sdk.tar.xz
rm openwrt-sdk.tar.xz
mv openwrt-sdk-* openwrt-sdk

Now you have the official SDK in folder openwrt-sdk

Now we need to install some prerequisites

apt-get install make libncurses5-dev ccache subversion build-essential zlib1g-dev gettext libssl-dev

I tried to keep them at minimum, but there might be more needed.
If you will encounter any errors, try to install some more as described here:
[OpenWrt Wiki] Build system setup
Let’s download official Turris OS 3.xx repository from NIC

Go to website Tags · Turris / openwrt · GitLab
Pick and click on your Turris Os version, (I will pick v3.11.16)
Click on the Download icon, right click on “tar.gz”, and Copy link location, in my case:
https://gitlab.labs.nic.cz/turris/openwrt/-/archive/v3.11.16/openwrt-v3.11.16.tar.gz

Download and unpack:
```
wget https://gitlab.labs.nic.cz/turris/openwrt/-/archive/v3.11.16/openwrt-v3.11.16.tar.gz
tar xvf openwrt-v3.11.16.tar.gz
cd openwrt-v3.11.16
```

Now you are in the openwrt-v3.11.16 directory, and we will do everything from here as our base directory

And here starts the part in which I’m not entirely sure if all the following steps are needed, and if their order has a meaning

Run the official compiling script - this will probably be not successful at the end, so don’t worry.
This process will configue our compiler to a proper state, and will prepare packages we will need in the next steps
```
./compile_fw omnia`
```
It will start compilation of all the packages, but we only need Nginx package for now, not all of them.
This process almost always failed for me. If you will wait and it will compile for you successfully, then you might not get any errors in points 10. 11.
But if it will fail or you don’t want to wait for the whole process to finish, you can interrupt it in the middle.
It should be safe to cancel the process after the line ‘make[3] -C tools/libtool install’ is finished. Just press CTRL+C whe you will see: make[3] -C tools/gmp compile

This is a mystery for me. I have a feeling that there is a few packages crucial/needed for compilation which can be
picked in menuconfig. And without them successfully compiled, there will be errors in points 10 an 11. I don’t know. Anyway let’s continue.

Now let’s prepare nginx package

./scripts/feeds update -a
./scripts/feeds install nginx
make package/nginx/download
make package/nginx/prepare

Then open menuconfig to choose which nginx modules you want to compile:

make menuconfig

You should see there, first two fields set as shown below:

Target System (Marvell Armada 37x/38x/XP)
Target Profile (Turris Omnia)

Now, go to: Network -> Web Servers/ Proxies and find nginx.
Press Space on it (it will get M flag), then go to its Configuration, enable any modules you need, then exit, exit, exit, and save the configuration at the end
Compile
```
make V=s package/nginx/compile
```
You might get an error here (if not, jump to the next point)::
cc/openwrt-v3.11.16/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.15_eabi/lib/libgcc_s.so.': No such file or directory*

This is the moment I didn’t know how to get these libraries (there would be more of them) by “normal way”. At that point, ‘compile_fw omnia’ never finished successful for me.
So I came with idea that I will take it from openwrt-sdk-18 (the one we downloaded at the beginning).
I don’t know how safe and future-proof this approach is. Anyway let’s copy all libraries from the openwrt-sdk:
```
 cp ../openwrt-sdk/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-7.3.0_musl_eabi/lib/* staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-4.8-linaro_musl-1.1.15_eabi/lib/
```
Compile again
```
make V=s package/nginx/compile
```
And here you might get another error (if not, jump to the next point):
configure: error: C compiler cannot create executables
See `config.log’ for more details

If we will look into build_dir/target-arm_cortex-a9+vfpv3_musl-1.1.15_eabi/pcre-8.41/config.log
We will see:
configure:3823: checking whether the C compiler works
./configure: line 3847: arm-openwrt-linux-muslgnueabi-gcc: command not found

So again, I don’t know what is the proper way to deal with that, but I found out that if I will use a toolchain bin directory from the original openwrt-sdk it will work, so:
```
echo PATH=\$PATH:/root/cc/openwrt-sdk/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-7.3.0_musl_eabi/bin > addPATH.sh
```
This will create a script addPATH.sh that will add our binaries to the $PATH every time you will need it in the future.
You can also add this path to your $PATH permanently, but I will not cover it here.

Now run it:
```
source addPATH.sh
echo $PATH
```
The result should be similar to this:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/root/cc/openwrt-sdk/staging_dir/toolchain-arm_cortex-a9+vfpv3_gcc-7.3.0_musl_eabi/bin

Now this magical path will be used as a source for binaries that the compiler is looking for.

Here is the point whereI consider cross-compilation environment ready and prepared.

A. Final compilation
Here you can probably add -jx parameter to increase the number of job threads, to make the compilation process faster
```
make -j5 package/nginx/compile
```
And as result (hopefully) you will find your compiled package(s) here:
```
ls -al bin/mvebu-musl/packages/packages/
```

B. To cleanly recompile Nginx again:

source addPATH.sh (optional - needed if you opened a new terminal and $PATH is not set)
make package/nginx/clean
make package/nginx/prepare
make -j5 package/nginx/compile

C. Steps A and B should be the same or similar for other packages you will be compiling.
Sometimes binaries will end in different directories somewhere inside bin/mvebu-musl

Hopefully this was helpful. Enjoy.

sik0vny

Twinkie · May 10, 2020, 5:36am

I am very happy to see guide like this as more and more people trying co compile packages that are not included themselves. I went slightly different path and went bit more far as I also needed to compile whole openwrt image and packages for x86 architecture so I will comment few things and add my experience as well.

As it was noted here it is impossible to compile this TOS3 on Debian 10 buster but it was possible on Debian 9 stretch and also on Debian 8 jessie. At first I compiled whole toolchain and sdk following guide here https://doc.turris.cz/doc/cs/howto/turris_os_build with that I was able to make toolchain on LXC debian on Turris directly as I used 64GB SDHC card. Yes it took long, yes there were many retry of make -j1 to compile toolchain itself and at some point some binaries and files could not be assmbled at powerpcspe so I also compiled toolchain on x86 debian 9 and few needed files that could not be assembled on Turris directly transferred there. Even it is not recommended to do such file I/O intensive task on SDCARD in Turris I got it working and still using it and it is practically only option for me now as I already upgraded to Debian 10 buster and don’t want to have Debian 9 somewhere aside just in order to compile package for Turris.
In 2. you compile everything in root folder under root permissions which works perfectly in this situation, my experience is that sometimes when I try to compile whole toolchain for openwrt master it was impossible under root and it was necssary to change all files ownership with chown -R user:user ./openwrt-sdk/ . At some point when compilation failed it also helped to switch back to root with chown -R root:root ./openwrt-sdk/ probably as some specific package needed some include or library from build system. So when compilation fail this could help at some point.
As pointed in point 8. If you need just one package it could be fine but sometimes it could be tricky as in cases of some packages there are more dependencies for another packages with libraries and then it is usefull to compile whole toolchain and packages. At first I was unable to compile packages that I added manualy from different feeds or when I added specific package Makefile manually without running command
make menuconfig and then find package in menu and marking it for compilation (*) as it fix some dependencies and make whole config aware of newly added package. So make menuconfig is really important command to run.
Now my goal in coming days as we are probably near to TOS5 HBS final release my effort will go to compile packages for TOS5 so it would be great to share experience of compiling packages for TOS5. I tried that myself by compiling fresh whole toolchin from master and changing target for Powerpc architecture. I been able to succesfully compile and create packages that should be installable on TOS5 but when I finally tried it with opkg install, it failed due to unsatisfied libraries or packages dependencies. Shortly my fresly created package for master used much newer libraries than TOS5 that is based on 19.07. So it is necessary to go back in time and checkout same openwrt version that was TOS5 taken from with something like git fetch --tags git checkout v19.07.2. There will be probably some more easy way to checkout directly TOS5 version from cz.nic git but following this forum briefly and not much recently I had not seen any guide or information how to assmbly TOS5 yet as it is still under development.

Finally I would be very glad if more people share their experience with compiling as it could be a bit tricky and very time demanding to resolve compilation errors and problems so any piece of information will make our life better.