PCB manufacturing issues

As one engineer to another, how did you trace down the WAN instability to the PCB? It got me thinking, because we never had issues with trace width control. Or we might just not have noticed it.
Is the trace width non-uniform or visibly wrong? Or did you notice reflections in waveforms that you were not expecting?

2 Likes

It was visibly wrong. When today’s presentation from IT16 conference will be on YouTube, I will send you a link where you can see it.

To make the reply more specific, at first we have seen a large number of checksum errors on the WAN interface. This made us think that there is a problem with soldering, because we have seen such errors in the past. Also the problem proved temperature dependent - when we cooled the WAN PHY down, the problem got worse. Because of this, we tried to “reflow” the PHY and other things, but nothing helped. Only then did we spot the trace width inconsistency between an earlier prototype (interestingly from the same manufacturer) and the current one.

2 Likes

Interesting. Have you scoped the signals on a working and a non-working board for comparison?

I guess it’s this picture Filip was talking about: https://youtu.be/_IaLOaOoSTA?t=1385

Can anoyone give a rough summary of the talk? I only understand half of it, but i’m really interested in the last part, the things that went wrong, and so on.

1 Like

Quick notes from 2nd half of YouTube presentation (sorry for quick english)

  1. variants:
  • router - 3 colors
  • wifi/no wifi
  • 2 memory options => 12 variants
  • power options - 4 main standards - 48 variants
  • stickers, photo, wall mount, LTE - 1152 variants
  • T-Shirts (size vs men/female) - 23040 variants
  1. manufacturing:
  • they had experience from Turris 1.0/1.1

  • little complicated by stretched goals and by 8000 pcs

  • timing (everybody want it right now)

  • Prototypes:
    – manufacturer in CZE - 60 pcs for development - 44 pcs received, 16 lately, problems with manufacturing (technology and problems with vias), problems with surface mount - from 60 orders were 27 functional but some of them just for a while - very badly manufactured - board are dying continously
    — some founders bought prototypes and they needed support, problems solved by replacing
    — problem is being solved with manufacturer…
    – manufacturer in GB - 10 pcs - 6 pcs arrived - machines in GB died - very bad surface, very badly manufactured, it was impossible to operate them - problem is being solved with manufacturer
    – a lot of delay 4-5 weeks
    – another manufacturer from China - alredy tested with good reputation, with good experience from previous successfull projects
    — 30 pcs ordered, 30 received before deadline
    — problems with paths on WAN - HF signal with precise impedance on paths - difference between documentation and manufactured product
    — some crosstalks on WAN etc.

  • this week - they wanted to make order on 8000 PCBs then make surface mount, testing etc. …

  • next steps - they need to make another testing cycle that means 4-6 weeks of delay for PCB manufacturing, delay for surface mount, shipping delays, testing delays, etc.

7 Likes

Insanity. I would expect China to provide bad pieces, but CZ and GB manufacturers ?

Anyway, people can interpret these results as very bad, but I want to express my support for Turris team.
It’s not every day that I spend such amount of money on router, but looking on other companies doing routers, you surpassed them by quite a margin.
And I intend to use that router to its maximum, so I want some fine piece of hardware from you, and I do NOT mind waiting longer than was forecasted.

Keep up good work,
Bobi

2 Likes

I have the similar needs and have ordered two omnia with NAS for two work places.
I need them to be very reliable so I am very happy to wait. There is no other device available that can fulfil the requirements so I would still be waiting for something else…and yes, cz.nic, please keep up the wonderful work!

Thanks for this transparency about the prototyping process, and relations with manufacturers (there is a word similar to glasnost in czech ?). I appreciate.
And as other posters have said : better late than sorry.
I hope you can solve those issues in the near future.

Can’t be stressed too much:

  • Reliability > Time to market!

Thanks cz.nic for not rushing it to the market! I assume that the large majority of people who support the Omnia would choose stability / security over delivery date as well! Personally I’d rather have you guys encountering issues on your desks than discovering it myself when the router is running on my desk (and having an unstable router is not an option for me)! :smirk:

So: keep up the good work; take your time to (have it all) fix(ed) and/or further test for other issues!!

We can wait (albeit being hyped)! :slight_smile:

2 Likes

absolutely interesting information.

@turris: why is this info not in a indiegogo update?? i’m sure a lot of backers would appreciate the transparency and be more open/kind to delays if they knew this!
sidenote: guess shipping will definitely be later than july first, so i can go and change my shipping adress?!?

but i do wonder, as you have already produced a lot of original turris routers:
is this kind of shitty prototype quality expected or is it just bad luck?

i mean, i’d expect 60 out of 60 prototypes made to spec (unless the fab doesn’t want the job…), thus if something doesn’t work it’s a design flaw and entirely the customer’s (in this case turris) fault… or am i just naive?

1 Like

This matter has been laid out quite clearly in this discussion and related topics, please read the prior discourse, including the images regarding the faults shown in the subject video presented in czech.

1 Like

If you stay within the design rules of the manufacturer with regard to trace width, trace distance and via diameter and placement there should be no problems.

As the current (bad) boards are pre-production i think the pcb producer use production settings and machines. These may be quite different from the ones used for small prototype runs and have different design rules.

As the traces are 76µm wide and 124µm apart it is quite possible these will not be possible on standard multilayer technology. KSG for example gives the trace width with 100µm and spacing with 120µm for 18µm copper on the outer sides. You would need 9µm copper on the outer sides to get down to 80µm and 100µm. Inner layers have lower values though. There is always some space to negotation.

So the chinese boards seem to be well within the manufacturers capability but outside of the needed specs.

1 Like


Boards are in production!

I have a to curb your enthusiasm here – the only the replacement boards for testing are in production. There will be a QA done on them and only if no problems are found with them, the boards for final product will go to production.

Still good news though!

I just do hope smaller discovered issues are not ignored or fixes for them rushed from now on (for a shorter time to market in an acknowledgement of the delays so far).

I just want my p-e-r-f-e-c-t router (rather a month later than incomplete / including (small) defects).

2 Likes

Same here, better later but solid hardware instead of rushed product.