torstai 25. toukokuuta 2017

Access point recommendations?

With some upgrade of my Internet connection, it seems that my trusty WRT-54GL wireless network is now a bottleneck. I would love to get a recommendation for 802.11ac etc access points. I'll be operating them strictly in access point bridged mode, and I'll need several so cost is a factor. But the WRT-54GLs have been spectacularly reliable and did not get confused and need reboots like many other products. That is something that I absolutely need.

lauantai 20. toukokuuta 2017

More disk!

I am in Canada, and have bought once again more hard drives! C-Ordinateurs Canada, the local Fry's equivalent, supplied the goods!

tiistai 16. toukokuuta 2017

If you think about IOT security, think broad enough!

Internet of Things security issues are serious, and are often the focus of discussions. The discussion is much needed. How can we make our IOT devices safe?

How can we prevent attacks similar to those that last year caused many popular Internet services to be unavailable, with badly secured IOT devices being used as a part of the attack?

This is a very important topic.

However, I would like to argue that people often think about this in a too narrow manner. First off, we  have a tendency to focus on visible, concrete things. However, there's more to IOT than the gadgets, and I think the other parts deserve equal scrutiny.

The IOT is not in the gadget, it is in the cloud.

We have to secure the gadgets, but we also have to secure the rest of the system. And more broadly, it benefits the consumers and users to have secure, interoperable, and open solutions for both the gadgets and other parts of the IOT ecosystem. We need data that is in well-specified format, we need data that is under user control, we need systems that you and I can compose from components. But we do not need closed ecosystems.

torstai 11. toukokuuta 2017

Internet and Societies

Today we have an interesting panel discussion organised by ISOC and Chatham House on the effect of the Internet on societies. Is the Internet helping bring societies together, or creating more divisions? With the increased criticism against globalisation, fake news, and the emergence of closed social circles for like minded (and often misguided) fragments of society, it is easy to be worried about this.

But, it is also easy to focus on the most visible issues. When looking at the Internet and societies, one needs to consider the full scope of human interaction, andconsider human, technical and commercial aspects together.

What issues are affecting our ability to connect together? I want to start with five points:

Human interaction is broad, and we need to look at the whole picture. It is easy to focus on the most publicly visible forms of media, and see how the news media for instance has, to put it kindly, become more diverse.

But whole picture is broader and more nuanced, and the concepts of togetherness and divisions may not be so applicable. For instance, the Internet has made it much easier for various smaller groups to connect where they perhaps had no ability to do so before. Communities working on Wikipedia, people with special interests being able to connect, minorities connecting to their culture, and so on.

Human interaction is both about tech and human abilities. It would be a mistake to think about solely technical solutions for problems involving, say news or social media.  Our technical capabilities advance at an incredible speed, but humans are also very good at learning new skills in new environments.

But, clearly critical media reading and communication skills are needed even more in today's world. These topics need to become even more central in our schools and continuing education programs.

Division vs. unification goes beyond people discussions. The Internet continues to be embedded in the fabric of our societies. We need to consider not just the people's discussions, but how well the Internet supports all the other interactions from personal gadgets to managing cities' traffic to running businesses.

Technical and commercial considerations. TCP/IP and the web provide a platform where we have almost universal interconnectivity and lack of technical barriers.

Still, as the IAB's IOT semantic interoperability workshop pointed out last year, interoperability at the level of applications can still be a problem. Can you buy Apple lightbulbs for a house that has Microsoft light switches?

And more broadly, are commonly used Internet services such as social networks erecting borders that restrict efficient connection, for instance due to their deployment patterns as is shown in the image further down?

And, is our increasingly centralised "winner takes it all" Internet economy driving a model where it becomes difficult to switch social network/search/video/mail/application store providers?

Finding broader consensus is hard, but rewarding. As those of us who work in standards or open source realise, finding agreements in broad, diverse communities is hard and time-consuming. Yet, we find the motivation to do so because if we succeed, the benefits are much greater than with everyone running their own things. We've obviously done this not just with technical developments like the Internet, but also to a large extent with our societies, building their infrastructure and rules. And I believe we will continue to be able to do that.

And where does all this leave us? Clearly, there is a lot of work ahead of us. But that work is not merely about the public sphere of news media or social media discussions, it is also about our ability to offer communications tools for all groups, regardless of their size. Our continuing education of the human parts of the system. Our drive to improve standards so that the technology allows connections. Our drive to ensure that the business system provides the possibilities for evolution and connection.

I would also like to point to my other article for a discussion of why IOT security is a much broader topic (inline with the thoughts in this article) than people usually focus on.

What do you think? Leave a comment below! You can also follow our panel discussion online.

Jari Arkko

Acknowledgments: I would like to thank all my friends and colleagues at the Ericsson, IAB, ISOC, and Chatham House for interesting discussions in this problem space.

Picture credits: 1/ Jari Arkko 2/ Evi Nemeth for the original picture, edits by Jari Arkko 3/ World Map of Social Networks from Vincos.It. How divided is this world, even at this level? And I was surprised to find out that there are places in the world where the most popular social media application is LinkedIn :-)

lauantai 29. huhtikuuta 2017

More 10G cards

Received my 2nd 10G Ethernet card, and successfully inserted it to the router. 3rd card is on order...

I've started testing the cards, and can get 9.3 Gbits/s speed! That does feel fast. This number is from iperf. Using SSH to copy files I get a smaller number, however, around 1.2 Gbits/s to 1.8 Gbit/s depending on which crypto is being used. The smaller number is on chacha20, the faster on aes128-ctr. Still investigating what the bottlenecks here are, trying to understand what iperf measures, for instance. Preliminary results seem to indicate that a CPU core is operating at a high load when it is doing encryption for SSH, but that disks are not the source of the delay.

More research needed... but this is already a 12-18 fold increase from my earlier servers who were only able to do about 100 Mbit/s while using SSH. In this case that speed was very clearly due to the CPU being unable to do crypto at a faster speed.

With regards to getting these cards to work, my only complaint is that it is difficult to manage Linux devices when the number of type of interfaces change. The interface names change... and for some reason I don't get accurate information about link status from ethtool, and some of my interfaces seem to not work well with a /etc/network/interfaces-based definition, but rather need explicit commands to be brought up. Odd. Maybe I've misconfigured something, or maybe there's some issue with these specific cards.

Photos (c) 2017 by Jari Arkko

tiistai 18. huhtikuuta 2017

Native IPv6!

I have once again full IPv6 connectivity on ALL of my uplinks, and this time natively! Thanks to my great ADSL ISP, Nebula, who offers IPv6 as standard service for everyone, and also my great mobile network provider, DNA, who do the same. IPv6 life is good in Finland!

Nebula gives you a /56 for your own networks, and a /96 for the router-to-router interface.

Their default sales and support guys understand and know IPv6 well. All you have to do is to ask for the /56 address and it will be given to you.

DNA IPv6 comes on, completely automatically, for every user with a capable device. Which is most devices by now. Both DNA and Nebula have been providing this service for many years.

My previous setup was through a tunnel service, interrupted due to addressing changes, and now gone forever; good riddance :-) Native is the way to go! I'm now fully dual stacked natively for all my networks, be it ADSL or LTE.

A couple of observations:

  • I was so happy to find out that while on my previous setup I had to resort to hacks to do firewalling on IPv6, all functionality is now there for even dynamic connection tracking. Great!
  • Once again, the simplification of my network to offer the bare essential services only has made things like firewall configuration much easier.
  • The router advertisement daemon, RADVD, has a bug on Ubuntu 16.04. The installation scripts do not create the pid directory /var/run/radvd, and this causes the startup scripts to fail. Silently... but you can do "sudo mkdir /var/run/radvd" and everything works after that.

Copyright (c) 2017 by Jari Arkko

lauantai 15. huhtikuuta 2017

New Router

I have finally replaced my trusty old main router with a new one. The old one has been going strong, but it was running on an old Pentium II platform from 1997 that I paid 10€ sometime in the early 2000s... for that investment, it has paid off phenomenally!

Not only that, but the old machine was badly maintained and stopped accepting updates without complete reinstall which I never found time to do. I was running kernel from 2005 for twelve years! Not just bad, this was a security nightmare!

I guess I don't have a lot of attack surface even if you get past the router/firewall, but I have been at least the target of DoS attacks on the router. Here is the old router, with its proud Pentium II CPU:

But, the hardware for the new router arrived earlier this year, and now I had time to set it up properly, and disconnect the old router. While the new router is not the newest gear still either, it has a modern architecture, hopefully better settings and maintenance, and much simplified configuration. The new machine is running an ASUS CS-B motherboard and the Intel Celeron G1850 CPU in a stylish but simple Bitphoenix Phenom micro-ATX case. There's a medium-sized SSD for the machine but no other disks. The machine will not run any other services than forwarding packets, firewalling, and DHCP for the internal network. And my OS is still Ubuntu, but this time version 16, not 4.

At the same time, I've reorganised my entire network around the following principles:
  • Right things in the cloud: Keep as much of the functionality in the cloud as possible. But do not lose control of your own systems or materials. I rent my own space in the cloud and keep file storage in my own servers at home.
  • Just make it fast: Build a fast, general-purpose and simple network that supports any new service that might come up in  the future.
  • Keep it simple: No unnecessary services, no extra complications, no complex architectures.
More specifically, what I have done is this:
  • Move all external-facing web services to the cloud. With one exception, all my websites -- such as -- are now hosted by Linode, and provide TLS certificates via Letsencrypt. I have yet to move, because that is the only domain that handles e-mail, and I haven't found a reasonable, free alternative to hosting that outside our lab server at work.
  • Simplify internal network organisation. I've disabled much of the old hardware and special purpose networks. I won't be needing NAT64 any longer, and I will work with a simpler network that doesn't require the HOMENET automatic routing setup. I will still maintain two special networks, for internal and visitor networks. But I've divided the two networks to use the two redundant uplinks that I have, on ADSL and LTE Advanced. This also allows easy (but manual) switching from one uplink and router to another when something breaks.
  • Turn off dozens of services for which I had no use, or which were only partially functional.
  • Upgrade the internal network to 10G. This is still in progress, as only one of my file servers has the necessary network card. Other cards have been ordered, but I'm still searching for a reasonably priced 10G switch with at least 3 but preferably 8 10GBase-T connectors. Pointers welcome.
  • Employ IPv6 as a means to access individual services from elsewhere in the Internet.
  • Employ smaller number but larger file servers. In my case it is still beneficial to have multiple physically separate devices for safety, but they need to be appropriately dimensioned. I.e., n * 10TB rather than measly 2-4 TB each as previously.

    The primary new file server runs on a similar new computer as the router, but with the MSI A88XM-E45 motherboard and the AMD Athlon x4 760K black edition CPU. This particular CPU unit is by the way a world record holder for the Athlon x4 760Ks; it used to be overclocked up to 7.1 GHz with liquid nitrogen, but it is now enjoying retirement at a more relaxed 3.7 GHz.

  • Employ redundant disk clusters. I've turned on ZFS on my new file server, running currently 2x10TB disks in mirroring mode, so being able to provide 10TB of storage. The really excellent thing with this is that I can add more storage on the go while keeping the same logical disk structure for users, even if I run out of the 10TB. Of course, redundancy within the same case is not sufficient for problems, so in addition to having manual backups I'm also considering hosting backup servers at alternate locations, with automatic network sync.

Not everything is quite up and running yet, in particular I spent five hours last night just getting the router to work. Turned out that the mere existence of a DHCP client package affected network interfaces that had been defined as static ones.

Setting up IPv6 to work with my ADSL connection to Nebula is the next step. The LTE side of the network already has it. There's also a couple of old laptops still running something that I need to figure out what it exactly is :-) as well. One of those laptops also drives the display to the sauna and its broken display... that needs replacement.

Here's the communications closet. The new router, file server, and old computations server are sitting side by side at the far end (this whole space is under a staircase), next to the new small rack that I had built earlier.

Photos (c) 2017 by Jari Arkko