NodeBlog

Network topologies have traditionally been constrained by many factors. Some of these factors are the physical or logical bounds of the primary technology used, specific vendor choice or implementation, the original architect/designer, space, power, budget etc. Sometimes there is a deep focus on network architecture in organisations yet more often than not growth is reactive and ad hoc. It's been bothering me for a while now that no standard language or grammar exists (backed by standardised metrics and design patterns) to support design communication, decision making, and/or topology choice. Sometimes we're lucky enough to have an awesome, influential, cohesive, and long term network team in an enlightened organisation that understand the network as both a foundational and advanced platform for practically everything digital. Lucky enough perhaps to have a team that is (cap)able and motivated to apply consistent engineering practices even without a common dictionary or taxonomy of network design patterns to fall back upon. As such, enterprises and organisations are at the whim or behest of either an individual or team's disparate experiences which, as with most human aggregates, are subject to standard deviations in perspective and normal irrational human flux. In comparison to other engineering disciplines one might be so bold as to suggest that this a strange community of practice with no agreed common tongue?

A simple network language borrowing from graph theory and comprised of visual representations of nodes(vertices), edges, and (potentiallty) overlay flows should exist such that simplicity, efficiency, consistency, continuity, scalability, and of course cost effectiveness are more easily discernable.

Will a bus, star, tree, CLOS, ring, mesh, or other new/hybrid topology suit the operational, situational or cost landscape?

When message passing; concepts of data confidentiality, integrity, availability, velocity, pathing, reachability, and dissemination do come in to play (as does the robustness, resiliency, efficiency and intelligence(1) of the underlying fabric). Tradtional concepts of parallelism, serialization, and optimal flows are applicable to both the endpoints and the fabric, yet few actually seem to calculate component or basic node availability metrics. Availability for one is dependent upon parallel or serial paths including metrics such as MTBF(Mean Time Between Failure) and MTTR(Mean Time to Repair) whereupon one seeks to either optimse or minimise each. Have we invoked ORM(Ostrich Risk Management) as all this maths and hoopla is too complex to calculate for a service due to additional vertical dependencies?

When redutionism is applied to find the base units in play e.g. atomic data blobs, their states (at rest/in motion -> messages), potential vectors (paths), actual telemetry (flows), delay (latency), bandwidth (throughput), cartesian coordinates/location (including multi-state), value (in some form of currency)... one quickly realises that data-in-motion and data-at-rest are complex entities and exhibit complex state conditions and vectors. Complexity is especially compounded when attributes mentioned previously such as confidentiality, integrity, and availabilty are attempted to be factored in to design patterns. Initially there is a leaning towards an adherence to OSI layer diagrams in the network community to help facilitate more simple layered models however many networks are built with minimal knowledge of future application flows, weightings, classifications or expected lifetimes. We are simultaneously using and evolving client/server hierarchies (with fan-in/fan-out ratios), peer-to-peer protocols, large scale distributed compute clusters with reflexive backpressures etc. One wonders what's coming next as endpoints and human clients seem to assume abundance rather than scarcity. Can our network commons absorb over-grazing?

So where is our specific network design and diagrammatic language, patterns, taxonomies, dictionaries, and grammar? And how can we simplify some of the conceptual building blocks to facilitate makers, consumers, and society at large? Can we build and maintain the most appropriate and adaptable structures while evolving them to satisfy pre-stated and emergent requirements? "Isn't that what we've always done?", a network engineer in the back of the room asks,"we've always built for the most use-cases knowable with the fattest and best quality we can squeeze out of or in to the budget?".

I propose a new interim domain specific language which is above math logic/graph theory and below traditional network designs to formalise prototyping and measurement. This may have utility across disciplines as we slowly awaken to a telecsopic and fractal like network paradigm at difference scales.

What would it look like? Well, shapes and lines of course

Think of a standardised CAD(Computer Aided Design) environment/language for networks! Here's an (A) and a (B) to get you started, do you have a (C)?

Note: I may be totally re-inventing the wheel (let me know if this is a subset/superset of something not called out here) but methinks it's plausible that a more well understood and easily used intermediate language is required for architects and engineers (and the masses) as we more deeply explore, augment, and enhance our natural and 'artificial' networks.

(1): If the network does no translation or policy enforcement it should be flat and localised including infrastructure node independence.
(2): If the population cannot be surveilled or policed by either intrinsic or extrinsic logic the fabric is deemed anarchic.

Something we @nodecity folk talk about from time to time; Cosmic Bit Flipping. With last week's large solar flare came discussions of bits flipping. On AusNOG, Mat Walker confirms seeing a wide variety of Real-World Cisco Routers being taken out by bit flips. His full post is below;

Concur.

Over the past few years we've seen a few;

%SYSTEM_CONTROLLER-3-ERROR: Error condition detected: TM_NPP_PARITY_ERROR

errors come up in anything from C2911, C6500 to C7600's, which results in a system reset.

All perfectly healthy devices, having never exhibited any problems ever before, and haven't since they day they threw the above error and reset.

In 99% of cases, I was able to correlate the above event with a large spike in solar flare activity on NOAA's website.

Oh look, its 16:00 AEST. Happy CME everyone!

Mat Walker
Network Operations Manager
iseek Communications

The IRISSCERT Cyber Crime Conference took place in the D4Berkely Hotel, Dublin, Ireland, on 23 November 2011.

The Irish Reporting and Information Security Service (IRISS) is a voluntary body created to to "aid Irish based organisations and citizens to better secure their information technology". The organisers put a huge amount of their free time in to the conference. Consequently, donations are appreciated and can be made through their website. As well as the gargantuan work that went into organising the conference itself, a further 500+ man hours were spent setting up the HACKEIRE Capture The Flag (CTF) competition.

As a non-security-professional, the most interesting presentation for me was Dale Pearson's tutorial on how to get inside an organisation via the physical route. He outlined how criminal hackers study social media to build up a profile of a company's employees, their activities and relationships. He explained the advantages of targetting the newest employees by, for example "accidentally" meeting up with them at their favourite lunchtime haunts. He showed how to gain access to buildings, including keeping an eye out of misplaced usb sticks. And how to virtually steal the furniture if you had a mind to. Fascinatingly dangerous guy. Incidentally he may have inadvertantly provided me with an explanation for the locked wheelie bins in Leeson St. which may well be worse than I originally thought.


Mikko Hypponen's contribution was a fascinating, if helter skelter, journey through the evolution of the virus. He showed examples of the, neutered, visual content of the early viruses. Some of these reminded me of the old game of tennis which was the first contact many of us had with the computer, and this at a time when "hacker" was a respectable term and these guys were doing it just for fun. The byte-a-time animations reminded me of Clive Sinclair's ZX Spectrum (16KB to those of you who can still think so small). Mikko then took us through the more destructive phase when malicious hackers just wanted to screw up your computer. And on to the time when criminal hackers figured they were on to a financial goldmine. Some of these created criminal cyber empires, outsourced the coding and advertised widely for virus code which the authors had finished with and which they could use in their wider field of operations. He finished up with the latest phase of intergovernmental cyber war, Stuxnet and Duqu – which he said came from the same stable. He also covered the recent Certificate Authority (CA) breaches which in part arose from the refusal of the international community to grant CA rights to Iran and led to that country hacking an existing Dutch CA. He observed that the current CA system was breaking up. He also mentioned that there were relatively less viruses out there for mobile devices as the guys were all busy doing the easier and more remunerative traditional hacking and there were a lot of different mobile systems. As Android looks like becoming the predominant operating system here, it is likely to be targeted by criminal hacker activity before long. Mikko's contribution was the only overtly political one. From his opening shot of the East German Typewriter ⁽¹⁾ to that of the latest laser printer, he stressed how the individual citizen's privacy was being eroded by government and how what originally brought howls of protest was now meekly accepted and proceeding apace behind the scenes (or inside the black box, so to speak).

Stephen Bonner was a real communicator, with high PR skills. He gave a rundown on the user awareness (TH!NK PRIVACY) campaign which they had run in Barclay's Bank. This campaign was very effective, and he gave examples of how, and how not, to measure effectiveness. The campaign had resulted from some serious brainstorming within the company. Some of the ideas they had come up with were so scary that they had to be scaled down in implementation. His illustrations were great and his technique for getting audience participation was a little unusual. Anyone asking or answering a question, or making a remark, from the floor during the presentation had a Ferrero Rocher chocolate pegged at them along with an invitation to their nearest neighbours to try and intercept it in flight. I was first to put up my hand in response to an invitation to receive a paperback book which was part of the Barclay's campaign and promptly had a copy pegged at me from the stage. I've just started reading it. One of his observations was certainly an indication of some of the lateral thinking that went into the campaign. He observed that arts graduates could be employed for peanuts compared to what IT/Security and other professionals demanded. So low budget arts graduates were extensively used in the campaign, along with authors/writers who were only too glad to see their work in print and who wrote the short stories in the campaign booklet which I had caught in mid-air. Great example of how to do a presentation. Serious material entertainingly presented.

Ryan Jones took us through an incident where a seller's online cart stopped working at at 2am and it was not clear what was going on. On closer investigation it turned out that the whole database had got dropped. He gave two different versions of an incident follow-up investigation. In the first there there were no proper database logs, so the investigator was relying on whatever else was on the system. The results were inconclusive. The second was where much more comprehensive logging had been undertaken and this revealed that the system had been compromised for some time and credit card and other details had been siphoned off. The company meanwhile, and unrelated to any incident, had decided to outsource the purchase module and as a result the hackers information stream dried up. So he came back and maliciously dropped the database. I asked Ryan was this not a silly thing for him to have done as it only drew attention to the fact that the system had been compromised. He replied that the hacker would have sold on the information and by that stage people should have been getting unusual entries on their credit card statements. So the hacker had nothing to lose. On reflection, though, there had been no mention of any complaints about this to the company. Probably the card holders had no idea where in cyberspace their details had been compromised. Interesting. In relation to the more comprehensive logging, Ryan made the point that, despite apparently huge storage space requirements, this could nevertheless be relatively selective – for example, you didn't have to store all the actual video material in the case of a Youtube account. He recommended looking up Trustwave's Global Security Report.

Robert McArdle demonstrated features from HTML5 which seems to have integrated almost anything you might wish to do into a HTML environment. The problem is that because this is all in realtime and under the contol of and within the browser it bypasses a lot of the traditional security mechanisms. We are obviously coming to the point where not only will new mechanisms be required but you will need to shut down all browser windows/tabs when not actually in use. The background is becoming a dangerous place. Still, the stuff looks magic, right up to 3D arcade/playstation interactive cyber gaming. Bob is now blogging his IRISS talk in three parts at Trend Micro.

John Burroughs (standing in for Rik Ferguson) gave a very clear exposition of the security problems arising from the move to cloud computing. The distributed nature of cloud computing over virtual machines, storage sharing, and the need for speedy security updates in a rapidly changing realtime environment, not to mention controlling the “hyper visor”, all pose complex security problems, which is one of the reasons for many people delaying migration to the cloud. There are also issues of control of, and responsibility for, security which have not been fully resolved. I can empathise with a certain suspicion of the cloud and a reluctance to cede all computation to a source outside my own controlled environment. I have no wish to be reduced to the status of a dumb terminal. I've been there in the early 1980s and I have no wish to go backwards.

Dave Venman's presentation, on Fun and Games with IPS, went a bit over my head both due to the content (I am not competent at the packet level) and the style of presentation. He did, however, recommend checking out the Verizon Data Breach Report. Eoin Keary dealt with mobile devices but again content and presentation meant I didn't really get a grip on this presentation.

Brian Honan and his team, are to be congratulated on a great piece of organisation, from getting the speakers, to running the day itself. Gordon did a great job as MC.

The material above is supplementary to irlpol's post
_________________________________________________________________

⁽¹⁾ Manual typewriters had to be registered in East Germany, along with a sample typed page, so that any (subversive?) literature produced on them could be subsequently traced to the source. Modern laser printers carry id coding which is transferred to documents to enable subsequent tracing. At the Q&A session I mentioned my experience from another former Communist country in 1991. In Vilnius (Lithuania) the internal phone directory in the hotel room was a vast matrix of apparently unrelated numbers. I enquired about this and was told that they were all direct phone lines. I really thought this most inefficient and couldn't understand why, even under the Soviets, this apparently modern hotel didn't have a normal automatic switchboard. However its (political) efficiency was another matter as all the direct lines went through the nearby police station.

Last week Nodecity provided the wired and wireless network for over 1000 devices across two concurrent web conferences in Sydney, Australia. The primary conference was Web Directions South by John Alsopp and Maxine Sherrin which welcomed more than 650 attendees. The second conference was WISE with approximately 70 or so attendees. Both conferences were running at the Sydney Convention and Exhibition Center where Nodecity built a hybrid network using portions of the convention center's network (including one of their internet links) but mostly with our own network equipment.

Interconnectedness
At Nodecity we intrinsically understand feedback loops and the benefit of interconnected open systems (partly to do with some of us being network engineers, programmers, designers, Zen Buddhists, and community builders!). As such, we wanted to ensure that the network facilitated message flows between entities and envoys akin to that of other mediums such as water, air, and light. This was the third year running we endeavoured to enhance the attendee experience by frictionlessly connecting human to machine, machine to machine, and human to human.

Read on for some insights, flow data, and metrics!

Traffic and Devices
Overall we saw about 135GB of data usage (which could have been greater had Nathan not quickly rate-limited apple.com on our wireless edge to defend our bandwidth from a possible iOS5, iCloud and Lion update stampede!) on our 40mbps symmetric link. We actually could push closer to 70mbps UP as the venue's rate-limiting was applied in one direction only. At WebDirections there is normally a heavy weighting towards Apple devices as you can see from the Meraki sourced metrics below). Unfortunately we did not have device fingerprinting capabilities on the WISE network and as such rate-limited their conference to a symmetric 10mbps pipe across the two days.

Entities, Envoys and Flows
The title of the post is "88 million cybernetic flows" and this is derived from Nodecity having sampled every 50th packet that traversed our core router whereby the netflow sampling saw 4,286,399 IP packets and 1,772,775 related flows. If we divide the packets by the flows we get a ratio of ~ 2.42 . Once we multiply back out the packets * 50 (the sampling rate) we get 214,319,950 packets and divide by the flow ratio (2.42) to arrive at 88,561,963 actual flows. Overall we saw 1,435,350 million DNS packets/flows which have a packet to flow ratio of 1 (rather that 2.42).

If you want us to data mine the traffic flows for you or even poke around the data set yourself then just email us at flow@nodecity.com and we can send you the raw anonymised flowset (including some simple commands using flow-tools)!

Accelerometers and OODA Loops
During the conference Andrew Fisher tried two cool live demos which both made use of attendees accelerometers in their iPhones and/or iPADs. Unfortunately muggins here even after much preparation, gave him the wrong subnet mask for his server right before the Arduino 'Tug of War'.. (we were going to separate broadcast domains) and after Andrew's second experiment crashed part of the RF network right before the closing keynote (Day 1) with his MMWTG (Massively Multiplayer Wifi Tank Game) we weren't looking too shiny around Nodecity's ad hoc network operation centre

Day 2 we tried again to let the pre-keynote audience play the tank game over wifi and ensured we had more radios covering the 2.4Ghz range. These smaller cells thankfully worked quite well without crashing the radios! 182 source IP addresses in the 10.16/16 range tried to send packets to 10.17.0.31/32 i.e. Andrew's wired laptop/server (which we had moved to the WISE conference subnet) though the server only registered 61 active connections. It was fun to see a nearly full auditorium trying to drive their wifi tanks on the main screen by moving their phones around and shooting at one another!

Radios
From the outset we knew we had a noisy environment to work within as the RF(radio-frequency) space used by most iPhones and handhelds (802.11b/g => 2.4Ghz) happens to be quite polluted these days. This spectrum gets utilised extensively at conferences whereby attendees enable and share their personal 3G connections via what's been dubbed MiFi. We began by asking the convention center to switch off all their nearby wireless access points however they had to leave the foyer and lobby access points running for operational reasons. At certain times throughout the event we were seeing 100% channel utilisation of 2.4Ghz in certain rooms whereupon the 5Ghz range and channels were hardly under any load at all. You can see some of the utilisation and physical location in the below images.

Building Blocks
Our network consisted of a Juniper SRX650 router (which was sampling network flows by looking at every 50th packet), Meraki MR14/MR24 Wireless Access Points, Ruckus 7962 Wireless Access Points and a Ruckus Zone Director 1100.

Team
This year our team was comprised of irldexter,nathan_scott, pat, incorrect and wadeis.

Special thanks to Santi Sthawornwiphat of the convention center tech team who always goes above and beyond for us each year!

Philosophy
More musings on interconnectedness in the form of Überplasticity and Novelocity.

Culture is expressed uniquely by each of us and when we get together we create a shared culture; not you, not me, but us! At this point it immediately develops momentum of its own. Once existent, shared culture is believed to be hard to change but this isn't exactly true.

I've read much about the importance of a good work culture but little about effective methods to help bring it about. Often presented solutions are management or Human Resources 'initiatives' such as frequent team meetings and annual team-building exercises. Translated, this means culture is driven 'Top Down'. I call bullshit. Every individual's action and interaction either reinforces existing ideas, protocols and behaviour, or is a step towards a new culture. The potential for evolution exists constantly, moment by moment. We often outsource our capacity to make a difference in the work place, yet, 'out-of-hours' or during weekends many fight to bring about change in their locality or the wider world.

The workplace doesn't have to be a battle ground where one constantly fights for change. The point here is that we do create our own culture. Accept responsibility for this fact and you take ownership of your ability to change your environment. Bottom up and inside out, be the change you wish to see. But how?

Seeding, signals, and feedback loops.

Changing systems can be hard. Seeding ideas and bouncing them off others is a great way to initialise and signal intent. Seeds start small and seeding lets an idea breathe. Throughout this time signals and feedback loops can go crazy, and with time they can take root. Work from inside the system and allow others to work with you.

The hardest thing about an active role in any change is that for those who are passive or already stagnant, change can be uncomfortable, challenging and hard. Habits once formed require mental energy to break. It's a case of re-embracing the unknown again once the change has embedded itself.

Remember: Culture is our shared responsibility and we can either reinforce the status quo or contribute by creating something better.

Nodecity's Culture:

Our culture is both emergent and architected. It exists as a living expression of individuals. These free agents embody a shared ethos and simple belief that in enabling entities to communicate freely and more effectively, that a universal net positive will ensue. We do this by building and cultivating logical and physical networks. We do this by seeking to understand ourselves and our role, influence and impact in the wider networks of society, nature and the cosmos. We investigate and challenge our interconnectedness and interrelationships. Nodecity culture is an antidote to legacy modes of thought but also an exploration of open systems where conflict is embraced and liberation sought. We aspire to making the invisible visible such that transparency in all areas facilitates the network to evolve, heal faster and be more resilient.

Nodecity believes humans have the right to unfettered private and anonymous connectivity in cyberspace. In light of recent events we've decided to explain our stance on digital freedom, the current climate of censorship, and our action.

Our Stance

As per the United Nations Declaration of Human Rights Articles 19 and 20:

we seek to ensure basic human rights for all our brothers and sisters traveling together on the spaceship planet earth.

Climate of Internet Censorship

Egypt is not the first, nor will it be the last country to be removed from the internet. In recent years governments throughout the globe, have routinely forced ISP's to drop the internet in times of unrest. Internet Censorship at country levels has recently surfaced in China, Germany, New Zealand, and the UK. It can safely be said that internet censorship is coming, like it or not.

Internet in times of Censorship, The Future

Taking action with Tor is a great step to stop filtering in a time of need, but it requires access to the internet to work. We would like to get the ball rolling on more Peer-to-Peer communications networks including ad-hoc nodes and local networks for when access to the internet (backhaul) is unavailable.

Question How do we bring up local networks across a mass populous and allow users to communicate and coordinate without relying upon any centralised supply chain?

Action

It's a small gesture but the node/server you are currently viewing Wiki on, is now also a node in the Tor projects network which allows for anonymous surfing of the internet using a technology called Onion Routing. (We are serving hundreds of concurrent connections every second and many gigabytes of data each day).

For more information on Tor http://www.torproject.org/

Notes and links

• Tor blog of recent events in Egypt: https://blog.torproject.org/blog/recent-events-egypt
• UN declaration of human rights: http://www.un.org/en/documents/udhr/index.shtml

Updates

• 16 May 2011 UN Report from the Office of the High Commissioner for Human Rights echos our post above (does not advocate these exact means however) http://www.ohchr.org/EN/Issues/FreedomOpinion/Pages/Annual.aspx
• 03 June 2011 Tor http://www.torproject.org/ are sending us t-shirts for running our high speed exit node.
• 17 March 2012 Tor weather https://atlas.torproject.org/#details/3E96079B64D7D0ED602C9245772B491362B83B31

Feedback

(Wade, Donal and all the other Nodals thank you for your attention!) What do you think about recent events?