Intro to GIN Systems, Applications, and Software

Intro to GIN Systems, Applications, and Software

Configuration Management in the NTT Com Global IP Network ITNOG 5 Shawn Morris VP - Systems, Applications, and Software Who We Are Global IP Network (GIN) AS2914 NTT Coms wholesale IP backbone

105 POPs in 28 countries on 5 continents Offering IP transit and L2 pseudo-wire services Bulk of customer traffic is on 10GE and 100GE Using Juniper MX2k, PTX1k, and Cisco ASR9000

2 Are we talking about SDN? What is SDN? Software-defined networking (SDN) is an approach to networking that centralizes control of the network by separating the control logic to off-device compute resources. This enables operators to use programmable control to orchestrate and automate network services

without having to physically access the networks hardware. SDN Central 4 Traditional Network 5 Software Defined Network 6

Hybrid SDN 7 Is SDN Important? Automation Matters Reduces training burden Minimizes peer review Aids in cost control Leads to higher quality of service

9

Lower error rates (particularly catastrophic errors) Consistent service delivery Faster moves, adds, and changes Customer-initiated service delivery GIN 2009 10 GIN 2019

11 GIN Network Automation GUMS Overview GIN Unified Management System Homegrown Started in late 1990s at Verio Originally developed by the IP engineering staff Currently manages 500+ network elements Driving fully automated (not autonomous) network configuration

Development is now done by a dedicated team Currently version is 5.13 13 fully automated network configuration?

14 Database-driven configuration management Device-ready configurations built from the DB Brute force configuration deployment

Server-side configuration is canonical No persistent manual configuration on the devices Developing NETCONF-based configuration for next-gen DWDM network Brute force configuration management Entire device configurations are generated on the server Configurations are pushed to the router via scripts built on RANCID commit replace is used on IOS-XR

load update is used on JUNOS Each device has a .conf file in revision control where manual configuration can be stored 15 NETCONF? It didnt exist It doesnt solve the hardest problem in network automation

(How do I ensure Ive gone from current state to desired state) 16 Tech Stack

Postgres M4 Templates C binary with custom M4 processor for building configurations Makefiles to drive configuration builds CVS repository per config group to store .conf files Web application built on perl and catalyst Read-only REST API Scripts built upon RANCID for deploying configurations

Still running RANCID Running configurations All GUMS-built configurations 17 GUMS Workflow SQL Database M4 Macros

Config File Router User enters database changes via Web UI User initiates config build via make command on server User initiates config push via loadconfmem command on server Router is contacted by script via SSH Router requests configuration file from server via FTP

Configuration is committed 18 GUMS Evolution 19

Original purpose was using m4 to build BGP policy First use of a database was to store configuration knobs Much of the config was typed freehand into text files Certain repeated config bits were migrated into macros With GUMS v2 interfaces moved into the database include(`JNX.m4')dnl define(`myLOOP',`129.250.0.45')dnl PLATFORM(juniper,martini)dnl # # Verio / PAIX Palo Alto, CA Unauthorized Access is Prohibited

# pao6.verio.net 2000.05.17-0 For Service Call (800) 551-1630 # @`SERVICES'(myHOST) SERVICES()dnl NAMESERVERS()dnl LOGGING()dnl USERS()dnl SNMP(,`PAIX')dnl # interfaces { so-1/0/1 {

description "BB: pvu0 p1-0-0-0 - PAIX c34-r4-s3-s-b2b-b3-19-20/MFS o2-brt-u88-0001/Q spa-3003095/ELI oc-obgl-105143-003-elg"; clocking external; encapsulation cisco-hdlc; sonet-options { fcs 16; payload-scrambler; } unit 0 { point-to-point; family inet { no-redirects; address 129.250.3.25/30;

} COST(13, `so-1/0/1', `BB: pvu0 p1-0-0-0')dnl PIMMODE(`sparse-dense', `so-1/0/1.0', 1) } } @DEVICE(myHOST())dnl PLATFORM(hfr,mcast)dnl dnl ! @BANNER(myHOST())dnl !

SERVICES(`loopback0')dnl ! dnl ENABLE()dnl ! @R_POLICY(myHOST())dnl ! dnl NETFLOW must be defined before INTERFACES _NETFLOW(_COLLECTOR1())dnl ! @INTERFACES(myHOST())dnl ! @CLNS(myHOST(), `verio',12,`wide')dnl

! dnl @MPLS(myHOST())dnl ! @STATICS(myHOST())dnl _BLACKHOLE()dnl ! @L2VPNU(myHOST())dnl ! IPEERS(myHOST())dnl ! GUMS Evolution (cont.)

We were able to abstract our configurations so that they 22 could be moved between platforms seamlessly. We moved from knowing router configuration commands to knowing database tables.

We made many fewer configuration errors. We still did not handle concurrent configurations gracefully. We still had to write the templates in M4 We still had to click through dozens of screens to configure anything moderately complex GUMS Present and Future We have deployed service-based provisioning tools We have started development of GUMS v6 Python-based config generation engine Templates will not be in M4 (currently experimenting with Jinja2) Queue-based management of builds and deployments

23 LAG Tool The basis of most GIN infrastructure is Link Aggregation Groups Configuring an infrastructure LAG used to take many screens LAG interface (IPv4, IPv6, ISIS, description, etc.) Member interfaces (MTU, LACP, description, etc.) Much of this information was repeated (LACP, parts of the description, etc.)

Much of the information is pre-determined (e.g. MTU) So we built a tool where it all could be configured or edited in a single screen. 24 Before 25 Before 26

After 27 Another Problem GIN has an extremely large volume of Move, Add, Change activity This work happens in the middle of the night when minds are not the sharpest. It is not uncommon for a maintenance to involve moving

dozens of connections in a 3-hour window Changing configurations was becoming a gating factor during maintenance. 28 Migration Tool We built a tool so that migration engineers could do the majority of their configuration work during daylight hours when theyre at their sharpest Configuration changes can be queued during the day

Open ports where services will be moved are reserved An already fast MAC process has been further improved. Can now move hundreds of ports in a 4-hour maintenance window 29 What About Your Network? Basic Facts There will be development work

There has to be organizational buy-in Powerful open-source infrastructure is available (e.g. OpenDaylight, NAPALM) You dont have to do it all at once 32 Start Simple 1. Go watch Leslie Carrs talk at NANOG about NetDevOps 2. 3.

4. 5. 33 (http://bit.ly/2o8KYw0) Run RANCID Put your configurations in revision control (e.g. git, svn) Learn some scripting/coding (bash, python) Start researching open-source projects Name

Title E-mail Web Twitter 34 Shawn Morris VP Systems, Applications, and Software [email protected] http://www.gin.ntt.net/ @GinNTTnet

Recently Viewed Presentations

  • HZt - Test #2 Revision - Ms. McDonagh-Vella's Classes

    HZt - Test #2 Revision - Ms. McDonagh-Vella's Classes

    Idealism - Dualism & Phenomenalism. Approaches to Theories of RealityZatesky, etc. The Mind/body question. Rene Descartes (1596-1650), Cartesian Doubt and the Search for Foundational Knowledge & Dualistic Interactions. Why doubt everything, including your senses?
  • Rotational Motion

    Rotational Motion

    Angular Variables and Tangential/Linear Variables. In the ice-skating stunt known as "crack-the-whip," a number of skaters attempt to maintain a straight line as they skate around one person (the pivot) who remains in place.
  • Legacy Indication in IBSS - IEEE Standards Association

    Legacy Indication in IBSS - IEEE Standards Association

    IBSS Legacy Information Jeyhan Karaoguz Matthew Fischer Henry Ptasinski Broadcom Extension of Legacy Indication to IBSS Mixed IBSS networks (e.g. 802.11g + 802.11b) Discovery and response to 802.11b STA Any individual STA may discover 802.11b STA, others may be unaware...
  • Color palette

    Color palette

    Template will be damaged._x000d_ _x000d_ For more section color options, see expanded version of template. Florida Health Care Transparency InitiativeNiall Brennan Executive DirectorHealth Care Cost InstituteJune 23, 2017 Consumer Health Information and Policy Advisory Council .
  • The Frontiers of Technology in Warhead Verification Henrietta

    The Frontiers of Technology in Warhead Verification Henrietta

    The Frontiers of Technology in . Warhead Verification. Henrietta Toivanen. Project on Nuclear Issues Capstone Conference 2016. Center for Strategic and International Studies
  • 2013 NRCS/IPM Workshop Norm Leppla UF, IFAS, IPM

    2013 NRCS/IPM Workshop Norm Leppla UF, IFAS, IPM

    Featured Creatures provides in-depth profiles of insects, nematodes, arachnids and other organisms. The site is a cooperative venture of the University of Florida's Department of Entomology and Nematology and the Florida Department of Agriculture and Consumer Services' Division of Plant...
  • Chapter 8 Thin-Layer Chromatography Thin-Layer Chromatography (TLC) TLC

    Chapter 8 Thin-Layer Chromatography Thin-Layer Chromatography (TLC) TLC

    Comments Do not let the plate sit in the solvent chamber after the solvent front reaches the top of the plate. Why? When spotting, the solution typically adsorbs very quickly and the spot can easily get larger than desired (1...
  • What We Must Show The World - Simple Bible Studies

    What We Must Show The World - Simple Bible Studies

    These things should motivate the lost to obey, but they should also motivate the believer to continue steadfastly in the faith, remembering from whence we came. What We Must Show The World is: THE LOVE OF CHRIST.