Network: Cisco Catalyst разное

Cisco live sessions – catalyst switches troubleshooting (разбирал обе презы подробно)
- Troubleshooting Cisco Catalyst 3850 and 3650 Series Switches
- Catalyst 3560/3850/9×00 Troubleshooting
Cisco live session – Cisco Enterprise Silicon (не разбирал презу подробно)

Еще куча неразобранных през

BRKARC-3438 - Cisco Catalyst 3850 and 3650 Series Switching Architecture 
BRKARC-3300-Troubleshooting Cisco Catalyst 3650,3850 & 9000 Series Switches 
BRKARC-2810 - DNA Campus Fabric Automation - A Look Under the Hood 
LTRNMS-2500 - A Practical Look at DNA Center: Hands-On Lab 
LTRCRS-2450 - IOS XE Programmability and Automation for Network Engineers

(security, cisco catalyst, fortigate) SMI (Smart Install) нужно отключать/или защищать строгими ACL, так же как и зачастую другие управляющие интерфейсы (SNMP/WEB/SSL VPN и даже SSH) сетевых устройств, есть уязвимости и на SNMP и на SMI (port 4786) и на SSL VPN (Fortigate)
IOS XE основан на Linux (подробнее ниже) + IOSd + LXC (Linux Containers) + Apps (Wireshark, Wireless, etc) + Crimson DB (собственная БД Cisco с работой по ряду традиционных и современных автоматизируемых интерфейсов)

Control, management и data plane не разграничены и используют одни ресурсы – как итог высокая утилизация data plane может приводить к проблемам с control/management plane.

Why Should I be concerned about high CPU utilization ? It is very important to protect the control plane for network stability, as resources (CPU, Memory and buffer) are shared by control plane and data plane traffic (sent to CPU for further processing).

 What are the usual symptoms of high CPU usage ? • Control plane instability e.g., OSPF flap • Reduced switching / forwarding performance • Slow response to Telnet / SSH • SNMP poll miss

Почему важно следить за загруженностью по памяти

Why Should I be concerned about high memory utilization?
It is very important have enough free memory to support features and network convergence events that
require transient memory.

What are the usual symptoms of high memory usage ?
• Memory utilization of process(es) keeps increasing
• System runs out of buffers and software packet forwarding stops
• Memory allocation failures are reported
• System crashes after reporting out of memory

Лучше использовать рекомендованное ПО (капитаним). Список рекомендованного ПО на Catalyst можно посмотреть на отдельной странице.
- Evaluated by Cisco for longevity & stability.
- Optimizations, critical fixes & hardening.
Встроенный tcpdump (называют Wireshark) в Catalyst позволяет собирать трафик проще

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2018/pdf/BRKARC-3190.pdf
• Ability to capture both Data and Control plane packets on the switch 
• Live capture for quick diagnoses and storage capabilities 
• Captures can be saved and analyzed on box or saved off box to be open using wireshark application 
• CPU & DRAM resources are limited, captures limited to 1000 pps

Switch#monitor capture CLEUR interface GigabitEthernet 1/0/2 both
Switch#monitor capture CLEUR match ipv4 any any
Switch#monitor capture CLEUR limit duration 60
Switch#monitor capture CLEUR file location flash:cleur.cap
Switch#monitor capture CLEUR start display brief

100MB может работать на 1/10G медных портах, причем в некоторых случаях только при использовании согласования

Can an mGig port work at 100 mbps if end device cannot work at a higher speed? Yes it can. Try using auto-negotiation instead of hard coding speed to 100.

Protocol not in this image – в образе нет нужного протокола (напр. OSPF для IP BASE под платформу Cisco 2960/3750).

S != E
- WS-C3850-24XS-S – IP Base feature set
- WS-C3850-24XS-E – IP Services feature set
S != X (‘S’ series is an older series)
- WS-C2960S-48TD-L (lan base) – C2960S-UNIVERSALK9-M – up to 255 vlans
- WS-C2960X-48TD-L (lan base) – C2960X-UNIVERSALK9-M – 1023 vlans

C2960S(config)#vlan 1054
C2960S(config-vlan)#exit
Proposed configuration exceeds the limit of 255 VLANs that can be supported on this platform. Reduce the number of VLANs proposed to be within this limit.
C2960S(config)#vlan 555
C2960S(config-vlan)#exit
Proposed configuration has too many VLANs for this device's core memory
resources. Either reduce the number of VLANs proposed or use another VTP
device to configure database. (The latter will cause this device to enter
TRANSPARENT mode automatically if propagated to it.)
APPLY VLAN changes failed.
VTP Operating Mode : Off

C2960X(config)#vlan 1024
C2960X(config-vlan)#exit
C2960X(config)#vlan 555
C2960X(config-vlan)#exit

Разные интересности с Cisco 2960X и 10G линками
- Cisco может менять тип интерфейса при установке DAC кабелей – у меня на глазах на Cisco 2960X исключался Te1/0/1 в пользу Gi1/0/25.
  
  На Cisco Te портах может работать только два одинаковых кабеля, при установке двух неодинаковых фиксируются проблемы.

errdisable

Отключение errdisable:
no errdisable detect cause gbic-invalid
no errdisable detect cause link-flap
no errdisable detect cause sfp-config-mismatch

Настройки flap settings errdisable
Cisco(config)#errdisable flap-setting cause link-flap flap-count ? 
<1-100> flap count 
Cisco(config)#errdisable flap-setting cause link-flap flap-count 100 time ? 
<1-1800> flap count time

Вместо уже устаревших Cisco Catalyst предлагает закупать новые Cisco Catalyst 9×00.
- old 2960 – new 9200
  - 2960: доступный 10G (2x10G) – WS-C2960X-24TD-L
  - 2960: доступный 1G (2x1G) -WS-C2960-24TC-L
- old 3850 – new 9300 and 9500 (10G 3850 models)
- old 4500/6500 – new 9400

Лицензии на Catalyst по уровням сервиса:
- WS-C3850-24T-L – Catalyst 3850 Models with LAN Base Software
- WS-C3850-24T-S – Cisco 3850 Models with IP Base Software
- WS-C3850-24T-E – Catalyst 3850 Models with IP Services Software

Catalyst 3850/3650, 9300/9400/9500 Architecture

Catalyst 9600 Architecture

Catalyst 4900

В 4900 – 8 фиксированных портов 10G (X2), 2 слота под half slot модули (до 24х10G)

MULTICAST: IP MROUTE/MFIB/IGMP

CEF/FED/punt L2-L3

FMAN/MATM/FED/UADP

TCAM

• Features that need packet forwarding at line rate program entries in TCAM

• TCAM is partitioned in several banks and regions

• Features use a Hash Table Manager (HTM) to select and configure region

• Entries wrongly programmed in TCAM will lead to wrong or unexpected forwarding decisions

TCAM используется можно сказать во всех задачах коммутатора (routing, multicast, security, qos, pbr; включая NAT, хотя он по факту на 3850 не поддерживается):

Потребление ресурсов TCAM отличается для разных ACL:

- IPv4 PACL, RACL need one VMR/ACE.
- IPv4 VACL/vlmap needs two VMR/ACEs.
- MAC PACL, RACL need two VMR/ACEs.
- IPv6 PACL, RACL need two VMR/ACEs.
- MAC VACL/vlmap needs three VMR/ACEs.

Stacking/Стекирование

И VSS и StackWise – проприетарные технологии Cisco и они обе могут работать на оборудовании серии Catalyst.
VSS (Virtual Switching System) – технология Cisco для стекирования коммутаторов (объединение двух или более физических коммутатора в один логический). Но в отличии от StackWise, для связи коммутаторов между собой используются не какие-то специализированные кабели, а Ethernet порты.
Подробно про стекирование
Стекирование настолько отвратительно, что мало того, что у одного вендора есть разный набор технологий, которые между собой несовместимы, в том числе по hardware (3 rings 3850 vs 1 ring 3650), так еще и не весь hardware ряд одной модели устройств поддерживают один набор стек-технологий

3850 StackWise-480 NOT backward compatible with currently fielded stacking
technologies, most notably StackWise Plus.

Can I connect Stackwise-480 cables on 3850- 48XS? No, 3850-48XS does not support Stackwise-480. It’s a high end model with 640G switching capacity & supports Stackwise Virtual using front panel ports.

RAM (Memory)

Почему может быть нагружена память:

CPU

Catalyst 3850 Series Switch High CPU Usage Troubleshoot

Лишь некоторые интересные причины утилизации и полезные команды

show process cpu sort | ex 0.00 # TOP на 3850 (c IOX XE 16)
show platform software fed switch active punt cause summary # причины обработки на CPU пакетов
show process cpu platform history 1min

QOS

QoS 3850 Into тут, подробнее тут:
- поддерживается настройка разных схем DSCP весов по типу трафика: 8Q3T, 1P7Q3T or 2P6Q3T
- Default Behavior Change
  - 3750, With “mls qos” enabled at global level all the ports are untrusted and DSCP/precedence/COS of the incoming packets are reset to 0. “mls qos trust” is needed at the interface level to change the trust mode
  - 3850, port is trusted by default, DSCP/precedence/COS values are retained
  - Больше фич поддерживается на egress в сравнении с 3750

Linux

Traditional IOSd is a process running on the kernel (linux_iosd)

Поддерживаются linux приложения: top, wireshark, nginx (видел на cisco 3850), netns (видел на cisco 3850)

с3850#show processes cpu platform monitor cycles 20 interval 5 lines 12
top - 16:33:25 up 266 days, 1:49, 0 users, load average: 0.22, 0.17, 0.18
Tasks: 298 total, 1 running, 297 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.0 us, 0.5 sy, 0.0 ni, 97.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 3976872 total, 46904 free, 1617376 used, 2312592 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 1977388 avail Mem

Много sh приложений в образе (подробно ниже), основные процессы IOS XE можно посмотреть тут

psvp.sh
iptbl.sh
pman.sh
bexec.sh

Можно запустить guest shell и в нем в том числе python скрипты

3850#guestshell run python flash:asic_drop_check.py loop 10

Пример полного списка запущенных процессов на Cisco 3850.

с3850#show processes cpu platform monitor
top - 16:33:51 up 266 days, 1:50, 0 users, load average: 0.29, 0.19, 0.19
Tasks: 300 total, 1 running, 299 sleeping, 0 stopped, 0 zombie
%Cpu(s): 2.0 us, 0.5 sy, 0.0 ni, 97.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 3976872 total, 47188 free, 1616784 used, 2312900 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 1977952 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
16716 root 20 0 4776 1472 1100 R 28.6 0.0 0:00.11 top
12403 root 20 0 2128468 672516 226048 S 9.5 16.9 17308:51 linux_iosd+
12751 root 20 0 2060892 333824 48564 S 4.8 8.4 34516:00 fed main e+
1 root 20 0 8864 4492 2604 S 0.0 0.1 12:25.59 systemd
2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
3 root 20 0 0 0 0 S 0.0 0.0 4:08.90 ksoftirqd/0
5 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/0:+
7 root rt 0 0 0 0 S 0.0 0.0 1:36.15 migration/0
8 root 20 0 0 0 0 S 0.0 0.0 0:00.00 rcu_bh
9 root 20 0 0 0 0 S 0.0 0.0 18:06.73 rcu_sched
10 root 20 0 0 0 0 S 0.0 0.0 68:06.13 rcuc/0
11 root 20 0 0 0 0 S 0.0 0.0 1239:46 rcuc/1
12 root rt 0 0 0 0 S 0.0 0.0 1:32.17 migration/1
13 root 20 0 0 0 0 S 0.0 0.0 3:04.51 ksoftirqd/1
15 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/1:+
16 root 20 0 0 0 0 S 0.0 0.0 1321:58 rcuc/2
17 root rt 0 0 0 0 S 0.0 0.0 1:34.50 migration/2
18 root 20 0 0 0 0 S 0.0 0.0 2:42.43 ksoftirqd/2
20 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/2:+
21 root 20 0 0 0 0 S 0.0 0.0 1326:16 rcuc/3
22 root rt 0 0 0 0 S 0.0 0.0 1:34.75 migration/3
23 root 20 0 0 0 0 S 0.0 0.0 2:59.10 ksoftirqd/3
25 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/3:+
26 root 20 0 0 0 0 S 0.0 0.0 1279:57 rcuc/4
27 root rt 0 0 0 0 S 0.0 0.0 1:34.62 migration/4
28 root 20 0 0 0 0 S 0.0 0.0 3:32.57 ksoftirqd/4
30 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/4:+
31 root 20 0 0 0 0 S 0.0 0.0 1272:43 rcuc/5
32 root rt 0 0 0 0 S 0.0 0.0 1:34.52 migration/5
33 root 20 0 0 0 0 S 0.0 0.0 5:42.76 ksoftirqd/5
35 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kworker/5:+
36 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 khelper
37 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kdevtmpfs
38 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 netns
39 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/60-oct+
40 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/89-oct+
41 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/90-oct+
42 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/94-oct+
43 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/91-oct+
44 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/92-oct+
45 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/93-oct+
46 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/95-oct+
47 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/127-oc+
48 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/96-oct+
49 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/98-oct+
50 root -51 0 0 0 0 S 0.0 0.0 0:00.73 irq/99-oct+
51 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 writeback
52 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 bioset
53 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 crypto
54 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 kblockd
55 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 ata_sff
56 root 20 0 0 0 0 S 0.0 0.0 0:00.00 khubd
57 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 md
58 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 edac-poller
59 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 rpciod
60 root 20 0 0 0 0 S 0.0 0.0 4:20.78 kworker/4:1
61 root 20 0 0 0 0 S 0.0 0.0 0:54.11 kworker/0:1
62 root 20 0 0 0 0 S 0.0 0.0 1:44.27 kworker/1:1
63 root 20 0 0 0 0 S 0.0 0.0 1:59.86 kworker/3:1
64 root 20 0 0 0 0 S 0.0 0.0 0:01.16 kswapd0
65 root 25 5 0 0 0 S 0.0 0.0 0:00.00 ksmd
66 root 20 0 0 0 0 S 0.0 0.0 0:31.49 fsnotify_m+
67 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 nfsiod
74 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0
79 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 octeon-eth+
80 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/97-oct+
91 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 deferwq
92 root 20 0 0 0 0 S 0.0 0.0 1:56.64 kworker/2:1
103 root 20 0 0 0 0 S 0.0 0.0 2:04.94 kworker/5:1
104 root 20 0 8692 4232 3924 S 0.0 0.1 0:02.51 systemd-jo+
108 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/5:2
113 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kauditd
124 root 20 0 6048 2044 1500 S 0.0 0.1 0:00.42 systemd-ud+
201 root 20 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
202 root 20 0 0 0 0 S 0.0 0.0 0:52.01 usb-storage
204 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/0:2
224 root 0 -20 0 0 0 S 0.0 0.0 0:00.87 kworker/0:+
225 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/3:2
271 root 0 -20 0 0 0 S 0.0 0.0 0:01.45 kworker/3:+
278 root 0 -20 0 0 0 S 0.0 0.0 0:01.23 kworker/5:+
279 root 0 -20 0 0 0 S 0.0 0.0 0:01.71 kworker/1:+
280 root 0 -20 0 0 0 S 0.0 0.0 0:01.54 kworker/2:+
281 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/2:2
287 root 0 -20 0 0 0 S 0.0 0.0 0:01.28 kworker/4:+
326 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/1:2
365 message+ 20 0 7948 2256 1872 S 0.0 0.1 0:00.16 dbus-daemon
622 root 14 -6 5792 2808 1596 S 0.0 0.1 0:00.26 pman.sh
626 root 20 0 223452 46296 29812 S 0.0 1.2 52:31.72 pubd
662 root 0 -20 0 0 0 S 0.0 0.0 0:00.00 loop0
711 root 0 -20 0 0 0 S 0.0 0.0 0:01.11 loop1
732 root 0 -20 0 0 0 S 0.0 0.0 0:08.03 loop2
794 root 20 0 9284 1988 1432 S 0.0 0.0 0:00.01 rotee
827 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 loop3
848 root 0 -20 0 0 0 S 0.0 0.0 0:00.11 loop4
881 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 loop5
903 root 14 -6 5792 2812 1596 S 0.0 0.1 0:00.26 pman.sh
952 root 0 -20 0 0 0 S 0.0 0.0 0:00.01 loop6
973 root 0 -20 0 0 0 S 0.0 0.0 0:00.02 loop7
1059 root 20 0 46728 13316 6196 S 0.0 0.3 35:33.73 syncfd
1093 root 20 0 9284 1988 1432 S 0.0 0.0 0:00.00 rotee
1194 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.26 pman.sh
1315 root 20 0 44200 10388 6080 S 0.0 0.3 36:11.15 nesd
1350 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
1429 root 14 -6 5792 2812 1596 S 0.0 0.1 0:00.26 pman.sh
1602 root 20 0 9284 1988 1432 S 0.0 0.0 0:00.00 rotee
1661 root 20 0 202324 46440 30608 S 0.0 1.2 37:06.74 ndbmand
1817 root 20 0 71652 16396 9748 S 0.0 0.4 55:21.85 dmiauthd
1864 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.25 pman.sh
1913 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
1995 root 20 0 139624 12016 8480 S 0.0 0.3 4:07.13 nginx
2001 root 20 0 148148 12384 6720 S 0.0 0.3 36:46.52 nginx
3731 root 20 0 5412 2356 1592 S 0.0 0.1 0:00.10 oom.sh
3733 root 20 0 7796 4728 1600 S 0.0 0.1 0:03.74 droputil.sh
3735 root 20 0 7792 4792 1616 S 0.0 0.1 36:43.95 reflector.+
3736 root 20 0 5016 1612 1348 S 0.0 0.0 0:00.01 libvirtd.sh
3751 root 20 0 17792 1860 1508 S 0.0 0.0 0:00.02 virtlogd
3763 rpc 20 0 6712 772 444 S 0.0 0.0 0:23.09 rpcbind
3780 root 20 0 3692 872 704 S 0.0 0.0 0:00.00 xinetd
3784 root 20 0 214828 4212 3400 S 0.0 0.1 0:00.13 libvirtd
3797 root 20 0 5016 736 436 S 0.0 0.0 0:00.00 boothelper+
3798 root 20 0 4160 1308 664 S 0.0 0.0 0:00.00 rpc.statd
3808 root 20 0 3000 660 528 S 0.0 0.0 0:00.00 inotifywait
3839 root 20 0 4516 804 192 S 0.0 0.0 0:00.00 rpc.mountd
3850 root 20 0 0 0 0 S 0.0 0.0 0:00.00 lockd
3851 root 20 0 0 0 0 S 0.0 0.0 0:00.02 nfsd
3852 root 20 0 0 0 0 S 0.0 0.0 0:00.03 nfsd
3853 root 20 0 0 0 0 S 0.0 0.0 0:00.02 nfsd
3854 root 20 0 0 0 0 S 0.0 0.0 0:00.02 nfsd
3855 root 20 0 0 0 0 S 0.0 0.0 0:00.03 nfsd
3856 root 20 0 0 0 0 S 0.0 0.0 0:00.02 nfsd
3857 root 20 0 0 0 0 S 0.0 0.0 0:00.03 nfsd
3858 root 20 0 0 0 0 S 0.0 0.0 0:00.03 nfsd
3933 root 20 0 9160 1888 1352 S 0.0 0.0 0:00.00 rotee
3996 root 20 0 2836 396 304 S 0.0 0.0 0:00.00 sleep
4069 root 20 0 9604 1984 1424 S 0.0 0.0 0:00.08 rotee
4077 root 20 0 9604 1984 1424 S 0.0 0.0 0:00.08 rotee
4119 root 20 0 3000 660 540 S 0.0 0.0 0:00.08 inotifywait
4137 root 20 0 6476 3520 1596 S 0.0 0.1 0:00.27 iptbl.sh
4246 root 20 0 9160 1888 1352 S 0.0 0.0 0:00.00 rotee
4345 root 20 0 3000 660 540 S 0.0 0.0 0:00.00 inotifywait
4470 root 15 -5 0 0 0 S 0.0 0.0 42:12.65 lsmpi-refi+
4471 root 15 -5 0 0 0 S 0.0 0.0 90:00.54 lsmpi-xmit
4472 root 15 -5 0 0 0 S 0.0 0.0 16:07.08 lsmpi-rx
5122 root 20 0 3692 1108 912 S 0.0 0.0 0:00.00 xinetd
6162 root 20 0 3692 1080 904 S 0.0 0.0 0:00.00 xinetd
6167 root 20 0 3692 1192 1012 S 0.0 0.0 0:00.00 xinetd
6182 root 20 0 9400 6396 1600 S 0.0 0.2 0:00.66 rollback_t+
6311 root 20 0 9160 1888 1352 S 0.0 0.0 0:00.00 rotee
6324 root 20 0 3000 664 540 S 0.0 0.0 0:00.00 inotifywait
6391 root 20 0 7884 4956 1628 S 0.0 0.1 0:00.47 psvp.sh
6428 root 20 0 9280 1888 1352 S 0.0 0.0 0:00.00 rotee
6434 root 20 0 3000 664 540 S 0.0 0.0 0:00.00 inotifywait
6439 root 20 0 8072 5084 1636 S 0.0 0.1 0:15.04 pvp.sh
6483 root 20 0 9400 1976 1424 S 0.0 0.0 0:00.08 rotee
6560 root 20 0 3000 664 540 S 0.0 0.0 0:00.00 inotifywait
6667 root 14 -6 5792 2816 1608 S 0.0 0.1 0:00.28 pman.sh
6780 root 20 0 9164 1956 1404 S 0.0 0.0 0:00.01 rotee
6816 root 14 -6 5796 2820 1604 S 0.0 0.1 0:00.26 pman.sh
6992 root 20 0 9164 1956 1404 S 0.0 0.0 0:00.00 rotee
7022 root 20 0 11520 8500 1624 S 0.0 0.2 0:01.09 auto_upgra+
7058 root 14 -6 5796 2816 1604 S 0.0 0.1 0:00.26 pman.sh
7243 root 20 0 9164 1956 1404 S 0.0 0.0 0:00.00 rotee
7283 root 20 0 13208 2288 1676 S 0.0 0.1 0:00.01 auto_upgra+
7309 root 14 -6 5796 2820 1604 S 0.0 0.1 0:00.25 pman.sh
7501 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
7542 root 20 0 64400 14484 10812 S 0.0 0.4 94:19.75 bt_logger
7658 root 14 -6 5796 2816 1604 S 0.0 0.1 0:00.26 pman.sh
7704 root 20 0 9284 1980 1424 S 0.0 0.0 0:00.00 rotee
7776 root 20 0 46396 10308 7020 S 0.0 0.3 55:22.08 btman
7829 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
7957 root 14 -6 5792 2816 1608 S 0.0 0.1 0:00.26 pman.sh
8104 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
8118 root 20 0 51744 13872 9000 S 0.0 0.3 2355:26 hman
8166 root 14 -6 5796 2808 1596 S 0.0 0.1 0:00.25 pman.sh
8326 root 20 0 11384 8416 1624 S 0.0 0.2 0:01.04 issu_stack+
8340 root 20 0 9164 1956 1404 S 0.0 0.0 0:00.00 rotee
8463 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.26 pman.sh
8581 root 20 0 30740 6040 4264 S 0.0 0.2 21:47.01 keyman
8626 root 20 0 9404 1980 1424 S 0.0 0.0 0:00.03 rotee
8729 root 14 -6 5792 2816 1604 S 0.0 0.1 0:00.25 pman.sh
8887 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
8895 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
8968 root 14 -6 5792 2824 1608 S 0.0 0.1 0:00.26 pman.sh
9123 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
9158 root 20 0 58344 12428 8720 S 0.0 0.3 69:51.96 lman
9264 root 14 -6 5792 2812 1596 S 0.0 0.1 0:00.26 pman.sh
9300 root 20 0 11552 8540 1624 S 0.0 0.2 0:01.14 ncd.sh
9405 root 20 0 9164 1956 1404 S 0.0 0.0 0:00.00 rotee
9491 root 14 -6 5792 2824 1608 S 0.0 0.1 0:00.26 pman.sh
9649 root 20 0 9164 1892 1352 S 0.0 0.0 0:00.00 rotee
9650 root 20 0 45788 6468 4856 S 0.0 0.2 0:00.09 nif_mgr
9746 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.25 pman.sh
9760 root 20 0 0 0 0 S 0.0 0.0 0:00.04 kworker/4:0
9843 root 20 0 9956 6912 1612 S 0.0 0.2 123:52.20 periodic.sh
9935 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.01 rotee
10028 root -51 0 0 0 0 S 0.0 0.0 0:00.00 irq/100-oc+
10030 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
10274 root 20 0 9164 1892 1352 S 0.0 0.0 0:00.00 rotee
10387 root 14 -6 5792 2816 1604 S 0.0 0.1 0:00.26 pman.sh
10587 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
10788 root 14 -6 5792 2824 1608 S 0.0 0.1 0:00.26 pman.sh
10934 root 20 0 9616 1988 1424 S 0.0 0.0 0:00.01 rotee
11023 root 14 -6 5792 2816 1604 S 0.0 0.1 0:00.26 pman.sh
11115 root 20 0 5532 2312 1468 S 0.0 0.1 97:18.02 sort_files+
11187 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
11414 root 20 0 56176 12772 9328 S 0.0 0.3 88:19.64 stack_mgr
11544 root 20 0 955512 76992 22512 S 0.0 1.9 2277:11 sif_mgr
11603 root 20 0 7684 4760 1636 S 0.0 0.1 0:02.25 pvp.sh
11795 root 20 0 9280 1888 1352 S 0.0 0.0 0:00.02 rotee
11863 root 20 0 695728 81208 26916 S 0.0 2.0 940:58.00 platform_m+
11885 root 20 0 3000 664 540 S 0.0 0.0 0:00.00 inotifywait
12091 root 14 -6 5784 2812 1604 S 0.0 0.1 0:00.26 pman.sh
12246 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
12271 root 14 -6 5784 2812 1604 S 0.0 0.1 0:00.25 pman.sh
12443 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.01 rotee
12491 root 20 0 45968 9912 6680 S 0.0 0.2 153:07.59 btman
12937 root 14 -6 5792 2816 1604 S 0.0 0.1 0:00.26 pman.sh
13050 root 20 0 9164 1956 1404 S 0.0 0.0 0:00.00 rotee
13104 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.25 pman.sh
13278 root 20 0 9164 1956 1404 S 0.0 0.0 0:00.01 rotee
13287 root 20 0 27296 5788 4016 S 0.0 0.1 21:50.82 tam_svcs_n+
13344 root 14 -6 5792 2816 1604 S 0.0 0.1 0:00.25 pman.sh
13452 root 20 0 9164 1956 1404 S 0.0 0.0 0:00.00 rotee
13457 root 20 0 28288 6584 4500 S 0.0 0.2 47:47.14 tamd_proc
13677 root 20 0 28820 6796 4496 S 0.0 0.2 67:29.40 tams_proc
13749 root 14 -6 5784 2808 1604 S 0.0 0.1 0:00.26 pman.sh
13904 root 20 0 9164 1956 1404 S 0.0 0.0 0:00.00 rotee
14111 root 20 0 38200 7232 4968 S 0.0 0.2 72:49.78 epc_ws_lia+
15312 root 20 0 0 0 0 S 0.0 0.0 0:00.01 kworker/u1+
15895 root 20 0 3008 668 544 S 0.0 0.0 0:00.00 inotifywait
15896 root 20 0 11384 7352 600 S 0.0 0.2 0:00.00 issu_stack+
15901 root 20 0 3008 668 540 S 0.0 0.0 0:00.00 inotifywait
15902 root 20 0 11384 7500 740 S 0.0 0.2 0:00.00 issu_stack+
15911 root 20 0 3008 668 540 S 0.0 0.0 0:00.00 inotifywait
15954 root 20 0 3008 668 540 S 0.0 0.0 0:00.00 inotifywait
16054 root 20 0 3000 660 540 S 0.0 0.0 0:00.07 inotifywait
16505 root 20 0 2844 400 300 S 0.0 0.0 0:00.00 sleep
16600 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u1+
16601 root 20 0 5020 1808 1452 S 0.0 0.0 0:00.01 bexec.sh
16603 root 20 0 5020 1812 1460 S 0.0 0.0 0:00.04 mcp_monito+
16710 root 20 0 2844 396 296 S 0.0 0.0 0:00.00 sleep
16712 root 20 0 2844 400 300 S 0.0 0.0 0:00.00 sleep
16713 root 20 0 5020 1808 1452 S 0.0 0.0 0:00.01 bexec.sh
16715 root 20 0 5020 1648 1332 S 0.0 0.0 0:00.04 mcp_monito+
17388 root 14 -6 5792 2812 1604 S 0.0 0.1 0:00.26 pman.sh
17567 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
17692 root 20 0 5020 1808 1472 S 0.0 0.0 0:00.00 stack_sntp+
17710 root 20 0 4656 840 700 S 0.0 0.0 0:00.00 sntp
18415 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.28 pman.sh
18491 root 20 0 390904 161716 40796 S 0.0 4.1 150:17.65 fman_fp_im+
18602 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.01 rotee
18738 root 14 -6 5792 2816 1604 S 0.0 0.1 0:00.26 pman.sh
18912 root 20 0 9284 1980 1424 S 0.0 0.0 0:00.00 rotee
18914 root 20 0 38952 8252 5192 S 0.0 0.2 80:53.58 plogd
18937 root 20 0 9972 3572 3308 S 0.0 0.1 0:00.40 journalctl
19082 root 20 0 54368 13764 10444 S 0.0 0.3 73:16.30 psd
19086 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.27 pman.sh
19196 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
19349 root 14 -6 5792 2812 1596 S 0.0 0.1 0:00.26 pman.sh
19463 root 20 0 9284 1980 1424 S 0.0 0.0 0:00.01 rotee
19477 root 20 0 147328 26920 18708 S 0.0 0.7 262:57.16 repm
19599 root 14 -6 5792 2816 1604 S 0.0 0.1 0:00.25 pman.sh
19652 root 20 0 3008 672 540 S 0.0 0.0 0:00.00 inotifywait
19713 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.01 rotee
19918 root 20 0 232460 72288 12684 S 0.0 1.8 209:18.61 smand
20047 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.26 pman.sh
20150 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.01 rotee
20357 root 20 0 52136 15308 6704 S 0.0 0.4 73:02.04 tms
20446 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.27 pman.sh
20550 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.01 rotee
20633 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.26 pman.sh
20767 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
20801 root 20 0 772100 48952 29892 S 0.0 1.2 74:55.19 cli_agent
20867 root 20 0 335200 70756 47740 S 0.0 1.8 74:06.86 sessmgrd
20904 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.28 pman.sh
21080 root 20 0 41508 7580 5232 S 0.0 0.2 74:25.64 cmm
21150 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
21285 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.26 pman.sh
21380 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.01 rotee
21396 root 20 0 252204 56468 40048 S 0.0 1.4 92:50.84 dbm
21934 root 20 0 333060 79340 65496 S 0.0 2.0 109:32.64 fman_rp
27396 root 14 -6 5792 2816 1604 S 0.0 0.1 0:00.27 pman.sh
27510 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
27593 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.24 pman.sh
27668 root 20 0 37512 6920 4800 S 0.0 0.2 55:01.45 ncsshd_bp
27688 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
27770 root 20 0 32504 6932 5416 S 0.0 0.2 0:00.36 ncsshd
28341 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u1+
28602 root 14 -6 5792 2816 1600 S 0.0 0.1 0:00.24 pman.sh
28650 root 20 0 9284 1988 1432 S 0.0 0.0 0:00.00 rotee
28729 root 20 0 6368 3356 1624 S 0.0 0.1 0:00.24 confd-star+
28783 root 20 0 9668 1892 1352 S 0.0 0.0 0:00.00 rotee
28798 root 20 0 813624 138652 3608 S 0.0 3.5 119:54.30 confd
28889 root 20 0 4124 440 312 S 0.0 0.0 0:00.00 inet_getho+
28890 root 20 0 4548 712 540 S 0.0 0.0 0:00.00 inet_getho+
32232 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.27 pman.sh
32348 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
32413 root 14 -6 5792 2820 1604 S 0.0 0.1 0:00.27 pman.sh
32581 root 20 0 9284 1892 1352 S 0.0 0.0 0:00.00 rotee
32583 root 20 0 22096 4972 3540 S 0.0 0.1 16:35.65 pttcd

TCL scripting

EEM

Config:
event manager applet CLEUR
description Test Applet
event neighbor-discovery interface GigabitEthernet1/0/46 cdp delete
action 1.0 syslog msg "CDP neighbor lost on Interface"

*Jan 30 09:31:45.211: %HA_EM-6-LOG: CLEUR: CDP neighbor lost on Interface

DPI/NBAR

TROUBLESHOOTING

Вверху две презы Cisco Live
Интересная тулза – Cisco CLI analyzer (Smart CLI) с фичами System Disagnostics, Crashinfo analyzer

Inventory Info

#show module

#show inventory

#show idprom interface tenGigabitEthernet 3/3

ONEX Serial EEPROM Contents:
Non-Volatile Register (NVR) Fields
ONEX Version :0x1E = MSA Version 0x1E
NVR Size in bytes :0x100
Number of bytes used :0x100
Basic Field Address :0xB
Customer Field Address :0x77
Vendor Field Address :0xA7
Extended Vendor Field Address :0x100
Reserved :0x0
Transceiver type :0xF1 =ONEX
Optical connector type :0x0 =Unspecified
Bit encoding :0x1 =NRZ
Normal BitRate in multiple of 1M b/s :0x2848
Protocol Type :0x1 =10GbE
Standards Compliance Codes :
10GbE Code Byte 0 :0x0 =Unspecified
10GbE Code Byte 1 :0x0
SONET/SDH Code Byte 0 :0x0
SONET/SDH Code Byte 1 :0x0
SONET/SDH Code Byte 2 :0x0
SONET/SDH Code Byte 3 :0x0
10GFC Code Byte 0 :0x0
10GFC Code Byte 1 :0x0
10GFC Code Byte 2 :0x0
10GFC Code Byte 3 :0x0
Transmission range in 10m :0x0
Fibre Type :
Fibre Type Byte 0 :0x0 =Unspecified
Fibre Type Byte 1 :0x0 =Unspecified
Centre Optical Wavelength in 0.01nm steps - Channel 0 :0x0 0x0 0x0
Centre Optical Wavelength in 0.01nm steps - Channel 1 :0x0 0x0 0x0
Centre Optical Wavelength in 0.01nm steps - Channel 2 :0x0 0x0 0x0
Centre Optical Wavelength in 0.01nm steps - Channel 3 :0x0 0x0 0x0
Package Identifier OUI :0x78787878
Transceiver Vendor OUI :0x78787878
Transceiver vendor name :CISCO-FOXCONN
Part number provided by transceiver vendor :U58H135.00
Revision level of part number provided by vendor :00
Vendor serial number :FOX1731XZHJ
Vendor manufacturing date code :2013080300
Current Reference :
5V Stressed Environment Reference 100 percent=1A :0x4
3.3V Stressed Environment Reference 100 percent=2A :0x1
APS Stressed Environment Reference 100 percent=2A :0x0
Normal APS Voltage :0x0
Digital Optical Monitoring Capability Byte :0xC1
Digital Optical Monitoring :Implemented
Low Power Start-up(LPS) Mode Capability Bit :0x0
Reserved :0x0
Basic Field Checksum :0x2B
Customer Writable Area :
0x00: 43 56 52 2D 58 32 2D 53 46 50 31 30 47 20 20 20
0x10: 00 56 30 33 20 46 4F 58 31 37 33 31 58 5A 48 4A
0x20: 37 34 2D 36 33 32 31 2D 30 33 20 20 41 30 20 20
Vendor Specific :
0x00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x20: 00 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0x30: 00 00 08 00 07 4E 7D D8 FD 62 EB E3 F5 C8 4B 5A
0x40: F9 1B 1E E6 C5 00 00 00 00 00 00 00 00 00 E1 A1
0x50: 30 F1 00 00 00 00 00 00 00
SFP+ Serial EEPROM Contents:
Common Block:
Identifier = SFP+ [0x3]
Connector = Copper Pigtail [0x21]
Transceiver
Type = SFP+ H10GB-CX1 Passive CU3M
Vendor name = Amphenol
Vendor OUI = 7907092
Vendor Part No. = NDCCDA-I202
Vendor Part Rev. = B
Vendor Serial No. = APF192920258JV
Diag monitoring = Not implemented
SEEPROM contents (hex) size 256:
0x0000 03 04 21 00 00 00 00 00 04 00 00 00 67 00 00 00 ..!.........g...
0x0010 00 00 03 00 41 6D 70 68 65 6E 6F 6C 20 20 20 20 ....Amphenol
0x0020 20 20 20 20 00 78 A7 14 4E 44 43 43 44 41 2D 49 .x'.NDCCDA-I
0x0030 32 30 32 20 20 20 20 20 42 20 20 20 01 00 00 E7 202 B ....
0x0040 00 00 00 00 41 50 46 31 39 32 39 32 30 32 35 38 ....APF192920258
0x0050 4A 56 20 20 31 39 30 37 32 34 20 20 00 00 00 04 JV 190724 ....
0x0060 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................
0x0070 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................
0x0080 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................
0x0090 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................
0x00A0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................
0x00B0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................
0x00C0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................
0x00D0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF ................
0x00E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................
0x00F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................

buffers, Bursts

DEBUG

Queue, drops

C3850#sh platform hardware fed switch 1 qos queue stats interface te1/0/4
DATA Port:4 Enqueue Counters
---------------------------------------------------------------------------------------------
Q Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2 Qpolicer
(Count) (Bytes) (Bytes) (Bytes) (Bytes)
- ------- -------------------- -------------------- -------------------- --------------------
0 6762 0 12949390972499 0 0
1 0 0 0 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
6 0 0 0 0 0
7 0 0 0 0 0
DATA Port:4 Drop Counters
-------------------------------------------------------------------------------------------------------------------------------
Q Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop QpolicerDrop
(Bytes) (Bytes) (Bytes) (Bytes) (Bytes) (Bytes)
- -------------------- -------------------- -------------------- -------------------- -------------------- --------------------
0 0 1147818248 0 0 0 0
1 0 0 0 0 0 0
2 0 0 0 0 0 0
3 0 0 0 0 0 0
4 0 0 0 0 0 0
5 0 0 0 0 0 0
6 0 0 0 0 0 0
7 0 0 0 0 0 0