PostCheck will also reformat an address to conform to the United States Post Office standards. Full Specifications. What's new in version 2.5. New Gatekeeper-compatible installer app, rather than. Posted 4/6/03 8:51 AM, 9 messages. The (expression) after the while won't be evaluated until all the code inside the curly braces has been executed. That means this do while loop is a post-check example. A pre-check looks like.
Introduction
This document describes the steps to troubleshoot Application Centric Infrastructure (ACI) upgrade issues and best practices to be followed before and during the upgrade process.
ACI upgrade involves update of Application Policy Infrastructure Controller (APIC) software and switches(leaf and spine) update. Switch upgrade is usually very straight forward, however APIC upgrade may involve some cluster issues. Here are a few pre-checks Cisco recommends to prepare before an upgrade is started.
Before Upgrade
Before you start the ACI upgrade, make sure to perform some pre-checks to avoid any unexpected behaviours.
Things to Do Before APIC Upgrade
- Clear All the Faults
Many faults in ACI fabric state that there are invalid or conflict policies or even disconnected interfaces etc. Please understand the trigger and clear them before you start the upgrade. Be aware, the faults such as encap already been used or Routed port is in L2 mode could result in unexpected outage. When you upgrade the switch, it would download all the policies from APIC from scratch. As a result, the unexpected policies may take over the expected polices which could cause an outage. - Clear VLAN Pool Overlap
VLAN pool overlap means the same VLAN ID is part of two or more VLAN pools. If the same VLAN ID is deployed on multiple leaf switches which is part of different VLAN pools, it would have a different VXLAN ID on these switches. Since ACI uses the VXLAN ID for forwarding, traffic destined to a particular VLAN may end up in different VLAN or get dropped. Since the leaf downloads the configuration from APIC after its upgrade, the order in which VLAN gets deployed has a major role. So, this could result in an outage or intermittent connectivity loss to endpoints in some VLANs.
So, it is important to check for VLAN ID overlap and correct them before you start the upgrade. It is recommended to have one VLAN ID be part of one VLAN pool only and reuse the VLAN pool where needed. - Confirm Supported Upgrade Path
The APIC upgrade involves the data conversion from one version to other which is done internally. For data conversion to succeed, there are some version compatibility that needs to be taken care of. So, always make sure to check if Cisco supports the direct upgrade from your current ACI version to the new target version you are upgrading to. Sometimes, you would have to go through multiple hops to reach the target version. If you upgrade to a non-supported version, it could result in cluster issues and configuration issues.
The supported upgrade paths are always listed on Cisco ACI Upgrade Guide - Backup APIC Configuration
Make sure to export a configuration back up to a remote server before you start the upgrade. This exported back up file can be used to get the configuration back on APICs if you have to lose all configuration or a data corruption after the upgrade.Note: If you enable encryption for the backup, make sure to save the encryption key. Otherwise, all the user account passwords including the admin password wouldn't be imported properly.
- Confirm APIC CIMC Access
Cisco Integrated Management Controller (CIMC) is the best way to get the remote console access to the APIC. If the APIC doesn't come back up after a reboot or the processes are stuck, you may not be able to connect to the APIC through out of band or in band management of the APIC. At this stage, you can login to CIMC and connect to the KVM console for the APIC to perform some checks and troubleshoot the issue. - Check and Confirm the CIMC Version Compatibility
Always make sure to run the Cisco recommended CIMC version compatible with the target ACI version, before you start the ACI upgrade. Refer to Recommended APIC and CIMC Version - Confirm APIC Process is not Locked
The process called Appliance Element(AE) which runs in the APIC is responsible to trigger the upgrade in the APIC. There is a known bug in CentOS Intelligent Platform Management Interface (IPMI) which could lock the AE process in APIC. If AE process is locked, the APIC firmware upgrade will not kick in. This process queries the chassis IPMI every 10 seconds. If the AE process has not queried the chassis IPMI in the last 10 seconds, that could mean the AE process is locked.
You can check the status of AE process to know the last IPMI query. From the APIC CLI, run the command date to check the current system time. Now run the command grep 'ipmi' /var/log/dme/log/svc_ifc_ae.bin.log | tail -5 and check the last time when the AE process has queried the IPMI. Compare the time against the system time to check if the last query was within the 10 second window of the system time.
If the AE process has failed to query the IPMI in the last 10 seconds of the system time, you can reboot the APIC to recover the AE process before starting the upgrade.Note: Please do not reboot two or more APICs at the same time to avoid any cluster issues
- Check and Confirm the NTP Availability
From each APIC, ping and confirm the reachability to the NTP server to avoid known issues due to APIC time mismatch. More details on this can be found in the troubleshooting section of this article. - Check APIC Health State
Check and confirm the health status of the APIC in the cluster before you start the upgrade. The health score of 255 means the APIC is healthy. Run the commandacidiag avread | grep id= | cut -d ' ' -f 9,10,20,26,46 from any APIC CLI, to check the APIC health status. If the health score is not 255 for any APIC, don't start the upgrade.
- Evaluate the Impact of New Version
Before you start the upgrade, please review the Release Notes for your target ACI version and understand any behavioural changes that are applicable to your fabric configuration to avoid any unexpected results after the upgrade. - Stage the Upgrade in Lab
Cisco recommends to try the upgrade in a lab or test fabric before the actual production fabric to familiarise yourself with the upgrade and behaviours in the new version. This also helps to evaluate any possible issues you could run in to after the upgrade.
Things to Do Before Switch Upgrade
- Place Virtual Port Channel (vPC) and Redundant Leaf Pairs in Different Maintenance Groups
ACI APIC has a mechanism to check and defer the upgrade of vPC pair leaf nodes from a certain version and beyond. However, it is best practice to put vPC pair switches in different maintenance groups to avoid both the vPC switches reboot at the same time.
In case of non vPC switches which are redundant, like border leaf, make sure to put them in different port groups to avoid any outages.
Troubleshoot Upgrade Issue
Always start to troubleshoot APIC1 if the upgrade gets stuck or fails. If APIC1 upgrade is not finished yet, don't do anything in APIC2 and APIC3. The APIC upgrade process is incremental and hence APIC2 will upgrade only after APIC1 completes upgrade and notifies APIC2 about it and so on. So, violating this rule may put the cluster in to a broken state with corrupt database and you may require to re-build the cluster.
Scenario : APIC ID 2 or Above Stuck at 75%
In this scenario, you wold see that APIC1 is upgraded successfully, but APIC2 is still stuck at 75%. This problem happens if the APIC1 upgrade version information is not propagated to APIC2 or above. Please be aware, svc_ifc_appliance_director process is in charge of the version sync between APICs.
How to Troubleshoot
Step 1: Make sure APIC1 could ping rest of the APICs with their Tunnel End Point (TEP) IP, this will determine whether we need to troubleshoot from leaf switch or continue from APIC itself. If APIC1 cannot ping APIC2, you may want to call TAC to troubleshoot the switch. If APIC1 could ping APIC2, then move to second step.
Post Checkpoint Codes
Step 2: Since APICs can ping each other, which means APIC1 version information should have been replicated to peer, but somehow was not accepted by the peer. The version information is identified by a version timestamp. We can confirm the version timestamp of APIC1 from the CLI and APIC2 CLI which is waiting at 75%.
On APIC1
apic1# acidiag avread | grep id=1 | cut -d ' ' -f20-21
version=2.0(2f) lm(t):1(2018-07-25T18:01:04.907+11:00)
On APIC2
apic2# acidiag avread | grep id=1 | cut -d ' ' -f20-21
version=2.0(1m) lm(t):1(2018-07-25T18:20:04.907+11:00)
As you see, version timestamp of APIC2(18:20:04) which is running version 2.0(1m) in this example is higher than the version timestamp of APIC1(18:01:04) that is running version 2.0(2f). So, the APIC2 installer process thinks the APIC1 upgrade is not complete yet and waits at 75%. APIC2 upgrade will kick off when the version timestamp of APIC1 goes above the version timestamp of APIC2. However, this could be lot of waiting based on how much is the time difference. To recover the fabric from this state, you can open a TAC case to get assistance to troubleshoot and fix the issue from APIC1.
Dinner delivery only
We have announced the website to serve you better. If you have any problems or feedback please contact us at: contact@alfornoeastcoast.com.sg or call at 91886372.
Thank you and have a pleasant day.
Delivery is available ONLY FOR DINNER FROM 6.30 to 10 (Mon. to Friday) & 5.30 to 10 (SAT & Sun).
As our food is prepared from scratch, please allow us 45 minutes to maximum of 150 minutes from the time of your order during operational hours. Any order delivered above 150 minutes will be free of charge.
Please note, delivery fee of S$3 will be added for orders below S$40.
Post Checklist
Please view the delivery map below.
No refund will be given.