Performing a DNS Migration with Reduced Risk

Updated May 26, 2024
Created March 24, 2022

Read through this post fully before getting started. Plan 2 days minimum to complete all the steps with the least amount of risk and stress.

DNS Migrations can be stressful for mulitple services and although careful planning and preparation can mitigate mosts risks, issues do occur and should be planned for.

Pre-Migration Testing

This is to reduce the errors in the DNS migration. Test as much as possible locally.

Tools

Steps

  1. Setup the new production server. Note any routes or APIs to be tested.

  2. Get the IP address of the new server. This can be achieved with ping.

    ping example.com

  3. Optionally, create a backup of your existing hosts file.

    sudo cp /etc/hosts /etc/hosts.backup-$(date '+%Y-%m-%d-%Hh%Mm')

  4. Append to your hosts file using either Switchhosts or manually

  5. Flush local DNS cache.

    sudo killall -HUP mDNSResponder && echo "done"

  6. Test the site with your browser. If this doesnt work as expected continue with the next step to debug.

  7. Test API(s) with hoppscotch.io / postman (Start building up a collection of the endpoints for testing if you havent already). Debugging may need to a host header added to the request.

  8. Check the SSL certificate being issued covers the expected domain. ie if our final domain is example.com, the SSL certificate needs to specify that (not a default AWS domain)

    # From Stackoverflow https://stackoverflow.com/a/34812039
curl --insecure -vvI https://www.example.com 2>&1 | awk 'BEGIN { cert=0 } /^\* SSL connection/ { cert=1 } /^\*/ { if (cert) print }'

Pre-Day Planning

  1. Look at analytics and advise the client when the best time would be. Plan to experience downtime and you may need to inform users. Prefer early mornings or early in the week. This is a gateway to the clients system and regardless of planning, changes will carry risk.
  2. Plan a brief session with the product owner, ideally two technical members who will be performing the DNS migration, and invite the client/stakeholder for transparency.
  3. Get access to the production DNS host or inform the client that a team member with access will need to attend the briefing call.
  4. Reduce the DNS TTL for minimal downtime.
  5. Schedule 3 sessions with the client for the migration day. At least 30mins each, and at least 2hrs apart. The sessions will be for rollout (±5mins) & testing (±10mins - adjust as needed), and if any issues, rollback (±5mins). The planned gaps are for debugging and making appropriate changes to code and systems before the next attempt.
  6. Discuss and agree with the client the bug triaging process. See this triaging table as a starting point.

Rollout Process

Tools

Steps

  1. Ensure all host files have been restored from prior testing.

  2. Flush local DNS cache.

    sudo killall -HUP mDNSResponder && echo "done"

  3. Talk through the steps and the plan. Invite all relevant parties and stakeholders, but set the precedent its for transparency and that not to add pressure. Acknowledge any previously failed attempts, the fixes that have been implemented, and the next steps.

  4. Create a backup of the values you plan to change and store then in a shared location. Consider doing an export via the existing provider. Be aware nameserver transfers will require more records to be moved.

  5. Update the DNS values in the DNS hosting provider.

  6. Flush Public Cache.

  7. Open the page via VPN in an Opera incognito window.

  8. Ensure there are no unexpected SSL issues.

  9. Test the routes and APIs previously noted. Run any Hoppscotch collections, e2e test suites, or a smoke tests.

  10. If there are any issues note them down. Proceed to gather as much information as possible. Aim to test the whole system - consider other paths around the failiures to see if there are other areas overlooked. ie homepage may not work but API's do.

  11. Assess if the errors experienced are critical, as per triage table.

  12. Move to Rollback Process if needed.

  13. Use dnschecker to check if propogation has finished.

  14. Write out and send a report with the results, steps followed, errors experienced, and next steps.

  15. If successful, increase the DNS TTL.

Rollback Process

Tools

As per rollout process.

Steps

  1. Most importantly; if there are any issues note them down. Proceed to gather as much information as possible. Aim to test the whole system - consider other paths around the failiures to see if there are other areas overlooked. ie homepage may not work but API's do.
  2. Update the DNS values in the DNS hosting provider to the original values.
  3. Flush Public Cache.
  4. Open the page via VPN in an Opera incognito window.
  5. Ensure there are no unexpected SSL issues.
  6. Test the routes and APIs previously noted. Run any Hoppscotch collections, e2e test suites, or a smoke tests.
  7. Flush Public Cache.
  8. Use dnschecker to check if propogation has finished.
  9. Write out and send a report with the results, steps followed, errors experienced, and next steps.

Further Info

Further Tools

Comments?