Open Heart Surgery – Rules for Working on Live Production Systems

Dimitris Kokoutsidis 15 mins0

Chris Moyer, dotFMP Berlin 2021

In this blog post, we explore Chris Moyer’s insightful talk on working directly on live production systems, humorously dubbed “Open Heart Surgery.” Chris shares his framework and decision-making process for safely performing live updates, highlighting when and how live modifications can be a practical necessity in FileMaker development.

This guide delves into key considerations, best practices, and mitigation techniques for reducing risk and disruption while working on live systems. Whether you’re managing a small project or a multi-user environment, this post will help you make informed decisions when navigating the intricacies of live system updates.

Table of Contents

Introduction

To top

During the dotFMP Berlin conference in October 2021, Chris Moyer shared an in-depth session on best practices for working directly on live production systems in FileMaker. He humorously dubbed this type of development “open heart surgery,” underscoring the critical nature of making live changes without disrupting active users. His talk explored the rationale behind live development, when it’s appropriate, and detailed strategies for minimizing risk while maximizing efficiency.

Reasons for Live Development

To top

Moyer identifies several scenarios where working on live systems can be essential:

  • Immediate Problem Resolution: Critical issues in production can demand immediate intervention. In such cases, working live is unavoidable.
  • Client Expectations for Rapid Updates: Clients often prefer fast iterations, particularly for smaller feature adjustments.
  • Real-Time Data Validation: Working with live data allows developers to validate changes in real-world scenarios.
  • Developer Productivity: Developing live can sometimes be faster due to reduced need for testing environments and deployment processes.

Assessing Risks Before Making Live Changes

To top

Before making any live changes, it’s crucial to evaluate potential risks and impacts on:

  • System Stability: Could this change destabilize core functionality?
  • User Experience: How many users might be disrupted? Is it possible to limit impact during non-peak hours?
  • Data Integrity: Could the change impact existing data relationships or constraints?

Performing live changes isn’t advisable if the potential disruption outweighs the benefits. A robust decision-making process is key to safely navigating these challenges.

Common Scenarios Where Live Changes Are Required

To top

Some scenarios necessitate live changes, including:

  • Emergency Fixes: In situations where the system’s stability is compromised, immediate fixes are often required.
  • Time-Sensitive Feature Additions: For instance, an executive request during a board meeting may require immediate updates, like adding a reporting capability on the fly.
  • Iterative Development Based on Client Feedback: Frequently, clients refine their requirements over time. In these cases, developing directly on live systems allows for more effective real-time iterations.
  • High-Cognitive Load Projects: For developers managing multiple concurrent projects, working live reduces the overhead associated with deploying changes across multiple environments.

Mitigation Techniques for Live Changes

To top

To minimize risks while working on live systems, Moyer suggests a suite of mitigation strategies:

  • Use Ad-Hoc Backups: Always create a backup before making major updates. This is especially vital for schema or data integrity-affecting changes.
  • Testing on Layout Duplicates: For UI changes, duplicate layouts to test modifications before going live. This prevents users from experiencing intermediate changes.
  • Prefer LAN over WAN: Remote work over WAN can be unstable, especially for critical changes. Remote into a local client on the same LAN as the FileMaker Server when possible.
  • Avoid Using FileMaker Client on Windows Servers: FileMaker recommends against using the client on Windows Servers due to known instability issues that can lead to server crashes.

Step-by-Step: The Live Change Decision Flowchart

To top

Here is Moyer’s decision flow for assessing whether a live change can proceed:

  1. Is it a Schema Change?
    • If No: Proceed with live changes (details on value lists, scripts, and layouts below).
    • If Yes: Determine if the table or relationships are actively used.
  2. If Table is Active: Assess if it’s feasible to temporarily restrict access. If yes, then proceed with modifications. If no, the change should be deferred until after hours when the system can be taken offline.
  3. Can Users be Temporarily Blocked? This may involve temporarily hiding navigation options or restricting access to certain areas of the system.
  4. Offline Preparation: For complex changes, prepare on a duplicate or offline environment. Once tested, use minimal downtime to implement changes by quickly copying and pasting.

This flowchart ensures minimal disruption and considers the need for subsequent data cleanup to restore consistency.

Handling Non-Schema Changes: Value Lists, Scripts, and Layouts

To top

  • Value Lists:
    • For Simple Additions, live changes are generally safe.
    • For Modifications, check dependencies to ensure no scripts or reports rely on specific values.
    • ID-Based Value Lists: If value lists use IDs, the display name can change without impacting underlying logic, providing flexibility.
  • Scripts:
    • Adding New Scripts: Safe to add directly.
    • Modifying Existing Scripts: Duplicate scripts first and use conditions, such as Get(ActiveModifierKeys), to test changes live with minimal disruption.
    • Script Dependencies: Be cautious of dependencies within loops or scripts that call other scripts. Ensure that updates won’t interrupt users mid-process.
  • Layouts:
    • New Layouts: Adding new layouts typically has minimal impact.
    • Modifying Existing Layouts: Always duplicate layouts and make changes offline first. Ensure consistent element placement to avoid user confusion.
    • Field Management: Removing unused fields is safe if they aren’t actively referenced. However, caution is needed if the field plays a role in calculations or data relationships.

Best Practices for Schema Changes

To top

Schema changes are among the riskiest updates. When they are required, adhere to the following:

  • Backup Before Changes: Always secure a backup, as schema changes can affect multiple tables.
  • Use Shadow Tables for Major Updates: When possible, add a shadow table with the updated schema. This minimizes the need to modify the core schema live.
  • Coordinate with Users: If the change is critical, notify users and schedule a downtime window.
  • Limit Schema Changes During Peak Hours: If a schema change is necessary, attempt to implement during low-usage hours to minimize impact.

Managing Legacy Data Bulk Updates

To top

When performing bulk data updates:

  • Review Modification Timestamps: Many compliance standards like HIPAA and ISO require logging modification timestamps. Understand how bulk updates impact these logs.
  • Conditional Timestamping: If timestamp suppression is used, understand the implications for data audit trails. In highly regulated environments, you may need to run updates without suppressing timestamps.
  • Use Server-Side Scripts: Bulk updates should ideally be run as server-side scripts to reduce data exposure and ensure consistency across records.
  • External Logging for Compliance: If compliance is a concern, consider maintaining an external log of changes to supplement or verify FileMaker’s internal timestamps.

Working with Layout Modifications

To top

Modifying layouts while maintaining a consistent user experience requires careful attention:

  • Use Layout Duplicates for Testing: This prevents users from encountering incomplete modifications.
  • Consider Layout Dependencies: Especially when dealing with portals or linked data, ensure that all related objects are compatible with the new layout structure.
  • Preserve User Navigation Familiarity: Avoid moving fields or buttons significantly, as this can disrupt user workflows and increase cognitive load.

Technical Tips for Reducing Risk in Live Changes

To top

In addition to the strategies above, Moyer discussed these specific techniques:

  • Serial Number Impact: Changing serial numbers or auto-entered fields can lock records, impacting data entry across multiple tables. Always validate serial numbers post-update.
  • Field Definitions: Changes to field definitions can disrupt entire tables. When necessary, apply changes first to non-critical fields to test impact.
  • Script Triggers: Review script triggers associated with layouts or fields being modified. Ensure that changes don’t inadvertently trigger scripts in production.

Advanced Techniques for Field Definitions and Record Creation

To top

If changes to field definitions or record creation are necessary:

  • Duplicate Record Scripts: Ensure your new record or duplicate record scripts contain error checks. Catching errors early prevents issues like empty serial numbers from propagating across related records.
  • UUIDs over Serial Numbers: When possible, use UUIDs for record creation. This reduces the likelihood of duplication or lock errors, particularly in networked environments.
  • Handling Weak Connections: Weak connections can lead to serial number issues. In environments with mobile or remote users, UUIDs help ensure that records are unique and data integrity is maintained.

Concluding Insights

To top

Working on live systems demands a structured approach and an acute awareness of potential risks. By following a decision-making flowchart, implementing risk mitigation techniques, and adhering to best practices, FileMaker developers can safely and effectively manage live changes. While live development offers speed and efficiency, the need to protect data integrity and user experience remains paramount.

Through careful preparation and adherence to these principles, developers can navigate the complexities of live systems and perform “open heart surgery” with confidence.

https://dotfmp.berlin/sharedfiles/328/download

Dimitris Kokoutsidis

Related Articles