Emergency Procedures for FileMaker Server Crashes

Jacob Taylor, April 21, 2021, FileMaker Training Videos

Introduction

FileMaker Server crashes are inevitable in high-demand environments. A quick and structured response is key to minimizing downtime, safeguarding data, and ensuring that such incidents don’t repeat. This guide provides a deep dive into diagnosing the cause, executing a proper recovery, preventing future crashes, and completing essential post-recovery tasks. Following these procedures will help maintain server performance and avoid data loss during crashes.

Menu

1. Diagnosis
   • 1.1 Thorough Log Analysis
     • FileMaker Server Logs
     • Common Error Codes
     • System Logs (OS-Level)
   • 1.2 Resource Usage Monitoring
     • CPU Monitoring
     • Memory Monitoring
     • Disk I/O Monitoring
   • 1.3 Network and Hardware Issues
     • Network Latency
     • RAID and Storage Health
     • External Devices
   • 1.4 Disable and Test Plugins
     • Sequential Plugin Re-enabling
     • Plugin Compatibility
   • 1.5 Test Database Integrity
     • File Verification
     • Consistency Check at Startup
2. Recovery
   • 2.1 Graceful Server Shutdown and Restart
     • Command-Line Restart
   • 2.2 Database File Integrity Check
     • Check for Auto-recovery
     • Manually Open Files
   • 2.3 File Restoration from Backup
     • Incremental Restore
   • 2.4 Gradual User Reconnections
   • 2.5 Communicate with Stakeholders
3. Prevention
   • 3.1 Server Health Monitoring
     • Automated Monitoring
     • Use Performance Metrics
   • 3.2 Optimize Scripts and Workflows
     • Script Logging
     • Transaction Management
   • 3.3 Regular Backup Scheduling
     • Automated Incremental Backups
     • Test Backup Restores
   • 3.4 Apply Regular Software and OS Patches
   • 3.5 Hardware and Network Maintenance
     • RAID Testing
     • Network Redundancy
4. Post-Recovery
   • 4.1 Review the Crash Incident
     • Document the Timeline
     • Identify Key Failures
   • 4.2 Refine Crash Recovery Procedures
   • 4.3 Schedule Future Maintenance
     • Server Stress Tests
     • Database Maintenance
   • 4.4 Plan for Long-Term Upgrades
     • Hardware Upgrades
     • Software Upgrades
   • 4.5 Continue Server Testing
     • Simulated High-Traffic Tests
     • Routine Performance Benchmarks
     • Monitor User Feedback
5. Check List
6. Videos

1. Diagnosis

To top

The first step to resolving a FileMaker Server crash is identifying its cause. Without a proper diagnosis, recovery efforts may fail, and future crashes are inevitable. Here’s an expanded breakdown of critical diagnostic steps:

1.1 Thorough Log Analysis

Start by thoroughly reviewing all log files. Pay attention to patterns that could lead to future issues.

• FileMaker Server Logs:
  • Event Log: Provides insights into server crashes, shutdowns, restarts, and critical failures. Look for patterns in error codes and times when crashes occur.
  • Access Log: Investigate whether multiple connection failures or unauthorized access attempts preceded the crash. Sudden disconnections of clients can indicate network or resource overload.
  • TopCallStats Log: Offers detailed information on long-running scripts and intensive database operations. Excessive processing time can result in server overloads.
  Common Error Codes:
  • 701: Disk I/O failure—could indicate disk corruption, RAID failure, or insufficient disk space.
  • 802: File open failure—often points to file corruption or access permissions issue.
  • 1409: Memory allocation error—this suggests that the server ran out of memory, often due to a memory leak or improperly optimized scripts.
• System Logs (OS-Level):
  • Check the system’s Event Viewer (Windows) or Console Logs (macOS) for network, hardware, or OS issues. Look for disk errors, failed hardware components, or driver issues that could affect FileMaker performance.
  • Verify whether the OS applied any automatic updates just before the crash. Unintended restarts or incompatible patches can destabilize the FileMaker Server.

1.2 Resource Usage Monitoring

Monitor the system’s resource usage leading up to the crash. Heavy resource consumption can provide important clues:

• CPU: FileMaker Server can consume excessive CPU if handling too many simultaneous processes, poorly optimized scripts, or unbalanced server configuration.
  • Track CPU spikes or constant high usage leading up to the crash. Use tools such as Windows Task Manager, macOS Activity Monitor, or third-party monitoring tools like Nagios.
  • Analyze active processes to see if specific FileMaker scripts or third-party plugins are responsible for the excessive load.
• Memory: High memory usage or memory leaks in plugins or FileMaker scripts can be a significant contributor to crashes.
  • Investigate memory consumption trends over time.
  • Look for errors such as Error 401 (Out of Memory) or Warning: High Memory Usage notifications in system logs.
• Disk I/O: Disk space and I/O performance are critical for stability, especially if databases or container fields are large.
  • Check disk space: Ensure there’s adequate space for FileMaker’s cache, temp files, and backup operations.
  • Use tools like I/O performance monitors to determine if slow disk access, failing hard drives, or insufficient write speeds caused the crash.

1.3 Network and Hardware Issues

Hardware and network stability are crucial for FileMaker operations, especially when hosting multiple users or integrating third-party systems.

• Check Network Latency: Network interruptions, packet loss, or high latency can cause client connections to drop or corrupt data transfers, leading to server instability. Tools like PingPlotter, Wireshark, or your network’s built-in monitoring system can help diagnose network performance.
• RAID and Storage Health: For RAID configurations, check the health of each drive. A failing RAID controller or degraded drive will not only impact FileMaker performance but also increase the risk of data corruption.
• External Devices: Ensure external devices like storage arrays or backup servers are functioning correctly. A sudden disconnection or malfunction of external storage devices may lead to corrupted databases or failed backup operations.

1.4 Disable and Test Plugins

Third-party plugins can be a common source of instability if not properly optimized or if they conflict with FileMaker’s native operations.

• Disable plugins temporarily: Start by disabling non-essential plugins to isolate the problem. If the server crash stops after disabling plugins, you’ve likely found the source of the problem.
• Sequential re-enabling: After the initial test, re-enable each plugin one-by-one to determine which plugin is causing the issue.
• Ensure plugin compatibility: Always verify that plugins are up to date and fully compatible with your version of FileMaker Server. Check for any deprecated functions or unhandled exceptions in the plugins’ logs.

1.5 Test Database Integrity

A corrupted database file can cause persistent crashes. Here are ways to check file integrity:

• File Verification: Use the built-in FileMaker Pro tools to run consistency checks on databases:
  • File > Manage > Verify will ensure file consistency.
  • Use FileMaker’s Recovery Tool to repair any corrupted files, but do not use recovered files in production without thorough testing.
• Consistency Check at Startup: Configure FileMaker Server to run a consistency check every time it restarts. This is critical for identifying hidden corruption early.

2. Recovery

To top

Once the source of the crash has been identified, focus on recovery. A successful recovery ensures minimal data loss, uptime, and secure restoration of services.

2.1 Graceful Server Shutdown and Restart

Before attempting recovery, always attempt a graceful shutdown of the server:

• Use Admin Console: Initiate a shutdown from FileMaker’s Admin Console whenever possible. This ensures that open database files are closed properly, reducing the risk of data corruption.
• Command-Line Restart: If the Admin Console is unresponsive, use terminal or command-line tools:
  • On macOS, use sudo fmsadmin restart.
  • On Windows, stop and restart the FileMaker service through Services.
  If the server hangs and refuses to shut down, avoid forced reboots, as this increases the likelihood of damaging open database files. Always attempt to terminate non-critical processes first.

2.2 Database File Integrity Check

Upon restart, you should always check that database files are intact. If any databases were open during the crash, the risk of corruption is high.

• Check for Auto-recovery: After the server restarts, FileMaker may attempt to auto-recover databases that were affected by the crash. Review Admin Console messages to see if any databases were recovered.
• Manually Open Files: If any files failed to open automatically, manually open them in FileMaker Pro and run a Recover operation to ensure data integrity.

2.3 File Restoration from Backup

If the database files are corrupted beyond repair, restoration from backup becomes necessary.

• Backup Check: Ensure that your backup files are not compromised. Review backup logs for any errors that occurred during the backup process. If the backups failed due to server stress, you may need to use an older version.
• Incremental Restore: In some cases, only portions of your database may be corrupted. Use incremental backups to restore only the affected parts. Always cross-reference data from multiple backups to ensure nothing critical is lost.

2.4 Gradual User Reconnections

Once the server is restored, start by allowing access to a small group of users. This controlled environment helps detect any lingering performance or stability issues.

• Test key functions: Ensure basic scripts, workflows, and database searches perform as expected.
• Monitor Performance: Check for early signs of instability such as memory spikes or error logs during the first few hours after restart.

2.5 Communicate with Stakeholders

It’s essential to communicate any downtime or performance issues with the relevant stakeholders:

• Send status updates to users, explaining what has been done, and what steps are being taken to avoid future crashes.
• Document the incident fully, noting the time of the crash, diagnostic steps taken, and the recovery procedure, as this can guide future incidents.

3. Prevention

To top

After recovery, focus shifts to minimizing the chances of future crashes. Proactive management ensures system stability and longevity.

3.1 Server Health Monitoring

Continuous monitoring of your FileMaker Server’s health is crucial for preventing future crashes:

• Set up Automated Monitoring: Use tools like Zabbix, Nagios, or built-in FileMaker monitoring tools to track CPU, memory usage, disk I/O, and network traffic. Set automated alerts to trigger whenever resource thresholds are exceeded.
• Use Performance Metrics: Periodically review FileMaker’s Top Call Stats to identify scripts or processes that consume excessive resources. Optimizing these can prevent future crashes.

3.2 Optimize Scripts and Workflows

Poorly optimized scripts are a common cause of crashes, especially in complex databases with multiple users:

• Script Logging: Review Top Call Statistics to identify scripts that take longer to execute or use excessive resources. Rework these scripts to be more efficient.
• Transaction Management: Use commit records and transactional processing for larger workflows to reduce the risk of crashes during peak usage.

3.3 Regular Backup Scheduling

A reliable backup system ensures that, should a crash occur, data recovery is swift:

• Automated Incremental Backups: Set up hourly incremental backups for databases with frequent changes and daily full backups for larger databases. Ensure backups are kept on a separate, reliable storage medium.
• Test Backup Restores: Regularly test backups by performing simulated restores on a test server. Verify data consistency, and ensure the integrity of the backups.

3.4 Apply Regular Software and OS Patches

Ensure that both FileMaker and the server’s OS are up to date:

• Scheduled Updates: Set up regular maintenance windows where server software and operating systems can be updated without impacting users. Check for any critical security patches or bug fixes that could prevent future crashes.

3.5 Hardware and Network Maintenance

Hardware and network infrastructure must be continuously maintained to ensure stability:

• RAID Testing: Perform regular RAID array checks to ensure that your server’s disks are functioning correctly. A failing RAID drive can lead to performance degradation or complete crashes.
• Network Redundancy: Implement network redundancy protocols to avoid downtime from network issues. Using failover clustering or load balancing ensures that your server remains online even if part of the network fails.

4. Post-Recovery

To top

After the server has been restored and preventive measures implemented, the focus shifts to post-recovery monitoring and optimization.

4.1 Review the Crash Incident

Once the crisis is under control, conduct a post-mortem of the crash to document all actions taken and lessons learned.

• Document the timeline of the incident, including when the crash occurred, what recovery steps were taken, and when normal operations resumed.
• Identify Key Failures: Highlight any shortcomings in server setup, monitoring, or response procedures and use this knowledge to update your crash response protocol.
• Internal Audit: Conduct an internal audit with key IT staff, database administrators, and stakeholders to determine the cause of the crash and identify any process improvements.

4.2 Refine Crash Recovery Procedures

With every incident, there’s an opportunity to improve your response plan:

• Update the Server Recovery Guide: Adjust your FileMaker Server recovery guide based on the lessons learned from this crash.
• Train Staff: Ensure that all IT staff are trained on the updated recovery procedures. Provide clear documentation, so team members can respond swiftly to future crashes.

4.3 Schedule Future Maintenance

Set up a maintenance schedule that ensures continuous optimization of your FileMaker Server:

• Server Stress Tests: Perform regular stress tests on your server to simulate high-traffic conditions and ensure it can handle peak loads without crashing.
• Database Maintenance: Periodically optimize databases by rebuilding indexes, removing unused fields, and performing consistency checks.

4.4 Plan for Long-Term Upgrades

In the long term, consider upgrading server hardware and software:

• Hardware Upgrades: Increase CPU, memory, or storage capacity to handle increased traffic or data load as your business grows.
• Software Upgrades: Keep your FileMaker Server and third-party plugins updated to the latest versions to avoid compatibility issues or security vulnerabilities.

4.5 Continue Server Testing

For a period after the crash (1-2 weeks), continue running tests to monitor server health.

• Simulated high-traffic tests: Stress-test your server to ensure stability under load.
• Routine performance benchmarks: Periodically run performance tests on the server, tracking changes in speed or resource usage.
• Monitor user feedback: Encourage users to report any performance anomalies immediately.

5. Check List

To top

Effectively handling FileMaker Server crashes involves preparation, a thorough recovery process, and ongoing server health monitoring. With the right strategies in place, you can ensure data integrity, minimize downtime, and prevent future crashes. By continuously improving your crash response procedures, your team will be better equipped to handle unexpected events with confidence and speed.

FileMaker Server crashes:

6. Videos

To top