FileMaker Server Monitoring with Zabbix

Wim Decorte, Mislav Kos, FileMaker DevCon 2019

Monitoring the health of your FileMaker Server is essential for ensuring optimal performance, security, and stability. With the removal of built-in monitoring tools from the FileMaker Admin Console in recent versions, developers and administrators need a reliable alternative to track key server metrics, manage alerts, and troubleshoot issues proactively. Zabbix, an open-source monitoring tool, fills this gap with robust capabilities to monitor, alert, and even take corrective action automatically. This comprehensive guide will walk you through setting up Zabbix for FileMaker Server, how it works, and best practices to get the most out of it.

Menu

1. Introduction to Zabbix
2. The Problem We’re Solving
3. How Zabbix Works: An Overview
4. Step-by-Step Guide to Installing Zabbix Server and Agent
5. FileMaker Server Metrics You Can Monitor with Zabbix
6. Setting Up Triggers and Actions for Proactive Monitoring
7. Dashboards, Visualization, and Advanced Data Analysis
8. Advanced Monitoring Techniques and Best Practices
9. Key Takeaways and Final Thoughts

Introduction to Zabbix #tag1

Zabbix is an open-source, enterprise-level monitoring solution designed to track the performance, availability, and status of IT infrastructure, including servers, databases, applications, and network devices. Its powerful feature set allows administrators to collect data from multiple sources, create customized triggers to detect problems, visualize performance trends, and take automatic actions to resolve issues.

Zabbix can be used to monitor the health of FileMaker Server installations, helping to prevent performance bottlenecks, security risks, and downtime. It integrates seamlessly with FileMaker, offering insights into server health, database performance, API usage, and more, all in real-time.

Why Choose Zabbix for FileMaker Monitoring?

• Open-source and Free: Zabbix is completely free and supported by a vibrant open-source community. No license fees are required.
• Flexible Deployment: Zabbix can be deployed on-premise or in the cloud, and it supports a wide range of operating systems, including Linux, Windows, macOS, and more.
• Customizable Triggers and Alerts: Administrators can create custom triggers that fire alerts when specific thresholds are crossed. For example, you can receive a notification when the FileMaker scripting engine goes down or the server’s CPU load spikes.
• Automation: Zabbix can automatically take corrective actions, such as restarting the FileMaker Data API or increasing cache size, without manual intervention.
• Scalability: Whether you’re monitoring one server or hundreds, Zabbix scales to meet the needs of small businesses and large enterprises alike.

Back to Menu

The Problem We’re Solving #tag2

In earlier versions of FileMaker Server (e.g., versions 16 and earlier), the Admin Console featured basic built-in tools for monitoring the server, such as live statistics and log viewers. These tools allowed administrators to view metrics in real-time and receive email notifications when errors or warnings occurred. However, in more recent versions, these tools were removed, leaving a significant gap in monitoring capabilities.

For developers and administrators, this removal has created several challenges:

• Lack of Real-Time Monitoring: Without the built-in live statistics chart, there’s no easy way to monitor server performance metrics like cache usage, API response times, or database load.
• No Built-in Alerts: In earlier versions, the Admin Console could send email alerts based on event log entries, such as errors or security issues. The new versions do not offer this functionality out-of-the-box.
• Manual Troubleshooting: In the absence of monitoring tools, administrators must manually check logs and troubleshoot issues after they arise, often leading to delays in problem resolution.

The Role of Zabbix: Zabbix offers a comprehensive solution to these challenges by providing live monitoring, alerting, and automation features. It allows administrators to track key performance metrics in real-time, receive notifications when problems occur, and automatically take corrective action to minimize downtime.

Key Problems Addressed by Zabbix:

1. Proactive Monitoring: Track key server metrics continuously and detect potential issues before they impact end users.
2. Automated Actions: Automatically restart critical services like the FileMaker scripting engine or Web Publishing Engine (WPE) when they fail.
3. Detailed Reporting: Generate detailed reports on performance issues, server downtime, and error patterns, allowing for data-driven decisions.
4. Unified Dashboard: View all server metrics, logs, and alerts in one unified dashboard, simplifying the monitoring process for administrators.

By addressing these key pain points, Zabbix empowers administrators to maintain a stable and secure FileMaker Server environment.

Back to Menu

How Zabbix Works: An Overview #tag3

Zabbix operates using a client-server model, where the Zabbix server collects data from multiple hosts, which could be FileMaker Servers, databases, network devices, or other systems. These hosts communicate with the Zabbix server using agents, lightweight software that collects data on the host machine and sends it back to the Zabbix server.

Passive vs. Active Monitoring

Zabbix supports both passive and active monitoring modes, giving you flexibility in how you collect data from your FileMaker servers.

1. Passive Monitoring: In passive mode, the Zabbix server periodically requests data from the agent running on the host. The agent does not take any action until the Zabbix server explicitly asks for information, such as CPU usage or free disk space.
2. Active Monitoring: In active mode, the Zabbix agent on the host collects data autonomously based on a pre-configured schedule. The agent then sends this data to the Zabbix server without waiting for a request. Active mode is useful when you want to avoid opening inbound network ports on the monitored system.

Key Components of Zabbix

• Items: An item is a specific data point that Zabbix monitors. For example, an item could be “CPU load” or “Free disk space.” Each item is linked to a host (the server being monitored) and assigned a collection interval.
• Triggers: A trigger is a logical expression that evaluates the data collected by items and determines whether a problem exists. For example, a trigger could check if free disk space drops below 10% and flag this as a problem.
• Actions: When a trigger detects a problem, Zabbix can automatically execute actions. Actions can include sending email notifications, restarting services, or posting alerts to a Slack channel.
• Graphs and Dashboards: Zabbix provides powerful visualization tools, allowing you to create real-time graphs and dashboards to monitor server performance. You can display multiple metrics simultaneously and zoom in on specific time periods to analyze issues.

Data Flow in Zabbix

1. Data Collection: The Zabbix agent collects data from the host (FileMaker Server) based on pre-configured items (e.g., CPU load, cache usage).
2. Evaluation: Zabbix evaluates the data against pre-defined triggers to determine if there’s a problem.
3. Alerts and Actions: If a problem is detected, Zabbix triggers an alert and optionally takes corrective action (e.g., restarting a service).
4. Visualization: The data is displayed in real-time on Zabbix dashboards, providing administrators with a complete view of system health.

Zabbix’s flexibility allows administrators to configure a wide range of custom monitoring scenarios, from simple CPU load tracking to complex multi-server health checks with automatic failover.

Back to Menu

Step-by-Step Guide to Installing Zabbix Server and Agent #tag4

To monitor FileMaker servers with Zabbix, you’ll need to install both the Zabbix server (which collects and stores data) and the Zabbix agent (which collects data from the FileMaker server). Here’s how to set them up.

1. Installing the Zabbix Server

The Zabbix server can be installed on a variety of platforms, including Linux, Windows, and macOS. However, for the best performance and compatibility, Linux is the recommended platform.

Steps to Install Zabbix Server:

1. Choose Hosting Environment: Decide whether you want to host Zabbix on-premise (e.g., on a dedicated Linux server or a virtual machine) or in the cloud (e.g., AWS, Google Cloud, or Azure).
2. Download the Zabbix Server: Go to the Zabbix website and download the appropriate version of the Zabbix server for your operating system.
3. Install the Zabbix Server: Follow the installation instructions for your chosen platform. Zabbix also offers pre-built appliances for virtual machines like VirtualBox or VMware, which come pre-configured with Zabbix and make installation easier.
4. Database Setup: During installation, Zabbix will require a backend database to store the collected data. Supported databases include MySQL, PostgreSQL, and Oracle. Install and configure your chosen database, then connect it to the Zabbix server.
5. Start the Zabbix Server: Once installed, start the Zabbix server service and access the Zabbix web interface to configure monitoring.

2. Installing the Zabbix Agent on FileMaker Server

The Zabbix agent collects data from the FileMaker server and sends it to the Zabbix server for analysis. Zabbix provides agents for most operating systems, including Windows, macOS, and Linux.

Steps to Install Zabbix Agent:

1. Download the Zabbix Agent: Visit the Zabbix download page and select the appropriate agent for your FileMaker Server’s operating system (e.g., Windows or macOS).
2. Install the Agent: Follow the installation instructions for your operating system. On Linux systems, you can install the agent using package managers like apt or yum. On Windows, use the pre-built installer.
3. Configure the Agent: Edit the agent configuration file (zabbix_agentd.conf) to specify the Zabbix server’s IP address and hostname. This allows the agent to communicate with the Zabbix server.
4. Start the Zabbix Agent: After configuring the agent, start it as a service (or daemon on Linux/macOS). The agent will now continuously collect data from the FileMaker server and send it to the Zabbix server.
5. Verify the Installation: In the Zabbix web interface, add the FileMaker server as a host and link it to the Zabbix agent. If everything is configured correctly, you’ll start seeing metrics in the dashboard.

Back to Menu

FileMaker Server Metrics You Can Monitor with Zabbix #tag5

Once Zabbix is set up, it can track a wide range of metrics that are critical to maintaining the performance and stability of FileMaker Server. These metrics can be visualized in real-time and used to trigger alerts or corrective actions when thresholds are breached.

1. Server Performance Metrics

Monitoring the general health of your FileMaker Server is essential for identifying bottlenecks and ensuring smooth operation. Zabbix can track:

• CPU Usage: Monitor the CPU load on the FileMaker server to ensure it doesn’t become a performance bottleneck.
• Memory Usage: Keep an eye on RAM consumption to avoid memory exhaustion, which could lead to crashes.
• Disk Space: Monitor free disk space on critical partitions, ensuring the server has enough room for logs, backups, and data files.
• Network Traffic: Track inbound and outbound network traffic to detect any unusual activity that could indicate a security breach or network bottlenecks.

2. FileMaker-Specific Metrics

Zabbix can also track metrics specific to FileMaker Server, such as:

• Cache Hit Percentage: A critical metric for FileMaker performance, the cache hit percentage indicates how often FileMaker can retrieve data from its cache without hitting the disk. A low cache hit percentage can lead to performance degradation.
• Elapsed Time Per Call: This metric tracks how long it takes for FileMaker to execute database calls. A high value could indicate server overload or inefficient queries.
• FileMaker Data API Usage: Monitor the health of the FileMaker Data API, which is used to access FileMaker data via web requests. You can track the number of API requests, response times, and any errors.
• Scripting Engine Health: Ensure the FileMaker scripting engine (FMSE) is running smoothly. Zabbix can automatically restart the scripting engine if it crashes.

3. Log Monitoring

FileMaker Server generates several logs that contain valuable information about its performance and health:

• Event Logs: Track important events like server restarts, crashes, and error messages.
• Statistics Logs: FileMaker Server generates detailed performance statistics that can be monitored to detect trends or anomalies.
• Access Logs: Monitor who is accessing the server and what actions they are taking. This can help detect unauthorized access or unusual activity.

4. Advanced Metrics

Zabbix can also monitor more advanced metrics by using custom scripts or external tools:

• Crash Dumps: Zabbix can detect when a crash dump file is generated, alerting administrators that a crash has occurred.
• Custom FileMaker Commands: You can create custom items that run FileMaker-specific commands (e.g., using fmsadmin to check server status) and collect the results in Zabbix.

By monitoring these metrics, Zabbix provides deep visibility into the performance of FileMaker Server, allowing administrators to detect issues early and take action before they affect end users.

Back to Menu

Setting Up Triggers and Actions for Proactive Monitoring #tag6

Monitoring data alone is not enough—you need a way to detect problems and respond to them quickly. Zabbix uses triggers to evaluate the data collected from FileMaker Server and determine whether a problem exists. If a trigger condition is met, Zabbix can execute actions, such as sending alerts or restarting services.

1. Creating Triggers

Triggers are logical expressions that evaluate the data collected by Zabbix items. For example, you might create a trigger that evaluates whether free disk space is below a certain threshold or whether the FileMaker Data API process has crashed.

Example of a Trigger:

{FileMakerServer:vm.memory.size[free].last()} < 5000000000

This trigger checks if free memory on the FileMaker server is less than 5 GB. If the condition is true, Zabbix considers this a problem.

Triggers can be customized based on your specific needs. For example:

• Cache Hit Percentage: Trigger an alert if the cache hit percentage drops below 95%.
• Elapsed Time Per Call: Generate a problem if the elapsed time per call exceeds 10 seconds for more than one minute.
• Disk Space: Send an alert if free disk space on the server drops below 10%.

2. Configuring Actions

When a trigger detects a problem, Zabbix can automatically execute actions. Actions can include:

• Email Notifications: Zabbix can send an email to administrators when a problem is detected. You can configure different email recipients based on the severity of the issue.
• SMS Alerts: For critical issues, you can configure Zabbix to send SMS alerts, ensuring that administrators are notified even when they are away from their desks.
• Restart Services: Zabbix can automatically restart services when they fail. For example, if the FileMaker scripting engine crashes, Zabbix can use the fmsadmin command to restart it.
• Run Custom Scripts: Zabbix can execute custom scripts to handle more complex scenarios. For example, you could create a script to back up log files and email them to an administrator when a problem is detected.

Example of an Action: Let’s say you want Zabbix to restart the FileMaker Data API process if it crashes. You would:

1. Create a trigger to detect when the Data API process is down.
2. Configure an action that runs the following command when the trigger fires:
   fmsadmin start fmrest
   
3. Optionally, configure Zabbix to retry the command up to five times if the process does not restart on the first attempt.

By configuring triggers and actions, Zabbix allows you to automate much of the monitoring and troubleshooting process, reducing the time it takes to resolve issues and ensuring that problems are addressed as soon as they arise.

Back to Menu

Dashboards, Visualization, and Advanced Data Analysis #tag7

One of Zabbix’s strengths is its ability to visualize the data it collects, making it easier to monitor your FileMaker server in real-time and spot trends or anomalies. Zabbix offers several ways to visualize data, including graphs, screens, and dashboards.

1. Creating Graphs

Zabbix can automatically generate graphs based on the data it collects from items. For example, you can create graphs to track:

• CPU Usage: Visualize how CPU usage changes over time, helping you detect spikes that could indicate performance issues.
• Cache Hit Percentage: Monitor the cache hit percentage to ensure that your FileMaker server is utilizing its cache effectively.
• Elapsed Time Per Call: Track how long database calls are taking and detect performance bottlenecks.

Zooming and Custom Time Periods: Zabbix allows you to zoom in on graphs to see data for specific time periods. You can also customize the time range (e.g., last hour, last 24 hours, last week) to analyze different periods of server activity.

2. Creating Screens

Screens are a collection of graphs, charts, and other visual elements that are displayed together. Screens allow you to create a single view that combines multiple metrics, making it easier to monitor the overall health of your FileMaker server.

For example, you could create a screen that displays:

• Real-time CPU usage
• Cache hit percentage
• Network traffic
• Active FileMaker clients
• Elapsed time per call

By displaying these metrics side-by-side, you can get a complete picture of your server’s performance at a glance.

3. Dashboards

Dashboards are similar to screens, but they provide more flexibility and customization options. Dashboards can display:

• Problem Overview: See a summary of all active problems, grouped by severity (e.g., disaster, high, warning).
• Triggers: Display the status of all triggers and whether any have fired recently.
• Graphs: Display key performance metrics, such as CPU usage, memory usage, or disk activity.

Dashboards are ideal for providing high-level overviews of system health. You can also configure Zabbix to cycle through multiple dashboards automatically, displaying each one for a set amount of time.

4. Integration with Grafana

If you need more advanced visualization options, you can integrate Zabbix with Grafana, an open-source analytics and monitoring platform. Grafana allows you to create highly customizable dashboards and graphs, and it can pull data directly from the Zabbix database.

With Grafana, you can create complex visualizations that combine data from multiple sources, such as FileMaker Server, network devices, and other servers. This is especially useful if you’re managing a large and diverse IT infrastructure.

Back to Menu

Advanced Monitoring Techniques and Best Practices #tag8

To get the most out of Zabbix, consider these advanced techniques and best practices for monitoring FileMaker Server.

1. Customize Triggers for Your Deployment

While Zabbix provides default triggers for common issues (e.g., low disk space, high CPU usage), it’s important to customize these triggers to fit your specific deployment. For example:

• Adjust Thresholds: If your FileMaker deployment typically runs at 90% CPU usage, you might want to raise the CPU trigger threshold to avoid false positives.
• Set Different Severities: Assign different severities to triggers based on their importance. For example, you might assign a “warning” severity to low disk space but a “disaster” severity to a crashed Data API process.

2. Automate Corrective Actions

Zabbix’s ability to automate corrective actions is one of its most powerful features. Consider automating actions such as:

• Restarting Services: Automatically restart the FileMaker scripting engine, Web Publishing Engine (WPE), or Data API process if they crash.
• Increasing Cache Size: If the cache hit percentage drops below a certain threshold, Zabbix can automatically increase the FileMaker server’s cache size using the fmsadmin command.
• Rebooting the Server: In extreme cases, Zabbix can automatically reboot the FileMaker server if it becomes unresponsive.

3. Use Templates for Easy Configuration

Zabbix allows you to create templates, which are pre-configured sets of items, triggers, actions, and graphs that can be applied to multiple hosts. For example, you could create a template specifically for monitoring FileMaker Server and apply it to all of your FileMaker hosts.

Templates simplify the setup process and ensure consistency across multiple servers. Zabbix provides a template library, and you can also create your own templates based on your specific needs.

4. Perform Regular Audits

Even with automated monitoring in place, it’s important to perform regular audits of your Zabbix setup. Check to ensure that:

• All critical metrics are being monitored: Make sure you’re tracking key metrics like cache usage, API health, and disk space.
• Triggers are firing as expected: Test your triggers periodically to ensure they’re working correctly and firing alerts when thresholds are breached.
• Actions are taking the correct steps: Verify that actions (e.g., restarting services or sending notifications) are being executed properly when a problem is detected.

By regularly auditing your monitoring setup, you can ensure that Zabbix continues to provide reliable and accurate monitoring for your FileMaker deployments.

Back to Menu

Key Takeaways and Final Thoughts #tag9

Zabbix is a powerful, flexible, and open-source monitoring tool that provides deep insights into the performance and health of your FileMaker Server. By leveraging Zabbix’s advanced monitoring, alerting, and automation capabilities, you can ensure that your FileMaker deployments remain stable, secure, and high-performing.

Key Benefits of Using Zabbix:

1. Proactive Monitoring: Zabbix allows you to detect and address issues before they impact users, minimizing downtime and improving reliability.
2. Customizable Alerts and Actions: Configure Zabbix to send notifications or automatically restart services when problems occur, ensuring rapid response to critical issues.
3. Comprehensive Dashboards: Use Zabbix’s built-in visualization tools to create real-time dashboards that display key metrics and trends, providing a complete view of system health.
4. Scalability: Whether you’re monitoring a single FileMaker server or an entire network of servers, Zabbix can scale to meet the needs of your organization.
5. Open-Source and Free: Zabbix is completely free to use, making it an ideal solution for businesses of all sizes.

By implementing Zabbix for your FileMaker deployments, you can gain complete visibility into server health, improve performance, and reduce the risk of downtime or security breaches.

Back to Menu

Zabbix FileMaker Server Monitoring Session at DevCon 2019

At FileMaker DevCon 2019, Mislav Kos and Wim Decorte presented a session on Server Monitoring with Zabbix. The FileMaker community has good information around this topic but most of it relates to how to install Zabbix.

In this short preview,they fast-forwarded a bit and showcased what the end result looks like. Specifically, we demonstrated the power of Zabbix. It can keep an eye on your FileMaker Server, show you its vitals, and send notifications when things go wrong. However, more importantly, it can also proactively step in and fix important issues.

Restarting the Data API Process with Zabbix

For this article, we will show how Zabbix can restart the Data API process if it stops. You can apply this same logic to the other processes that make up all of FileMaker Server: the server-side scripting engine, the web publishing engine, the ODBC/JDBC process, etc…

When any of these components crash or stop, Zabbix can step in and take corrective action.

How Zabbix Works with FileMaker Server

Zabbix works as follows:

1. It collects data on items that you have defined. In the Zabbix template released with our DevCon presentation (get them here), we include over 150 relevant items related to FileMaker Server’s health.
2. For those items that you deem important, you define a trigger that evaluates the data for a particular item. Examples include: process running or not running, cache hit % below 95%, number of Remote Calls in Progress higher than 10, etc…
3. When Zabbix finds that the data for anfdevcon item matches one of your triggers, it raises a problem event. Zabbix automatically tracks these problems and their durations, which you can see in the Zabbix dashboard.
4. Additionally, you can have Zabbix take action. Most typically choose to send out a notification by email or SMS. However, Zabbix can also execute a command on the host machine where the problem happens.

FileMaker Server Monitoring with Zabbix

One of the servers we track is a FileMaker Server 17 running on macOS. Using the macOS Activity Monitor we can see the various FileMaker Server processes including “fmwipd,” which is the Data API process.

There are a few different ways that we can track if it stops working; for instance, we could rely on the FileMaker Server event log where an event with ID 701 will be raised when FileMaker Server detects that its Data API has terminated abnormally. Zabbix can certainly (and easily) keep an eye on the event log.

But we will work with the operating system and use the OS tools to track the process. On Windows, we would do this by using the built-in Windows Perfmon and ask for the Thread Count on the process. If it is 0 or there is no data for that counter, then we would know that the process is not running.

But this is macOS, so we will just use the built-in “ps” command to get a list of the running processes and their state.

In our Zabbix template we have an item for just that purpose (note that it filters the output of the ‘ps’ command to just processes owned by ‘fmserver’):

This item will list all of the fms processes every 30 seconds. We have a dependent item that uses this output to look specifically for the ‘fmwipd’ process. For this we make use of Zabbix’s JavaScript pre-processing functionality:

This item will return a simple ‘1’ if the process is running and ‘0’ if it is not.

As you can tell from the screenshot, we have a trigger defined for the item to raise a ‘Disaster’ problem when the last value returned for the item is ‘0’. We check this every 30 seconds.

At this point, the problem visibility is limited to the Zabbix dashboard. We need an action to be notified automatically and for Zabbix to take action:

We have configured the action to do two things: email us immediately to make us aware of the problem and also immediately try to restart the Data API through the FileMaker Server admin command line:

fmsadmin restart fmdapi

What happens if we simulate a Data API crash? From the macOS activity monitor, we force quit the Data API process:

When we turn to Zabbix and look at the data for the Data API item, we see that it was showing ‘1’ (aka ‘running just fine’). It then dipped to ‘0’ (aka ‘not running) and fairly immediately went back to the running state:

A quick look at our FileMaker Server shows that the process has indeed returned. Note that it has a different process ID (PID) than it had before, an indication that the process has restarted.

In Zabbix, the problem event was updated to ‘RESOLVED’, and we can see the actions that Zabbix took and how fast it did it:

• At 8:53 and 47 seconds it caught the problem.
• One second later it executed the command.
• 30 seconds later (the next data collection interval for the item), it determined that the process was running and the problem was closed.

Other Zabbix Actions

Zabbix triggers and actions have much more power than what we are showing here, including escalation procedures. However, we hope this shows how Zabbix can give us both monitoring and healing capabilities beyond anything we ever had with just the FileMaker Server Admin Console.

Next Steps with Zabbix FileMaker Server Monitoring

The functionality to allow Zabbix to execute commands on your FileMaker Server is ready for you. We hope this convinces you to consider adding Zabbix to your toolbelt and fortifying your FileMaker Server deployments. 

DevCon 2019 SHA02 – Server Monitoring with Zabbix – Wim DeCorte