How does the EU AI Act affect FileMaker Developers?

Dimitris Kokoutsidis, Oct 9, 2024, CyberFM

Key Provisions of the EU AI Act

1. Risk-Based Classification System

The EU AI Act categorizes AI systems into four risk levels based on their potential impact on individuals’ safety and fundamental rights:

Unacceptable Risk: AI systems that are deemed harmful to safety or infringe on fundamental rights are banned. Examples include AI systems that manipulate human behavior (e.g., social scoring by governments) or exploit vulnerabilities of specific groups.
High Risk: AI systems that pose significant risks to safety or fundamental rights must comply with strict regulations. This category includes AI used in critical areas such as:
- Healthcare (e.g., AI systems for diagnosing diseases)
- Recruitment (e.g., AI used for hiring decisions)
- Law enforcement (e.g., AI used for criminal risk assessment)
- Education (e.g., AI used for grading exams)
These systems must undergo rigorous conformity assessments, ensure robust risk management, and maintain data governance practices.
Limited Risk: AI systems with limited risks, such as chatbots, will be subject to transparency obligations. For instance, users must be informed when they are interacting with an AI system, but compliance requirements are less stringent compared to high-risk systems.
Minimal or No Risk: AI systems that pose minimal or no risk (e.g., spam filters, AI-enabled video games) are not subject to specific obligations under the Act.

2. Transparency and Disclosure

AI systems must meet transparency obligations to ensure users understand when they are interacting with an AI system. For instance:

AI systems that interact with humans must clearly disclose they are AI-powered (e.g., chatbots).
Users must be informed when they are interacting with AI in certain applications (such as deepfakes or image recognition).

3. Human Oversight

For high-risk AI systems, the EU AI Act requires human oversight to ensure that AI does not act autonomously in ways that could harm individuals or society. Human operators must be able to understand and, when necessary, intervene in the operation of AI systems.

4. Data Governance and Quality

The Act emphasizes the importance of high-quality datasets to ensure that AI systems are reliable and do not result in discriminatory or biased outcomes. Specific provisions include:

Datasets used to train AI systems must be representative, free of errors, and complete to avoid discrimination.
AI developers must implement risk management systems, and the data used in AI systems must be documented and auditable.

5. Conformity Assessment and CE Marking

AI providers must conduct conformity assessments for high-risk systems before they are deployed. These assessments are designed to ensure compliance with safety, transparency, and fundamental rights standards. After passing the conformity assessment, high-risk AI systems will be granted a CE marking, indicating that the AI system complies with EU standards.

6. AI Governance and Enforcement

The Act mandates that each EU member state establishes a national supervisory authority responsible for overseeing AI compliance. Additionally, the European Artificial Intelligence Board will coordinate and facilitate cooperation between these national authorities to ensure consistent enforcement across the EU.

7. Post-Market Monitoring #post_market_monitoring

For high-risk AI systems, providers must implement systems for post-market monitoring, including incident reporting. If an AI system causes harm or fails to meet its regulatory requirements, it can be subject to corrective actions or penalties.

8. Penalties for Non-Compliance #penalties

Non-compliance with the EU AI Act can result in significant penalties, depending on the severity of the breach:

Up to €30 million or 6% of global annual revenue for serious breaches, including violations of prohibited practices (e.g., deploying AI systems that are banned).
Up to €20 million or 4% of global annual revenue for violations related to high-risk AI systems.
Up to €10 million or 2% of global annual revenue for lesser violations, such as failing to meet transparency requirements.

9. Prohibition of Specific AI Practices #prohibition

The Act bans certain AI practices deemed unacceptable, including:

Social scoring: AI systems that score individuals based on their behavior, which could lead to unfair or discriminatory treatment.
Real-time biometric identification in public spaces: This includes facial recognition technologies used for law enforcement purposes in public spaces, with limited exceptions (e.g., preventing terrorist attacks).

10. Security and Robustness #security_robustness

AI systems, especially high-risk ones, must be secure and robust, with measures in place to prevent unintended outputs or misuse. The systems must function in a predictable and consistent manner, even in unexpected situations.

Implications for FileMaker Developers #implications_filemaker

The EU AI Act affects FileMaker and its community primarily through the compliance and regulatory framework it imposes on the use and integration of artificial intelligence (AI) technologies. While FileMaker is a versatile low-code platform focused on building custom applications for businesses, it increasingly integrates AI-driven tools, services, and workflows. As AI adoption grows within the FileMaker ecosystem, organizations using it in the EU or serving EU citizens need to be aware of how the AI Act impacts AI features or integrations.

1. Risk-Based Classification #filemaker_risk

The AI Act classifies AI systems into four risk categories: unacceptable, high, limited, and minimal risk. Any FileMaker application or solution that uses AI will need to be assessed under these classifications, particularly:

High-risk AI systems: FileMaker solutions that incorporate AI in high-risk areas (such as healthcare, recruitment, or financial services) must comply with strict regulatory obligations, including risk assessments, documentation, and monitoring.
Limited-risk AI systems: If FileMaker apps leverage AI features like chatbots, recommendation engines, or simple data classification tools, these are subject to transparency requirements. For instance, FileMaker developers must inform users that AI is in use and ensure it doesn’t infringe on users’ rights.

2. Transparency and User Awareness #filemaker_transparency

Any FileMaker solution that uses AI models, such as chatbots, natural language processing, or machine learning integrations, will need to comply with transparency rules. This includes:

User notification: If a FileMaker app leverages AI (e.g., for customer interactions or workflow automation), the user must be clearly informed that they are interacting with an AI system.
Data usage clarity: FileMaker applications must disclose how AI systems use personal data, ensuring that it adheres to data governance and GDPR standards.

3. Human Oversight in High-Risk Applications #filemaker_oversight

For FileMaker solutions classified as high-risk AI (e.g., in sectors like finance, HR, or law enforcement), human oversight becomes crucial. FileMaker developers and organizations integrating AI must:

Ensure human review over automated decision-making processes.
Build workflows within FileMaker that allow users to override AI-based decisions when necessary.
Maintain control and the ability to explain how the AI arrived at its conclusions, ensuring that AI outcomes can be scrutinized or modified by human operators.

4. Data Governance and Quality Standards #filemaker_data

Many FileMaker apps rely on external AI services through API integrations (e.g., AI models for predictions or analysis). Under the EU AI Act, FileMaker developers must ensure that:

Data quality: AI systems in FileMaker must use high-quality, non-biased data sets to avoid discriminatory outcomes. This applies to any machine learning model or AI tool integrated into FileMaker through APIs or custom web services.
Auditability: FileMaker applications with AI functionality must document data flows and processes. If AI models are used for decision-making (e.g., predictive analytics or automated classification), the process must be auditable, and clear documentation must be maintained.

5. Post-Market Monitoring #filemaker_monitoring

For any FileMaker solution involving high-risk AI, the EU AI Act mandates post-market monitoring. FileMaker developers and businesses must:

Implement processes for tracking and auditing the performance of AI systems integrated into their applications.
Set up mechanisms within FileMaker to report issues or AI malfunctions to the appropriate authorities.
Maintain compliance logs, ensuring that AI systems used within FileMaker apps are performing as intended and do not cause harm to users.

6. Security and Robustness of AI Systems #filemaker_security

If a FileMaker application integrates AI models that make critical business decisions, the security and robustness of those systems become a key concern:

Secure AI integration: FileMaker apps must ensure secure API connections when integrating AI services to avoid tampering or breaches.
Reliable performance: FileMaker apps must ensure that the integrated AI models function reliably and don’t behave unexpectedly in real-world use. For example, if FileMaker pulls AI insights from third-party tools for business intelligence, the results should be consistent and accurate.

7. Penalties for Non-Compliance #filemaker_penalties

Non-compliance with the EU AI Act can lead to significant fines and penalties, which directly impacts any business using AI within FileMaker solutions. Companies and developers using AI-driven features in FileMaker applications must:

Ensure their solutions meet EU AI Act compliance, particularly when working with high-risk AI systems.
Implement robust data protection, transparency, and human oversight mechanisms to avoid penalties.

8. Custom AI Integrations #filemaker_custom

FileMaker’s flexibility allows developers to create custom solutions and integrate AI services via APIs, such as OpenAI’s GPT models or other machine learning services. However, the EU AI Act creates challenges:

API-driven AI: Any AI integration through APIs (e.g., sentiment analysis, image recognition) must comply with transparency, data governance, and risk assessment rules.
Custom AI models: FileMaker solutions with custom-trained AI models for specific industries (like financial forecasting or HR) must meet conformity assessment procedures if used in high-risk environments.

Practical Steps for FileMaker Developers #filemaker_steps

FileMaker developers working with AI integrations should consider the following to comply with the EU AI Act:

Evaluate AI Risk Levels: Understand where your AI systems fit within the EU AI Act’s risk-based classification. For high-risk systems, prepare to implement additional security, transparency, and human oversight measures.
Ensure Transparency: When integrating AI into FileMaker, especially for customer-facing applications, make it clear to users when AI is in use. Provide necessary disclosures on how their data is being used.
Build Human Oversight Mechanisms: For high-risk AI use cases, ensure that human operators can review and override AI decisions if necessary. Design workflows in FileMaker that allow human intervention in critical decisions.
Comply with Data Governance Rules: Ensure that the AI models you integrate or build within FileMaker applications are trained on high-quality, unbiased data. Document all data flows to ensure compliance with the Act’s requirements.
Post-Market Monitoring: Implement monitoring processes in FileMaker to track the performance of AI systems, ensuring they operate within the safety and compliance guidelines set by the EU AI Act.

By following these guidelines, FileMaker developers can ensure their AI integrations are compliant with the EU AI Act, keeping their solutions safe, transparent, and legally sound.

Semantic Find Functionality in FileMaker 2024 #semantic_find

The introduction of the Semantic Find functionality in FileMaker 2024 brings immense potential to revolutionize data searches. However, with this capability comes a set of risks, particularly when applied in sensitive areas such as financial decision-making, production planning, and healthcare. These risks stem from the potential for misinterpretation of results, reliance on AI-generated suggestions, and the unintended consequences of actions taken based on those suggestions.

Financial Risks #financial_risks

Automated Searches in Financial Data: The ability to perform semantic searches on financial data can streamline finding insights or identifying trends, but this can also lead to errors if searches retrieve misleading or incomplete information. For example, a search for “high-value clients” could yield results based on outdated or incorrect criteria, potentially influencing decisions like loan approvals or credit assessments.
Investment and Spending Decisions: AI-based findings might suggest a certain pattern in investment data, leading businesses to make high-risk financial decisions based on AI interpretation, which could amplify losses if the AI misinterprets the data trends.

Production Planning #production_risks

Over-Reliance on AI for Supply Chain Optimization: If the semantic find feature is used to predict supply chain needs or production rates, a wrong prediction could lead to stockouts, overproduction, or delays. For example, searching for “low inventory” could miss contextual factors like pending shipments or seasonal demand spikes, leading to incorrect production planning.
Operational Efficiency Missteps: Decisions about scaling operations based on data extracted through semantic search could result in increased operational costs if the AI does not properly contextualize search results, such as production capacity or labor requirements.

Healthcare Risks #healthcare_risks

Medical Record Searches: In healthcare, searching through patient records semantically could yield erroneous results, especially in complex cases. For example, a search for “patients with heart disease” might pull in patients with unrelated conditions if the AI misinterprets medical terminology, leading to inappropriate treatments.
Compliance Issues: The incorrect retrieval of sensitive health data could result in compliance breaches with regulations like GDPR, leading to legal and financial penalties.

FileMaker 2024 Potential Risks #filemaker_2024_risks

In addition to the Semantic Find feature, other functionalities within FileMaker 2024 introduce potential risks, especially when applied in high-stakes contexts like finance, healthcare, or critical infrastructure. Let’s explore some of the features that carry risk and the specific challenges they present.

1. Automated Decision-Making and AI-Driven Insights #ai_decision

FileMaker’s growing support for integrating AI into workflows—whether for generating reports, analyzing data, or providing predictive insights—poses risks when these systems are trusted with high-stakes decisions without human oversight.

Risk in Finance: AI-driven insights based on transactional history or financial modeling could inadvertently promote biased financial decisions. For example, an algorithm might recommend investment portfolios based on incomplete data, resulting in financial loss or unequal opportunities for certain demographics.
Risk in Healthcare: If medical professionals over-rely on AI-generated insights, such as patient diagnoses or treatment recommendations, it could lead to incorrect diagnoses or inappropriate treatments. There are risks of automation bias, where users trust the AI’s recommendation without question.

2. Data Automation and Scripting #data_scripting

FileMaker’s ability to automate repetitive tasks and workflows through scripting is a powerful feature, but in sensitive industries, it poses several risks.

Financial Automation Risks: Automating tasks such as transaction approvals, payroll processing, or inventory purchasing might lead to significant financial losses if an automation script misfires or is executed based on incorrect assumptions. An unchecked script could over-purchase inventory or incorrectly process financial data, causing losses or fraud.
Healthcare Risks: Automated workflows involving patient data management, billing, or lab report generation must be carefully configured. Incorrect scripting or automation errors could lead to patient harm (e.g., delivering incorrect medications or generating the wrong dosage instructions).

3. Cross-System Integration #cross_system

FileMaker’s ability to integrate with APIs, external databases, and third-party systems provides immense flexibility but opens up significant risks related to data integrity, security, and compliance.

Security Risks: Cross-system integrations could expose sensitive data if not properly secured. An integration with external systems, particularly if they are outdated or vulnerable, could lead to data breaches, unauthorized access, or the leaking of proprietary information. This is particularly concerning for financial institutions handling transaction data or healthcare organizations storing patient information.
Compliance Risks: Misconfigured integrations may lead to violations of data protection laws such as GDPR or HIPAA. For instance, if patient health data or financial data is transferred across borders without adequate protection, this could result in hefty fines and legal consequences.

4. Cloud Hosting and Data Storage #cloud_risks

FileMaker’s cloud-hosting features offer convenience and scalability, but storing sensitive data in the cloud poses risks related to data breaches and mismanagement of access controls.

Risk in Financial Services: Hosting financial data in the cloud without proper encryption or access control policies could expose sensitive transactional data to unauthorized actors, leading to breaches, financial fraud, or theft.
Risk in Healthcare: Storing electronic health records (EHRs) in the cloud presents risks if proper safeguards (such as encryption, role-based access control, and audit trails) aren’t in place. A breach of EHRs could lead to legal consequences under regulations like HIPAA.

5. AI-Assisted Data Analysis and Natural Language Processing #nlp_risks

FileMaker 2024’s potential for AI-assisted data analysis, natural language processing (NLP), and the ability to understand and generate text-based insights introduce risks of incorrect conclusions, misuse of sensitive information, or unintentional bias.

NLP in Financial Reporting: AI-powered analysis of financial documents may miss nuances, leading to incorrect financial statements or investment guidance. For example, NLP might misinterpret legal or contractual obligations, resulting in decisions that negatively impact businesses or investors.
Healthcare Documentation: NLP could misinterpret medical notes, leading to incorrect patient treatment plans. Misunderstanding context or medical terminology in patient records could result in inappropriate treatments or procedures.

6. User-Defined Privileges and Access Controls #access_controls

FileMaker provides detailed user-defined privileges to control access to data and system functionality. However, improper configuration of these privileges can lead to unauthorized data access or insider threats.

Insufficient Privilege Management: If access controls are not rigorously implemented, employees might gain access to confidential information they don’t need to perform their duties. In finance, this could lead to insider trading or fraud, and in healthcare, it might result in violations of patient privacy.
Misuse of Admin Privileges: Users with admin access might accidentally or maliciously alter sensitive data or workflows, which could cause system-wide disruptions or data loss.

7. Predictive Analytics and Machine Learning Integrations #predictive_risks

As FileMaker becomes more capable of integrating with machine learning tools and predictive analytics, there’s a risk of deploying models that are unvetted, biased, or trained on incomplete or incorrect data sets.

Predictive Models in Finance: Predictive analytics could lead to incorrect forecasting or budgeting decisions. For example, faulty predictive models could recommend excessive risk-taking in investments or poor cash flow management, resulting in substantial financial losses.
Healthcare Predictive Models: Predictive analytics used to forecast patient outcomes or treatment success may be biased or inaccurate if trained on incomplete datasets. This could result in medical misdiagnosis or inappropriate care decisions, leading to patient harm.

Summary of Risks in High-Stakes Domains #risk_summary

Financial Risks: Automated decision-making, inaccurate predictive models, incorrect semantic searches, and poorly secured data integrations may lead to significant financial harm or regulatory violations.
Healthcare Risks: Misinterpretation of patient data, incorrect diagnoses, privacy breaches, and over-reliance on AI-driven decisions can jeopardize patient safety and violate healthcare regulations.
Operational and Compliance Risks: Misconfigured automation, access controls, and cloud storage solutions can expose businesses to legal, financial, and reputational risks in various regulated industries.

Mitigation Strategies #mitigation

To mitigate these risks, FileMaker users should:

Audit and Monitor AI Outputs: Establish human oversight and quality control checks, especially in critical decisions like finance or healthcare.
Secure Integrations: Ensure that all external systems and APIs are securely integrated, with encryption and compliance protocols in place.
Train Users: Provide comprehensive training to ensure that users understand how to use these features responsibly, especially in high-risk areas.
Implement Strong Access Controls: Regularly review and update access privileges, and use multi-factor authentication and role-based access to minimize insider threats.

In conclusion, while FileMaker 2024 offers groundbreaking features, organizations must remain cautious in how they implement them, especially in industries with high regulatory standards and potential for significant consequences. Proper governance, human oversight, and security measures are essential to unlocking the full potential of these technologies without exposing the organization to undue risk.