Who is using MFTF?

First, the Magento Functional Testing Framework

When Magento 2 came out in 2015, there were already a lot of things covered with tests. Magento included unit tests, integration tests, static analysis checks (like PHPCS and PHPStan) and even functional tests with PHPUnit. But navigating the frontend with some browser as a means of testing was not yet included, until MFTF came along.

MFTF bundled Codeception with the Selenium browser automation tool, plus a Magento-specific XML-layer to generate tests in a granular way. Using XML makes a lot of sense: When two Magento modules shape up a certain page, a third party module could be used to change it. With MFTF, the core modules could include XML-based functional tests, while the third party module could simply extend upon this XML, similar to how the third party module was extending the system. Yes, it is XML, but it makes sense because it is extensible.

The complexity is overwhelming. And the performance sucked

MFTF has been difficult to adopt: It's XML includes so many aspects - actors, sections, metadata, pages, data providers and then the real tests - that it is hard to get started with even a simple test like navigating to a certain page. Next to that, the procedure to get started with MFTF has been cumbersome all a long. Every time that I installed it to run it, it took me easily one hour. Usually, I tried to setup a single Magento instance with MFTF installed, to reuse that instance for extension testing over-and-over again.

Next to that, the performance is also bad. Underneath, MFTF uses the Java-based Selenium tool, which used to be great, until other tools like Playwright and Cypress came alone that are way faster. Adding to this, MFTF needs to parse all of the XML files and then generate actual PHP tests, before you can run the tests. It feels all a bit slow.

Running the entire MFTF suite for all core modules took days.

Some history

Years back I was part of ExtDN (which is now in a peaceful passive state), the extension developer network. And there, the role of MFTF was seen as quite important: The only way that extension vendors could claim stability was by focusing upon the interoperability between their module, the core and perhaps other modules. And the only framework flexible enough to allow for this was MFTF.

We worked together with great people from Magento, including Tom Erskine (who maintained and further developed MFTF). For extensions, it made sense.

However, after organizing MageTestFest twice, I saw that MFTF was also being adopted by agencies for custom projects. I always found that strange, even though Magento advocated that MFTF could be used for that purpose too: If you would have a Magento shop with 20 modules and only 1 module would not comply to MFTF already, then it could break the system.

So, who is using MFTF?

The LinkedIn poll showed that 95% of the respondents did not use MFTF. MFTF is not very popular these days. It is almost absent with agencies, it is only used by some extension vendors.

MFTF for whom?

As I have seen it, MFTF was great for extension vendors, but horrible for vendors. MFTF combines tests for the core with tests for third party modules. But as an agency, you don't want to test the core or third party modules, you want to test the customization of the shop. End-to-end tests are therefore much more common with agencies - Cypress, Playwright, etc.

However, with an E2E test, you are only testing the system from the outside. There are still certain parts of the application that you would want to test from the inside: Price calculators, product configurations, checks, validations, filters. For this, functional tests are the way. However, with MFTF, the functional tests are run from the outside (in which they are similar to E2E tests). And I'm referring to the functional tests that are more similar to integration tests.

What is functional testing anyway?

There seem to be multiple definitions of functional testing, but I personally always define it as the assurance that various aspects of the delivered product actually work. From the perspective of Magento (and Magento being the product), MFTF is great for that. If the product is a third party module, an integration test is great. But for custom projects it is either E2E tests or PHPUnit-based tests that are simply run on a copy of the live database.

So, for me, functional tests are very similar to integration tests - they could even include the same code. However, with integration tests, you are taking the core and only a limited subset of modules to see if things are integrating correctly. And a dummy database is handy with that. And with functional tests, you are taking the entire project (with the core and all of the modules installed on top of it). And with that, an actual database makes more sense.

No more MFTF

In the past, I have been explaining people how to use MFTF. However, in real life, nobody used it. Because of this, I'm skipping MFTF in all my trainings and advocating not to use it anymore. There are other better ways.