ExtDN - Do use composer

The rule of thumb: Do use composer

To quote from our ExtDN whitepaper first: "Use composer packages to distribute (especially commercial) extensions. For a local environment, it is fine to develop your own code under app/code. However, once you distribute your module to other environments, it should be through composer as otherwise dependencies are left unmanaged. In a production environment, the app/code folder should therefore ideally be empty."

Devil in the details

So the first rule we wanted to lay down is simple: Do use composer. It seems obvious, right? This might seem like a really basic thing to say, maybe something for newbies: But there is more to say about this. The devil is in the details.

Distribute extensions via composer

The first sentence starts boldly: Use composer packages to distribute extensions. There is no discussion here. All third party extensions should be distributed through composer. If you download a free extension from somewhere, there should be absolutely no reason why there is no composer package attached to it.

No more copying

If you bump into a free extension that requires you to copy the code to app/code, it tells you that the extension developer makes a crucial mistake (and there might be other mistakes as well).

Benefits

This is important: Use composer. Why? Because it gives so many benefits. Every extension is always depending on the Magento core. This might be just the framework, but there might well be dependencies with specific core modules or perhaps even PHP modules.

Those dependencies can be tracked using composer. With composer, you can guarantee those dependencies are working for you - both in development and in production.

The only thing that an extension developer needs to do is add a composer.json and you're done.

... especially commercial ones

However, the phrase also includes the phrase "especially commercial ones". This makes it more difficult. Commercial extensions are (usually?) not placed on GitHub or BitBucket, but their code is hidden by default. Perhaps you purchase a lifetime license to access its source code, perhaps it is accessed through a support subscription. But it does not mean that commercial extensions are not bound to our composer rule.

No negotiation

To the contrary: Exactly because it is a commercial extension, composer should be used. Not using composer could easily lead to more support issues - frustrating for both the buyer and the seller. And if the extension would be encrypted (which is yet a completely different story to discuss), using composer becomes even more important.

To sell extensions means that you should add quality service for that extension as well. Not offering composer with your commercial extension means that your extension tracks dependencies in a worse way than free extensions. How weird is that?

Why is composer not used for commercial extensions?

Unfortunately, in the field, this seems pretty hard to solve. Composer allows you to download packages from a specific resource. But commercial extensions require that resource to be protected by some kind of mechanism, otherwise the sources would be given away for free. And in practice, a lot of Magento extension developers do not go the extra mile to make sure a composer mechanism exists.

To give extension providers some inspiration, here are some options to offer commercial extensions via composer:

Solution: Use Magento Marketplace

The most obvious solution: Use Magento Marketplace. When you install Magento, it will connect to the Magento Marketplace by default through specific API credentials. If you want to distribute your own commercial extension through Marketplace, you will need to comply with their revenue sharing model and go through their Extension Quality Program (which is a good thing anyway).

Once you work along with Magento on this, they sort out all of the other details. Installing an extension that is hosted on Marketplace is peanuts.

Solution: Download artifacts

One way is to let customers download a ZIP archive of the extension. The ZIP includes composer.json file. Next, this ZIP archive is added to the Magento git repository (for instance var/packages). And then the composer configuration uses that var/packages as an artifact repository. Fooman is using this approach.

Solution: Subscription-based composer URLs

Yireo itself is using another solution: Customers buy software subscriptions to gain access to extension sources. This is done through an API key which is used to generate a subscription-specific URL. And that URL is simply used as another composer repository.

To make this happen, we have built our own API gateway (based on a couple of PHP scripts) to mimic the behaviour of something like Packagist. I believe the Magento Marketplace is using something similar.

Solution: Use SSH public keys

Another method of authentication might be SSH public keys. Using SSH public key authentication is among System Integrators a proven way of dealing with deployment of projects and packages. It is supported by GitHub, BitBucket, GitLab and others. The difficulty here is that commercial extensions might be sold quite often, meaning that you don't want to manually add keys.

This could be solved by creating your own SSH public key upload form (perhaps making sure keys are correct through validation using the PHP SSH2 extension). Or use the APIs of GitHub, BitBucked and GitLab to assign read-only members, create access tokens or something similar. It is really cool to build something like this, but hard work.

Solution: Private Packagist or Satis

Instead of building a private composer repository yourself, you can also build it yourself using Satis. Out of the box, Satis does not offer that much for creating private repositories (SSH public key authentication, tokens). It is kind of a variation of the previous suggestions. But there are options.

Another option might be Private Packagist (built by the same guys that give you composer and Satis): A paid solution which gives you a complete API to handle access in a really solid way. This is awesome stuff.

Solution: Let your customers sort it out

Finally, an option might be to let your customers sort it out themselves. If you simply offer a ZIP file for download, include a composer.json file with a proper version in it. And then instruct your customers on how to create their own git repository, add your extension sources to it, perhaps tag it with a specific git version and then use it through composer.

This is not user-friendly and it is actually requiring the System Integrator to fix a problem that you created. I would definitely not recommend this approach. However, I do instruct my own customers how they can use this approach, on top of the solution of using the Yireo API (as mentioned above).

Local development under app/code?

Moving on with the ExtDN rule. It mentions that "for a local environment, it is fine to develop your own code under app/code". Many developers do this. Heck, I do this myself as well. Especially with code that I'm not sure enough about. I often create litle modules that might sometimes be deleted again after some time. It is fine to develop code in app/code.

Distribute through composer only

"However, once you distribute your module to other environments, it should be through composer as otherwise dependencies are left unmanaged." Do you get why? If you build your own local module under app/code and it depends on the Magento_Wishlist module, how do you document this dependency? If you also deploy the same module in app/code to production, but at the same time a collegue disables the Magento_Wishlist module, it might break things.

This is exactly the reason why I believe we should use composer.

Using app/code is cheap. But if your code grows, sooner or later more dependencies are added (Magento core modules, your own submodules, PHP modules, PHP versions) and those dependencies become a risk if you keep that code in app/code.

Make changes within the folder vendor?

An alternative is to develop under vendor. This is a bit harder, because now, every time you want to push to production, you need to commit your code in your own vendor folder, push back to the repository, update the composer.lock file, commit that. At least, this used to be my own stupid workflow.

There are smarter ways of dealing with this (like submodules, composer path, symbolic links). My main point is: You can actively develop your own code and still distribute it to production using composer.

An empty app/code

"In a production environment, the app/code folder should therefore ideally be empty."" Perhaps also in your local environment (because they should be duplicates?). But if you see code in app/code in production, you can bet on it that sooner or later this will bite you. And why take that risk?

Specifically this point lead to discussions in the community. Some system integrators build so many custom modules for specific projects that managing composer dependencies actually becomes a nightmare. In those specific situations, it might be easier to have custom modules placed in app/code. However, the main point still stands: Ideally, it is empty so that composer is doing its jobs. Having so much custom code leads to the less ideal situation that the tool composer becomes too hard to use.

Conclusion

I hope you appreciate this blog. It is a long read. But it also shows how powerful and flexible composer can be. And why there should be little reason to distribute extensions in another way. The ExtDN flyer of 8 points contains yet 7 other interesting points. I will be blogging about these soon!