JetBrains ships three major releases a year, and each version can break plugin APIs. I maintain 22 plugins. That means roughly 132 version combinations to keep working. Most of this is automated, but the automation took a while to build.
Compatibility ranges
Every JetBrains plugin declares a version range in plugin.xml - the oldest and newest IDE version it supports. I target six versions back from the current release. That covers developers who update immediately and developers who stay on last year’s IDE because their company locks the version.
The intellij-platform-gradle-plugin handles most of the build configuration. You set platformVersion in your build script, and Gradle downloads the right SDK. But “it compiles against version X” and “it works in version X” are two different things.
The CI matrix
Each plugin runs tests against six IDE versions in CI. The matrix is defined in the build configuration - one axis for the IDE version, one for the plugin variant when a plugin targets multiple IDEs (IntelliJ, WebStorm, PyCharm, etc.). A full matrix run for a single plugin takes 15 to 20 minutes. Across all plugins, I batch the runs and trigger them per-plugin when a change lands.
The tests are headless. The IntelliJ test framework spins up a minimal IDE instance without a UI, loads the plugin, and runs JUnit tests against real PSI trees. You parse a test file, assert the structure, trigger a completion, check the results. It is slow compared to regular unit tests, but it catches real compatibility issues.
The undocumented breaks
Some API changes are documented in the JetBrains Platform changelog. Many are not. I discovered that a test utility class moved packages between two releases when my CI went red. No migration guide, no deprecation warning - just a compile error.
Another favorite: a method in LightPlatformCodeInsightTestCase changed its return type from a concrete class to an interface. Tests compiled, ran, and then threw ClassCastException at runtime because my test code was casting the result. That one took me two hours to find because the stack trace pointed to the wrong line.
The only reliable defense is running the full test suite against every version. Reading the changelog is not enough.
Marketplace compatibility verification
JetBrains runs their own compatibility check when you upload a plugin. It verifies that your plugin does not use any APIs that are missing in the declared version range. This catches the obvious problems - calling a method that does not exist in an older version. It does not catch behavioral changes, renamed parameters, or methods that exist but behave differently.
I treat the marketplace check as a safety net, not a test suite. By the time I upload, my CI has already tested against every version in the range.
What I would do differently
If I started today, I would set up the CI matrix on day one, before the first release. I built my first plugins without version matrix testing and spent more time debugging user-reported compatibility issues than I spent writing features. Six parallel test runs in CI cost a few minutes of compute. A compatibility bug that reaches users costs a review, a forum post, and a reputation hit on the marketplace.