Skip to main content
  1. Posts/

Plugins in .NET: Creating the plugin host

Preface #

This is section relates to Coral, a self-hosted music streaming platform I’m open-sourcing in May-June 2023. Feel free to skip the preface if you just want to know how the plugin system works.

One of the gripes I had with using Plex as a music player, was that it by default, matched my music collection with existing releases on MusicBrainz and used their metadata instead of mine. I’ve spent countless hours curating and tagging my music collection just the way I want it so it was quite annoying to see how it had been butchered by MusicBrainz’s metadata. Tags had been ruined and a few releases were incorrectly matched. I’m all for sane defaults that work for the majority of people, but I would’ve appreciated a heads-up before matching my library with MusicBrainz.

I believe that users should be able to explicitly opt-in to using their third-party integration of choice for both privacy and usability reasons. That is why I’m creating a plugin system for Coral, which ensures that users are able to pick exactly what integrations they want to run - or even make their own should they wish to.

Introduction #

What are plugins? Plugins are extensions that are dynamically loaded into an application to extend its features.

Microsoft’s documentation on how to create a plugin system will only take you so far, once you start introducing custom service providers and dynamic loading of ASP.NET Core controllers, you’re on your own. This is going to be quite a long article with a lot of code, so get comfy.

Let’s take a deep dive into how it all works.

System requirements #

There are a few things to consider when you’re building a plugin system.

Your plugins will need to know about what’s going on within your application to be able to provide supplemental features. How will they do that?

What services should the plugins be able to access - do you want them to be able to read your application database directly? How are your users going to interact with the plugins once they’re loaded? Do you want to be able to dynamically load and unload plugins at runtime or are application restarts OK? How should the plugins store their configuration? The list goes on.

In Coral, I want to be able to dynamically load and unload plugins without needing to restart the application. I also wanted plugins to be configured and interfaced with via ASP.NET Core controllers, so I could dynamically build configuration pages for the plugin in my frontend. I initially don’t want the plugins to be able to tamper with a user’s database before I have a permission system in place - and rather create their databases should they need to store data.

Security caveats #

There are a few security caveats to keep in mind when you’re supporting plugins, especially when it comes to dynamically loaded controllers. Once the plugin controller is loaded into the host, it has access to every single service the host is able to access via ASP.NET Core’s HttpContext.

However, implementing mitigation tactics are beyond the scope of this article.

Figuring out how to solve our system requirements #

To start off, let’s look at what we’ll need to build a plugin that logs information about the currently playing song and sends it to Last.fm. This logging operation is called a “scrobble”. What information does the Last.fm API need to be able to register a scrobble?

According to their API docs, we need the following data:

From the Coral API:

  • Artist name
  • Track name
  • Timestamp of when the playback started

From the user:

  • An API key

From the plugin itself:

  • A user session
  • An API request signature generated by the plugin

This means that we need 3 things:

  1. A way for the host to communicate with the plugins
  2. A way for users to configure the plugin
  3. A way for plugins to persist configuration data, in this case a user session to Last.fm

Now that we know our requirements, let’s take a look at the build plan.

Build plan #

This is what we’re going to be building today:

  • A plugin loader which will be responsible for dynamically loading and unloading plugins
  • A plugin host, which keeps track of all the loaded plugins as well as their dependencies
  • An event system that is used to communicate between the host and its plugins
  • Finally, the Last.fm plugin itself

Plugin loader #

The plugin loader is responsible for finding the compiled plugins are loading them into the application. Plugins are distributed as compiled assemblies. What are assemblies and how are they loaded?

What are assemblies and how are they loaded? #

Assemblies are the fundamental units of deployment, version control, reuse, activation scoping, and security permissions for .NET-based applications. An assembly is a collection of types and resources that are built to work together and form a logical unit of functionality. Assemblies take the form of executable (.exe) or dynamic link library (.dll) files, and are the building blocks of .NET applications. They provide the common language runtime with the information it needs to be aware of type implementations.

Every .NET 5+ and .NET Core application implicitly uses AssemblyLoadContext. It’s the runtime’s provider for locating and loading dependencies. Whenever a dependency is loaded, an AssemblyLoadContext instance is invoked to locate it.

  • AssemblyLoadContext provides a service of locating, loading, and caching managed assemblies and other dependencies.
  • To support dynamic code loading and unloading, it creates an isolated context for loading code and its dependencies in their own AssemblyLoadContext instance.

Taken from Assemblies in .NET and Understanding AssemblyLoadContext.

So, to recap, our plugins are compiled into .dll files and we can create a class that inherits from AssemblyLoadContext load them.

Loading plugins #

To be able to detect if an assembly contains a plugin, we need to create a common interface for all plugins that contains some metadata about them and some setup logic.

public interface IPlugin
{
    string Name { get; }
    string Description { get; }

    public void ConfigureServices(IServiceCollection serviceCollection);
}

The plugin host will maintain a service provider for each plugin to be able to support dependency injection within the plugin itself. We’ll explore how that works later.

Plugins and their dependencies are placed into sub-folders within the main plugin directory, which is how the PluginLoader can find plugins and their dependencies.

➜ Local/Coral/Plugins tree
├── LastFmConfiguration.json
├── LastFmUser.json
└── lastfm
    ├── Coral.Plugin.LastFM.deps.json
    ├── Coral.Plugin.LastFM.dll
    ├── Coral.Plugin.LastFM.pdb
    ├── Coral.Plugin.LastFM.runtimeconfig.json
    ├── Microsoft.Extensions.DependencyInjection.Abstractions.dll
    ├── Newtonsoft.Json.dll
    └── RestSharp.dll

Now that we have a plugin interface, let’s take a look at how we load them.

public class PluginLoader : AssemblyLoadContext
{
    private readonly ILogger<PluginLoader> _logger;
    public PluginLoader(ILogger<PluginLoader> logger) : base(true)
    {
        _logger = logger;
    }

    public (Assembly Assembly, IPlugin Plugin)? LoadPluginAssemblies(string assemblyDirectory)
    {
        (Assembly Assembly, IPlugin Plugin)? assemblyGroup = null;
        // load the plugins with their dependencies
        foreach (var assemblyPath in Directory.GetFiles(assemblyDirectory, "*.dll"))
        {
            // if PluginBase is present, skip loading
            if (Path.GetFileName(assemblyPath).StartsWith("Coral.PluginBase"))
            {
                _logger.LogWarning("Coral.PluginBase assembly detected, please remove from plugin folder. " +
                                                    $"Skipping load of: {assemblyPath}" +
                                                    " to ensure plug-in can load.");
                continue;
            }

            var assembly = LoadFromAssemblyPath(assemblyPath);
            try
            {
                var types = assembly.GetTypes();
                // if assembly has more than 1 plugin,
                // throw exception about poor design.
                var pluginCount = types.Count(t => typeof(IPlugin).IsAssignableFrom(t));
                if (pluginCount > 1)
                {
                    throw new ConstraintException("Cannot load assembly with more than 1 plugin." +
                                                    " Please separate your plugins into multiple assemblies");
                }

                // if assembly has no plugins, continue, it's a needed dependency
                if (pluginCount == 0)
                {
                    _logger.LogDebug("Loaded plugin dependency: {assemblyName}", assembly.GetName().Name);
                    continue;
                }

                var pluginType = types.Single(t => typeof(IPlugin).IsAssignableFrom(t));
                var plugin = Activator.CreateInstance(pluginType) as IPlugin;
                if (plugin != null)
                {
                    _logger.LogInformation("Loaded plugin: {name} - {description}", plugin.Name, plugin.Description);
                    assemblyGroup = (assembly, plugin);
                }
            }
            catch (ReflectionTypeLoadException)
            {
                _logger.LogWarning("Exception thrown loading types for assembly: {AssemblyName}", assembly.GetName().Name);
            }
        }
        return assemblyGroup;
    }
}

You may have noticed that the PluginBase assembly is ignored by the plugin loader. The assembly loader views types based on which assembly they come from. If the type belongs to an assembly that was loaded outside the default load context, it is seen as a separate type from a separate assembly and cannot be resolved as a type belonging to an assembly loaded by the default load context.

TL;DR: don’t load the assembly holding the base types and we’ll be OK.

Now that the plugin assembly is loaded, how do we use the types within the assembly?

Plugin host #

The plugin host is fairly big and complex. Let’s break it down into smaller digestable pieces, starting with its interface.

public interface IPluginContext
{
    public void UnloadAll();
    public void LoadAssemblies();
    public TType GetService<TType>()
        where TType : class;
}

The PluginContext class is responsible for loading and unloading plugins via the PluginLoader and providing access to dependency injection containers for each plugin.

private readonly ConcurrentDictionary<LoadedPlugin, ServiceProvider> _loadedPlugins = new();
private readonly ApplicationPartManager _applicationPartManager;
private readonly MyActionDescriptorChangeProvider _actionDescriptorChangeProvider;
private readonly IServiceProvider _serviceProvider;

The class has the following class members:

  1. ConcurrentDictionary which keeps track of the loaded plugins and their respective ServiceProviders
  2. The ASP.NET Core ApplicationPartManager to load new controllers dynamically
  3. An internal ASP.NET Core mechanism to notify the framework of controller changes - magic
  4. The host’s configured ServiceProvider instance, this’ll make more sense in a second

Let’s look at how the LoadAssemblies method is implemented.

public void LoadAssemblies()
{
    // load plugin via PluginLoader
    var assemblyDirectories = Directory.GetDirectories(ApplicationConfiguration.Plugins);
    foreach (var assemblyDirectoryToLoad in assemblyDirectories)
    {
        var pluginLoader = new PluginLoader(_pluginLoaderLogger);
        var loadedPlugin = pluginLoader.LoadPluginAssemblies(assemblyDirectoryToLoad);
        if (!loadedPlugin.HasValue)
        {
            continue;
        }

        var storedPlugin = new LoadedPlugin()
        {
            LoadedAssembly = loadedPlugin.Value.Assembly,
            Plugin = loadedPlugin.Value.Plugin,
            PluginLoader = pluginLoader
        };

        var serviceCollection = ConfigureServiceCollectionForPlugin(storedPlugin.Plugin);

        // get controller from plugin
        // note that if a plugin has multiple controllers, this will allow them all to load
        // even if only one of them is a subclass of PluginBaseController
        var controller = loadedPlugin.Value.Assembly.GetTypes().SingleOrDefault(t => t.IsSubclassOf(typeof(PluginBaseController)));
        if (controller == null)
        {
            return;
        }
        // load controller assembly
        // build service provider for assembly
        var serviceProvider = serviceCollection.BuildServiceProvider();
        // load assembly into MVC and notify of change
        _applicationPartManager.ApplicationParts.Add(new AssemblyPart(storedPlugin.LoadedAssembly));
        _actionDescriptorChangeProvider.TokenSource.Cancel();
        _loadedPlugins.TryAdd(storedPlugin, serviceProvider);
        // finally, register event handlers
        RegisterEventHandlersOnPlugin(serviceProvider);
    }
}

This method is responsible for loading the plugin assembly, creating a service provider for the plugin and then dynamically loading the controller using the ActionChangeDescriptorProvider. Finally, it registers the event handlers described by the plugin.

I found a clever solution online to handle dynamic controller loading, but it didn’t exactly explain how it all works, so here I shall try my best to do so.

Loading API controllers at run-time #

// http://msprogrammer.serviciipeweb.ro/2020/09/28/asp-net-core-add-controllers-at-runtime-and-detecting-changes-done-by-others/
public class MyActionDescriptorChangeProvider : IActionDescriptorChangeProvider
{
    public static MyActionDescriptorChangeProvider Instance { get; } = new MyActionDescriptorChangeProvider();

    public CancellationTokenSource TokenSource { get; private set; } = default!;

    public IChangeToken GetChangeToken()
    {
        TokenSource = new CancellationTokenSource();
        return new CancellationChangeToken(TokenSource.Token);
    }
}

Which is then registered with the main application’s ServiceCollection.

builder.Services.AddSingleton<IActionDescriptorChangeProvider>(MyActionDescriptorChangeProvider.Instance);
builder.Services.AddSingleton(MyActionDescriptorChangeProvider.Instance);

This class is then triggered in the plugin host like this:

_applicationPartManager.ApplicationParts.Add(new AssemblyPart(storedPlugin.LoadedAssembly));
_actionDescriptorChangeProvider.TokenSource.Cancel();
The creator of ASP.NET Core, David Fowler, shows a different way of doing this exact thing in a GitHub Gist written in November 2020.

This all seems like a ton of magic, so let’s break it down.

How does this thing work? #

A method that the framework can expose as a route is an action. These actions commonly return IActionResult.

An ActionDescriptor can be described as follows:

Provides information about an action method, such as its name, controller, parameters, attributes, and filters. https://learn.microsoft.com/en-us/dotnet/api/system.web.mvc.actiondescriptor?view=aspnet-mvc-5.2

Describes an MVC action. https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.mvc.abstractions.actiondescriptor?source=recommendations&view=aspnetcore-7.0

These ActionDescriptor instances are cached by ASP.NET Core once controllers are loaded. This cache can be invalidated and re-created using IActionDescriptorChangeProvider.

An IActionDescriptorChangeProvider is described as follows:

Provides a way to signal invalidation of the cached collection of ActionDescriptor from an IActionDescriptorCollectionProvider.

https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.mvc.infrastructure.iactiondescriptorchangeprovider?view=aspnetcore-7.0

This is the key to dynamically adding new controllers. By adding a new application part via the plugin assembly, we tell ASP.NET Core to load the plugin controller. Then via _actionDescriptorChangeProvider we tell the framework to invalidate its ActionDescriptor cache, making the new controllers accessible.

This was one hell of a puzzle to figure out, but so satisfying when it finally worked! My friend Kieran Foot came up with an idea of writing a method that would proxy routes for the plugin’s controller actions, which I implemented and got to work, but I wanted to figure out if I could load the whole controller and also get an updated OpenAPI definition that I could consume in my frontend.

Conclusion #

To recap, to load and write plugins, you must implement a common interface that all your plugins implement. Then, create a class inheriting from AssemblyLoadContext to load your compiled plugin assemblies, which is managed by a plugin host, which keeps track of every load context to ensure unloadability. The plugin host keeps track of all the loaded plugins and maintains service providers for their dependencies.

There are still huge parts of the plugin system that I have yet to cover, but this should give you enough information to build your own system. If you’re still interested in how the rest of the system is built, let’s dive into the second part of the series, creating a plugin. It’ll answer burning questions such as “why the heck was the host ServiceProvider injected into the plugin host??” and “why even maintain multiple ServiceProviders?”.

I hope you liked what you saw so far, see you in the next article!