Plugin Mechanism

[delvh]

What should it be?

• it should allow adding additional functionality to Gitea without bloating the main binary or being responsible for maintaining it
• it should not allow to modify the core functionality of Gitea

Architecture

• Plugins are called using Gos plugin package. (see discussion below, was generally disliked)
• Plugins are called over RCP using the go-plugin package.
• Reasons:
  • active community to maintain the package
  • works on every platform and even across languages
  • minimalistic and understandable -> no technical debt to have to consider
  • OS is responsible for setting plugin resource limits, not Gitea
  • Protocol versioning built-in
• Reasons against that architecture:
  • requests over RCP make everything slow
  • extra dependency
  • vendor lock-in for plugins to go-plugin

Communication: Gitea ➡️ plugin ➡️ Gitea

• common data structures and methods needed
• recommendation: an extra package that both Gitea and the plugins use
  • during application startup, Gitea should use the package from the receiver side and the plugins from the sender side
  • while dispatching calls to the plugins, the roles should be switched around so that Gitea becomes the sender and the plugins the receivers
• things that plugins, as well as Gitea, should have access to (at best only as a wrapper that is set once at runtime because the underlying implementation can change quite often):
  • especially things located in modules/:
  • logger
  • context
  • i18n
• again, to avoid vendor lock-in, this package cannot depend on anything that it doesn't define on its own, or is provided by Go itself
• to reduce the chance of plugins breaking with a new Gitea release, this extra package should be developed to be "all-inclusive" from the beginning meaning that it entails everything a plugin might wish to configure
  • it is recommended that the extra package uses easily extendable data structures, i.e. map[string]?, where possible to avoid breaking changes

Communication: plugin ➡️ Gitea ➡️ plugin

• Let Plugins use the Gitea API when requesting additional information from Gitea. Reasons:
  • easy to implement (literally NO work at all 😃 )
  • easy to maintain (again, because there is literally NO code to maintain 😃 )
  • safe to use for the plugins because the API guarantees stability
  • API gets used more, hence more people want to see it well-maintained and there is an actual incentive to add an API for a feature that at that moment only exists for the web UI
• Reasons against this approach:
  • slow, because it will always require HTTP requests
  • plugins need to parse the answer instead of simply being able to use a predefined method/ datatype

Plugin Lifecycle (What the plugins can set/ "use")

• Plugin Initialization
• Application is completely initialized and running - Hook
• Plugin Dependencies (to Gitea (versions), to other plugins?)
• Setting Plugin-wide settings as well while parsing the server-wide settings (requires (a) section(s) reserved for plugins)
• Custom translation keys to allow for localized new features
• Adding custom tabs to different units:
  • most basic:
  • user
  • org
  • user+org
  • all repos
  • public repo
  • private repo
  • additionally possible:
  • issue
  • PR
  • projects
  • packages
  • ...
  • to make parsing this easier, it would probably be good to use an approach that allows setting a list of tags where this tab will be shown. Problem: How to handle OR/ AND relations (depending on the chosen algorithm)?
• Adding consistency checks
• Cron Job Hooks?
• Shutdown Hook

What data is needed by every/ some plugin(s)?

• unique plugin name (i.e. by following reverse naming such as io.gitea.example_plugin)
• human printable plugin name (i.e. Example Plugin V1)
• plugin version in semver form
• list of core units affected by this plugin (rather as a code of honor the plugin intends to keep than a security feature)
• dependencies to other plugins
• custom translations/ translation files
• custom settings/ setting files
• custom web routes
• custom API routes (problem: how to update the swagger file with the new operations?)
• provided hooks
• minimum Gitea version the plugin needs

Why this long text rambling?

1. Everyone should know without a doubt how it should look like in the end
2. To make the work for the (eventual) reviewers easier by simply letting them cross-check the actual functionality with its intention
3. Most of the text I wrote here can be copy-pasted into the docs, making it easier for me to write the docs
4. To allow for discussion if I missed something while planning this feature
5. To see if the scope others expect from this feature differs from what I imagine from it

[42wim]

I think https://github.com/hashicorp/go-plugin should also be taking in consideration, afaik the "normal" plugin system has a lot of shortcomings (eg crashing the whole process because of issues with the plugin), that the Hashicorp one does not suffer from.

See also:

• https://www.youtube.com/watch?v=SRvm3zQQc1Q
• https://alperkose.medium.com/things-to-avoid-while-using-golang-plugins-f34c0a636e8
• https://www.sobyte.net/post/2022-03/golang-plugin/

[delvh]

Yeah, I've also seen their implementation when looking for a plugin system.
However, to me, it appears as if they don't support executing dynamic functions as I think I read the sentence sending functions is not possible anywhere, and then I stopped reading and didn't investigate further as that is something we most likely need.
After having read through their docs for a little, it seems I was a bit too fast with that.

Regarding the unexpected crashes: I intended to call every plugin by using something like

func PluginCall(function func(plugin *Plugin)err) {
  for _, plugin := range plugins {
    executePluginCall(plugin)
  }
}

func executePluginCall(plugin *Plugin, function func(plugin *Plugin)err) {
  defer func(){
    if err := recover(); err != nil { // Error handling
    }
  }()
  function(plugin)
}

and thus mitigating the potential for those errors.

So, the question is now, should we use the RPC version which might offer a few benefits and higher latency, or do we stay with the Go native approach that we can completely customize to our liking?

[Gusted]

However, to me, it appears as if they don't support executing dynamic functions as I think I read the sentence sending functions is not possible anywhere, and then I stopped reading and didn't investigate further as that is something we most likely need.

Do you want to compile/execute Go code on-the-fly?

[delvh]

Not really, but we certainly need to handle the custom routes which can best be achieved dynamically, and making this interface-based might be possible, but (a little bit) more work.

[Gusted]

Wish you many luck with this advanced logic.

[lunny]

I prefer the plugin system based on javascript or nodejs because that will allow it include frontend code.

Possbile library be used. https://github.com/dop251/goja, there is also a plugin support nodejs .

[delvh]

Interesting how many different views there are regarding this feature.
Now we have three completely different options and no agreement on what to use...

[lunny]

Interesting how many different views there are regarding this feature. Now we have three completely different options and no agreement on what to use...

Use javascript based plugin system, we can implement a plugin market and one click installation. That's impossible for Go based plugin system.

[delvh]

The problem I can see is: We do not even have the same definition of what we want in the end:
What you want seems to be a per-user plugin mechanism, what I thought of was a per-instance plugin mechanism...

[lunny]

The problem I can see is: We do not even have the same definition of what we want in the end: What you want seems to be a per-user plugin mechanism, what I thought of was a per-instance plugin mechanism...

I don't think so. I think I'm also talking about per-instance plugin. If I made you confusing, I'm sorry.

[jolheiser]

-1 for the go plugin package.
It can't be removed per compatibility promise, but it is absolutely not stable, and there are far too many caveats to get it working properly.

[lunny]

I think we can begin from a theme-kind plugin, which is simple and useful. A plugin could be a zip/tar file with a main.js or index.js and other files like theme.sass, theme.css and images and etc.

[wxiaoguang]

I do not think the Go's plugin package is the future. Is there any success story of using it?

Most large Go projects uses gPRC for inter-process commutation, including plugins, but it's also heavy.

[delvh]

Ok, seems as if the plugin package is generally disliked, so I'll exclude it from the possible options.

I'm still not a fan of a purely frontend-based plugin mechanism as that will have many shortcomings of its own, and I don't know if

plugin market and one click installation.

should be the deciding factor for how to implement the plugin system...
Server admins should be sure of what they want for their instance, and not install it with a simple misplaced click.
Also, even with the Go-based approach, a plugin "market" would be possible as a third-party tool that simply stores the executables and perhaps some other information about the plugin...

So, as I can see that now, the two possible options are

• a backend-based plugin mechanism (most likely using go-plugin)
• a frontend-based plugin mechanism (no idea how to implement that - I didn't see anything practical in the link you sent, @lunny ?)
• Did I miss something else?

[luwol03]

Isn't this library lunny suggested meant for a backend-based plugin mechanism? But instead of writing go you can write js? Or would that be to difficult to build a bridge between the Gitea go code and js code?

[lunny]

Yes, it could expose structs, methods to frontend. i.e. If we expose getdb method, then we can operate database with javascript.

[lafriks]

I think the best way to go would be using grpc so that plugins could be implemented using any language. Plugin could return asset files on init (JavaScript etc) that would need to be injected/used in specific places

[jolheiser]

I would also like to throw yaegi into the mix.
I think gRPC is probably the only way to go (unfortunately), but figured yaegi is worth considering as well.

[Mai-Lapyst]

Honestly, I would like to see the native go plugin implementation.
Yes it has drawbacks, but it would be fast and admins had full controll over what to use.

But since thats off the table for now, I would rather go with the gRPC approach, since its more open (i.e. you can choose more languages) and imo more secure then javascript.

The problems I see with js plugins is, that it lead to a huge security risk, since js can be so minified or encrypted that you not even need any letters (only parentheses) to write correct js. We then would need an very strict system so no js can accidentally just execute "DROP ALL TABLES;" (or similar) on startup.

Also an interpreter (of any kind) in gitea would bring back the risk of one plugin crashing the whole instance, or to eat up so much performance that gitea stops responding (i.e. a infinite loop in a script somewhere). To mitigate this we then would need to sandbox the plugins into own processes, which would nearly the same as the gRPC approach but with less freedom.

[delvh]

To be fair, you can also DROP ALL TABLES; in Go or consume all available memory.
But yeah, I can see your point.

[luwol03]

Then you have to think twice about an official plug-in market like lunny suggested. Official also indicates that they are save for usage and have no malicious intentions. An easy one click install could potentially break servers from not experienced users who don't understand what the plugin is doing and just pressing install.

To prevent this, this would at least requires some reviewing and analysis of newly published plugins, which is time expensive.

[lafriks]

We could probably get some ideas on how mattermost have implemented their plugin system.

[jolheiser]

We could probably get some ideas on how mattermost have implemented their plugin system.

For posterity:
mattermost plugin package
It looks like they use go-plugin

[mscherer]

Another inspiration could be woodpecker proposal around extensions woodpecker-ci/woodpecker#915

[Trendyne]

Since nobody has mentioned it yet, projects like Adobe Lightroom, MPV, or Wireshark use Lua for this. You'd read and run Lua files from a directory where they would register events with callbacks or maybe return properties and functions. It's very light. simple, and fast but would make it very easy to make plugins and shouldn't be any harder to implement

Made up possible example:

-- Greet users on their first repo
gitea.on("newRepo", function(repo)
	if #repo.owner.repos == 1 then
		local msg = ("Welcome to our Gitea, %s!"):format(repo.owner.name)
		repo.owner:sendNotifacation(msg)
	end
end)

[KN4CK3R]

The Lua integration could be plugin too which forwards callbacks to Lua.

[Lord-Leonard]

Are there any updates on this one? ^^

[lunny]

For most backend tasks, Gitea Actions can be considered as somewhat plugins.

[Lord-Leonard]

Yeah for my usecase the webhooks are sufficient. I will just have to host another small webserver alongside Gitea.
But it would have been nice to just write a plugin and call it a day :)

[Sharaf5]

Then there are 4 point of views could be considered for plugins

• Compile directly inside source coed
• RCP called via go-plugin
• Gitea Actions can be considered as plugins
• Webhooks

and I will add the 5th

• APIs or Sockets with separate frontend from back "PWA Frontend"

[lunny]

Looks like Typescript 7 will be migrated to Go. So that maybe we can use typescript as the plugin language. https://github.com/microsoft/typescript-go

[wxiaoguang]

Looks like Typescript 7 will be migrated to Go. So that maybe we can use typescript as the plugin language. https://github.com/microsoft/typescript-go

typescript-go is just a TypeScript compiler implemented in Go with a LSP. Where do you see that "use typescript as the plugin"?

---

To be frank:

For the "plugin" proposal, I do not see anyone is really working on it, or whether it is feasible at current stage (my answer is "no": either gPRC or js/wasm/lua would involves a lot of work to design the structs and interfaces, no manpower).

Mission Impossible

So I would suggest to mark the proposal as "close as not planned" since it doesn't seem able to get real progress (reopen at any time when there is a feasible design and manpower)

For some existing requirements:

• Frontend customization: include custom JS scripts into pages by "custom template"
• Backend customization: webhook & API, or build from source with custom code

[wxiaoguang]

Quote above: So I would suggest to mark the proposal as "close as not planned" since it doesn't seem able to get real progress (reopen at any time when there is a feasible design and manpower)

Golang eco-system is naturely not easy with plugins .... unfortunately.

[hiifong]

Maybe we can learn from git's plugin model, where commands that start with git- can be called freely by git, e.g. I have a git-bug plugin, and I can call my git-bug plugin by calling git bug.

[Sharaf5]

most top named Go/C/C++ projects support plugins including

• CoreDNS : pre-compiled, selectively activated Go plugins
• Traefik : Transpiling GO on the fly ( not recommended for performance ), and WASM ( may be considerd )
• Nginx : pre-shared lib .so
• PgSQL : plugable extensions ( citus, timescaleDB, ... )
• WebHooks could be considerd a plugin mechanism for both front and backend
• Blender, GIMP, VSCode, ... had plugin/estension systems

there were many success stories and a cost effective announce using plugins opening the way to be integrated into other projects
Really, closing it disappointed me, but hope this is in mind for future at least
I wished this project could grow beyond it's limit
hope you best luck

[wxiaoguang]

most top named Go/C/C++ projects support plugins including

These examples you shown, are either desktop/traffic-server or C++ applications (Golang plugin doesn't work on Windows, but Gitea still needs to support), or the plugin doesn't need to heavily depend on the host application (traffic server: CoreDNS, Traefik). And either they have a large company & team, or a lot of active contributors to maintain.

For Gitea's plugin: Gitea itself is a heavy web application (but not a traffic server), there are more business logic than others, the proposed "Plugin Lifecycle" is already more complex than the traffic server examples. And Gitea community is not that large as others (that's what I meant "my answer is no: either gPRC or js/wasm/lua would involves a lot of work to design the structs and interfaces, no manpower").

---

WebHooks could be considerd a plugin mechanism for both front and backend

Actually it doesn't need a general plugin system, but you see that the webhook related PRs are still stale for long time (WIP: Custom Webhooks #19307). As simple as this one doesn't get success, how would a general plugin proposal succeed?

I don't want to disappoint anyone, but I think we need to face reality.

And see my comment above: reopen at any time when there is a feasible design and manpower

[wxiaoguang]

Let's keep it open and don't disappoint anyone. Let's hope there would be some real progresses.

[hiifong]

Maybe we can learn from git's plugin model, where commands that start with git- can be called freely by git, e.g. I have a git-bug plugin, and I can call my git-bug plugin by calling git bug.

A successful case of a go project: github cli, But this plugin mode only supports command line calls

[wxiaoguang]

Maybe we can learn from git's plugin model, where commands that start with git- can be called freely by git, e.g. I have a git-bug plugin, and I can call my git-bug plugin by calling git bug.

A successful case of a go project: github cli, But this plugin mode only supports command line calls

You need to find some "rich web application" to learn.

[Sharaf5]

And either they have a large company & team, or a lot of active contributors to maintain.

I'm mainly a biology teacher and server admin, while I'm tring to learn Go I go slowly so my kwnolage is under just entry level or below ( gpt aided ), so I understand your point here, unfortunatly for now I can only help with ideas but we cant stop building next roadmap steps that it's just somehow far ( you may consider good suggestion for juat undetermined version )

plugin doesn't need to heavily depend on the host application (traffic server: CoreDNS, Traefik)

you are right, I previously raised an issue here to enhance Frontend ( #31125 ) and separate it from backend so the backend just work independantly and could be integrated in any frontend - this may make the application focus more on functionality especially if PWA / SPA is considered for frontend as it will reduce SSR ( server side rendering ) and minimize validation to AuthN, AuthZ

• maybe version 3 or 4

"Plugin Lifecycle" is already more complex

yes but may introduce many panels to go for git workflow based deployments ( https://chatgpt.com/share/67f13ee5-3334-8005-82e8-7d087c1cb51b ) integrating gitea and making more developer wanting to contribute and develop more ( separate plugins )

• I really have some ideas about a good plugin system but that involves making gitea a core project for git as it is and any additional thing is a statically loaded dynamically enabled plugin but I didn't have success on implementing that myself ( Nginx plugins is the most neer to that )

I don't want to disappoint anyone, but I think we need to face reality.

And see my comment above: reopen at any time when there is a feasible design and manpower

you are thinking well,
I just expressed what in mind "when I said disappointed", as I realy love this project and feal it's mostly the future of git servers