Thanks for logging this one, I have subscribed to the post and will follow along.

I am not at that scale yet, but researching how well ArgoCD handles these kinds of numbers for when that time comes. Beyond the slow UI loading have you found it handles 2000+ without too many issues? What sort of system resources does your instance use?

Server-side pagination was implemented by Akuity (see their blog post https://akuity.io/blog/akuity-july-2023-update/). When we asked on Slack, we were told that the team at Akuity was planning on upstreaming it in around a month (see discussion here.

That's great news, really!
Currently with 4K applications, the UI (/applications) is clearly no longer usable.

@alexymantha Looking forward to the upstreaming 🎉

@alexmt's original estimate would put this in time for 2.9, so setting it for that milestone for now. :-)

I really hope to see it backported for 2.7 release

@Whisper40 I haven't seen the code, but my guess is that it was fairly involved. I kind of doubt we'll be able to backport, but will certainly consider it.

In good news, 2.8 seems to be pretty solid, and I expect 2.9 to be a relatively light release. So hopefully the upgrade path will be painless.

@crenshaw-dev @alexymantha Hi, it seems that the 2.9 release has been published but i don't see any mention about pagination. Is it normal ? (Oh it seems it's for 2.10 :( )

@Whisper40 I don't think this has been announced to be included in any particular release yet.

It looks like it is going to be released in v2.10 Argo CD Roadmap (view)

The Akuity folks have told me they plan to get this into 2.10, so I've added the ticket to that column in the project. It's not a promise, just an indication of the current plan. :-)

Any progress on this from Akuity?

I'd say, if they won't merge their patch soon it can be someone else with a completely different implementation and then Akuity would have a hard time to maintain their fork.

@dyatlov are you signing up for the task?

Sorry, for the delay with the PR. We ran into several issues along the way and had to take a step back to ensure good quality. As a short summary:

• CLI changes were absent. Akuity doesn’t provide customized Argo CD CLI to customers, so no related changes were implemented.
• Pagination is leveraging direct DB query to the etcd database. This is possible only if K8S is using KINE with DB as a backend. So this part has to be reworked as well.

Since the implementation is more involved than expected, I’ve started with a proposal that documents required changes: #17222

Hello ArgoCD Team, @alexmt . What's the latest on when we think this feature will be released? Thanks.

Hi ArgoCD TEAM, @alexmt is there an update about this?

Any idea when this is going to be implemented?

I found out that during the list application phase, we're getting back a bunch of data we don't really need, like operationState and resources. Since we're gonna fetch these again from the server once someone clicks on a specific application, how about we trim down the data during the list application phase? Could this also speed up the page response time? I'm tweaking and testing stuff locally right now.

updates 2024.07.02:
i add a function to prune application contents, and it looks like working well, In my ArgoCD test environment (v2.9.0), the original API response returned a JSON with 15.5k lines and a size of 616KB. After modifications, the JSON now has 4.45k lines and a size of 128KB. This represents a 71.3% reduction in the number of lines and a 79.2% decrease in data volume.

update 2024.12.25:
add zstd support on argocd 2.11.5, in my prod enviroment, list all applications cost 46.8mb, and now it's only 384kb! reduced 99.2%.

func pruneApplication(a *appv1.Application) appv1.Application {
   a = a.DeepCopy()
   defer func() {
   	if e := recover(); e != nil {
   		log.Errorf("[edenz] error: %v\n%s", e, debug.Stack())
   	}
   }()
   if a.Status.OperationState != nil && a.Status.OperationState.Operation.Sync != nil && len(a.Status.OperationState.Operation.Sync.Resources) > 0 {
   	a.Status.OperationState.Operation.Sync.Resources = nil
   	fmt.Println("[edenz] Pruning OperationState", a.Status.OperationState.Operation.Sync.Resources)
   }
   if a.Status.OperationState != nil && a.Status.OperationState.SyncResult != nil && len(a.Status.OperationState.SyncResult.Resources) > 0 {
   	a.Status.OperationState.SyncResult.Resources = nil
   	fmt.Println("[edenz] Pruning SyncResult", a.Status.OperationState.SyncResult.Resources)
   }
   if len(a.Status.Resources) > 0 {
   	a.Status.Resources = nil
   	fmt.Printf("[edenz] Pruning Resources: %v\n", a.Status.Resources)
   }
   if len(a.Status.History) > 0 {
   	a.Status.History = nil
   	fmt.Printf("[edenz] Pruning History: %v\n", a.Status.History)
   }

   a.Annotations = nil
   fmt.Println("[edenz] Pruning Annotations")

   return *a
}

Hi guys, any update on this? would really help us a out.

This is much awaited feature we are waiting for.
As we are maintaining 4000 apps and the UI becomes slow and sometimes it is dead slow.

i add a function to prune application contents, and it looks like working well, In my ArgoCD test environment (v2.9.0), the original API response returned a JSON with 15.5k lines and a size of 616KB. After modifications, the JSON now has 4.45k lines and a size of 128KB. This represents a 71.3% reduction in the number of lines and a 79.2% decrease in data volume.

@ezxfv to prioritize that change, I think we'd need reason to believe that payload is a significant performance bottleneck. For now, it seems like CPU is the more significant bottleneck.

@crenshaw-dev yeah TTFB (Time to first byte) should be a metric of interest.

@crenshaw-dev Indeed, this method of reducing irrelevant information can only temporarily alleviate my current predicament, allowing the Argo CD homepage to barely load all applications (8000+). Upon a cursory examination of the performance on Chrome, the primary time consumers are script execution (9s) and rendering (5s). Would leveraging Web Workers and WASM contribute to accelerating this process? It appears that pagination remains the least affected solution by the growth in the number of applications.

@ezxfv I imagine there are a lot of things that could be tuned on the frontend to speed things up. But even if you manage to half the script execution/rendering time, you're probably not going to get as much a win as you would be cutting the number of processed applications by 90%.

There are a lot of ways we could improve performance. Pagination seems like the highest payoff for the least effort / maintenance burden.

Any news on this?

Any news on this?

The proposal Alex wrote is now merged: #17222

Implementation is pending.

Any update on this ?
Maintaining a lot of apps (above 3-4k) start to feel painful with the UI
I think having argo cd ui running on things like Nextjs with a full server side first approach would be a good thing
But this is probably bigger workload too

I made the following two small changes: by delaying the rendering of non-essential components, the entire page now loads much faster, and memory usage has been significantly reduced. I have 9500 applications, and now the homepage opens in just 1-2 seconds, with memory usage significantly reduced. Search, filtering, and other functions have become incredibly smooth. Previously, it took over 20 seconds, and the page could even crash. In addition to this, there were some other minor changes, but these are the main improvements.

@ezxfv where PR

@ezxfv where PR
This is just a temporary version for review. If needed, I’ll organize it further later. I generated 10w applications in local cluster for testing, and it still works well!

@tooptoop4 @alexmt @crenshaw-dev #21334

Any update on this ? Maintaining a lot of apps (above 3-4k) start to feel painful with the UI I think having argo cd ui running on things like Nextjs with a full server side first approach would be a good thing But this is probably bigger workload too

@LeoAnt02 Maybe you can build a new image with my pr, it supports 10w+ applications

@ezxfv Hi! In the company I work for we have an ArgoCD environment of over 3200 apps, and your solution sounds great to resolve the UI slowness.

We are now considering to split ArgoCD to multiple instances in order to solve the slowness issue. Will this solution be released in a new version?
If so, do you have an ETA?

I just wanted to contribute some things I have experienced at multiple customers regarding UI slowness of ArgoCD. There are two different scenarios where customers experience slowness in the UI. The first one is the initial loading time of the webinterface of ArgoCD here I have seen loading times of up to a minute at around (3000-4000 ArgoCD apps). The second is the general responsiveness of the ArgoCD UI when filtering or searching. From my experience this change described here will only bring changes to the second scenario. The first one is mainly caused by inefficient rbac processing and can be improved by better rbac configuration. You can get 3000-4000 Argo apps easily in only one Argo instance with an initial loading time of like 3-5 seconds if you configure rbac more effectively. Maybe this helps to point you in the right direction @netashiloh

@naturboy Our issue is mostly the first one you specified- the initial loading time of the webinterface of ArgoCD.
It actually improved for us after some server optimizations we did:

1. Increase the number of ARGOCD_API_SERVER_REPLICAS
2. Increase ARGOCD_GRPC_MAX_SIZE_MB value (we increased it to 1600).
3. Updated the resources limits and requests to limit: cpu 4, memory 12Gi and requests: cpu: 3, memory 8Gi

Note: Don't take our values as best practice, it's still a trial an error to see what's working for us. For reference, we work in a growing environment of 3000-7000 apps (we have dynamic envs).

There is still room for improvement and I'm sure the server side pagination will help. I plan to review high availability argocd doc for more improvements we can do. Can you share the recommended configurations of RBAC too?

@netashiloh Split into multiple instances doesn't resolve the ui problem, the problem is many unnecessary react dom node created in the background. I don't know whether my pr would be accepted or not. You can build a image from my branch and test it.

i maybe add virtual page soon，when list more than 1000 applications at one time

We are also interested in this, we run 5k applications in ArgoCD and the UI is impossible to use

We have 4000+ apps and the UI does not work now. Hopefully we can have this sooner

does @ezxfv PR fixes this ?

Any update on this?

Hi! Any update on this? Thank you :)