Stream repo zip/tar.gz/bundle achives by default #35487

ChristopherHX · 2025-09-14T14:27:46Z

Initial implementation of linked proposal.

ChristopherHX · 2025-09-14T14:39:32Z

Improve initial download performance as the POST+GET negotiation on UI goes away and it will be pure GET.

Here I kept the POST+GET negotiation, where the POST does nothing when enabled

silverwind · 2025-09-14T21:02:27Z

Here I kept the POST+GET negotiation, where the POST does nothing when enabled

Will check later, I think it should be possible to turn into a GET. Benefit is that these downloads will be curl-able.

ChristopherHX · 2025-09-14T21:12:16Z

They are curl-able for some amount of time already, just not via html inspect. I believe in 2021 this didn't work in gitea.

e.g. curl -L https://gitea.com/gitea/act_runner/archive/main.zip just like on github.com

We would get a cleaner html page if we remove the POST as long this setting on.

silverwind · 2025-09-14T21:57:40Z

I wonder if we even need the setting at all. Are there any benefits to non-streaming downloads?

The only one I can think of is reduced CPU usage for instances that receive a lot of downloads, but it's rather niche I think, especially because if a repo is pushed to often, that cache will invalidate every push.

ChristopherHX · 2025-09-14T22:16:46Z

Are there any benefits to non-streaming downloads?

I would not require any setting, but I also do not host gitea productively anywhere. Let's hear others opinions, given I assume this would be more a change for a future release like 1.26.0.

If we decide to removing the repo-archive cache, then we also no longer need the repo-archive settings. I assume this way of doing this has been inherited from Gogs.

silverwind · 2025-09-15T00:34:12Z

I guess non-streaming does have a benefit for archived or otherwise rarely pushed-to repos. FWIW, Github does only implement streaming and I think this is also the only mode that can work on big instances without consuming copious amounts of disk space (think crawlers crawling archive links for all commit hashes).

silverwind · 2025-09-16T03:45:25Z

I pushed the UI change which will omit the POST when streaming is enabled. The archive links work natively without JS when streaming is enabled.

* origin/main: Clean up npm dependencies (go-gitea#35484) Update eslint to v9 (go-gitea#35485) Revert the rspack change (go-gitea#35482) Replace gobwas/glob package (go-gitea#35478) Fix various typos in codebase (go-gitea#35480) Fix different behavior in status check pattern matching with double stars (go-gitea#35474) Replace webpack with rspack (go-gitea#35460) Don't store repo archives on `gitea dump` (go-gitea#35467)

routers/web/repo/repo.go

services/repository/archiver/archiver.go

wxiaoguang · 2025-09-16T07:16:30Z

web_src/js/features/repo-common.ts

+  // when archive streaming is enabled, the links will work natively without JS
+  if (!window.config.streamArchives) {
+    queryElems(document, 'a.archive-link[href]', (el) => el.addEventListener('click', onDownloadArchive));
+  }


It's too hacky and fragile, there will be bugs in the future if somebody touches the code.

So what do you suggest? We could alternatively just remove non-streaming mode entirely which would make all the JS and probably a good chunk of Go code deletable (my preference).

Why you need this JS? "GET" request can always start the downloading. So you can just make POST check return immediately for streaming?

It's still a unneccesary round-trip to the server. It's better to not tamper with these links if we don't need to.

BTW I think we could eliminate the POST (and the JS) in both cases if we just delay the GET response in non-streaming mode.

Any objection?

For me would be your plan ok as well.

On the other hand the current js change makes the file download start sightly faster without right click menu and stops showing me http 400 codes shortly after restarts (invalid csrf tokens? auto fixes itself with later page refresh, GET is not protected)

I will make some changes according to my plan. I think eventually we will get rid of the repo archive and remove a lot of code.

the current js change ....

I would prefer not going with current JS. It couples a lot of modules (source files) together, a single "true/false" value is passed everywhere, which is not ideal. We should avoid such fragile "global variable".

And eventually, we will remove all the legacy JS

showing me http 400 codes shortly after restarts (invalid csrf tokens?

IIRC CSRF token is preserved between restarts, can you reproduce the CSRF token problem by other forms? For example: a comment form?

Go ahead and revert the JS if you like. I still hope we eventually can change all use cases to GET only, but the POST+GET isn't bothering me too much I guess.

Edit: I see you did already.

Thank you all. Let's keep the unnecessary "POST" for a short time, then completely remove it in next release 🎉

ChristopherHX

After simplifing Archiverequest.Stream, I decided to call this as well when caching the repository but use the pipe as destination

services/repository/archiver/archiver.go

routers/web/repo/repo.go

wxiaoguang · 2025-09-18T06:57:39Z

Made some changes, and I merged the legacy copy-pasted code (since long time ago ...) into one ServeRepoArchive, then no duplicate download/prepare/Add(link)

Now +110 −156 , less code, better result.

There are tests covering "repo archive download", so as long as tests pass, I think it is good enough.

Stream repo zip/tar.gz/bundle achives by default

45f6c20

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Sep 14, 2025

github-actions bot added modifies/api This PR adds API routes or modifies them modifies/go Pull requests that update Go code docs-update-needed The document needs to be updated synchronously labels Sep 14, 2025

Remove unnecessary line in file.go

9142a38

make UI use GET only when StreamArchives is enabled

3545ba0

github-actions bot added modifies/templates This PR modifies the template files modifies/frontend labels Sep 16, 2025

silverwind added 3 commits September 16, 2025 05:45

update comment

2ab754a

fix lint

f5648ae

silverwind approved these changes Sep 16, 2025

View reviewed changes

GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Sep 16, 2025

silverwind added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label Sep 16, 2025

hiifong approved these changes Sep 16, 2025

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Sep 16, 2025

wxiaoguang reviewed Sep 16, 2025

View reviewed changes

routers/web/repo/repo.go Outdated Show resolved Hide resolved

wxiaoguang reviewed Sep 16, 2025

View reviewed changes

services/repository/archiver/archiver.go Outdated Show resolved Hide resolved

wxiaoguang reviewed Sep 16, 2025

View reviewed changes

services/repository/archiver/archiver.go Outdated Show resolved Hide resolved

wxiaoguang reviewed Sep 16, 2025

View reviewed changes

wxiaoguang marked this pull request as draft September 16, 2025 07:16

Apply feedback

d447b7c

ChristopherHX commented Sep 16, 2025

View reviewed changes

services/repository/archiver/archiver.go Outdated Show resolved Hide resolved

services/repository/archiver/archiver.go Outdated Show resolved Hide resolved

routers/web/repo/repo.go Outdated Show resolved Hide resolved

ChristopherHX and others added 6 commits September 16, 2025 10:52

fix lint

d58de54

dedup downloadName

9f48582

Fix missed method DownloadArchive

6d710e7

Merge branch 'main' into stream-repo-archive

001e4cf

reset

8523363

merge

cc5713d

github-actions bot removed modifies/templates This PR modifies the template files modifies/frontend docs-update-needed The document needs to be updated synchronously labels Sep 18, 2025

wxiaoguang marked this pull request as ready for review September 18, 2025 06:59

wxiaoguang added this to the 1.25.0 milestone Sep 18, 2025

use correct middleware

759abab

silverwind approved these changes Sep 18, 2025

View reviewed changes

wxiaoguang approved these changes Sep 18, 2025

View reviewed changes

lunny added the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Sep 19, 2025

wxiaoguang merged commit 9a0ec53 into go-gitea:main Sep 19, 2025
26 checks passed

GiteaBot removed the reviewed/wait-merge This pull request is part of the merge queue. It will be merged soon. label Sep 19, 2025

silverwind mentioned this pull request Sep 19, 2025

Stream zip downloads to the browser #25002

Closed

Uh oh!

Stream repo zip/tar.gz/bundle achives by default #35487

Stream repo zip/tar.gz/bundle achives by default #35487

Conversation

ChristopherHX commented Sep 14, 2025 • edited by wxiaoguang Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChristopherHX commented Sep 14, 2025

Uh oh!

silverwind commented Sep 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChristopherHX commented Sep 14, 2025

Uh oh!

silverwind commented Sep 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ChristopherHX commented Sep 14, 2025

Uh oh!

silverwind commented Sep 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

silverwind commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wxiaoguang Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

silverwind Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wxiaoguang Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

silverwind Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

silverwind Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wxiaoguang Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

ChristopherHX Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

wxiaoguang Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

silverwind Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wxiaoguang Sep 18, 2025

Choose a reason for hiding this comment

Uh oh!

ChristopherHX left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wxiaoguang commented Sep 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ChristopherHX commented Sep 14, 2025 •

edited by wxiaoguang

Loading

silverwind commented Sep 14, 2025 •

edited

Loading

silverwind commented Sep 14, 2025 •

edited

Loading

silverwind commented Sep 15, 2025 •

edited

Loading

silverwind commented Sep 16, 2025 •

edited

Loading

silverwind Sep 16, 2025 •

edited

Loading

wxiaoguang Sep 16, 2025 •

edited

Loading

silverwind Sep 16, 2025 •

edited

Loading

silverwind Sep 16, 2025 •

edited

Loading

wxiaoguang Sep 18, 2025 •

edited

Loading

silverwind Sep 18, 2025 •

edited

Loading

wxiaoguang commented Sep 18, 2025 •

edited

Loading