Forge software/host

[VlinkZ]

From another thread:

@VlinkZ

Create a new repo that depends on nixpkgs as a flake input and work towards migrating everything into the new repo (or set of repos) to the point that we can drop the nixpkgs dependence

I feel like if we take this approach, it may make moving easier either now or later since we wouldn’t need to move an entire fork of nixpkgs, rather just the parts we are working on (although it may grow to nixpkgs size in time). I feel like starting off directly with a fork of nixpkgs will be too much to maintain, especially starting out now. I imagine the core repo importing nixpkgs as a flake input, with other package groups importing the core, and then have a top-level repo (auxpkgs or something else) exporting everything. I think building something from the ground up might be easier than breaking nixpkgs apart to fit our new mold, and could also allow us to more easily move away from GitHub.

[_deleted]

You can certainly attempt that, but from my own personal experience, sooner or later you’ll run into issues that cannot be addressed through overlays, overrides, etc.
Personally, I’d not attempt this without resolving Support flake references to patches · Issue #3920 · NixOS/nix · GitHub

Your experience will not be great.

[VlinkZ]

Sorry I don’t think I was clear. I’m not proposing using overlays or overrides, rather creating a new monorepo (or group of repos), but cheat by being able to rely on nixpkgs (flake input) packages in case they aren’t already packaged within the new repo itself. I’m worried that if we just fork and try to start maintaining all of nixpkgs off the bat, we’ll be overwhelmed just maintaining let alone restructuring it

[algernon]

@jakehamilton

• Fork Nix on the same forge (GitHub) today
• Build the foundation of Aux, working through the roadmap
• Once sustainable, transition to a self-hosted forge

I like this plan. I am also happy to assist in making sure Forgejo becomes a suitable forge for Aux to self-host with. It’s not terribly hard to get there, it just isn’t there today.

Forgejo has a built-in CI that is very similar to GitHub Actions (it is using act as a runner, under the hood), so the transition there wouldn’t be very painful. Not a drop in replacement, but it’s close-ish, and the transition is usually not too painful in my experience.

@_deleted

Gitea has been trying for years to rid themselves of Github and become self hosted

…they have not been trying too hard. If they closed PRs and issues on GitHub for about a week, they could have migrated by now. Trying to migrate while issues and PRs are open is never going to work.

@_deleted

They do, but only by burning all the bridge, treating Gitea as a hostile fork.

No? Forgejo was self-hosting from day one (perks of forking off), except for CI (Forgejo initially used Woodpecker before switching to Forgejo Actions). No bridges needed to be burned, and Forgejo continued to try and collaborate with Gitea for two years. Some Forgejo PRs even landed in Gitea during that time. Even after the hard fork, we do not treat them as a hostile fork. We cherry pick weekly from Gitea, and they’re free to cherry pick from us too.

[_deleted]

Forgejo was self-hosting from day one (perks of forking off), except for CI (Forgejo initially used Woodpecker before switching to Forgejo Actions).

I never claimed any different. My point was, forgejo never tried to migrate out of any forge, trying to keep metadata, issues, wiki etc. In tact. They forked the Gitea code base, treated upstream pretty much as hostile, and started fresh.

[algernon]

Yes. But there was no reason to migrate. It has nothing to do with hostility.

On the other hand, I helped companies migrate off of GitHub, repos and projects larger than Gitea. It’s not easy, but it is definitely doable and does not take years.

[_deleted]

But there was no reason to migrate.

Basically my point, hence it does not compare.

[Psentee]

Before NixOS I was using FreeBSD. The scope and size of FreeBSD (including ports) is definitely comparable to Aux, and I believe we can learn a log from the BSD community. The BSDs are structured for independence and being self-sufficient. They’re very conservative and I can’t tell much about governance structure, but I was there as a user during migration from Subversion to Git (which happened in 2020, this is how conservative the BSDs are; I think OpenBSD still uses CVS), and I can talk a bit about their code hosting.

All is self-hosted, no reliance on any third party forge or service.

Canonical central repositories are at https://cgit.freebsd.org/ – no forge. No forks either, there’s one central repo with branches, so the issue with forks taking up space doesn’t apply. I’m not sure how the repositories themselves are hosted – I expect either custom solution with plain ssh access, or something as simple as gitosis/gitolite. On top of that, cgit for read-only web access.

All the contributions I made were simple, so I sent them as patches to Bugzilla. For more complex changes there’s a Phabricator. I’m not sure how does the workflow look like once you’re a comitter, TBH. I can try to look it up if anybody’s interested.

This is simple, but very flexible – since Git itself is distributed, committers can work on branches in the central repo or maintain the fork however they see fit. There’s a read only GitHub mirror that can be forked. Some subystem maintainers organize their work around GitHub – they create a central GH fork for their workstream, accept PRs from other contributors, and then push their integration branch to the central repository to upstream. DRM (graphics subsystem) effort was initially organized this way.

I see that the freebsd GitHub org is used for canonical hosting of some smaller components that aren’t as stable as the core repos. Seems like DRM is now an out-of-tree kernel module (GitHub - freebsd/drm-kmod: drm driver for FreeBSD) – this repository is used as a source for a port (i.e. external package out of the base system tree).

If this is helpful at all, I’ll try to look for descriptions of the development workflows [EDITED TO ADD: Committer's Guide | FreeBSD Documentation Portal documents the development workflow in detail, it’s not as sophisticated as we’d need, but still might be valuable for inspiration]. I feel that BSDs’ focus on self-reliance and self-hosting is something we can learn from.

Does anybody have insight how are other projects of similar scale hosted and structured? (Other BSDs? Debian? Arch? Slackware? Something large that’s not a os/distro and not a de facto corporate project?) Let’s try to learn from other projects and avoid repeating their trials and errors if we can.

[srtcd424]

@toastal

Alternatively, Radicle (the forge, don’t want to hear about the independent cryptocurrency project) approaches distribution differently akin to Bitorrent–where the D in distributed version control system (DVCS) starts to mean something.

I’d love to see a gateway to/from Radicle at some point, but I agree for the foreseeable future pragmatism and getting up and running quickly is probably the most important. Also, I hate myself for it, but I do tend to feel mild irritation when I have to deal with a non-Github hosted project - apparently I’m getting old and can no longer cope with learning multiple UIs So perhaps concerns about about developer friction have some validity.

[minion]

For more complex changes there’s a Phabricator. I’m not sure how does the workflow look like once you’re a comitter, TBH. I can try to look it up if anybody’s interested.

My employer hosts phabricator and uses it for some things, so I’m familiar with it.

I know that phabricator is unmaintained since 2021, see https://phacility.com/

There is a community fork, https://we.phorge.it/, but I don’t know much about it

[Jeff]

@_deleted

Excuse me for being blunt here, but that is just incredibly naive. Claiming that “Git makes it easy to switch code hosting”

No problem, you and I are on the same page. When I said code hosting I meant it very literally: it makes just switching storage locations easy. Not issue tracking, actions, webhooks etc. Sorry that wasn’t clear, I went and edited to be more explicit.

[jakehamilton]

Update:

I have heard back from Codeberg and I am setting up a call with them to discuss some specifics. I may also ping the Lix folks in the mean time to see how we can collaborate since they have their own Forgejo instance.

[danderson]

(reposting something that came up on Matrix, to put it in a place where long-form discussion is easier)

In On the future of our nixpkgs fork, there’s a proposal to abandon forking nixpkgs as-is, and focus on building an alternative directly instead.

@crebwyllion made the connection that if we do go that route, forges other than github become a lot more feasible. Nixpkgs’s massive size and the integration of nixpkgs tooling with github are a technical obstacle to moving elsewhere if we’re trying to fork and maintain nixpkgs as-is.

OTOH, if we’re going straight to the smaller, modular repos and doing our own thing… We’re going to need a bunch of original tooling anyway, and we should end up with sensible repo sizes. Which makes moving off github a lot more tractable.

So… yeah, something to consider in this discussion

[jakehamilton]

I think the biggest consideration is going to be bandwidth since each user clones the repositories at least once (but realistically every update). Not sure if we could reasonably self host that right now, but if we could then I think it would be cool to set up our own forgejo instance. Otherwise Codeberg may still be the best option.