We are using kustomize.toolkit.fluxcd.io/ssa: merge annotation so that I can
add our own spec.sync to the FluxInstance.
This is not known to be reliable and you may cause a major outage, already I've seen my Flux config nuked once I'm not sure how.
Beware!
Cozystack now (as of 0.8.0) optionally provides a flux-operator installation
as a Cluster addon, and installs Flux as a FluxInstance in the cozy-fluxcd
namespace. The spec.sync section is not written by the system, so users can
override it at runtime.
Add a FluxInstance in clusters/test-cluster/flux-system/flux-instance.yaml
as I have done, leaving alone the cluster and distribution settings which
are pre-configured.
Add your own config to the sync section:
...
metadata:
...
annotations:
kustomize.toolkit.fluxcd.io/ssa: merge
spec:
cluster: {}
distribution: {}
sync:
kind: GitRepository
url: "ssh://[email protected]/kingdon-ci/cozy-fleet.git"
ref: "refs/heads/main"
path: "clusters/harvey"
pullSecret: "flux-system"Use the flux CLI to create a secret and add the secret as a Deploy Key:
flux create secret git flux-system --url ssh://[email protected]/kingdon-ci/cozy-fleet.git -n cozy-fluxcdSee the Flux documentation on flux create secret git for more details.
This would be better with direct configuration and not merged like this, but I don't think there should be any issues. (We'll find out! YOLO...)
One cluster definitely had some issues, may be it was not Flux or Cozystack's fault. It seemed to be a node trouble condition. I do not know how node issue like this can result in deleting a secret or a namespace.
I'm betting Cozystack installer did it somehow.
I will keep testing and update this repo with better configs as I learn.
I was able to clear the issue by scaling the MachineDeployment and deleting
the trouble node. Not only was the secret adjacent to my FluxInstance wiped
out, but also many pods kept returning to CrashLoopBackOff - I suspect this
type of issue is reproducible, but it is also the first time I've seen it.
So far since I deleted the trouble node and let another one replace it, there have been no further issues. I am very novice at KubeVirt and Kamaji!
My intention is to copy all of the configs from Cozystack down to this repo, sort of as a reverse-GitOps. The configs are all generated, but I want view access so I can inspect them.
One day I may add more ssa:merge instructions, but for now, the configs in
apps are meant to be referenced by Kustomizations in your clusters, like
the flux2-kustomize-helm example repo.
The tenants/ definitions are meant to be replayed in the event of total loss,
or used as reference to understand the arrangement of our system's tenants.
And the system/ directory reflects Cozystack core and system packages.
As of this writing, I am using a home-brew distribution of Cozystack built from some PRs that have yet to merge for release. You can follow my progress:
In a future release of Cozystack, perhaps users who select the Flux addon can
configure the sync right there, in the Kubernetes app. It makes sense to do
that, but an open question is where does the pull secret come from?
For now, the configuration has been kept minimal and only what is required, to preserve some 3-way merge opportunities like this one.
There are probably more changes coming. Cozystack users, we need your feedback.
Enjoy! (PRs welcome!)