Add ability to update scm project using github app #15472

TheRealHaoLiu · 2024-08-27T20:33:48Z

SUMMARY

Add github_app_id, github_app_installation_id and github_api_url to scm credential type
Add ability to generate github app token to clone project with git for github

ISSUE TYPE

New or Enhanced Feature

COMPONENT NAME

API

AWX VERSION

ADDITIONAL INFORMATION

- Add github_app_id, github_app_installation_id and github_api_url to scm credential type - Add ability to generate github app token to clone project with git for github

sonarcloud · 2024-08-28T13:39:10Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

chrismeyersfsu · 2024-08-28T13:43:35Z

This should be a different credential.

webknjaz · 2024-08-28T13:47:00Z

awx/main/models/credential/__init__.py

@@ -645,6 +645,9 @@ def create(self):
            {'id': 'password', 'label': gettext_noop('Password'), 'type': 'string', 'secret': True},
            {'id': 'ssh_key_data', 'label': gettext_noop('SCM Private Key'), 'type': 'string', 'format': 'ssh_private_key', 'secret': True, 'multiline': True},
            {'id': 'ssh_key_unlock', 'label': gettext_noop('Private Key Passphrase'), 'type': 'string', 'secret': True},
+            {'id': 'github_app_id', 'label': gettext_noop('GitHub App ID'), 'type': 'string'},
+            {'id': 'github_app_installation_id', 'label': gettext_noop('GitHub App Installation ID'), 'type': 'string'},


Are you trying to surface this into a UI field? This is usually discoverable automatically.

So ideally, there should be no UI inputs at all. Log into https://pre-commit.ci or Codecov / RTD to see their UX for working with repos accessible through GH app installations.

Whenever the app is installed, you already get an event with all the information needed via webhooks.

webknjaz · 2024-08-28T13:49:14Z

awx/main/models/credential/__init__.py

@@ -645,6 +645,9 @@ def create(self):
            {'id': 'password', 'label': gettext_noop('Password'), 'type': 'string', 'secret': True},
            {'id': 'ssh_key_data', 'label': gettext_noop('SCM Private Key'), 'type': 'string', 'format': 'ssh_private_key', 'secret': True, 'multiline': True},
            {'id': 'ssh_key_unlock', 'label': gettext_noop('Private Key Passphrase'), 'type': 'string', 'secret': True},
+            {'id': 'github_app_id', 'label': gettext_noop('GitHub App ID'), 'type': 'string'},


Is this also surfaced to the end-users? Why? It should be a platform-global value, it's not directly a credential.

webknjaz · 2024-08-28T13:50:44Z

awx/main/tasks/jobs.py

+            {
+                'iat': int(time.time()),  # Issued at time
+                'exp': int(time.time()) + (10 * 60),  # JWT expiration time (10 minute maximum)
+                'iss': project_update.credential.get_input('github_app_id', default=''),  # GitHub App's identifier


GH App should be global for the entire AWX. The whole idea is that the users don't need to set up and maintain multiple apps. They just need to click “Install” on GH UI.

webknjaz · 2024-08-28T13:51:26Z

awx/main/tasks/jobs.py

+
+        headers = {'Authorization': f'Bearer {jwt_token}', 'Accept': 'application/vnd.github.v3+json'}
+
+        github_api_url = project_update.credential.get_input('github_api_url', default='https://api.github.com')


This as well should be a deployment-global value rather than bound to a credential.

AlanCoding · 2024-08-28T13:53:51Z

On @chrismeyersfsu 's point, the intent was never for the Project.credential field to validate the literal "Source Control" credential type.

awx/awx/main/models/projects.py

Lines 163 to 164 in 79c1921

    
           elif cred.kind != 'scm': 
        
               raise ValidationError(_("Credential kind must be 'scm'."))

Instead, it only wants the kind of the credential to be "scm", that, I believe (incorrectly) maps to the credential type namespace

awx/awx/main/models/credential/__init__.py

Lines 135 to 137 in 79c1921

    
           @property 
        
           def kind(self): 
        
               return self.credential_type.namespace

Again, I think that's wrong. When the Project.credential field is validated, I think it should require that the related credential_type.kind is "scm".

awx/awx/main/models/credential/__init__.py

Line 362 in 79c1921

kind = models.CharField(max_length=32, choices=KIND_CHOICES)

you can see the choices:

awx/awx/main/models/credential/__init__.py

Lines 347 to 360 in 79c1921

    
           KIND_CHOICES = ( 
        
               ('ssh', _('Machine')), 
        
               ('vault', _('Vault')), 
        
               ('net', _('Network')), 
        
               ('scm', _('Source Control')), 
        
               ('cloud', _('Cloud')), 
        
               ('registry', _('Container Registry')), 
        
               ('token', _('Personal Access Token')), 
        
               ('insights', _('Insights')), 
        
               ('external', _('External')), 
        
               ('kubernetes', _('Kubernetes')), 
        
               ('galaxy', _('Galaxy/Automation Hub')), 
        
               ('cryptography', _('Cryptography')), 
        
           )

This CredentialType "kind" designation is like an intended use of that credential type. So any "scm" kind of credential type should be acceptable for use in a project.

webknjaz · 2024-08-28T13:54:30Z

awx/main/tasks/jobs.py

+        headers = {'Authorization': f'Bearer {jwt_token}', 'Accept': 'application/vnd.github.v3+json'}
+
+        github_api_url = project_update.credential.get_input('github_api_url', default='https://api.github.com')
+        installation_id = project_update.credential.get_input('github_app_installation_id', default='')


This should be discovered from the GitHub repository URL and querying https://docs.github.com/en/rest/apps/installations?apiVersion=2022-11-28#list-app-installations-accessible-to-the-user-access-token, and stored somewhere in the DB, not surfaced to the users.

webknjaz · 2024-08-28T13:55:07Z

awx/main/tasks/jobs.py

+            access_token = response.json()['token']
+            return access_token
+        else:
+            raise Exception(f"Failed to get access token: {response.status_code} {response.text}")


Any chance not to use an unspecified exception?

webknjaz · 2024-08-28T13:55:30Z

awx/main/tasks/jobs.py

+        url = f'{github_api_url}/app/installations/{installation_id}/access_tokens'
+        response = requests.post(url, headers=headers)
+
+        if response.status_code == 201:


If you invert it to !=, you can dedent a few lines of code.

TheRealHaoLiu · 2024-08-28T20:18:44Z

@webknjaz for this implementation we just want to scope this to specifically project update and also this will give the ability for people to have different github app for different projects if desired

webknjaz · 2024-08-29T23:11:23Z

also this will give the ability for people to have different github app for different projects if desired

That's kinda against the entire idea of GH Apps 🤷‍♂️

github-actions bot added the component:api label Aug 27, 2024

Add ability to update scm project using github app

f8e9862

- Add github_app_id, github_app_installation_id and github_api_url to scm credential type - Add ability to generate github app token to clone project with git for github

TheRealHaoLiu force-pushed the github-app-scm-project branch from 6187010 to f8e9862 Compare August 28, 2024 13:37

webknjaz reviewed Aug 28, 2024

View reviewed changes

TheRealHaoLiu mentioned this pull request Aug 29, 2024

GitHub app scm project alt #15481

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to update scm project using github app #15472

Add ability to update scm project using github app #15472

TheRealHaoLiu commented Aug 27, 2024

sonarcloud bot commented Aug 28, 2024

chrismeyersfsu commented Aug 28, 2024

webknjaz Aug 28, 2024

TheRealHaoLiu Aug 28, 2024

webknjaz Aug 28, 2024

webknjaz Aug 28, 2024

webknjaz Aug 28, 2024

webknjaz Aug 28, 2024

AlanCoding commented Aug 28, 2024

webknjaz Aug 28, 2024

webknjaz Aug 28, 2024

webknjaz Aug 28, 2024

TheRealHaoLiu commented Aug 28, 2024

webknjaz commented Aug 29, 2024


		headers = {'Authorization': f'Bearer {jwt_token}', 'Accept': 'application/vnd.github.v3+json'}

		github_api_url = project_update.credential.get_input('github_api_url', default='https://api.github.com')

Add ability to update scm project using github app #15472

Are you sure you want to change the base?

Add ability to update scm project using github app #15472

Conversation

TheRealHaoLiu commented Aug 27, 2024

SUMMARY

ISSUE TYPE

COMPONENT NAME

AWX VERSION

ADDITIONAL INFORMATION

sonarcloud bot commented Aug 28, 2024

Quality Gate passed

chrismeyersfsu commented Aug 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlanCoding commented Aug 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TheRealHaoLiu commented Aug 28, 2024

webknjaz commented Aug 29, 2024