All operations are performed in the VM

everything is on-premises and free (except VM)

The only thing you need is:

• a VM
  • Memory: 16G is recommended
  • Operating system: Ubuntu 22.04 (recommended, other operating systems have no difference except Docker installation)
  • Port: 3000 port needs to be released for Grafana to use, and 22 port can be determined by yourself.

Everything else is based on existing stuff, or based on open source software, no extra cost, for example:

• GitHub Organzations with Copilot enabled (I believe, you already have it, more to GitHub Copilot | Microsoft Azure or learn from How to subscribe for GitHub Copilot Business without GitHub Enterprise: A Step-by-Step Guide | All things Azure)
• Docker (Community Version is enough)
• Elasticsearch (Community Version is enough)
• Grafana (Community Version is enough, do not need Grafana Cloud Account)
• CPUAD-Updater build from this project (MIT license)

For installation instructions, refer to Install Docker Engine. For Ubuntu 22.04, you can use the following command

apt install docker.io

verify

docker version

The following content is obtained, indicating ok

Client:
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.21.1
 Git commit:        24.0.7-0ubuntu2~22.04.1
 Built:             Wed Mar 13 20:23:54 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.21.1
  Git commit:       24.0.7-0ubuntu2~22.04.1
  Built:            Wed Mar 13 20:23:54 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.12
  GitCommit:
 runc:
  Version:          1.1.12-0ubuntu2~22.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

Put all the work in the /srv directory, click download zip archive, unzip and rename the folder to copilot-usage-advanced-dashboard, or directly git clone

cd /srv
git clone https://github.com/satomic/copilot-usage-advanced-dashboard.git
cd copilot-usage-advanced-dashboard

verify

ls -ltr

The following content is obtained, indicating ok

total 64
-rw-r--r-- 1 root root   100 Dec 16 11:22 fetch.sh
-rw-r--r-- 1 root root    56 Dec 16 11:22 docker_build.sh
-rw-r--r-- 1 root root  1063 Dec 16 11:22 LICENSE
-rw-r--r-- 1 root root  1031 Dec 16 11:22 Dockerfile
-rw-r--r-- 1 root root   193 Dec 16 11:22 push.sh
drwxr-xr-x 2 root root  4096 Dec 16 11:22 mapping
-rw-r--r-- 1 root root    22 Dec 16 11:32 requirements.txt
-rw-r--r-- 1 root root   996 Dec 16 13:44 log_utils.py
drwxr-xr-x 2 root root  4096 Dec 17 00:18 grafana
-rw-r--r-- 1 root root  2571 Dec 17 00:18 gen_grafana_model.py
-rw-r--r-- 1 root root 22500 Dec 17 01:40 main.py

If you already have ES, you can skip this step and go directly to the next step.

ES will not be exposed to the outside of the VM, so there is no need to enable xpack.security.enabled

1. Create a data persistence directory and a configuration file directory for Elasticsearch:

   mkdir -p /srv/elasticsearch/data /srv/elasticsearch/config
2. Grant read and write permissions.

   chown -R 777 /srv/elasticsearch/
3. Create the elasticsearch.yml configuration file in the /srv/elasticsearch/config/directory:

   cat >> /srv/elasticsearch/config/elasticsearch.yml << EOF
   network.host: 0.0.0.0
   node.name: single-node
   cluster.name: es-docker-cluster
   path.data: /usr/share/elasticsearch/data
   path.logs: /usr/share/elasticsearch/logs
   discovery.type: single-node
   bootstrap.memory_lock: true
   EOF
4. Use the following command to start Elasticsearch and bind the data directory and configuration file:

   docker run -itd --restart always --name es \
     -p 9200:9200 \
     -e "xpack.security.enabled=false" \
     -e "ES_JAVA_OPTS=-Xms4g -Xmx4g" \
     -v /srv/elasticsearch/data:/usr/share/elasticsearch/data \
     -v /srv/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro \
     docker.elastic.co/elasticsearch/elasticsearch:8.17.0
   
5. verify

   curl http://localhost:9200

   The following content is obtained, indicating ok

   {
       "name": "single-node",
       "cluster_name": "es-docker-cluster",
       "cluster_uuid": "oO3mfjYWTZ6VZFSClDiSLA",
       "version": {
           "number": "8.17.0",
           "build_flavor": "default",
           "build_type": "docker",
           "build_hash": "2b6a7fed44faa321997703718f07ee0420804b41",
           "build_date": "2024-12-11T12:08:05.663969764Z",
           "build_snapshot": false,
           "lucene_version": "9.12.0",
           "minimum_wire_compatibility_version": "7.17.0",
           "minimum_index_compatibility_version": "7.0.0"
       },
       "tagline": "You Know, for Search"
   }

1. Confirm that you are in the correct path

   cd /srv/copilot-usage-advanced-dashboard
2. Execute the script and create an index

   bash create_es_indexes.sh

   The following content is obtained, indicating ok

   {"acknowledged":true,"shards_acknowledged":true,"index":"copilot_usage_total"}
   {"acknowledged":true,"shards_acknowledged":true,"index":"copilot_usage_breakdown"}
   {"acknowledged":true,"shards_acknowledged":true,"index":"copilot_seat_info_settings"}
   {"acknowledged":true,"shards_acknowledged":true,"index":"copilot_seat_assignments"}
3. verify

   curl -X GET http://localhost:9200/_cat/indices?v

   The following content is obtained, indicating ok

   health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
   yellow open   copilot_usage_total        XrOEfAngTS60VsuUz3Lbrw   1   1          0            0       227b           227b         227b
   yellow open   copilot_seat_info_settings WtOBdBNUQRqua7wi7VANeQ   1   1          0            0       227b           227b         227b
   yellow open   copilot_seat_assignments   lK5t4SwASZizPQ_W4NX4KQ   1   1          0            0       227b           227b         227b
   yellow open   copilot_usage_breakdown    xE6tkg5GQEOP-EP8pwAkYg   1   1          0            0       227b           227b         227b
   

If you already have Grafana, you can skip this step and go directly to the next step.

1. Creating a Data Path

   mkdir -p /srv/grafana/data
   chmod 777 /srv/grafana/data
2. run

   docker run  -itd --restart always --name=grafana \
     --net=host \
     -p 3000:3000 \
     -v /srv/grafana/data:/var/lib/grafana \
     -e "GF_LOG_LEVEL=debug" \
     grafana/grafana:11.4.0
3. Access Grafana
   • Access address: http://<PUBLIC_IP_OF_YOUR_VM>:3000
   • The default username and password are admin/ admin, please change the password

1. admin visit Administration → Users and access → Service accounts

   

2. input Display name，Role select Admin, Create

   

3. click Add service account token

   

4. click Generate token

   

5. Copy to clipboard and close

   

6. Now, you have obtained your Grafana Token "<your_grafana_token>", please save it and set it as an environment variable in the VM, which will be used in the next steps.

   export GRAFANA_TOKEN="<your_grafana_token>"

1. Confirm that you are in the correct path

   cd /srv/copilot-usage-advanced-dashboard

2. run script, add data sources

   bash add_grafana_data_sources.sh

3. Visit the Grafana UI to confirm that the addition was successful

   

1. Confirm that you are in the correct path

   cd /srv/copilot-usage-advanced-dashboard
2. Execute the script to generate a Grafana json model. Execute one of the following two commands

   # Generate Copilot Usage Advanced Dashboard
   python3 gen_grafana_model.py --template=grafana/dashboard-template.json
   
   # Generate Copilot Usage Advanced Dashboard Original
   python3 gen_grafana_model.py --template=grafana/dashboard-template-original.json

   Get the output

   Model saved to grafana/dashboard-model-2024-12-17.json, please import it to Grafana

1. Download the generated file to your local computer

   scp root@<PUBLIC_IP_OF_YOUR_VM>:/srv/copilot-usage-advanced-dashboard/grafana/dashboard-model-*.json .
   dashboard-model-2024-12-17.json                                                                                                                                                  100%  157KB 243.8KB/s   00:00
   dashboard-model-data_sources_name_uid_mapping-2024-12-17.json                                                                                                                    100%  210     1.1KB/s   00:00

2. Copy the generated json file and import it into Grafana

   

   Select the file to import, or paste the content directly

   

3. Import

   

4. Congratulations, you now have a complete dashboard, but there should be no data yet. Next, run the core program.

is the abbreviation of the first characters of Copilot Usage Advanced Dashboard Updater

Parameter description

• GITHUB_PAT: Your GitHub PAT, which needs to have Owner permissions for Organizations. Please replace <YOUR_GITHUB_PAT> with the actual value.
• ORGANIZATION_SLUGS: The Slugs of all Organizations that you want to monitor, which can be one or multiple separated by , (English symbol). If you are using Copilot Standalone, use your Standalone Slug here, prefixed with standalone:, for example standalone:YOUR_STANDALONE_SLUG. Please replace <YOUR_ORGANIZATION_SLUGS> with the actual value. For example, the following types of values are supported:
  • myOrg1
  • myOrg1,myOrg2
  • standalone:myStandaloneSlug
  • myOrg1,standalone:myStandaloneSlug
• LOG_PATH: Log storage location, not recommended to modify. If modified, you need to modify the -v data volume mapping simultaneously.
• EXECUTION_INTERVAL: Update interval, the default is to update the program every 1 hours.

docker run -itd \
--net=host \
--restart=always \
--name cpuad \
-e GITHUB_PAT="<YOUR_GITHUB_PAT>" \
-e ORGANIZATION_SLUGS="<YOUR_ORGANIZATION_SLUGS>" \
-e LOG_PATH="logs" \
-e EXECUTION_INTERVAL=1 \
-e ELASTICSEARCH_URL="http://localhost:9200" \
-v /srv/cpuad-updater-logs:/app/logs \
satomic/cpuad-updater

1. Confirm that you are in the correct path

   cd /srv/copilot-usage-advanced-dashboard
2. Install Dependencies

   python3 -m pip install -r requirements.txt
3. Setting Environment Variables. If you are using Copilot Standalone, use your Standalone Slug here, prefixed with standalone:, for example standalone:YOUR_STANDALONE_SLUG.

   export GITHUB_PAT="<YOUR_GITHUB_PAT>"
   export ORGANIZATION_SLUGS="<YOUR_ORGANIZATION_SLUGS>"
   
4. run

   python3 main.py
5. output logs

   2024-12-17 05:32:22,292 - [INFO] - Data saved to logs/2024-12-17/nekoaru_level3-team1_copilot_usage_2024-12-17.json
   2024-12-17 05:32:22,292 - [INFO] - Fetched Copilot usage for team: level3-team1
   2024-12-17 05:32:22,293 - [INFO] - Data saved to logs/2024-12-17/nekoaru_all_teams_copilot_usage_2024-12-17.json
   2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for organization: nekoaru
   2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level1-team1
   2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level1-team1
   2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level2-team1
   2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level2-team1
   2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level2-team2
   2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level2-team2
   2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level3-team1
   2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level3-team1
   2024-12-17 05:32:22,293 - [INFO] - Sleeping for 6 hours before next execution...
   2024-12-17 05:32:22,293 - [INFO] - Heartbeat: still running...
   
   

At this moment, in the VM, you should be able to see 3 containers running (if you have deployed them from scratch based on docker), as follows:

docker ps

CONTAINER ID   IMAGE                                                  COMMAND                  CREATED        STATUS        PORTS                                                 NAMES
1edffd12a522   satomic/cpuad-updater:20241221                         "python3 main.py"        23 hours ago   Up 10 hours                                                         cpuad
b19e467d48f1   grafana/grafana:11.4.0                                 "/run.sh"                25 hours ago   Up 10 hours                                                         grafana
ee35b2a340f1   docker.elastic.co/elasticsearch/elasticsearch:8.17.0   "/bin/tini -- /usr/l…"   3 days ago     Up 10 hours   0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp   es

At this point, return to the Grafana page and refresh. You should be able to see the data.

or