February 21st, 2025

Visualize ROI of your GitHub Copilot Usage, Deploy it!

Xuefeng Yin
Xuefeng Yin

Image HC0400 MS AzureDeveloperBlogSeries Banner 103124 DC V2 02 2

Tips

In today’s enterprise software development landscape, organizations are increasingly investing in AI-powered development tools like GitHub Copilot to boost developer productivity. However, the standard 28-day usage metrics often fall short of providing the comprehensive insights needed to truly understand ROI and adoption patterns across large organizations. The Copilot Usage Advanced Dashboard fills this critical gap by offering a powerful, enterprise-grade solution that captures and analyzes long-term usage data across multiple teams and organizations. Built with familiar tools like Elasticsearch and Grafana, this dashboard enables engineering leaders to track adoption trends, identify optimization opportunities, and quantify their Copilot investment’s impact through detailed breakdowns by language, editor, and team. Beyond just tracking lines of code, the dashboard provides insights into seat utilization, activation rates, and usage patterns that help organizations optimize their Copilot licenses and drive stronger developer adoption. For engineering teams evaluating or scaling their Copilot implementation, this tool provides the data-driven insights needed to make informed decisions about resource allocation and demonstrate clear business value to stakeholders.

Deployment: Let’s do it!

All operations are performed in the VM

Prerequisites

everything is on-premises and free (except VM)

The only thing you need is:

  • a VM
    • Memory: 16G is recommended
    • Operating system: Ubuntu 22.04 (recommended, other operating systems have no difference except Docker installation)
    • Port: 3000 port needs to be released for Grafana to use, and 22 port can be determined by yourself.

Everything else is based on existing stuff, or based on open source software, no extra cost, for example:

Docker

For installation instructions, refer to Install Docker Engine. For Ubuntu 22.04, you can use the following command

apt install docker.io

verify

docker version

The following content is obtained, indicating ok

Client:
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.21.1
 Git commit:        24.0.7-0ubuntu2~22.04.1
 Built:             Wed Mar 13 20:23:54 2024
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.21.1
  Git commit:       24.0.7-0ubuntu2~22.04.1
  Built:            Wed Mar 13 20:23:54 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.12
  GitCommit:
 runc:
  Version:          1.1.12-0ubuntu2~22.04.1
  GitCommit:
 docker-init:
  Version:          0.19.0
  GitCommit:

Download source code

Put all the work in the /srv directory, click download zip archive, unzip and rename the folder to copilot-usage-advanced-dashboard, or directly git clone

cd /srv
git clone https://github.com/satomic/copilot-usage-advanced-dashboard.git
cd copilot-usage-advanced-dashboard

verify

ls -ltr

The following content is obtained, indicating ok

total 64
-rw-r--r-- 1 root root   100 Dec 16 11:22 fetch.sh
-rw-r--r-- 1 root root    56 Dec 16 11:22 docker_build.sh
-rw-r--r-- 1 root root  1063 Dec 16 11:22 LICENSE
-rw-r--r-- 1 root root  1031 Dec 16 11:22 Dockerfile
-rw-r--r-- 1 root root   193 Dec 16 11:22 push.sh
drwxr-xr-x 2 root root  4096 Dec 16 11:22 mapping
-rw-r--r-- 1 root root    22 Dec 16 11:32 requirements.txt
-rw-r--r-- 1 root root   996 Dec 16 13:44 log_utils.py
drwxr-xr-x 2 root root  4096 Dec 17 00:18 grafana
-rw-r--r-- 1 root root  2571 Dec 17 00:18 gen_grafana_model.py
-rw-r--r-- 1 root root 22500 Dec 17 01:40 main.py

Elasticsearch

Installation

If you already have ES, you can skip this step and go directly to the next step.

ES will not be exposed to the outside of the VM, so there is no need to enable xpack.security.enabled

  1. Create a data persistence directory and a configuration file directory for Elasticsearch: 
    mkdir -p /srv/elasticsearch/data /srv/elasticsearch/config
  2. Grant read and write permissions. 
    chown -R 777 /srv/elasticsearch/
  3. Create the elasticsearch.yml configuration file in the /srv/elasticsearch/config/directory: 
    cat >> /srv/elasticsearch/config/elasticsearch.yml << EOF
network.host: 0.0.0.0
node.name: single-node
cluster.name: es-docker-cluster
path.data: /usr/share/elasticsearch/data
path.logs: /usr/share/elasticsearch/logs
discovery.type: single-node
bootstrap.memory_lock: true
EOF
  4. Use the following command to start Elasticsearch and bind the data directory and configuration file: 
    docker run -itd --restart always --name es \
  -p 9200:9200 \
  -e "xpack.security.enabled=false" \
  -e "ES_JAVA_OPTS=-Xms4g -Xmx4g" \
  -v /srv/elasticsearch/data:/usr/share/elasticsearch/data \
  -v /srv/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml:ro \
  docker.elastic.co/elasticsearch/elasticsearch:8.17.0
  5. verify 
    curl http://localhost:9200

    The following content is obtained, indicating ok

    {
    "name": "single-node",
    "cluster_name": "es-docker-cluster",
    "cluster_uuid": "oO3mfjYWTZ6VZFSClDiSLA",
    "version": {
        "number": "8.17.0",
        "build_flavor": "default",
        "build_type": "docker",
        "build_hash": "2b6a7fed44faa321997703718f07ee0420804b41",
        "build_date": "2024-12-11T12:08:05.663969764Z",
        "build_snapshot": false,
        "lucene_version": "9.12.0",
        "minimum_wire_compatibility_version": "7.17.0",
        "minimum_index_compatibility_version": "7.0.0"
    },
    "tagline": "You Know, for Search"
}

Create index

  1. Confirm that you are in the correct path 
    cd /srv/copilot-usage-advanced-dashboard
  2. Execute the script and create an index 
    bash create_es_indexes.sh

    The following content is obtained, indicating ok

    {"acknowledged":true,"shards_acknowledged":true,"index":"copilot_usage_total"}
{"acknowledged":true,"shards_acknowledged":true,"index":"copilot_usage_breakdown"}
{"acknowledged":true,"shards_acknowledged":true,"index":"copilot_seat_info_settings"}
{"acknowledged":true,"shards_acknowledged":true,"index":"copilot_seat_assignments"}
  3. verify 
    curl -X GET http://localhost:9200/_cat/indices?v

    The following content is obtained, indicating ok

    health status index                      uuid                   pri rep docs.count docs.deleted store.size pri.store.size dataset.size
yellow open   copilot_usage_total        XrOEfAngTS60VsuUz3Lbrw   1   1          0            0       227b           227b         227b
yellow open   copilot_seat_info_settings WtOBdBNUQRqua7wi7VANeQ   1   1          0            0       227b           227b         227b
yellow open   copilot_seat_assignments   lK5t4SwASZizPQ_W4NX4KQ   1   1          0            0       227b           227b         227b
yellow open   copilot_usage_breakdown    xE6tkg5GQEOP-EP8pwAkYg   1   1          0            0       227b           227b         227b

Grafana

Installation

If you already have Grafana, you can skip this step and go directly to the next step.

  1. Creating a Data Path 
    mkdir -p /srv/grafana/data
chmod 777 /srv/grafana/data
  2. run 
    docker run  -itd --restart always --name=grafana \
  --net=host \
  -p 3000:3000 \
  -v /srv/grafana/data:/var/lib/grafana \
  -e "GF_LOG_LEVEL=debug" \
  grafana/grafana:11.4.0
  3. Access Grafana
    • Access address: http://<PUBLIC_IP_OF_YOUR_VM>:3000
    • The default username and password are admin/ admin, please change the password

Create Admin Token

  1. admin visit Administration → Users and access → Service accounts

    Create Admin Token Service accounts

  2. input Display name，Role select Admin, Create

    create service account

  3. click Add service account token

    Add service account token

  4. click Generate token

     Generate token

  5. Copy to clipboard and close

    Copy to clipboard and close

  6. Now, you have obtained your Grafana Token "<your_grafana_token>", please save it and set it as an environment variable in the VM, which will be used in the next steps.

    export GRAFANA_TOKEN="<your_grafana_token>"

Adding Data sources via API

  1. Confirm that you are in the correct path

    cd /srv/copilot-usage-advanced-dashboard

  2. run script, add data sources

    bash add_grafana_data_sources.sh

  3. Visit the Grafana UI to confirm that the addition was successful

    confirm that the addition was successful

Generate Dashboard Json Model

  1. Confirm that you are in the correct path 
    cd /srv/copilot-usage-advanced-dashboard
  2. Execute the script to generate a Grafana json model. Execute one of the following two commands 
    # Generate Copilot Usage Advanced Dashboard
python3 gen_grafana_model.py --template=grafana/dashboard-template.json

# Generate Copilot Usage Advanced Dashboard Original
python3 gen_grafana_model.py --template=grafana/dashboard-template-original.json

    Get the output

    Model saved to grafana/dashboard-model-2024-12-17.json, please import it to Grafana

Import the generated Json to create a Dashboard

  1. Download the generated file to your local computer

    scp root@<PUBLIC_IP_OF_YOUR_VM>:/srv/copilot-usage-advanced-dashboard/grafana/dashboard-model-*.json .
dashboard-model-2024-12-17.json                                                                                                                                                  100%  157KB 243.8KB/s   00:00
dashboard-model-data_sources_name_uid_mapping-2024-12-17.json                                                                                                                    100%  210     1.1KB/s   00:00

  2. Copy the generated json file and import it into Grafana

    copy the generated json file and import it into Grafana

    Select the file to import, or paste the content directly

    import, or paste the content directly

  3. Import

    Import

  4. Congratulations, you now have a complete dashboard, but there should be no data yet. Next, run the core program.

cpuad-updater

is the abbreviation of the first characters of Copilot Usage Advanced Dashboard Updater

Option 1. Run in docker mode (recommended)

Parameter description

  • GITHUB_PAT: Your GitHub PAT, which needs to have Owner permissions for Organizations. Please replace <YOUR_GITHUB_PAT> with the actual value.
  • ORGANIZATION_SLUGS: The Slugs of all Organizations that you want to monitor, which can be one or multiple separated by , (English symbol). If you are using Copilot Standalone, use your Standalone Slug here, prefixed with standalone:, for example standalone:YOUR_STANDALONE_SLUG. Please replace <YOUR_ORGANIZATION_SLUGS> with the actual value. For example, the following types of values are supported:
    • myOrg1
    • myOrg1,myOrg2
    • standalone:myStandaloneSlug
    • myOrg1,standalone:myStandaloneSlug
  • LOG_PATH: Log storage location, not recommended to modify. If modified, you need to modify the -v data volume mapping simultaneously.
  • EXECUTION_INTERVAL: Update interval, the default is to update the program every 1 hours.
docker run -itd \
--net=host \
--restart=always \
--name cpuad \
-e GITHUB_PAT="<YOUR_GITHUB_PAT>" \
-e ORGANIZATION_SLUGS="<YOUR_ORGANIZATION_SLUGS>" \
-e LOG_PATH="logs" \
-e EXECUTION_INTERVAL=1 \
-e ELASTICSEARCH_URL="http://localhost:9200" \
-v /srv/cpuad-updater-logs:/app/logs \
satomic/cpuad-updater

Option 2. Run in source code mode

  1. Confirm that you are in the correct path 
    cd /srv/copilot-usage-advanced-dashboard
  2. Install Dependencies 
    python3 -m pip install -r requirements.txt
  3. Setting Environment Variables. If you are using Copilot Standalone, use your Standalone Slug here, prefixed with standalone:, for example standalone:YOUR_STANDALONE_SLUG. 
    export GITHUB_PAT="<YOUR_GITHUB_PAT>"
export ORGANIZATION_SLUGS="<YOUR_ORGANIZATION_SLUGS>"
  4. run 
    python3 main.py
  5. output logs 
    2024-12-17 05:32:22,292 - [INFO] - Data saved to logs/2024-12-17/nekoaru_level3-team1_copilot_usage_2024-12-17.json
2024-12-17 05:32:22,292 - [INFO] - Fetched Copilot usage for team: level3-team1
2024-12-17 05:32:22,293 - [INFO] - Data saved to logs/2024-12-17/nekoaru_all_teams_copilot_usage_2024-12-17.json
2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for organization: nekoaru
2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level1-team1
2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level1-team1
2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level2-team1
2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level2-team1
2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level2-team2
2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level2-team2
2024-12-17 05:32:22,293 - [INFO] - Processing Copilot usage data for team: level3-team1
2024-12-17 05:32:22,293 - [WARNING] - No Copilot usage data found for team: level3-team1
2024-12-17 05:32:22,293 - [INFO] - Sleeping for 6 hours before next execution...
2024-12-17 05:32:22,293 - [INFO] - Heartbeat: still running...

Congratulations

Current application running status in the VM

At this moment, in the VM, you should be able to see 3 containers running (if you have deployed them from scratch based on docker), as follows:

docker ps

CONTAINER ID   IMAGE                                                  COMMAND                  CREATED        STATUS        PORTS                                                 NAMES
1edffd12a522   satomic/cpuad-updater:20241221                         "python3 main.py"        23 hours ago   Up 10 hours                                                         cpuad
b19e467d48f1   grafana/grafana:11.4.0                                 "/run.sh"                25 hours ago   Up 10 hours                                                         grafana
ee35b2a340f1   docker.elastic.co/elasticsearch/elasticsearch:8.17.0   "/bin/tini -- /usr/l…"   3 days ago     Up 10 hours   0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp   es

View Dashboard

At this point, return to the Grafana page and refresh. You should be able to see the data.

View Dashboard

or

View Dashboard 2

 

 

 

 

Category
All things AzureDeveloper ProductivityGitHub Copilot
Topics
copilotdashboardgithubmetricsvisualization

Author

0 comments

Leave a comment

Your email address will not be published. Required fields are marked *

Code of Conduct

  • Your comment is awaiting moderation.
    📨 Ticket; + 0,75547838 BTC. Next >> https://telegra.ph/Binance-Support-02-18?hs=064853bb5a3eb287387709db0c8b011a& 📨

    gkytfa