CircleCI in daily life: how I improved Matrix stickerpicker

CircleCI is a great tool that everyone seems to skip and I don’t know exactly why. I’m gonna quickly show you how I’ve automated adding stickers to the Matrix messenger.

What is CircleCI?

CircleCI is a cloud platform that can run automated jobs when different events occur. The most common use-case is running tests in the cloud when you push a commit to the GitHub repo. But the possibilities are endless, you can literally automate everything with it - I even used it to compile .exe file in the cloud and include a zipped binary into the release package. Don’t ask me why I had to compile .exe, better ask why the guys from my University made a deal with the Devil.

Matrix

Matrix is a messaging protocol which I’m currently trying to replace Telegram with. It’s in itself a separate topic for discussion, but what’s important for today is that Matrix lacks such a huge collection of stickers as there is in Telegram. I’m a big fan of Wojak stickers and many other ones, so I had to do something about it.

stickerpicker

Matrix Stickerpicker

Stickers work in a slightly weird way in Matrix. Basically you have to deploy your own sticker “server” (don’t worry, it’s just a static site) where you can add your stickers - it will be used as a widget inside your client.

There’s an amazing tool for that - stickerpicker. It’s written in Python and the author of course took care of the import feature - you can provide a link to a Telegram sticker pack - it will re-upload it to your Matrix server and then generate a web UI using GitHub Pages that you can integrate as a widget in your client. Your friends can use it too.

The problem is that each time you want to add new sticker pack to your collection you have to run a Python script. I’m lazy and I don’t wanna do that - so let’s automate it!

Automation

Main idea of my solution is to have a single plaintext file telegram-packs.txt which will contain links to all sticker packs I import from Telegram. Then every time I update it CircleCI will run a job that will run this Python script for me. Deadly simple.

This approach ships with few bonuses by default:

You can edit the file from anywhere (even from phone 😆)
If someone from your friends uses your stickers collection and wants to add a new pack, they simply have to open a PR which adds the pack URL to telegram-packs.txt.

There are two problems of running this Python script automatically:

You have to log into your Matrix account
You have to log into your Telegram account

Config

Here’s my fork of the repo: https://github.com/eug-vs/stickerpicker

CircleCI is fully controlled by a config file .circleci/config.yml. It uses YAML which is really picky about indentation, so be careful with that.

Ok so let’s start outlining our job, we’ll be using default CircleCI Python image:

defaults: &defaults
  working_directory: ~/repo
  docker:
    - image: cimg/python:3.9.5

jobs:
  upload_stickers:
    <<: *defaults
    steps:

The defaults part might look scary, but you could just move everything from defaults directly under upload_stickers job. Defaults are useful when you have multiple jobs, and I planned to have 2 initially. Let’s now move to the steps:

- checkout

- run:
    name: Install python dependencies
    command: pip3 install .

Obviously we checkout our code first, and then install all the dependencies for our Python script.

Now let’s solve the first problem we found. Instead of logging into Matrix manually, you can supply a config.json file with your credentials. So I prepared a template for this file (config.template.json):

{"homeserver": "$HOMESERVER", "user_id": "$USER_ID", "access_token": "$ACCESS_TOKEN"}

You might think how am I gonna put the actual variables in here? Well, there’s a tool for that called envsubst. It does exactly what you think - substitutes environment variable names with the actual values. We will put the actual values of $HOMESERVER, $USER_ID and $ACCESS_TOKEN in our CircleCI environment (as secrets), and the template can be safely pushed to the GitHub repo.

There’s a little problem - envsubst is not installed in the image that we use (cimg/python:3.9.5). Solution to it is orbs - basically plugins for your CircleCI needs. Let’s add envsubst orb at the top of our config:

orbs:
  envsubst: sawadashota/envsubst@1.1.0

Now back to the steps. Let’s install the tool and use it to generate our config.json:

- envsubst/install

- run:
  name: Create config.json from template
  command: envsubst < config.template.json > config.json

Ok so we can now login into Matrix, but what about Telegram? The script uses Telethon package under the hood which works with a sticker-import.session file. This is basically an SQL database, and I don’t really wanna mess around with creating it from scratch (although I could!). Instead, I will just encrypt it with GPG and push to the repo. That’s just me being lazy, but you know, that is what automation is about! Of course, someone could try to decrypt my Telethon session since it’s now publicly available, but I can just revoke it at any time. I will probably improve this at some point, but now it’s not a big deal for me.

Our next steps in config will be to decrypt the session and finally pipe our telegram-packs.txt to the script using xargs:

- run:
    name: Decrypt telethon session
    command: gpg --batch --passphrase $PASSPHRASE --decrypt sticker-import.session.gpg > sticker-import.session

- run:
    name: Reupload and install stickers
    command: cat telegram-packs.txt | xargs sticker-import

At this point, the stickers are uploaded to the server, but the UI is updated via it’s own .json file which contains information about uploaded stickers. It’s already updated by the script, we just have to push it to our repo. For that I’ve created a new SSH key and added it to CircleCI secrets so that it can push commits to my repository. So the final steps would be:

- add_ssh_keys

- run:
    name: Add github to known_hosts
    command: |
      mkdir -p ~/.ssh
      ssh-keyscan github.com >> ~/.ssh/known_hosts
- run:
    name: Commit stickerpack JSON's
    command: |
      git config --global user.email "eug-vs@keemail.me"
      git config --global user.name "eug-vs"
      git add .
      git commit -m "feat: add stickerpacks with CircleCI [ci skip]"
      git push -u

The [ci skip] flag tells Circle to avoid running on this commit, otherwise it would have to run once again since we pushed a new commit with it.

One thing that I didn’t mention is that I did not actually need to check the new packs in the telegram-packs.txt - the script is actually smart enough to skip already existing ones. Otherwise I would have to use git diff to find only the new ones.

And that’s it! Now every time I add new URLs to the telegram-packs.txt, they will automatically appear in my widget within a minute or so. Enjoy your stickers!

wojak-brain-chair

You can find full version of the config here: https://github.com/eug-vs/stickerpicker/blob/master/.circleci/config.yml