QL's blog

How to run synchronous code with asyncio

Fri, 29 Sep 2023 12:56:00 +0200

[2023-09-29 Fri 12:17]

Suppose you’re experimenting with Playwright through Jupyter for HTML scraping, you’ll notice that Playwright offers two types of APIs: sync_api and async_api. (Playwright | Playwright Python)

from playwright.sync_api import sync_playwright

import asyncio
from playwright.async_api import async_playwright

The trend is towards using the async API, but several libraries are yet to catch up or might prefer to stick to sync. There’s no inherent issue with sync API. The idea is to understand and utilise the APIs differently to suit our needs.

Calling the sync API directly within the event loop isn’t advisable, since it will block the loop and hang the entire system, making it unresponsive to events like mouse movements, keyboard inputs, and incoming requests. When Playwright’s sync API refuses to function from an async loop, that’s a correct behaviour.

from llama_index import download_loader
ReadabilityWebPageReader = download_loader("ReadabilityWebPageReader")
loader = ReadabilityWebPageReader()
documents = loader.load_data(url='https://github.com/n8n-io/n8n/pull/7136')

Error: It looks like you are using Playwright Sync API inside the asyncio loop.
Please use the Async API instead.

So, we need to address this – how can we run synchronous code as needed? The strategy here is to launch it via a separate thread, and asyncio conveniently allows us to do that.

import asyncio
from llama_index import download_loader
ReadabilityWebPageReader = download_loader("ReadabilityWebPageReader")
loader = ReadabilityWebPageReader()
documents = await asyncio.to_thread(loader.load_langchain_documents, url='https://github.com/n8n-io/n8n/pull/7136')

We can also concurrently run multiple instances of the synchronous code, either via their independent threads or through a thread pool. Considering Playwright library’s operations are I/O bound, waiting for the response can be significant. However, bear in mind that Playwright itself can launch multiple browsers, which could be quite CPU intensive.

import asyncio
from llama_index import download_loader
ReadabilityWebPageReader = download_loader("ReadabilityWebPageReader")
loader = ReadabilityWebPageReader()

# Spawn one thread per request
documents = await asyncio.gather(*[asyncio.to_thread(loader.load_langchain_documents, url=url) for url in [
    'https://github.com/n8n-io/n8n/pull/7136',
    'https://github.com/n8n-io/n8n/pull/7136',
    'https://github.com/n8n-io/n8n/pull/7136',
    'https://github.com/n8n-io/n8n/pull/7136'
]])

# Spawn a pool of 2 threads and run 4 instances of the code in the pool
pool = ThreadPoolExecutor(max_workers=2)
documents = await asyncio.gather(*[asyncio.get_event_loop().run_in_executor(pool, lambda: loader.load_langchain_documents(url=url)) for url in [
    'https://github.com/n8n-io/n8n/pull/7136',
    'https://github.com/n8n-io/n8n/pull/7136',
    'https://github.com/n8n-io/n8n/pull/7136',
    'https://github.com/n8n-io/n8n/pull/7136'
]])
pool.shutdown()

scraped: https://github.com/n8n-io/n8n/pull/7136
scraped: https://github.com/n8n-io/n8n/pull/7136
scraped: https://github.com/n8n-io/n8n/pull/7136
scraped: https://github.com/n8n-io/n8n/pull/7136
scraped: https://github.com/n8n-io/n8n/pull/7136
scraped: https://github.com/n8n-io/n8n/pull/7136
scraped: https://github.com/n8n-io/n8n/pull/7136
scraped: https://github.com/n8n-io/n8n/pull/7136

You might notice that the first four requests might be quicker, being scraped simultaneously while the latter are processed in batches of two - slower, but using lesser CPU. This is the performance trade-off we need to account for.

Happy code experiments!

Firestore maintainance

Fri, 24 Mar 2023 17:58:00 +0100

Recently, while working on a Firebase project, I realized the need for a tool that can perform backup and restore for Firestore databases, as well as extract production data for local or staging environments. While there is a commercial project, Firefoo - The Powerful GUI Client for Firebase Firestore, which is easy to use, it’s quite slow and automation may not be possible.

However, there is an official way to achieve this by using the gcloud command. Essentially, you can export the data to a bucket, which can then be copied to local storage using rclone remote.

gcloud firestore export gs://REDACTED.appspot.com/backup-$DATE
rclone copy -P REDACTED:REDACTED.appspot.com/backup-$DATE/ .
firebase emulators:start --import backup-$DATE

But if you ever want to have actual control and do the filtering via the excellent jq for example, you are going to need a way to export the data to JSON. This script below handles the basic operations needed, which also includes handling sub-collections and can manage a very large collection thanks to the implementation via stream and JSONL.

To export the whole database, you can do it like this:

curl http://REDACTED/maintainance\?secret\=$SECRET\&action\=listCol | xargs -I{} bash -c "echo {} && curl REDACTED/maintainance\?secret\=$SECRET\&action\=getCol\&col\={}\&withSubCollections\=1 | jq -c > {}.jsonl" \;

/**
 *  Maintainances
 */
app.post("/maintainance", (req, res) => {
  (async () => {
    if (req.query.secret !== SECRET) {
      res.status(500).json({ error: "what do you want from me?" });
      return;
    }

    const allowActions = [
      "count",
      "listCol",
      "getCol",
      "importCol",
      "DANGEROUSLY_delCol",
    ] as const;

    const action = req.query.action as unknown as typeof allowActions[number];

    if (action === "count") {
      const cols = (req.query.col as string).split(","); // let it crash
      const count: { [k: string]: number } = {};
      for (const col of cols) {
        const snapshot = await db.collection(col).select().get();
        count[col] = snapshot.size;
      }
      res.json({ count });
    } else if (action === "getCol") {
      const rootCol = db.collection(req.query.col as string); // let it crash
      const withSubCollections = !!req.query.withSubCollections;

      /**
       * should be pairs of doc/col/doc/col... of the req.query.col as root collection
       */
      let subCollectionsQueue = new Set<string>();
      let pending: Promise<unknown>[] = [];

      const snapshot =
        rootCol.stream() as unknown as AsyncIterableIterator<DocumentData>;

      for await (const doc of snapshot) {
        res.write(JSON.stringify({ id: doc.id, data: doc.data() }));
        if (withSubCollections) {
          pending.push(
            (async () => {
              const subCols = await doc.ref.listCollections();
              if (subCols.length > 0) {
                for (const subCol of subCols) {
                  if (!subCollectionsQueue.has(subCol.id)) {
                    console.log("  queue subCollection", [
                      req.query.col,
                      subCol.id,
                    ]);
                    subCollectionsQueue.add(subCol.id);
                  }
                }
              }
            })()
          );
        }
      }

      if (withSubCollections) {
        if (pending.length > 0) {
          console.log(`wait for pending operations: ${req.query.col}`);
          await Promise.all(pending);
          pending = [];
        }

        while (subCollectionsQueue.size > 0) {
          const path = subCollectionsQueue.values().next().value;
          subCollectionsQueue.delete(path);
          const col = db.collectionGroup(path);
          const snapshot =
            col.stream() as unknown as AsyncIterableIterator<DocumentData>;
          for await (const doc of snapshot) {
            pending.push(
              (async () => {
                const subCols = await doc.ref.listCollections();
                if (subCols.length > 0) {
                  for (const subCol of subCols) {
                    if (!subCollectionsQueue.has(subCol.id)) {
                      console.log("  queue subCollection", [
                        req.query.col,
                        subCol.id,
                      ]);
                      subCollectionsQueue.add(subCol.id);
                    }
                  }
                }
              })()
            );
            let parent = doc.ref.parent;
            let paths = [parent.id, doc.id];
            while (true) {
              parent = parent.parent;
              if (!parent) {
                break;
              }
              paths.unshift(parent.id);
            }
            const rootName = paths.shift();
            if (rootName === req.query.col) {
              res.write(
                JSON.stringify({
                  id: paths.join("/"),
                  data: doc.data(),
                })
              );
            }
          }
          if (pending.length > 0) {
            console.log(
              `wait for pending operations: ${req.query.col},... ${path}`
            );
            await Promise.all(pending);
            pending = [];
          }
        }
      }

      res.end();
    } else if (action === "listCol") {
      const cols = await db.listCollections();
      res.end(cols.map((it) => it.id).join("\n"));
    } else if (action === "importCol") {
      const col = db.collection(req.query.col as string); // let it crash
      const withSubCollections = !!req.query.withSubCollections;
      const pipeline = req.pipe(
        jsonlParser()
      ) as unknown as AsyncIterableIterator<{
        value: { id: string; data: unknown };
      }>;
      let count = 0;
      let batch = db.batch();
      let cap = 0;

      for await (const item of pipeline) {
        count += 1;
        if (cap > 400) {
          await batch.commit();
          cap = 0;
          batch = db.batch();
        }

        cap = cap + 1;
        if (withSubCollections) {
          const path = item.value.id.split("/");
          let ref = col.doc(path.shift()!);
          if (path.length > 1) {
            ref = ref.collection(path.shift()!).doc(path.shift()!);
          }
          batch.create(ref, item.value.data);
        } else {
          const ref = col.doc(item.value.id);
          batch.create(ref, item.value.data);
        }
      }

      await batch.commit();
      res.json({ count });
    } else if (action === "DANGEROUSLY_delCol") {
      const cols = (req.query.col as string).split(","); // let it crash
      const count: { [k: string]: number } = {};
      for (const col of cols) {
        const snapshot = await db.collection(col).select().get();
        let batch = db.batch();
        let cap = 0;
        for (const item of snapshot.docs) {
          if (cap > 400) {
            await batch.commit();
            cap = 0;
            batch = db.batch();
          }

          cap = cap + 1;
          batch.delete(item.ref);
        }
        await batch.commit();
        count[col] = snapshot.size;
      }
      res.json({ count });
    } else {
      res.end(allowActions.join("\n"));
    }
  })();
});

Decluttering org headings: Removing extranous IDs

Sun, 26 Jun 2022 17:34:00 +0200

If you’ve ever tried to use dengste/org-caldav: Caldav sync for Emacs orgmode to synchronize your org calendar, you may have noticed that it adds an ID to all headings by design. For some, these extra PROPERTIES blocks can be quite distracting. To address this, I proposed a patch to create IDs for only relevant headings here: https://github.com/dengste/org-caldav/issues/227.

However, if it’s already too late, here is a script to clean things up. You can evaluate it to remove all IDs that are neither a target, have attachment(s), nor a timestamp (synced with CalDAV). It assumes everything is in the ~/org directory, including archived files, and requires the use of rg, jq, and awk. Please back up your org files before running it (presumably via git) and run org-lint to verify that nothing is broken.

(let* ((default-directory (expand-file-name "~/org"))
       (ids (-> (shell-command-to-string
                 "rg -a -g '*.org*' --json '^:ID:\s*([A-Za-z00-9\-])+\s*$'  | jq -r '.data.submatches | select(.) | .[].match.text' | awk '{ print $2 }' | sort")
                (string-trim)
                (split-string "\n")))
       (targets (-> (shell-command-to-string
                     (format "rg -a -g '*.org*' --json '\\[id:(%s)]' | jq -r '.data.submatches | select(.) | .[].match.text[4:-1]' | sort" (string-join ids "|")))
                    (string-trim)
                    (split-string "\n")))
       (not-targets (seq-difference ids targets)))
  (save-excursion
    (seq-each #'(lambda (id)
                  (condition-case nil
                      (progn
                        (org-id-goto id)
                        (unless (or (org-entry-get nil "SCHEDULED")
                                    (org-entry-get nil "DEADLINE")
                                    (org-entry-get nil "TIMESTAMP")
                                    (member "ATTACH" (org-get-tags nil t)))
                          ;; (message "DELETE %s" id)
                          (org-delete-property "ID")))
                    (error (message "FALSE POSITIVE %s" id))))
              not-targets)))

A fennel REPL for Hammerspoon

Tue, 29 Mar 2022 19:47:00 +0200

The Fennel programming language is one of my favorite programming languages. It gets a lot of things done right:

Modern syntax (Clojure-like, but better)
Immutable by default
Excellent pattern matching

If I ever find myself working with a Lua interpreter (Hammerspoon, Neovim, TIC-80, etc.), Fennel is often my go-to for concise and efficient code.

The question is, can I have a REPL for Hammerspoon? There are multiple options available, like nREPL via jeejah, but I find it limited compared to the REPL that comes with fennel-mode. Hammerspoon does come with an interactive shell, and I would like to adapt it to work with fennel-mode. In fact, it’s very possible. Hammerspoon comes with a hook, _consoleInputPreparser, that can be hacked to evaluate an expression. Essentially, we can run a REPL in a coroutine and resume it whenever the user inputs something.

(require :hs.ipc)

(local fennel (require :fennel))

(let [coro (coroutine.create fennel.repl)]
  (coroutine.resume coro {:readChunk (fn []
                                       (let [input (coroutine.yield)]
                                         (.. input "\n")))
                          :onValues (fn [xs]
                                      (print (table.concat xs "\t")))
                          :onError (fn [_ msg]
                                     (print msg))})
  (set hs._consoleInputPreparser (fn [s]
                                   (coroutine.resume coro s)
                                   "")))

Here is the result:

And of course, you can evaluate code and get completion with fennel-mode, too.

One-liner to enable Touch ID for sudo

Sat, 01 May 2021 20:23:00 +0200

You may already know how to enable Touch ID for sudo already, but it’s constantly getting disabled after each update of macOS. This one-liner is a no-brainer to enable it again. Basically, type it once in your terminal, and type ‘^R pam.d ENTER’ to enable it again once needed and it will be enabled again.

sudo sed -i '' '2i\
auth       sufficient     pam_tid.so
' /etc/pam.d/sudo

Backup all helm stacks

Tue, 29 Sep 2020 10:40:00 +0200

Recently we implemented CI/CD using linktohack/helm-stack: Ultimate Helm chart to deploy all kinds of stack and I find the need for a way to export and backup all the values and manifests of our applications .

This Makefile allows me to make snapshot and make template in a CronJob to back up all of our stack. It comes with an extra target template-latest to generate the latest manifest of the stack, but can be of course adapted to generate whatever version of the manifest (1.25, 1.22, 1.18 etc.).


snapshot:
        helm ls -A -o json \
                | jq -r '.[] | "echo \(.namespace)/\(.name) && mkdir -p values/\(.namespace) && helm -n \(.namespace) get values \(.name) | sed /USER-SUPPLIED/d > values/\(.namespace)/\(.name)-\(.chart).yaml"' \
                | sort \
                | sh -

template:
        helm ls -A -o json \
                | jq -r '.[] | "echo \(.namespace)/\(.name) && mkdir -p values/\(.namespace) && helm -n \(.namespace) get manifest \(.name) > values/\(.namespace)/\(.name)-\(.chart).spec.yaml"' \
                | sort \
                | sh -

template-latest:
        helm ls -A -o json \
                | jq -r '.[] | select (.chart | startswith("stack-")) | "echo \(.namespace)/\(.name) && mkdir -p values/\(.namespace) && helm -n \(.namespace) template \(.name) link/stack -f values/\(.namespace)/\(.name).yaml > values/\(.namespace)/\(.name)-latest.spec.yaml"' \
                | sort \
                | sh -

Options to migrate from Swarm to K8s

Wed, 01 Jul 2020 15:00:00 +0200

Back in 2017, I decided that our applications should be deployed in a Swarm cluster. Fast forward to 2020, we have more than 50 stacks running and the number is still counting on.

I have no regret of that decision: (1) The simplicity of docker-compose file format allows us to deploy a new stack very quickly, skipping the ops part. Any developer can have a look at the stack file and say exactly what’s going on. (2) The declarative nature of config files (minimal) and volumes allow us to backup and update the application with ease, basically create a new snapshot, try to change the image version and re-deploy the stack. With the help of network drive (and advance FS) like NFS with ZFS we, don’t have to worry about volume provision and applications can be scaled quickly within the limit of the stateful part (DB and Storage.)

As of 2020, the war between Swarm and K8s seems to be settled. Swarm seems to be more appropriate for the hobbyists (single node cluster) and/or RPi cluster etc. If I’m looking for the auto-scale ability of the cloud provider to prepare for bigger applications, the only solution seems to be K8s.

Current structure

A Swarm cluster of a bunch of VPSes for memory and CPU
A pool of NFS servers served as storage for application
Traefik as Ingress controller
Let’s Encrypt as TLS provider

A typical `docker-compose` file

version: '3.1'

services:
  redmine:
    image: redmine
    volumes:
      - /path/to/nfs/volume/tld.redmine.www/redmine/config/configuration.yml:/usr/src/redmine/config/configuration.yml
      - /path/to/nfs/volume/tld.redmine.www/redmine/files:/usr/src/redmine/files
    environment:
      REDMINE_DB_MYSQL: db
      REDMINE_DB_PASSWORD: example
    networks:
      - default
      - web
    deploy:
      placement:
        constraints:
          - node.hostname == prod2
      labels:
        - 'traefik.frontend.rule=Host:www.redmine.tld'
        - 'traefik.port=3000'

  db:
    image: mysql:5.7
    volumes:
      - /path/to/nfs/volume/tld.redmine.www/mysql:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: example
      MYSQL_DATABASE: redmine
    networks:
      - default
    deploy:
      placement:
        constraints:
          - node.hostname == prod2

networks:
  web:
    external: true

Idea

The idea of this post is to evaluate the possibility target K8s deployment, basically transform the above docker-compose file to a bunch of Service, Deployement and Ingress and so on.

Imagine a new structure of:

A K8s cluster with
NFS provisioner (that should be changed to cloud provider if necessary)
Traefik as Ingress controller
Let’s Encrypt as TLS provider

In order to migrate, each of the above two services should generate:

A static PV with Retain policy point to the correct location beneath the NFS mount point and a claimRef point to
A PVC, should be bound automatically to the above PV, serve as the volume for the Pod
An Ingress that exposes www.redmine.tld from the above labels via the port defined by (no for the db service: port-forward if we need to access the DB from outside)
A ClusterIP Service that expose the port defined by the label to the cluster
A Deployment to deploy the container. It should be able to handle at least replicas and placment properties of the yaml

Ideally this process should be done automatically from the above yaml file, allows us to use more advance features of K8s while keeping the simplicity of docker-compose file format.

I can go ahead and list all of them here (2 services * 5 files = 10 files in total) but let’s skip them for the sake of brevity.

Research

Write it manually

Too much, should be 10 times longer, 400 lines of code for a simple service, with a lot of duplication. We will compare the final result if one of anything below works…

Generator from `stack.yaml`

Sound like… Helm

Helm

Two possibilities:

Use stack file directly as values.yaml
A big helper file to extract data from each service of yaml file and extends the templates (10 files only 1, 2 line with per service/deployment etc from these values, or they can be merged into one single file)

Looking at a Helm chart example, it seems unimaginable but imagine we use only one chart for every stack, the trade of worth it. The helper file can be versioned and symlinked to the chart.

Resouces

Let’s Migrate Symfony Project to Kubernetes! Part 2: Publishing the Application with Helm.

Kustomization

Heard about it, but how to extract variables from a docker-compose file?

Compose-on-kubernetes

Looks like the holy grail. But…

Ingress seems to be missing

DONE Evaluation

Let’s go with Helm first…

(Somehow) it works for above docker-compose.yml. The final output of all the deployment object is about 240 lines, only 6 times bigger than the stack.yml.

The repo can be found here: linktohack/helm-stack: Deploy your `docker-compose` `stack` with Helm.

Next step

The next step is to evaluate other stack files. There are still some points need to be worked on:

Handle more labels property (custom header etc…)
Handle node constraint (nodeSelector)
Less assumption about the stack file: i.e. normalize it
- Separated volumes section
Network section

About me

Mon, 01 Jan 0001 00:00:00 +0000

I am a Director of Software Development at BFAST, a subsidiary of Europe Holding Group (a.k.a Groupe Transalliance).

Polygot in both speaking and programming languages. I like the nuance of Japanese, the two extremities of ML and Lisp.

Projects