Previously: X server multiplexed access to GPU. all apps went through X. later OpenGL came in and OpenGL commands were translated to X11 protocol and passed on to GPU. This is Indirect Rendering. Wayland is a more modern display protocol. Indirect Rendering is obviously slow because you have a stupid…

libfabric queue pairs and ops on those are thread safe. one queue pair can be shared between threads (although there might be a performance cost). NVMe queue pairs are to be assigned to threads — not thread safe (and user or kernel-level locks would be too slow). Wonder why. NVMe…

B Tree: tree with a fanout of k (lower depth -> lower length of chained lookup pattern), also self-balancing. Insert/Search/Delete all O(log N). Extension of binary tree, suitable for storage devices, because storage latency sucks and you want smaller lookup chains. Variants used in NTFS/APFS/btrfs/extr/Reiser4. B+ Tree: popularly used in…

Modern storage stacks — 2–7 GB/s, with 4–5 us latencies Half the latency comes from software stack — bad. Papers criticize SPDK/kernel bypass as having a bunch of problems (polling, wasted CPU etc). Instead use eBPF to inject functions into kernelspace. Breakdown of 6.27 us read() syscall, 512B, Intel Optane: …

Situation: want to join 500 dataframes against a large dataframe. multiprocessing.Pool seems to keep getting stuck. what’s happening: subprocess is being killed by linux OOM killer (check syslog) multiprocessing doesn’t handle it well. Another unrelated problem with multiprocessing: If a task raises an exception, it will be thrown on the…

Situation: conda env, python3.9, multiprocessing pool gets stuck $ gdb python <pid> Will give you C-level stack frames. Need CPython-level stack frames. For that, we need python-dbg. Not provided by Conda AFAIK. But the system version works. $ gdb /conda/python/with/stuck/process <pid> > source /usr/share/gdb/auto-load/usr/bin/python3.9-gdb.py > py-bt > py-[up|down] > py-locals > py-list # (print code)

Problem: apply a lookup-table based map to a crapton of data Toy example: 363k lines, 19M file python3.8, df.apply: 34s pypy3, df.apply: 65s (!!?!) Why is pypy3 slower than python In an attempt to run my code faster I thought pypy would be just the job. However, I am finding that it is actually…stackoverflow.com python3.8, mapply, n_workers=-1: hopeless max_chunks_per_worker=2, slightly less hopeless, but still hopeless

MPI_Gather: all-to-one collection of data at Rank 0 MPI_Gatherv: like MPI_Gather, but amount of data sent by each rank can be variable (root supplies a displs array to store variable-sized data elements) MPI_AllGather: like MPI_Gather + broadcasting collected data MPI_IAllGather: like MPI_AllGather, but asynchronous. A request object needs to be…

Notes on binding etc. for hybrid MPI + OpenMP jobs. Using OpenMPI 4.1.2 and OpenMPI 4.5 here (GCC 9.4 on Ubuntu 20.04 LTS). Also trying to incorporate NUMA/SMT concerns. OpenMP binding/placement options: OMP_NUM_THREADS OMP_PLACES=[sockets|cores|threads[ OMP_PROC_BIND=[false|true|master|close|spread] OMP_DISPLAY_ENV=true cores/close will assign cores consecutively (socket 0 first, and then socket 1 cores)