Open in app

Sign In

Write

Sign In

Ankush Jain
Ankush Jain

37 Followers

Home

About

Nov 18, 2022

On Taylor Swift, Tickets, and Scale

The purpose of this post is to explore the design options for a ticket booking service, and see how it aligns with what TicketMaster (TM) does. …

5 min read

5 min read


Aug 27, 2022

notes on linux graphics

Previously: X server multiplexed access to GPU. all apps went through X. later OpenGL came in and OpenGL commands were translated to X11 protocol and passed on to GPU. This is Indirect Rendering. Wayland is a more modern display protocol. Indirect Rendering is obviously slow because you have a stupid…

1 min read

1 min read


Aug 24, 2022

On doorbells/NVMe etc.

libfabric queue pairs and ops on those are thread safe. one queue pair can be shared between threads (although there might be a performance cost). NVMe queue pairs are to be assigned to threads — not thread safe (and user or kernel-level locks would be too slow). Wonder why. NVMe…

2 min read

2 min read


Aug 24, 2022

On B+ Trees vs LSM Trees

B Tree: tree with a fanout of k (lower depth -> lower length of chained lookup pattern), also self-balancing. Insert/Search/Delete all O(log N). Extension of binary tree, suitable for storage devices, because storage latency sucks and you want smaller lookup chains. Variants used in NTFS/APFS/btrfs/extr/Reiser4. B+ Tree: popularly used in…

2 min read

2 min read


Aug 24, 2022

notes on ebpf/kernel bypass/storage latency

Modern storage stacks — 2–7 GB/s, with 4–5 us latencies Half the latency comes from software stack — bad. Papers criticize SPDK/kernel bypass as having a bunch of problems (polling, wasted CPU etc). Instead use eBPF to inject functions into kernelspace. Breakdown of 6.27 us read() syscall, 512B, Intel Optane: …

2 min read

2 min read


Jul 22, 2022

Python multiprocessing OOM handling

Situation: want to join 500 dataframes against a large dataframe. multiprocessing.Pool seems to keep getting stuck. what’s happening: subprocess is being killed by linux OOM killer (check syslog) multiprocessing doesn’t handle it well. Another unrelated problem with multiprocessing: If a task raises an exception, it will be thrown on the…

1 min read

1 min read


Jul 21, 2022

Debugging conda-python with gdb

Situation: conda env, python3.9, multiprocessing pool gets stuck $ gdb python <pid> Will give you C-level stack frames. Need CPython-level stack frames. For that, we need python-dbg. Not provided by Conda AFAIK. But the system version works. $ gdb /conda/python/with/stuck/process <pid> > source /usr/share/gdb/auto-load/usr/bin/python3.9-gdb.py > py-bt > py-[up|down] > py-locals > py-list # (print code)

1 min read

1 min read


Jul 21, 2022

Some Pandas/Python benchmarks

Problem: apply a lookup-table based map to a crapton of data Toy example: 363k lines, 19M file python3.8, df.apply: 34s pypy3, df.apply: 65s (!!?!) Why is pypy3 slower than python In an attempt to run my code faster I thought pypy would be just the job. However, I am finding that it is actually…stackoverflow.com python3.8, mapply, n_workers=-1: hopeless max_chunks_per_worker=2, slightly less hopeless, but still hopeless

1 min read

1 min read


Jul 11, 2022

MPI Collectives: Notes, benchmarks etc

MPI_Gather: all-to-one collection of data at Rank 0 MPI_Gatherv: like MPI_Gather, but amount of data sent by each rank can be variable (root supplies a displs array to store variable-sized data elements) MPI_AllGather: like MPI_Gather + broadcasting collected data MPI_IAllGather: like MPI_AllGather, but asynchronous. A request object needs to be…

4 min read

4 min read


Feb 4, 2022

MPI/OpenMP binding

Notes on binding etc. for hybrid MPI + OpenMP jobs. Using OpenMPI 4.1.2 and OpenMPI 4.5 here (GCC 9.4 on Ubuntu 20.04 LTS). Also trying to incorporate NUMA/SMT concerns. OpenMP binding/placement options: OMP_NUM_THREADS OMP_PLACES=[sockets|cores|threads[ OMP_PROC_BIND=[false|true|master|close|spread] OMP_DISPLAY_ENV=true cores/close will assign cores consecutively (socket 0 first, and then socket 1 cores)

1 min read

1 min read

Ankush Jain

Ankush Jain

37 Followers

Umm…

Following
  • Intel

    Intel

  • Jon Alexander

    Jon Alexander

  • Bernard Woolley

    Bernard Woolley

  • Max Howell

    Max Howell

  • The Stoa

    The Stoa

Help

Status

Writers

Blog

Careers

Privacy

Terms

About

Text to speech