As of 2010-11-15 I do not plan to update these pages. Not that I have done so in the
past couple of years anyway. However, the pages will remain on ze interwebs for hysterical raisins.
Also, due to switching servers, some links might point to places that no longer
exist (I intentionally left out some cruft during the move, but am too lazy to
fixup the links). If you run across such a link, you may imagine the stuff that
was previously found the link (if you really want to acccess it, drop me a line and
I'll ask the mightly /dev/urandom how to deal with it).
Image galleries (very partial)
Semi-random collection of links to content on this "site" (aka "public projects finished")
Most projects listed here were developed during 2006-2007.
- Non-fiction book list.
- (in Finnish) Instructions on how to build custom RHEL4 kernels.
- (in Finnish) How to setup dynamic updates with BIND9 in targetted SELinux-environment.
- (in Finnish) Tietojärjestelmien luotettavuus, esitys Ohjelmistotuotannon menetelmät -kurssille. (Open Document Format, Presentation). PDF-versio. (English: Reliability in information systems, outlining the reasons why the black-box model is somewhat flawed when applied to modern systems and electronics.)
- Step-by-step guide on implementing network security on RHEL4/CentOS. Some of the text also applies to other Linux distributions.
- Step-by-step guide on how to build out-of-tree kernel modules for the Nokia N800.
- Improved two-phase syntax high-lighter for C (demo output).
- Why HTML-emails are a bad idea (excerpt of actual SPAM that I received).
- Reboots For Fun And Profit. (You could also use reboot -f now.)
- A script to send an email when filesystem usage has exceeded a limit.
- A document that started from describing the largest differences between HP-UX and Linux but since acquired a life of its own. Contains typos and other brainfarts.
- Getting root shell on UNIX-systems (not an exploit).
- Google is fun. Found an old asm program (size optimization contest which I started) floating around: SMALLPAL.ASM. I was 16 years old back then :-). Program compiles into 79 bytes (.COM) and requires DOS and VGA.
- Automatic L2-topology, port and host discovery (netmap) by using only BRIDGE-MIB. Sanitized example of a real network at off-office hours. Hybrid python/C.
- What exactly is disk speed? (full data area linear read measurements over commodity hard drives). Now also featuring measurements over flash devices, SD/MMC cards, CD-ROMs and DVDs as well as other interesting technical blip-blap.
- Somewhat newer graphs using the automagic grapher. (Includes USB "hard disks", some RAID-tests and the most of the original blockspeed images.)
- How to answer the question "How much memory is a process using now".
- Tool to report memory usage for larger systems. Example output for a single CPU desktop system. Implemented in python.
- Tool to display system wide memory usage once per second. Quite different goals compared to meminfo. Both python and C versions.
- (SIS-related) "Random Access into Compressed data"-format (aka RAC). Needed a mechanism to access the virtual machine images in random fashion without decompressing the whole thing at once so decided to implement a custom format.
Encoder works as filter (which is nice), decoder cannot read from stdin but can output to stdout. Internally implements desparsing so that finding out whether given logical sector(s) are zero or not is an operation
that doesn't require uncompression or I/O to the compressed files. The underlying compression "algorithm" is zlib (modified LZ77). Decompression speed is very good. Here is a table comparing different compressors/decompressors on Linux (I blame OpenOffice for the funky formatting). RAC is included (a first unoptimized version).
Test data is a bunch of VHD-format virtual machine disk files (base or delta). RAC-codecs implemented in C. Completed.
- Zerotools, a toolset that aids in keeping virtual machine disks cleaner by filling released virtual disk space with binary zeroes. Also acts as a template on implementing a generic LD_PRELOAD-based wrapper. Implemented in C.
- A small and simple utility to do chroot, cwd, drop-priv and exec all in one go. Implemented in C.
Current projects (in no particular order)
All computer/networking related stuff is done using Python/C/Asm on Linux (obviously).
- MoomEek: replacement for rlplot for blockspeed. Implemented in C with libfreetype and libpng as the only external dependencies. Semi-active (not feature complete).
- blockspeed: add RPM-measurement (rethink), try whether overlapping AIO ops will hide the cache mess, cache-banging tests (for HBA-tests), cache-mod-bang test (sector N stays constant), do some runs on SCSI stuff and PLs. On hold.
- A7 (multipurpose custom init-replacement and hw boot-strap mechanism, and much much more. Screenshot from PXE-booted version). Another screenshot in which regular top and bash are running. Footprint of A7 (a7slave) is pretty minimal as you can see (80KiB unique data and code, including dirty stack). Implemented in C as a single binary implementing functionality of udevd, modprobe, init and automatic hardware bootstrapping and network init, however the version in screenshot is linked against a regular dynamic glibc (hence the large size). Boots quickly (especially over network). On hold.
- Efficient storage and graphing of time-series measured sparse datasets (rrdtool is not efficient). Implemented in C, but prototypes currently in python. On hold.
- De-centralized multi-site L2-admission control system. Custom RADIUS/DHCP with topology tracking. PXE/WOL thrown in as a bonus. Hybrid python/C. On hold.
- L2-topology based host/service alerting (see netmap above) via SMS. Hybrid python/C. On hold.
- Trafstat2: Adding multi-VLAN support over mirror-switch ports into trafstat. On hold.
- Docs-1250: A docs-midway replacement that uses synhilite, generates automatic dependency info for make and outputs XHTML/Strict (zerotools and user-chroot pages are generated with this). Implemented in python. "Complete".
- snarfcontrol: making automatic config backups after switch (procurve) configs change. Integrated as part of ssh/menu-driven switch management system. Contains a single-shot tftp server, telnet scrubbing, as well as multiple control daemons and wrappers. Implemented in python and C. Active.
- spectre: network booted multicasting imaging system for large environments (NTFS primary target). Imaging clients will run Linux (as well as the server(s)). Will be released as GPL. Hybrid python/C. Waiting for sponsorship(s).
Projects on hold/completed non-public
These projects are still all useful/necessary/fun, but mostly they're waiting for better times or they're already complete, but I'm too lazy to make example outputs/screenshots.
Most projects listed here were started/developed during 2005-2007.
- ionos: combine trafstat2, scannet and real-time netmap. Hybrid python/C. Not properly started.
- Layer 2 Love Machine (L2LM): client-server solution to combine separate L2 LANs transparently across multiple geographical sites into one broadcast domain using a star-topology TCP-forwarding. Works but does things without MAC-filtering, so ends up sometimes forwarding packets unnecessarily. Implemented in C (server prototype in python, but was going to be reimplemented in C). Needs more loving. Abandoned (the problem that L2LM solved went away).
- which-driver: program to collect PCI vendor:device codes from a Linux source code tree in order to make it possible to quickly determine whether a particular version of Linux will support a specific device (without building the kernel). Seemed like a great idea, but turned out to be too difficult to implement (compared to the need). Abandoned (after 500 lines of python).
- Docs-1500: A middle-way version of ASCII -> OASIS XML -> scripted OO.o -> PDF pipeline driven by make. Implemented in python. On hold.
- GoSlo: L2 transparent traffic shaper with web-interface, remote control and failure simulation modes. Works, needs to integrate with A7 and virtual IP support. Implemented in C/Python (latter for web-frontend). On hold.
- MUPE/Chirp (Most Useless Project Ever): software build process audiolizer (suprisingly enough binary object files do not sound like white noise). On hold.
- Traffic priorization stuff (adaptive link compression with fast cut-through support for low-latency data). Implemented in C. On hold.
- Adding vmware video driver support into kdrive-drivers in order to complete a vmware-ready kernel image which contains vmxnet and all necessary drivers to run on any modern vmware system. On hold.
- Adding necessary ioctls into aoe-client driver to fool VMware to think that the device is IDE/SCSI block device. Makes vmbd-wrapper unnecessary when using AoE as raw-disks for VMware guest. Completed (also fixed a bug in AoE client), but fixes not submitted yet.
- Visualisation of block device operations over time as a useful graph. Necessary for SIS. Needs the aoe-client fix to be implementable. Implemented in python (SVG output). On hold (data collection with lockless side-daemon, SVG generation not implemented).
- SIS (Single Instance Storage), a way to save space when handling large blocked data (virtual machine images, ISO images, etc). Quite promising (beats rzip/bzip2/gzip, first two in speed as well while providing random access to stored blocks). Related post-projects are an AoE-emulator to access virtual images (using read/write cycles from virtual machines) as well as a pure user-space transparent bridge implemented as Samba-VFS-module (for those pesky windows-users). "Content hashing" is a term that some literature uses for this kind of storage. Prototype in python. On hold.
- (SIS-related) Sector remappers for Virtual PC/VMware sparse virtual disks. Sparse single images work, need to implement stacking. Implemented in C.
- (SIS-related) Filesystem scrubbing layer for NTFS. The NTFS-parser works, providing suprisingly many locations at which scrubbing could be done. Needs integration to the stacking layer so that scrubbing will be a sector-providing layer into images. The people who designed NTFS must have been smoking crack. Implemented in C.
- (SIS-related) Block-translation layer so that partitions will start virtually at a SIS-block granular positions. Flat partitions work, need support for hierarchial and "dynamic disks".
- (SIS-related) Tools to generate sparse raw images out of virtual images and partitions within. Implemented in C. Complete.
- (SIS/RAC-related) Various optimizations to zlib/deflate/inflate. Implemented in C and GCC asm/asm-intrinsics (x86/x86-64/SSE). Complete.
- (SIS/RAC-related) Memory efficient and mostly O(1)-lookup capable structures for building and searching large amounts of hash data. Implemented in C. Mostly complete (online version completed, offline version needs work).
- Pic1 Pic2 Pic3 Pic4. Silly little real-time rendering toy that I'll continue when I get decent OpenGL-hardware (full screen blitting in X is painfully slow at 1280x1024x32). The pictures are screenshots of an actual animation and are best viewed at 1:1 zoom (no interpolation) on a digital display. Worst screensaver ever (uses all available CPU, however Xorg ends up using 90% of it currently). Implemented in C.
- (Synhilite-related) Tools to extract C prototype information out of manual pages, as well as tools to generate untargetted structured representations out of programming man pages so that manual page access can be properly integrated in synhilite generated source code listings. (doclifter makes a mess). Implemented in Python. Active (and has produced a number of patches against linux-manpages).
- (Synhilite-related) Online caching/compressing XHTML generator for programming related pages using a lockless/synchronized/fast Python/C split-process design. Prototype complete.
- Harpie: A single sector (MBR) Linux boot loader. Implemented in x86/16-bit asm (using nasm). On hold.
- Nullplayer (high-performance multi-instance audio framework). Implemented in C. On hold.
- bpmcount: automatic BPM-measurement and beat-start detection for synthetic music. Will try to detect bar-starts and theme-changes as well. Does not use frequency-analysis. Looks promising, however unsuitable for acoustic music and rock. Aims for project are automixing, beat-locking, audio-peeling and possible DMX-control. Implemented in C. On hold.
- audio-pitcher: command line utility to speed up/slow down (linear speed change) between points in time of audio tracks. Useful to prepare tracks of different BPM before mixing them in audacity. Utility proved the necessity of bpmcount since if using manual beat-tapping to get track bpms, the results are still not accurate enough to mix more than 1-2 bars. Biggest problem was finding a suitable method of countering accumulated errors even when using 64-bit floats in resampling with variable speed. Speed changes when done over long periods of time seem to cause no tempo-related perceptional artifacts. Second project (eclectic melancholic electronic summer collection) in which bpmcount, audiopitcher and audacity were all used to do offline mixing). Implemented in C. Completed but needs a re-write (plus I need to implement freq-filters).
- GI (aka Greatness Interactive, Gay (as in happy) Informer, or just Gather Information). Started as a simple program to provide technical data related to I/O measurements (used on blockspeed page), but quickly became a monster with a will of its own (in order to support various kernel versions). Implemented in python. Completed.
- scannet: a program to collect information over IPv4 network and make a report of IPs in use (with history), MAC-owners, ranged-colors, NB-names, DNS-space (forward and backward). Implemented in python. In production (wasn't really meant to be production sw).
- trafstat: realtime traffic classifier, counter aggregator and dynamic rrdtool-based graph generator (in order to visualize traffic going over upstream Inet-links). Wasn't really meant to be to used in production, but rather to act as a test of one visualization idea. Unfortunately the idea was too successful, and the system ended up being in production use (still is). Implemented in C.
- onlinex: virtual training environment where each participant has root privileges. Also includes a web-front end to download files as well as control the "virtual machine". Implemented as a multi-level network arch in order to limit security implications (using UMLinux). Support tools implemented in C and shell. Web-frontend implemented in PHP/shell. In production.
- DOCument processing System (DOCS): a make driven toolset for generating HTML output from very light marked up input text files (custom markup, not HTML). Was used to prepare/generate a lot of training material (and developed for that exact use). Implemented in python. No development plans (it's too evil and I no longer train anything).
External links
Useful kernel related links:
Long-term interests related to computing, Linux and networking
- High-performance software on multi-core/NUMA systems.
- Micro-concurrency via automatic co-routines.
- Network and systems management and visualization.
- Performance measurement.
- System(s) reliability.
- Real-time audio.
- Virtualization.
