User:TTK Ciar

My name is TTK Ciar, and I am a happily married middle-aged Software Engineer / Architect living rurally in California, USA. My main interests are software development (particularly distributed ETL SOA and Artificial Intelligence), materials engineering, anarchism, and military technology (in particular AFV's and the GPC concept).

Software Projects

I have way too many of these, and have been narrowing my focus, trying to get some finished before spending more time on others.

OpenSSL/OpenSSH Patches

Status: Work in progress

In light of the NSA's surprisingly competent campaign to circumvent the security of every citizen (both in America and abroad), I am taking it upon myself to improve the security of my ssh/scp communications with a couple of patches. OpenSSL is the library which provides basic RNG and encryption functionality, and OpenSSH wraps this functionality to provide secure communications.

Once these improvements are complete, I will deploy them on my own systems, and offer them to the OpenSSL/OpenSSH development community to include in the mainstream project -- or not; getting in with that crowd can be tough, especially if one is perceived to be an "amateur cryptographer". I have kept a hand dipped in cryptography since 1996, but not professionally, and I haven't published anything, so I'm prepared to get snubbed.

OpenSSL/OpenSSH is implemented in C89-compliant ANSI C, which is proving quite a lot of fun. ANSI C used to be my bread-and-butter language, but I don't write just a whole lot of it lately, so getting my fingers back in C is a nice break from the usual.

RNG Patch

On Linux, my preferred platform, the OpenSSL random number generator implementation essentially reads bytes from /dev/urandom as needed. The Linux system RNG is not very good, with known problems both in the quality and rate of entropy collection and the LFSR-based algorithm used to recycle that entropy. Schneier pointed out a few flaws of LFSRs for cryptographic purposes in his excellent book, _Applied Cryptography_.

An adversary able to predict the output of the RNG used to construct public key pairs could conceivably derive the value of the private key, and thus undermine all RSA/DSA-based encryption and authentication using that key.

Therefore, my first patch wraps OpenSSL's low-level RNG with a strong counting cipher RNG, using OpenSSL's own SHA256 implementation as an HMAC, with a side-channel for injecting user-provided entropy into the initial state, and mixing periodically from /dev/urandom. The result is a RNG immune to correlation attack, and a more secure (less predictable) initial state. This is a work in progress, and I'm enjoying it rather a lot. I've promised my friends/co-conspirators Chris and Rob documentation and illustrations describing my system, and when those are written I will provide a link to them from here as well.

(My earlier plan was to use an NLFSR, but I realized recently that an NLFSR is needlessly complicated. A simple counting cipher should be sufficient.)

Threefish Cipher

The second patch is an OpenSSL implementation of the Threefish cipher. It would not be used directly, but rather as part of the AES256-Threefish-Counter cipher (below). Nonetheless, implementing it as a standalone capability would make it available to the OpenSSH "speed" utility, which will be nice. Quality implementations of the Threefish algorithm abound, so very little development will be necessary, leaving me free to focus on making that development high quality (ideally: clean, well-documented, bug-free, and without obvious vulnerabilities).

UPDATE 2015-05-26: I decided to mothball this effort since djb already got his chacha20-poly1305@openssh.com cipher incorporated into the main OpenSSH implementation. It is a fine cipher, and I am happy to use it.

AES256-Threefish-Counter Cipher

UPDATE 2015-05-26: I decided to mothball this effort since djb already got his chacha20-poly1305@openssh.com cipher incorporated into the main OpenSSH implementation.

ZACL

Status: Work in progress

ZACL is A Concurrency Language. Does the world need yet another programming language? Yes, frankly, it does. Or rather, it will.

Looking at trends in enterprise computer systems, some facts stand out:

• The number of cores per processor, processors per server, and servers per cluster, are all increasing. Enterprise computing is performed by vast datacenters filled with multi-core servers, not by individual computers.

• Though the amount of memory per server is increasing exponentially with time, it is increasing on a slower exponential curve than processing, storage, and networking capability. Datasets in large ETL systems are increasing in size roughly in proportion to storage and networking capacity. This implies that as datasets grow larger, less physical memory will be available per datum.

• Programming language popularity follows a fad pattern. Successful languages will exhibit a protracted period of unpopularity, followed by a rapid rise to prominence, a long or short plateau of ubiquity, and eventually a rapid decline to niche status. The TIOBE graphs of various languages' rise and fall demonstrate this. (A notable exception is C, which has always been and remains very special.)

So, what does this mean for future software development?

• Software will be distributed and concurrent.

• Software will need to make efficient use of memory.

• Software will not be written in the languages we use today.

At the same time, some things will not change:

• Rapid development will be necessary, so that problems may be solved and new products/services developed in a short timeframe.

• The object-oriented paradigm will remain popular -- even as individual languages rise and fall, the new languages are predominantly object-oriented. It is how software engineers think about software in the 21st century.

• The 80/20 rule seems eternal, which is pertinent to code optimization. Low-level optimization involves identifying the 20% of the "inner loop" code which comprises 80% of the runtime, and funneling engineer-hours into making it run faster (e.g., by rewriting it in C or C++).

This tells us what a future programming language needs to look like, and I am using it as a specification for the ZACL programming language, an imperative, object oriented, highly expressive, garbage collected language with a strong/weak static/dynamic hybrid type system, memory-efficient data structure primitives, and powerful tools for distributed and concurrent processing.

I have some practical experience and theoretical education with all of these technologies so I'm trying my hand at putting them together. So far I have most of a formal grammar written up (the expression grammar still needs some work) and some parts of the common runtime implemented in C. It will use a hand-written parser, and LLVM for its back-end.

At the same time, I am trying out some approaches to concurrency paradigms in my Parallel::RCVM::* perl modules (not in CPAN yet, but eventually). Getting real-world work done in a concurrency framework similar to ZACL's has been very educational, and continues to influence ZACL's direction.

UPDATE 2020-05-03: I have been learning and enjoying The D Programming Language quite a bit. It appears to have already achieved much of what I imagined ZACL to be, and it seems likely that ZACL will be implemented as a fork of D.

The main points of enhancement I'd like ZACL to offer over D are, ranked by priority:

• Even better practical expressiveness,

• A built-in "job" type allowing safe, transparent access to threads, processes, coroutines, interfaces,

• First-class regular expressions,

• The compiler as a .so library, to facilitate run-time code evaluation,

• Improved implementation of associative arrays,

• Lower compile-time memory overhead (seriously, dmd's symbol table is ridiculous)

UPDATE 2020-10-21: More and more of ZACL's goals are proving achievable in the form of D library implementations, rather than a fork of the language. At least for the foreseeable future, ZACL's A Concurrency Library for the D programming language.

A couple of sticky problems persist -- I'm having trouble making regular expressions act more like first-class language elements. String operator overloading was a bust. I'm looking at adding support for them to std.variant or arsd.jsvar next. Also, library implementations of associative arrays have shortcomings which the D community is struggling to amend. I'm punting on it for now, and depending on which way the language evolves I might implement AA based on cascading hash tables as a library (again, perhaps as part of std.variant or arsd.jsvar) or a dmd patch to be submitted upstream.

Forking the language is still on the table, but I'm putting off that decision for as long as possible, and doing everything I can within the confines of the existing D language.

Most of the work thusfar has been consolidating/encoding high-expressiveness tricks and shortcuts into relevant libraries, which the main "zacl" library provides as public imports.

Black Tea Project

Status: Work in progress

The Black Tea Project is a collection of other people's open-source software projects, aiming to provide Java-free alternatives to popular enterprise infrastructure.

I have seen several companies face the conundrum of needing infrastructure which depends on the JVM, but having no in-house Java talent.

These companies face the prospect of either learning JVM management skills, hiring new employees with Java experience and integrating them into the business, or using (or developing) alternative solutions which do not depend on the JVM.

These solutions take a lot of time, cost a lot of money, and introduce new risk to the business. I have seen all of these solutions attempted, and fail more often than not. The problem is that the Java ecosystem is at least as complex as the UNIX ecosystem, and Java-based technologies really need expert attention to get started and maintained effectively and efficiently. At the same time, many popular Java-based technologies represent hundreds or thousands of engineer-hours of value.

Furthermore, despite enjoying immense popularity today on a variety of high-visibility platforms, its long-term trajectory is clear: Java is on its way out, and by 2026 or 2028 will be less widely used than LISP.

These observations motivated me to start this project for collecting/developing alternatives to popular Java-based enterprise software (primarily: Cassandra, Lucene/Solr, Hadoop/HFS, and Zookeeper), which do not depend on the JVM. Some alternatives already exist, but might need further development (sometimes a lot of development) to be completely viable choices.

For instance, Lucy is comparable to Lucene in functionality and single-core performance, and is implemented in C (with perl bindings) so avoids the JVM dependency. It has roughly one tenth the memory overhead of Lucene for a given size dataset, which is great, but it is also single-threaded, which means it requires additional infrastructure layered over it to scale horizontally (much as ElasticSearch scales Lucene horizontally). It also needs bindings for languages other than perl (which is all but dead in the enterprise today).

At the moment I'm liking the following replacements, but ascertaining their suitability is an ongoing process:

• etcd (Go) replaces ZooKeeper (or not -- etcd has taken a bad turn)

• Dezi (C, Perl) replaces Solr

• ScyllaDB (C++) replaces Cassandra

• Redis (C) also replaces Cassandra -- sort of; see notes below.

• Hypertable (C++) replaces Hadoop

• GlusterFS (C) replaces HFS

Notably absent from this list is an ElasticSearch replacement. I've been tempted to take a stab at this, using an approach similar to Dezi's (Perl bindings to Lucy), but distributing documents across striped mirrors of Lucy instances, using ZeroMQ (ZMQ::LibZMQ3) to stream to/from remote instances. API compatability with ElasticSearch would be a primary goal, with robustness and scale-out performance and capacity as secondary goals.

etcd / redlock as zookeeper replacement notes

UPDATE 2020-10-21: The etcd project has careened off in an unfortunate direction, and is no longer an appropriate zookeeper replacement. For now I have been using Redis for zk-like roles, but this is an imperfect fit, lacking zookeeper's fault tolerance.

Redlock is a simple algorithm for using one or many Redis servers to make quorum-based distributed locks. It's not as airtight as etcd or zookeeper (as a Redis server can go down after a lock succeeded and come up again without the lock while the lock is believed to still be valid, under some conditions), but perhaps good enough.

I've been pondering writing my own DragonScale system which uses Redlock to protect federated services with functionality similar to zookeeper.

Redis notes

Redis lacks the scale-out capability and high-level (SQL-like) query interface necessary to be a perfect Cassandra replacement. Nevertheless, it is perfectly adequate for most use-cases. Abstraction layers over clusters of Redis or Tarantool instances might provide more complete replacements.

There is also a distributed Redis solution making the rounds called Redlock which may or may not prove a suitable replacement for Zookeeper. There are implementations available in several languages. I'll be putting it through its paces to see how it holds up.

Redlock is inappropriate when correctness is mission-critical, https://martin.kleppmann.com/2016/02/08/how-to-do-distributed-locking.html (archived)

ScyllaDB looks like the more appropriate Cassandra replacement "on paper", but I'm having trouble getting it to work. The project takes a rather immature approach to versioning and dependencies. If you're not using the latest Fedora, they don't think you exist. Possibly it can be made to run on more enterprise-appropriate environments, but it's a work in progress.

Time::TAI::Simple

Status: Committed to CPAN

I don't like NTPd and IETF leapseconds dorking around with the system clock my software depends on, and the existing Time::TAI and Time::TAI::Now modules are a bit impractical for everyday use, so I wrote my own Time::TAI::Simple module and contributed it to CPAN.

DrunkenFay

Status: Work in progress

DrunkenFay is my fork of a cute, simple wiki program called OddMuse. OddMuse is very fast to deploy, simple to use, and makes silly content-population/manipulation tricks easy because it uses flat files instead of a database for storing content. It seriously takes me two minutes to set up a new OddMuse wiki instance on my webserver.

OddMuse is great for small, simple tasks, but it has limitations making it impractical for bigger projects, and as it happens I have a bigger project. My MBT Website is hard to maintain by hand even at its modest size, and I've accumulated more than 20GB in about half a million additional documents I'd like to make available.

After turning over several options, I decided to modify OddMuse to make it work for this larger dataset, and allow for restricted access for a dynamic pool of administrators.

Scaling up its document capacity was fairly easy. Instead of storing documents in a subdirectory named after the document title's first letter, it now takes the MD5 checksum of the title and uses two of its three-digit substrings as subdirectory names for a multi-level directory storage hierarchy.

Adding real user accounts and authentication shouldn't be hard, but I just haven't gotten around to it. After that, I need to write a little script that populates its pages with copies of the content, breadcrumbs, and links to the original content, and it should be good to start .. then years of curation will follow.

Note 2014-08-11: It'd be a little more work to make an android app with a Google-Keep-like interface permitting population of a DrunkenFay page with text, photos, voice via Android's native capabilities. Use PhoneGap to avoid Java.

Porkusbot

Status: Work In Progress

I hang out on a realtime chat system called ICB, which is roughly contemporary with IRC but nowhere nearly as popular. Unlike IRC, there are relatively few bots which work with ICB. Porkusbot is my humble contribution to the toybox, incorporating features of infobot, moderator bot, madlibs (encryptobot), and others.

Three iterations of Porkusbot have been deployed, each better than the other in some respects, but none of them with particularly good security. This has inhibited me from releasing their source code. Not that I'm happy with security through obscurity, I just didn't spend the time and effort to do it right.

I've started version four recently because one geek group expressed a desire to replace its moderator bot with something like porkusbot, and it needs adequate security and transparency. This one's getting a good, clean implementation (of security and everything else) which people can read and easily understand.

UPDATE 2020-10-21: Though porkusbot hasn't been used much directly, I have been tearing pieces out of it and incorporating those pieces into the actively used "shurly" ICB bot and the "ravenbot" IRC bot in ##slackware-help freenode channel. I'm probably going to refactor it into a protocol-agnostic module which can be used directly by both "shurly" and "ravenbot" (and others).

Pinkeye

Status: Work in Progress

What started as an idle experiment with primal sketch algorithms turned into a serious machine vision exercise. The target application is realtime video interpretation of sufficient quality to provide automatic navigation in an embedded system.

This means feature identification can be sparse, as long as it is sufficient to provide points of reference useful for triangulation and self-location. This "just good enough" approach tremendously reduces the system's processing and memory requirements, compared to traditional implementations. I'm hoping this will reduce those requirements sufficiently to bring deployment into the reach of inexpensive embedded systems.

Since this is a practical project and not an academic exercise, I am totally cheating and depending on stereoscopic vision for locating objects in three dimensions, rather than overly clever (and insufficiently reliable) post-processing algorithms.

I've been developing and proving my algorithms in perl to figure things out, and then re-implementing them in C for performance once they're working right.

UPDATE 2020-10-21: I've picked this project up again of late, but am re-implementing it in D. My hope is that this will allow for both "thin" compilations which will fit on embedded systems like Raspberry Pi and "thick" compilations which will utilize the d-compute library to process video frames as CUDA kernels on GPUs.

Physics::Ballistics::*

Status: Published to CPAN

One of my side interests is military and defense technology, specifically armor (vehicular, personal, and fortification). To understand armor at a practical level, it is necessary to understand the threats it faces. To this end, I have accumulated a considerable library of physics functions related to ballistics and armor, written in perl for easy use from my perl-REPL scientific calculator.

These are mostly perl implementations of other people's formulae (some well-known, others esoteric), but some of them have been improved, while others are my own original work. They are organized into three modules, Internal, External, and Terminal. Within each, individual functions are documented as valid within the ballistic domain, hypervelocity domain, or both.

These modules have been released to CPAN: [1].

Future updates will incorporate some cleaning-up of parameter units (right now it's a pretty mixed bag of imperial and metric units), improvements to the "lethality" and "pc" functions, and a ballistic calculator which combines several of these functions to provide a graphing tool similar to Renegade.

Also, I am working on a separate Physics::Hardness module which will replace the half-assed material hardness unit conversion functions provided by Physics::Ballistics::Terminal.

Geekbench

Status: Backburnered

Because none of us have infinite time and money to throw at our computer hardware, it's nice to be able to look at a benchmark and gauge whether it's worth it to spend a little extra cash at a more powerful CPU, or one with a different memory subsystem, or a solid state disk, etc.

For this, we have system benchmarks.

For a system benchmark to be useful to a person, it must measure the system's capability to perform the kinds of tasks that person needs their system to perform. Because of this, the best benchmarks consist of several components, each exercising the system's ability to solve a certain kind of problem. Thus the prudent seeker can look at the performance numbers for the components most like their expected workload, and ignore the others as irrelevant.

Of this kind of benchmark, SPECcpu is probably the best, particularly for server type systems. It exercises sets of integer-intensive and floating-point-intensive components based on real-life solutions to popular problems, and it does so in two contexts: serial (running in one core on one processor) and parallel ("rate", running on as many cores and processors as the system has).

SPECcpu is wonderful. It is awesome. It has shortcomings:

• SPECcpu is for vendors to show off their most powerful systems. Because of this, only expensive, high-end systems get their benchmark results published.

• SPECcpu takes pains to exercise just the CPU and memory subsystems in isolation from I/O devices such as disks. In the real world, disk performance often has an impact on actual system performance, but SPECcpu cannot reflect this influence.

• SPECcpu published results generally represent the performance of aggressively optimized systems. Vendors are allowed to run the benchmarks, profile them to see where they are slow, tweak their system, and then run the benchmark again, repeating this cycle however many times they need to get the best possible results. The published results represent only the run on the most-optimized system configuration. Vendors will recompile their kernel, hand-tweak standard libraries, customize compiler optimization passes, and so on, to achieve the results which make their system look as good as possible.

That may be fine for the enterprise, where multibillion-dollar corporations can throw nontrivial resources at optimizing their production servers, but where does that leave the rest of us?

Small and mid-sized companies, universities, and enthusiastic amateurs typically don't have the time, budget, or expertise to hyper-optimize the most expensive hardware, and run in a pure RAMdisk environment. SPECcpu results do not represent the performance such people can expect out of their systems.

I would say they need a "benchmark for the rest of us", but that's not quite true. Professional system administrators and engineers are not "the rest of us". Neither is the home enthusiast mining dogecoin on a bookshelf full of second-hand laptops.

We need a benchmark for geeks. Hence, Geekbench.

Like SPECcpu, Geekbench consists of several components, each representing a common type of workload.

Unlike SPECcpu, Geekbench is for benchmarking all kinds of systems, from the high-end to the low. It uses whatever operating system, compiler, libraries, etc are installed on the tested system. It allows the disks and filesystem to impose themselves upon the workload.

Also unlike SPECcpu, Geekbench automatically uploads the results of a run to a central server, along with a description of the system (including its hardware and software), to be stored in a database. A website provides a public front-end to that database, so people can aggregate, segment, filter, and view benchmark results depending on their needs.

Thus it is hoped that Geekbench will measure the performance of more kinds of systems, with more kinds of subsystems (cpu, memory, disk, filesystem, and system software), and represent enough samples to provide a distribution curve of expected performance.

While it is certainly possible for bad actors to game the system and submit bogus results, the expectation is that many more people will submit good results than bad ones, and the latter can be filtered out as outliers.

That's the idea, but in practice I haven't been giving the project the time it requires. Geekbench has gone through a few revisions, using mostly microbenchmarks (which are not really what it needs).

It does automatically upload its results to my server, but I never wrote the web front-end to allow users to actually see the collected results.

Geekbench has languished. I haven't worked on it for years. It's still a good idea that I'd like to see happen, but I don't know when or if I'll get around to getting it in order.

Misc Info

This Wikipedia account established on 2006-01-13.

• Home Page

• Blog