The Chaocipher Clearing House

Progress Report #26

Moshe Rubin (mosher@mountainvistasoft.com)

This Progress Report is a compilation of Chaocipher-related information over the past two years since Progress Report #25 (March 2017).

Here's a brief listing of items:

Belated obituaries for two Chaocipher researchers: Mike Cowan (2015) and Jeff Hill (2016)
George Lasry's "A Methodology for the Cryptanalysis of Classical Ciphers with Search Metaheuristics"
Revisiting the Chaocipher characteristic "no hits < 9"
Chaocipher II 8086 assembly code program listings
Chaocipher Sightings on the Web

Belated obituaries for two Chaocipher researchers: Mike Cowan and Jeff Hill

We present belated obituaries of two prominent Chaocipher researchers, Mike Cowan And Jeff Hill.

Mike Cowan
Michael J. Cowan photo

Mike Cowan, a Chaocipher researcher, passed away on April 20, 2015, at the age of 78. At the time I posted an obituary on the Crypto Chaocipher Forum.

I first made Mike's acquaintance on April 16, 2009. At that time I was investigating solving Wheatstone cryptograph cipher messages using Simulated Annealing (SA). Mike had experience and publications regarding SA under his belt, so I dropped him a line. During our subsequent email thread he asked what else I was working on, I mentioned Chaocipher, Mike read the entire Chaocipher Clearing House web site, and the rest was history. From that point onwards he joined Jeff Hill, Jeff Calof, and myself as a Chaocipher researcher.

A member of the American Cryptogram Association (nom de plume: ANCHISES), Mike was co-editor of the Computer Column from 2003 to 2009. He had a penchant for using software programs to automatically solve ciphers in the ACA's bi-monthly publication, The Cryptogram. I assume it was his programming skills that led him to devise a method for solving small Playfair ciphers using computers. He published a paper in Cryptologia (2008) entitled "Breaking Short Playfair Ciphers with the Simulated Annealing Algorithm", one which is often quoted in other serious cryptanalytic publications. Another fine piece of research was his 2004 paper "Rasterschlüssel 44 – The Epitome of Hand Field Ciphers", also published in Cryptologia.

His web site, cryptoden.com, was a good place to browse whether you were looking information on Simulated Annealing or how to program in Python.

Most of the now-defunct cryptoden.com website is archived by the Wayback Machine project. The project, however, did not archive executables, ZIP files, and other non-displayed entities. For these items, you can search a bare-bones copy of cryptoden.com I archived on my private site.

In a Skype conversation we had years back, I seem to remember Mike's saying he had been an industrial chemist for many years, and had traveled the world to work with customers in different countries.

Mike was passionate in all his cryptological endeavors and will be greatly missed.

Update (28 May 2021):

In an email correspondence, Mike's son Will kindly sent me his father's photograph, and provided information about Mike's career.

"Dad was born in the Shanghai International Settlement in 1936 and lived there and in Australia and South Africa whilst growing up.
His family returned to England whilst he was in his late teens and he then attended Oxford University from which he graduated with a degree in Chemistry.
After university he joined Unilever and worked for that company throughout his career.
He had a very successful business career, running businesses all over the world and rising to be a Senior Vice President of Unilever.
He started programming for recreation on an Apple IIe in the late 1970s and always had a recreational interest in mathematics and computing.
In retirement he had two great interests: sailing his yacht on the Solent off the South Coast of England and, having rediscovered programming in retirement, he thoroughly enjoyed applying this to cryptography and participating in the ACA."

Jeff Hill

Jeffrey A. Hill passed away on August 8, 2016 at the age of 70 (his obituary can be read here). He was the first Chaocipher researcher to join me after The Chaocipher Clearing House (TCCH) was established, and remained involved in the effort to solve the cipher system.

I first made my acquaintance with Jeff back in 2008. At the time I was searching for anyone who had worked on Chaocipher in the past. Eleanor Joyner (HONEYBEE), the editor of the American Cryptogram Association (ACA), kindly sent me some possible leads for tracking down Chaocipher researchers. One of the leads was the address and phone number of former ACA member E. E. REMINGTON (Jeff Hill), who had researched Chaocipher together with PHOENIX (William G. Sutton) back in the 80's and 90's. I left a message on his voice machine and received an email from him on 31 July 2008, which was the beginning of our fruitful Chaocipher collaboration and friendship.

Jeff's major contribution was his then-unpublished paper entitled "Chaocipher: Analysis and Models". At the time there were independent observations of Chaocipher characteristics made by different people. With his paper, Jeff presented a hypothetical model that could explain many of the aforementioned observations. With his paper in hand, Chaocipher researchers could begin to focus their efforts on proving or disproving the hypothetical system. It had the effect of galvanizing our cryptanalytic work.

Jeff was a scholar, having received his Bachelors and Masters degrees in mathematics, and had also studied philosophy in university. Anyone who knew Jeff was aware of his painstaking accuracy and perfectionism when performing cryptological research or authoring a technical paper. He was a good person to bounce ideas and questions off of.

When the Chaocipher system was made public, Jeff retained his interest in Chaocipher, albeit on a lower flame. He had his first bout with cancer in 2011, but made an excellent recovery and was in remission. He retired from his last place of employment in December 2015. In 2016, however, the cancer returned.

I was unaware of Jeff's medical condition, as our last correspondence was in December 2015. I wasn't worried when short emails went unanswered, and I noticed that birthday greetings on Jeff's Facebook page were not acknowledged. In November 2018 I became concerned. A friend of Jeff's notified me that he had passed away in August 2016.

I am indebted to Jeff for being the first person who joined me with Chaocipher research, and who loved the topic and invested years in analyzing this fascinating cipher. I will miss him.

George Lasry's "A Methodology for the Cryptanalysis of Classical Ciphers with Search Metaheuristics"

TCCH readers may recognize George Lasry's name. George is a Staff Software Engineer and Manager at Google in Tel-Aviv, holds a Ph.D. in Information Security, and is an extremely talented and prolific cryptanalytical researcher.

I first became acquainted with the name after reading George's article in Cryptologia entitled "Solving the Double Transposition Challenge with a Divide and Conquer Approach" (June 2014). George and his co-authors had solved Klaus Schmeh's 2007 Double Transposition Challenge cipher in 2013, using a ciphertext-only hill climbing attack. This made quite a splash in the cryptologic world at the time. Since then, George has gone on to scaling unprecedented cryptanalytical heights. Honing his hill-climbing techniques, he has tackled and solved historical ciphers in system like Hagelin M-209, columnar transpositions with long keys, ADFGVX, diplomatic code systems, Enigma, and more.

In January 2015, a friend and former colleague of mine, Dmitry Rubinstein, forwarded my Chaocipher Cryptologia article to George, who was also a friend of his. George, always on the look-out for interesting cryptanalytic challenges, asked to be introduced to me. This was the start of an eye-opening experience for me. George immediately set to work attempting to solve Chaocipher Exhibit #6. Exhibit #6 consisted of 50 messages, each comprised of 25-30 ciphertext letters, with the additional information that they were all "in depth", i.e., all the messages start with the identical machine settings. The story of how he solved the 50 messages and discovered powerful techniques for solving Chaocipher messages can be read in his Ph.D. thesis (more on this in a moment!).

In between all of this cryptanalytical activity, George has succeeded in receiving his Ph.D. (summa cum laude) from Kassel University in Germany. His Ph.D. thesis, entitled "A Methodology for the Cryptanalysis of Classical Ciphers with Search Metaheuristics" is published by Kassel University Press. It is also available online (cached here and here), and is a goldmine for cryptanalysts who want to use hill-climbing to solve difficult ciphers. Chapter 8 is dedicated to how George solved Chaocipher Exhibit #6.

If you are a cryptanalytic researcher who is looking for powerful hill-climbing techniques to solve ciphers, this book is a must for you.

Revisiting the Chaocipher characteristic "no hits < 9"

When The Chaocipher Clearing House (TCCH) web site was inaugurated, the first Progress Report I posted presented a striking observation I had noticed in Exhibit #1 of John F. Byrne's Chaocipher challenge: a pt/ct identity never repeated at a distance less than nine (9). For example, here are some pt/ct equivalences, but the pt/ct pairs ("hits") are always at a distance of 9 or more apart::

Line 142: pt: U L D R E L I N Q U I S H
          ct: B F B W Y Z L O Y B S S T

Line 151: pt: P O W E R S Q I N C A P A
          ct: X Q Q R E X S U Z O X X U

Line 156: pt: N G T H E L A W S F O R N
          ct: D S G D Z X O R W J J V D

Line 162: pt: D E O F N E W O F F I C E
          ct: I I X J B C H V M Z C T I

Line 206: pt: I C E A N D M A G N A N I
          ct: R I G A V V X I G Y K F R

Line 215: pt: O F O U R I N T E N T I O
          ct: H H C Y W F P Y M E R D C

Throughout the period of researching Chaocipher, whatever system was presented and proposed had to explain this phenomenon. To date no one has presented the reason why this observation is true.

A few weeks ago I had a simple idea, one I should have thought of back in 2010. I basically generated seven sets of 100,000,000 random plaintext characters, encrypting them each time with the same right/left alphabet keys. The goal was to see whether nine (9) was indeed the minimal distance between pt/ct "hits".

To my surprise, distances of eight (8), and even of seven (7), were seen. Here are the results I obtained, laid out in table form:

Test ID	Sample Size	Distance
Test ID	Sample Size	8	7
A	100,000,000	1692	85
B	100,000,000	1799	87
C	100,000,000	1771	89
D	100,000,000	1709	84
E	100,000,000	1768	102
F	100,000,000	1831	64
G	100,000,000	1764	71
Total	700,000,000	12334	582

I think we are empirically safe in stating that, for classic Chaocipher, no pt/ct "hits" occur at a distance less than seven (7).

There is a cryptanalytic lesson to be learned here. I remember how my early observation of "no pt/ct hits less than a distance of nine (9)" had me thinking of possible encryption schemes that would mimic this important point. I spent scores of hours investigating possible systems, not realizing that the lower limit of hit distances was really seven, not nine.

Chaocipher II program listings

In 2013 I received files pertaining to an improved version of Chaocipher called "Chaocipher II". The background history is that following John Byrne's divulging how Chaocipher worked to Cipher Deavours and Lou Kruh, Deavours and John Byrne's nephew, Tony Bean, collaborated to improve the cipher system. Based on the original Chaocipher, the improved version ostensibly expanded the possible input and output characters to handle any 8-bit characters. The document entitled "Preliminary Instructions for Chaocipher II" on the NSA web site gives insights into what the program looked like and what its purpose was.

The National Cryptologic Museum (NCM) holds a 3.5" diskette containing files related to the Chaocipher II product. These files can now be perused on this web site. You can see the 8086 assembly source code (probably written by Cipher Deavours), the 16-bit version of Chaocipher II (chao.exe), 16-bit utility programs, and more:

File Name	Description	Comments
CHAO.ASM	The Intel 8086 assembly source code for Chaocipher II
CHAO.EXT	The Chaocipher II program compiled as a 16-bit executable program.	Change extension to "EXE" to execute
CHAOCAT.COT	A 16-bit utility program for listing encrypted, hidden Chaocipher II files in a folder	Change extension to "COM" to execute
CHAOSET.COT	A 16-bit utility program for setting the attribute of a Chaocipher II files in a folder	Change extension to "COM" to execute
DOS Files_Screenshots.pdf	Screenshots of the Chaocipher II files, as taken by the original developers	Not one of the files on the original 3.5" diskette
INSTRUCT.FW3	The Chaocipher II instruction manual	The original word processor file (which one?) that produced the preliminary instructions document.
TO_TONY.TXT	A short note from Cipher Deavours to Tony Bean	"Cy" refers to Professor Cipher Deavours

Note that the EXE and COM binary programs have been renamed "EXT" and "COT" respectively on the Chaocipher.com web site. This is to make it easier to download the programs without interfering with browser permissions.

For those who would like to run the CHAO.EXE implementation of Chaocipher II on their computer, I advise installing DOSBox, an Open Source DOS emulator to run old 16-bit DOS programs. Here is a screen capture running it in DOSBox on my 64-bit Windows 10:

Here is an open challenge to our readers: duplicate the features of Chaocipher II by converting the 8086 assembly source code to a modern day programming language (e.g., C++, Java, Python). If you succeed, I would be more than happy to publish it on TCCH.

Errata Page for "John F. Byrne's Chaocipher: An Historical and Technical Appraisal"

Chaocipher Revealed: Up-to-Date Reference List

As mentioned at the end of the Cryptologia article "John F. Byrne's Chaocipher: An Historical and Technical Appraisal", there are two dedicated pages that list errata and corrected links referred to in the article.

Chaocipher Sightings on the Web

Here is a list of Chaocipher-related pages on the web since Progress Report #25 in March 2017. It is probably an incomplete list, so we would appreciate comments from the readers about other sightings to add to it.

Schneier on Security

A few days after Chaocipher was revealed on the web in July 2010, Bruce Schneier posted a page to his blog about Chaocipher. The blog talkbacks provide fascinating insights into thoughts about Chaocipher then and now.

National Cryptologic Museum (NCM) library catalog shows 35 Chaocipher items

The National Cryptologic Museum has a library catalog listing about 20,000 items in the museum's library. Among the comprehensive and fascinating list are 35 Chaocipher-related items.

"Obscurity, Codes and Puzzles in Books: Ponderings" on the "Overweening Generalist" blog site

Ruminations about John F. Byrne and the Chaocipher.

The Technical University in Freiberg, Germany hosts interesting Chaocipher page

The Math Department of the Technische Universitat Bergakademie in Freiberg, Germany has a page dedicated to Chaocipher. The pages, in German, do not get translated automatically by Google's Chrome. Therefore, here is a list of the pages you can link to and translate. I believe you will find the step-by-step description of encryption to be of interest.

The first page says the following (as translated by Google Translate):

On the web pages to a rather unknown, but no less interesting method from the cryptography, the "Chaocipher". As part of an internship with Prof. Dr. Ing. Hebisch at the TU Bergakademie Freiberg. Have fun watching and trying out the ciphering!

Starting Page
History of the Chaocipher
Functionality
Safety of the Procedure
Imprint and Sources

PR Gomez on "The Scrabble Cipher"

Francisco Ruiz is a professor at the Illinois Institute of Technology, in Chicago, working on energy-related inventions and transportation, and writing fiction as PR Gomez. It is under this pen name that he publishes an eclectic and most interesting blog called "PR Gomez".

A post to the Chaocipher forum of the Crypto Forum tipped me off that Professor Ruiz had published a blog post entitled "The Scrabble Cipher", which is his attempt at creating a cipher based on Chaocipher but simplifying the mechanics. I heartily recommend reading his interesting discussion of the Chaocipher and to see my response in the talkback section.

I'm still wondering whether the fact that the plaintext (right) alphabet does not add significant security changes the bits of information used by the cipher. The classic Chaocipher may be equivalent to using a straight alphabet for the plaintext alphabet, performing the simple substitution at the very end. Nonetheless, using a 'random' plaintext alphabet complicates the cryptanalytic process (see George Lasry's use of the Index of Coincidence in his evaluation function rather than English letter statistics).

Moshe Rubin undergoes plastic surgery on "Kryptos - Beyond K4" :-)

You'll probably chuckle when you read this blog post on the "Kryptos - Beyond K4" blog. There are many people I admire and hold in hero status. I never thought I'd be reading one about myself. :-)

New York Times Review of John F. Byrne's "The Silent Years", by Frank O'Connor

Here is a link to a New York Times review of John F. Byrne's "The Silent Years", written by Frank O'Connor. It is interesting to read what a major literary critic had to write about Byrne's autobiography soon after it was published.

Geocaching Mystery Toolbox: Chaocipher

Geocaching is "an outdoor recreational activity, in which participants use a Global Positioning System (GPS) receiver or mobile device and other navigational techniques to hide and seek containers, called "geocaches" or "caches", at specific locations marked by coordinates all over the world."

The hints and coordinates to find geocaches are sometimes encrypted using known cipher systems. The Geocaching Mystery Toolbox web site provides geocachers with working programs to encrypt and decrypt cipher messages, and includes a working online version of a Chaocipher encrypter/decrypter.