2017-07-05 03:27 pm

ssl libraries

I only recently checked the OpenSSL site and found out that they've changed their license from 4 clause BSD to an Apache style license. Many sites mention a license conflict between BSD 4 clause licenses and GNU GPL licenses. Several programs I use try to avoid the issue by adding a waiver to their license to allow linking with OpenSSL. Some examples are Dillo and D+. The issue can also be avoided in many cases by using software with a license other than the GNU GPL. So, the new switch to an Apache style license solves the license conflict issue in many cases (although it may cause new license conflict issues).

There's a useful reference on license information at the curl site:

There was an interesting thread about the GPL/BSD 4 clause license conflict on the lynx development list. Various SSL library options such as OpenSSL, gnutls and nss were discussed. It was mentioned that parts of the gnutls library (the compatibility layer) was licensed under GPLv3 which is incompatible with GPLv2 only ( https://www.gnu.org/licenses/gpl-faq.html#AllCompatibility ). The thread concluded with a recommendation to use the nss library to avoid license conflicts. It also mentioned that contacting everyone who worked on the project to re-license lynx would be too difficult. I guess there was some accommodation made after the thread because the latest version of lynx includes a COPYHEADER file that includes permission to use libraries such as OpenSSL and gnutls.

With the license change to OpenSSL, I hoped that the various license issues one might run across when building and distributing software that need SSL support had been solved. I've been using OpenSSL for a long time and it's always been highly portable and worked on a variety of platforms. However, when I downloaded the latest version of OpenSSL, I found that was no longer the case. With the various OpenSSL security issues in the news, work has been done to update the library and make it more secure. It appears, in this particular case, the security enhancements have come at the cost of portability. I used to be able to build OpenSSL using the ActiveState version of Perl. Now, OpenSSL is designed to only allow use of ActiveState or Windows native Perl builds using the Visual C++ compiler. If I continue to use the mingw compiler, I need a msys or cygwin based version of Perl. I'd previously built OpenSSL for DOS with no issues. Looking through some of the code, I'm afraid to even give it a try with the latest version of OpenSSL. OpenSSL is also much more dependent on Perl than it used to be. If I'm building a C library, it would be nice if it was only dependent on C code to build instead of requiring an interpreted language that's rather complicated to build.

I found it was time to either redesign my build of Perl (since a native Perl could no longer be used with MinGW to build OpenSSL) or to look for a new SSL library alternative. I'll share some of the results in my search for SSL library alternatives.

The major SSL options are OpenSSL, gnutls (with GNU GPL license) and nss (from Mozilla with Mozilla Public License). There are OpenSSL compatibility layers for gnutls (GPLv3 license) and nss ( Nss_compat_ossl originally from Fedora and no longer supported ). From some of the articles I read online, OpenSSL and nss were mentioned as more secure than gnutls. I also read that gnutls was a popular option on certain Linux systems such as Debian.

However, these aren't the only SSL libraries. There are several other options for embedded systems and other specialty uses. It's actually hard to track the options since the names of some of them keep changing. Typically, a program needs to offer support specifically for one or more of these library options because there usually isn't a compatibility layer to use that SSL library in place of a more common SSL library. Curl is a great example of a program that supports multiple SSL library options. D+ included support for cyassl. However, cyassl is now known as wolfssl. axtls looked like an interesting possibility because of its BSD license. It supposedly builds on Windows, but I didn't have any luck with it. Of all the various lightweight and/or embedded options, the only one I was able to get to build out of the box on Windows was mbedtls (formerly polarssl). mbedtls is available under a GPL license. Rather than try to track all the options and library name changes, I'll refer to the Wikipedia list:

After the recent security issues with OpenSSL, there were also forks of OpenSSL. These include LibreSSL (a fork by the OpenBSD developers) and BoringSSL (by Google). I tried building BoringSSL, but it required an interpreter (similar to OpenSSL), so why not stick with OpenSSL instead of switching to BoringSSL. I also tried building some of the various SSL options with cmake and had no luck at all.

I'd tried building LibreSSL in the past with no luck either. The fork was originally just designed for BSD systems and support for other platforms was added later. There are even some posts on the Internet about how early versions of LibreSSL include issues that made it insecure on Linux and other non-BSD systems. Once support was added for other platforms, I ran into an issue with what platforms were supported. The developers only supported the MinGW64 project on Windows. I use a version of MinGW based on an earlier version of the original MinGW which has a more lenient license. I've been adapting the mingwrt and w32api libraries, but they're based on the original public domain licensed versions of the libraries. So, when I previously tried to build LibreSSL, it wasn't compatible with my MinGW compiler. However, after the various changes to OpenSSL, I decided to give LibreSSL another try. Was pleasantly surprised this time when it only required minor patching with my particular version of MinGW in order to get it to build. It also uses the standard GNU autotools style configure/make/make install rather than requiring Perl just to configure and build it. If anyone's interested in the patches, feel free to contact me about getting a copy.

At this point, I'm considering switching to the LibreSSL fork. Several Linux distributions that use musl as their main C library are using LibreSSL as their SSL library of choice. They included patches to get various libraries and programs to build with LibreSSL instead of OpenSSL. There are also some patches at the LibreSSL site.

However, there are still some issues with switching. Many of the programs that included a waiver for OpenSSL, do not provide a similar waiver for LibreSSL. I certainly be interested in hearing how others are handling this particular issue. It would be nice if LibreSSL users could contact some of these projects and ask about a waiver for LibreSSL as well as OpenSSL. Some of the projects that come to mind are D+, Dillo, lynx, opusfile. Curl, libarchive and libssh2 can also use LibreSSL instead of OpenSSL, but there aren't any issues with license conflicts unless one adds a GNU GPL licensed library or a similar incompatible license to the mix.

I'd be interested to hear what others use as their SSL library of choice when building cross-platform applications and why they chose that particular library. You can find further discussion of this topic on the CPPDesign mailing list ( http://www.distasis.com/connect.htm ).
2017-05-15 11:10 am

C runtime libraries

Was at a PHP meetup where the group was discussing Docker. Members and the presenter knew certain Linux distributions had a smaller footprint for use with Docker. I was surprised to find out they really didn't know why that was. One of the key factors is the C runtime library. Basic C runtime libraries just cover the functions and data structures that are part of the ISO C standard. Many C runtime libraries also add functions and data structures that are part of the POSIX standard as documented by the Open Group. Some C runtime libraries are rather bloated and provide a wide variety of functions (even beyond those documented by the ISO C and POSIX standards). Others provide a bare minimum. Some, especially those targeting embedded systems are designed for efficiency. Others are designed for functionality. Some provide no Unicode support (locale 'C' only). Some like musl, concentrate on UTF-8 support. Some try to support a large variety of characters sets and internationalization features. All these factors can affect code size and efficiency when compiling programs.

Alpine Linux previously used the uclibc runtime library and now uses the musl library. Most major Linux distributions use glibc. It was a big step, but a positive one when Debian (and Ubuntu) made the switch to eglibc. The choice of C runtime library can make a huge difference for the operating system.

For Windows developers, if you're using MinGW, the GNU compiler on Windows, you're using the Microsoft runtime libraries. The original version of MinGW used crtdll.dll but later versions use msvcrt.dll. Windows systems typically have one of these runtime libraries already installed. So while, you can't distribute Microsoft's libraries, you really don't have to. At least one is already on any particular Windows system. It gets even more complicated because different versions of Windows have different versions of msvcrt (such as msvcr70.dll, msvcr80.dll). MinGW uses a subset of the runtime functions based on Visual Studio 6.0. There are ways to access runtime functions from later versions of Visual Studio, but the application becomes less portable to various versions of Windows. Cygwin which also works on Windows, avoids the problem of dealing with multiple Windows runtime libraries and dlls and provides better POSIX compatibility for their runtime library by using a runtime library based on newlib.

There are a wide variety of runtime libraries designed for embedded systems. They're typical more compact and less bloated than a library like glibc and many are easier to port to various platforms. Of the runtime libraries available for embedded systems that are highly portable, I thought the newlib design was interesting. newlib is currently maintained by Red Hat. There are a limited number of syscalls (typically functionality provided by the kernel) that one needs to provide code for to get the library to work. Newlib uses a combination of public domain and BSD style licenses. Cygwin uses a runtime library derived from newlib, but adds several POSIX functions. It also uses a GNU GPL license. That means whenever you distribute programs linked with their runtime library you must also distribute the source code in order to be compliant with the GPL license.

As mentioned, the standard C library on most Linux distributions is glibc which was developed by the Free Software Foundation. The glibc project was slow to take patches and is a rather large library, so some distributions switched to eglibc which is binary compatible with glibc. The binary compatibility makes it easy to switch between the two. Some Linux distributions wanted to avoid bloated C libraries and work well on low resource or older systems. They chose uclibc which was designed for embedded systems. When I worked with a uclibc based distribution, I found myself making several patches to Open Source programs just to get them to compile. Many Linux distributions such as Alpine Linux are now using musl. It's designed for efficiency and standards compliance including full POSIX standards compliance.

musl was designed to work with a Linux kernel. So unlike choices such as newlib, it is not easy to port to other operating systems. The midipix project is endeavoring to create a POSIX compatible layer for Windows, so that musl will work out of the box on that system. Musl uses a MIT license. The midipix project will use a more restrictive license similar to Cygwin. Many basic embedded system libraries tend to use less restrictive licenses like MIT and BSD so that they will be adopted by companies. However, project or company adapts a C library to a particular system and adds a lot of functionality, they typically tend to use GNU GPL or other more restrictive licenses. Many projects dual license in hopes of selling companies a commercial license with less restrictions.

Google developed the Bionic C library for it's Android operating systems. It has a BSD license. It's also designed to work on low resource systems so it tends to offer less functionality than other C libraries such as glibc. It has partial POSIX support.

I've been hitting many limitations with the MinGW runtime libraries when it comes to porting. Alternatives such as Cygwin's or midipix's runtime libraries would overcome the issues. However, the licenses are much more restrictive. I know I definitely wanted more POSIX compatibility on Windows than MinGW offers out of the box. As a cross-platform programmer, it would be nice to take whatever C runtime library I end up with on Windows and reuse it on Linux and other systems. I looked at C runtime libraries for embedded systems which typically port well. Of those, newlib seemed the most interesting, because according to some of the documentation it only requires 17 syscalls to port it to an operating system. Some Windows CE compiler ports use newlib with added functionality for the Windows CE operating system to derive a working C/C++ compiler. When I investigated the newlib code, I did not particularly like the design, especially the way it handled threading by providing standard and threaded versions of functions. The implementation of file I/O looked like it had been modified several times and at this point could use a major refactoring. When I read some of the comments in that section of the code, I felt very uncomfortable using the library. I wanted a simple, clean, basic design that I can add to.

Another option I looked at was PDClib. I love the idea of a public domain library. MinGW original licensed their WIN32 and runtime code (which integrates with Microsoft's code) as public domain.
PDCLib was based on an earlier Public Domain library project originally at Sourceforge. I tried PDClib on Windows (which the developer says he uses it with), but I was unable to get file I/O to work properly. I contacted the developer to see if he needed with the project, but he really didn't seem to need any assistance at the time. PDClib only supports standard C functions. It does not provide any POSIX functionality. So, it would have limited usage as is for running most Open Source programs.

A number of original operating systems use their own C runtime libraries. I figure, if they can reinvent the wheel and create a C runtime library for their particular purposes, so can I.

I've been coding functions that are typically part of the C runtime library in order to provide better porting support for Windows. I wrote a C11 compatible thread library, several BSD and POSIX string functions, some POSIX file functions, etc. That left me with the dilemma of how best to integrate the additions with the runtime library. They really should be part of the library and part of the C standard headers. I currently have them implemented as supplemental libraries that have to be added separately. It's easier to test and integrate on various operating systems that way. Ideally, it would be nice to have everything accessible as one library though.

I'm very familiar in the various methods of connecting to the kernel in Windows. Some projects such as midipix just use ntdll.dll. Other projects connect to other Microsoft dlls such as kernel32.dll. One can use LoadLibrary, GetProcAddress to connect t a dll or if the library is already linked in, one can skip LoadLibrary. A few Open Source projects I've seen actually implement the LoadLibrary functionality from scratch. I'm not as familiar with the techniques to connect to the Linux, BSD and other kernels and would love to find more clear documentation on this subject. If you run across any good materials, please let me know ( http://www.distasis.com/connect.htm ). Linux uses techniques such as vdso, vsyscall, syscall to call kernel functions.

I find it fascinating to consider the design trade-offs of various C runtime libraries. With that in mind, here's a list of some of the C runtime library options:


Linux from Scratch build instructions (including musl)
BSD regular expression library
(Musl regular expression support was forked from this library.)

Original Public Domain C library:
libTom Public Domain libraries for math and cryptographics functions
(Some of musl's functions were forked from these.)



Windows CE cross-compiler:


2017-04-20 11:53 am

pdcurses/ncurses applications

I've covered SDL and FLTK based applications. Now I'd like to discuss pdcurses and ncurses based ones. ncurses is found on many POSIX systems especially Linux. pdcurses was used primarily on DOS and Windows systems. Support for X and SDL was later added. The X support uses some legacy functions, so it doesn't build well on modern Linux systems. However, the SDL support allows pdcurses to be ported to any system that supports the SDL library. That means it can run on systems like Syllable and Haiku not just POSIX systems with X Windows. ncurses worked only on POSIX systems for a long time, but more recently support was added for Windows and MinGW32.

So which is the better library to use? Depends on your needs. ncurses is very stable, provides good Unicode support on many operating systems and offers added functionality like panels, menus and forms. On POSIX systems, it runs in console mode, so you can run applications outside X Windows (or Wayland) or on systems with no X Windows installed. Applications built with ncurses can also be used via ssh and telnet. Unicode support for pdcurses is currently limited to UCS-2 (16 bits). It designed in such a way that expansion to UTF-8 or UTF-32 would mean major changes. It works in console mode, but on DOS/Windows only. You'll typically need X Windows if you run pdcurses on a POSIX systems. There are some work-arounds to this. You can use DirectFB or nano-x to run SDL in framebuffer and use pdcurses with the SDL backend. However, applications run in framebuffer mode aren't usable with ssh and telnet. You'll need something like vnc to remotely access them. There is also a pdcurses fork called win32a which provides native Windows support and better support for colors. pdcurses has some functions that are not typically standard with ncurses/curses such as specific mouse and clipboard functions. However, it lacks a lot of the extras that ncurses adds such as menu and form support. Both pdcurses and ncurses have functions for the standard ASCII character set and for wide characters (wchar). However, keep in mind that wchar is 16 bits on Windows (and DOS djgpp) systems and 32 bits on most POSIX systems (such as Linux and BSD). Personally, I prefer to use UTF-8 whenever possible for Unicode. I also prefer the new char32_t (or uint32_t if not available) in place of wchar_t in all my applications with internationalization support. However, pdcurses/ncurses aren't designed for these options. Of the two libraries, ncurses better supports characters in a Unicode character set beyond the initial 16 bits and is better designed to handle large character sets.

Even though ncurses has many advantages over pdcurses, I still like to work with pdcurses. It's a more compact library, with the backend code separated from the rest of the code to make it easy to port to other platforms. With this design and with its SDL support, it's easy to get pdcurses working on a wide variety of operating systems including some places ncurses has not been ported. As mentioned, libform and libmenu are not a part of pdcurses, but their functionality is available with ncurses. It turns out that other versions of curses did not originally have support for forms and menus and they were separate libraries. BSD still has libform and libmenu that work with its version of curses. I found it very easy to port libform and libmenu to work with pdcurses. There were some bugs in the libraries which I'm surprised no one has fixed yet, but I have patches for them. One other advantage of pdcurses built with SDL is that you can intermix SDL and pdcurses commands. pdcurses/ncurses are primarily console/text libraries. SDL offers graphics support. You can mix graphics from SDL with text from pdcurses and use pdcurses as a text user interface (GUI alternative) for SDL. That's a really nice option when developing SDL applications.

I've added SDL 2.x support to pdcurses, so it will port to systems that support SDL 1.x or SDL 2.x. pdcurses with the SDL backend, used bitmaps to represent characters. That curtailed Unicode support. I've added support to use SDL_TTF for drawing characters. Now the only limitation on characters that can be used with PDCurses is the 16 bit limit to the character size built into the system. It should be possible to expand that to 32 but it would require modifications that would affect the entire design. Also, since curses only supplies ASCII and wchar support for routines, a Windows (or DOS) system would still be limited to 16 bits. One would need a new set of routines with UTF-32 support or UTF-8 support. The current routines could not be used to support 32 bit characters on systems where wchar is limited to 16 bits. The ASCII functions are designed to work with one character at a time not several which UTF-8 might require.

Some of my patches for better internationalization support for pdcurses have been added back in to the official version of pdcurses. All of my pdcurses patches plus documentation on them and my build scripts can be found by following the archive link at http://www.distasis.com/cpp/lmbld.htm I'm interested in adding some of the win32a patches and continuing to improve support for internationalization in the future. If anyone wants to discuss design ideas for continuing to add better Unicode support for pdcurses, please contact me.

The good news with ncurses/pdcurses based applications is that it's relatively easy to port them from one version of curses to another in many cases. It may require changes in the names of header files. Non-portable functions such as those supporting a mouse may not be available on some systems. However, one can typically use any application designed for a particular version of curses with another version of curses. That does not mean ncurses/pdcurses applications can easily be ported to other operating systems. For instance, if an application uses functions that aren't very portable across platforms (such as fork which isn't easy to emulate on Windows and doesn't work on DOS), it will be very difficult to port that application. Unfortunately, many ncurses applications do just that and use POSIX functions that aren't widely ported to non-POSIX systems.

So, let's talk about some of the applications that I've found that do port across a variety of operating systems and can be built with pdcurses.

There are several ncurses based hex editors available. I personally like the interface for ncurses hexedit. It reminds me of the kzap program I used to use with DOS, so I find it user-friendly. shed is another interesting option. It handles large file sizes extremely well.
You can find the programs here:

I have patches/build scripts for ncurses hexedit to work with pdcurses and port to other platforms. I've built shed on Windows before, but haven't created my own build scripts/patch set for it yet.

There aren't a lot of music applications that work well in console mode and provide a text user interface, but there are some. The ones that come to mind are TiMidity++ and GramoFile. Make sure if you intend to work with TiMidity++ that you start with the source from version control not the very old, outdated tarball at Sourceforge. Many features, including better Karaoke midi support, have been added since the tarball was created. I find it discouraging that distributions such as Debian refuse to update to the latest supported version of this project. I always build TiMidity++ from source myself to make sure I have the updates I need. I'm currently working on a build script for TiMidity++ that will allow me to build it as a library so it can be used with other applications. I currently have build scripts for TiMidity++ and freepats (Open Source sound fonts used with TiMidity++) and gramofile.

I originally looked for library versions of libform and libmenu because I thought a lot of curses applications would make use of these features. Turns out, not as many use it as I originally estimated. One program that makes use of the functionality for these libraries is ckpass. It's a console based password keeper that is compatible with KeePass 1.x format. That means, if you want a lightweight alternative to the earlier versions of KeePass or KeePassX, you can still have console access to your passwords. The program looks great and it's easy to use. Unfortunately, it offers just read-only support for viewing password information. There's no easy way to modify password information using the program. However, it isn't that hard to add code to allow editing as well as viewing of passwords using libform. I'm working on some modifications to ckpass to give it more functionality, so it will become a useful, more lightweight replacement to other password storage alternatives. There are other command line based password storage options available and some even use portable formats so that you can find cross-platform solutions. However, I like the idea that ckpass supports the KeePass 1.x format which many other GUI password programs also use. This is the link to the original ckpass project:

The final category I'll cover is the text editor category. There are a great many text editors that work in console mode or build with ncurses as can be seen from the listings at TextEditors.org ( http://texteditors.org/cgi-bin/wiki.pl ). However, the large majority of them are not highly portable.

When the competition between vi and emacs comes up and people start debating what should be the default editor on a Unix or Linux system, I typically skip both and go with nano. nano was based on the pico text editor. It wasn't a fork. It was coded from scratch, so there would be no issues with code licenses. pico was the main editor for the alpine mail system. Compared to other editors with unusual key sequences that you need to memorize to get anything to work, I find it very user-friendly. It has a menu and easy access to help if you forget what command keys you need to perform a specific function. My personal preference is to use a programming editor that will allow you to map key commands however you see fit. That avoids large learning curves if you didn't originally use vi or emacs as your first text editor and memorize their commands. However, if a text editor with keymapping support is not available, a user-friendly text editor with easy to access command menus and help descriptions is a good second choice. Most POSIX systems I work with have either pico or nano available or they can be installed fairly easily. So, if I need to do a quick edit from the console on a POSIX system, my default editor is typically nano or pico. One major drawback to pico is that it only handles one file at a time. You cannot have multiple files open and flip between editing them or cut and paste between them. nano adds support for multiple files, but the commands to switch aren't particular easy to remember.

I did a port of nano to Windows several years ago. There's a lite way to build nano (tiny) and a way to build it with more functionality. Building it the lite way and adding some modifications for porting, I was able to compile it successfully on Windows. So, while it didn't provide all the functionality I used on POSIX systems, it did work on Windows. A few years ago, I found a Windows port of nano on github. It didn't provide extra functionality, but it was interesting to see how the author went about porting the application and the differences between my port and that one. According to the nano editor FAQ, it now looks like there's some official support for Win32. However, when I tried to build the latest version, I still ran across a lot of the same issues as when I built previous versions. Only the lite version is supported for Windows. There were some assumptions about MinGW types that aren't true for all versions of MinGW. Rather than using configure for what it was designed for and testing if there needs to be a work-around for a particular system, the code has ifdefs that assume MinGW always handles things wrong and tries to fix the situation. In the process, it causes errors in the build for MinGW systems that do handle the situation properly. Guess I'll need to stick with my port or a Windows specific port like the one at github if I want nano to build properly on Windows.

nano is a great lightweight alternative for a text editor that works on a large variety of operating systems. When I need to modify a file and want an editor to quickly call from the command line, it is often my editor of choice. You'll find the source code and more information on nano at:

As I mentioned, I prefer programming editors that have good support for key-mapping. The scintilla text editing component offers just that. A large variety of Open Source editors, including SciTE (my favorite GUI editor) and fxite (lightweight FOX Toolkit based GUI editor), use the scintilla text editing component. While fxite is lightweight and works well on older systems, it requires X Windows on POSIX systems and the FOX Toolkit to build it from source. I wanted something even more lightweight. I did a search of the various editors that make use of the scintilla text editing component and I ran across textadept. It is a GUI text editor, but there's a ncurses based version that works from the console. I found it relatively easily to build the editor on Windows using pdcurses. So, it's rather portable. If I had to pick a console based editor besides nano and pico, textadept would be it.

There are ncurses front ends to some applications I work with such as hunspell and prozilla, but I typically don't use the front ends. I prefer using hunspell via command line or integrated with a programming editor. There's a nice FLTK front end for prozilla that is highly portable. One other ncurses based application I'll mention is WordGrinder. I typically see it mentioned on sites and blogs that specialize in discussing console based applications. It has many of the features of a word processor. The current web site for it is:

If you have other recommendations for useful, cross-platform portable ncurses/pdcurses/curses based applications, I'd love to hear ( http://www.distasis.com/connect.htm ) about them. It would be great to add more helpful applications to this list. I'm also very interested in the possibilities of using pdcurses/libmenu/libform as a text user interface (GUI alternative) for SDL applications.
2017-03-28 07:59 am

My projects with rhyming and language translation and other dictionaries, word lists and thesauri

The last post mentioned other groups' projects with dictionaries and language resources. I thought I'd mention some of the projects I've been working on in this area.

I've been creating build scripts with the LM BLD project ( http://www.distasis.com/cpp/lmbld.htm ) so that I'll have automated, repeatable steps to build programs, libraries and other types of packages. Here are some of things I've been working on.

The Moby project is a very nice dictionary resource. Using their thesaurus, I was able to create a word list and a simple dictionary in stardict format. I use it with Open Source programs like scramble.

The Strongs concordance is in the public domain. I've created a translation dictionary in stardict format with it.

I happen to like the stardict dictionary format. There are several nice programs that can work with that format. I wanted something lightweight that would work well on older systems or let me create my own GUI interfaces. The closest thing I could find to what I wanted was sdcv. However, there were a few issues I had with it. The biggest is that it requires glib as a dependency and I didn't want to install GTK+ related dependencies on my systems. The second issue I ran into was that it couldn't handle some of the newer versions of the stardict format. Since the code is GNU GPL licensed, I started with it and made several modifications and customizations. The result is sdcv2 which can be linked to my own Unicode shared libraries in place of glib if desired and can work with dictionaries in more recent stardict formats. It may not make use of all the latest features in the newer formats, but it can at least access information from them.

I've seen other projects that use the sdcv library as a back-end and create their own GUI for a dictionary program. It makes sense if the program uses GTK+, but it seems awkward for Qt or other GUI programs to require GTK+ related dependencies. With sdcv2, there are no GTK+ related dependencies.

I would love to find a dictionary with a FLTK GUI, especially if it can handle stardict format. Since, I haven't been able to find one, I may try to write one at some point. I've also been thinking about creating a pdcurses front end. When I use sdcv (or sdcv2) from the command line, certain systems like Windows can't handle input or output of certain Unicode characters correctly. I've added support for SDL 2.x, SDL2_ttf and the ability to work with a range of Unicode characters within the UCS-2 character set to pdcurses. I think pdcurses would make an interesting front end for a program using the sdcv2 library. It would work on any system that supports SDL 1.x or 2.x, including more unusual operating systems like Syllable and Haiku. Would like to hear from others who may be interested in or are working on similar projects.

The dictzip program compresses dictionary files. It uses an extension to the gzip format with extra fields to include information about the compressed dictionary. Files compressed with this format often use the .dz extension. You can use dictzip with stardict files to save space. dictzip is primarily a POSIX compliant program, so it doesn't convert well to certain systems. I was able to find a Windows port that limited the program's functionality, but did enough to get the job of compression done. I've made some modifications to it and am using it as a cross-platform method of compressing stardict dictionary files.

Several utilities and conversion programs were created for stardict in the stardict-tools project. Similar to stardict and sdcv, glib is a dependency for stardict-tools. There are a few tools that use a GTK+ front-end as well. I personally only use the stardict-tools to convert tab delimited files and files in babylon format to stardict. So, I modified the command line tools that do those conversions to build without glib. I also created my own makefile just to build the tools I use.

I've searched and I've yet to find a rhyming dictionary in stardict format. So, I'm working on creating one. It's a slow process. I've taken a public domain rhyming dictionary as a starting point and I'm in the process of editing it and converting it to the format I need.

I've also been searching for an Open Source C/C++ grammer checker, but I've yet to find one that I like.

These are just some of the projects I'm working on. If you're interested in comparing notes on these topics or if you have recommendations of other dictionary and word related projects you like, feel free to contact me ( http://www.distasis.com/contact.htm ).
2017-03-23 09:25 am

Word lists, dictionaries and rhyming dictionaries, thesauri and language translation references

It's hard to find public domain, Open Source and Creative Commons licensed language resources in formats that are easy for programs to work with. There are growing resources of scanned public domain books. Among them, you can find all kinds of dictionaries and references. Some sites even use an OCR to translate scanned documents to text formats. However, the translated versions are usually full of typographical errors.

There are a few projects out there that use Free, Open Source or Creative Commons licensing and have a goal of creating dictionaries or other references in accessible and searchable digital format, but not a lot. It would be nice to see more projects of this sort. The results could be useful with word processors and editors (such as LibreOffice, Abiword, SciTE), electronic dictionaries (such as stardict) and games (such as anagramarama and scramble).

Here are some of the projects I've located:

XDXF, the XML dictionary interchange format, project had a collection of dictionaries and language translation word lists they were working with and converting to various formats.
They also have tools for converting between XDXF and other formats:

The Moby project is a wonderful public domain resource. It includes word lists, thesaurus and more.

The Free Dictionaries Project also provides downloads. If you want to translate one language to another, this is a useful, free resource:

SCOWL (Spell Checker Oriented Word Lists) and Friends has useful word lists and resources for spell checker utilities:

YAWL (Yet Another Word List) is based on the updated Public Domain ENABLE (Enhanced North American Benchmark Lexicon).
You can also find Libre licensed word lists in FLOSS games such as anagramarama.

Here's a rhyming dictionary (source code and online example) that uses Moby project resources to find rhymes:

If you know of other projects or developments in this area, I'd love to hear about them ( http://www.distasis.com/connect.htm ).",public,0,,
16253,2017-03-28 07:59:00,2017-03-28 11:59:20,"My projects with rhyming and language translation and other dictionaries, word lists and thesauri","The last post mentioned other groups' projects with dictionaries and language resources. I thought I'd mention some of the projects I've been working on in this area.

I've been creating build scripts with the LM BLD project ( http://www.distasis.com/cpp/lmbld.htm ) so that I'll have automated, repeatable steps to build programs, libraries and other types of packages. Here are some of things I've been working on.

The Moby project is a very nice dictionary resource. Using their thesaurus, I was able to create a word list and a simple dictionary in stardict format. I use it with Open Source programs like scramble.

The Strongs concordance is in the public domain. I've created a translation dictionary in stardict format with it.

I happen to like the stardict dictionary format. There are several nice programs that can work with that format. I wanted something lightweight that would work well on older systems or let me create my own GUI interfaces. The closest thing I could find to what I wanted was sdcv. However, there were a few issues I had with it. The biggest is that it requires glib as a dependency and I didn't want to install GTK+ related dependencies on my systems. The second issue I ran into was that it couldn't handle some of the newer versions of the stardict format. Since the code is GNU GPL licensed, I started with it and made several modifications and customizations. The result is sdcv2 which can be linked to my own Unicode shared libraries in place of glib if desired and can work with dictionaries in more recent stardict formats. It may not make use of all the latest features in the newer formats, but it can at least access information from them.

I've seen other projects that use the sdcv library as a back-end and create their own GUI for a dictionary program. It makes sense if the program uses GTK+, but it seems awkward for Qt or other GUI programs to require GTK+ related dependencies. With sdcv2, there are no GTK+ related dependencies.

I would love to find a dictionary with a FLTK GUI, especially if it can handle stardict format. Since, I haven't been able to find one, I may try to write one at some point. I've also been thinking about creating a pdcurses front end. When I use sdcv (or sdcv2) from the command line, certain systems like Windows can't handle input or output of certain Unicode characters correctly. I've added support for SDL 2.x, SDL2_ttf and the ability to work with a range of Unicode characters within the UCS-2 character set to pdcurses. I think pdcurses would make an interesting front end for a program using the sdcv2 library. It would work on any system that supports SDL 1.x or 2.x, including more unusual operating systems like Syllable and Haiku. Would like to hear from others who may be interested in or are working on similar projects.

The dictzip program compresses dictionary files. It uses an extension to the gzip format with extra fields to include information about the compressed dictionary. Files compressed with this format often use the .dz extension. You can use dictzip with stardict files to save space. dictzip is primarily a POSIX compliant program, so it doesn't convert well to certain systems. I was able to find a Windows port that limited the program's functionality, but did enough to get the job of compression done. I've made some modifications to it and am using it as a cross-platform method of compressing stardict dictionary files.

Several utilities and conversion programs were created for stardict in the stardict-tools project. Similar to stardict and sdcv, glib is a dependency for stardict-tools. There are a few tools that use a GTK+ front-end as well. I personally only use the stardict-tools to convert tab delimited files and files in babylon format to stardict. So, I modified the command line tools that do those conversions to build without glib. I also created my own makefile just to build the tools I use.

I've searched and I've yet to find a rhyming dictionary in stardict format. So, I'm working on creating one. It's a slow process. I've taken a public domain rhyming dictionary as a starting point and I'm in the process of editing it and converting it to the format I need.

I've also been searching for an Open Source C/C++ grammer checker, but I've yet to find one that I like.

These are just some of the projects I'm working on. If you're interested in comparing notes on these topics or if you have recommendations of other dictionary and word related projects you like, feel free to contact me ( http://www.distasis.com/contact.htm ).
2017-03-20 08:56 am

How to find lightweight applications

It's fun to discover new lightweight applications. They work well on newer computer systems as well as older or slower computers and low resource machines like many mobile devices. You can run more of them at once. If they're not well-known, they can actually be more secure sometimes (using the security through obscurity principle). I also personally prefer portable applications. That way, you can use the same programs on any operating system. You don't have to relearn new programs for each system you work with.

It can be quite a challenge to find new lightweight applications. I've read several threads on forums where users post their favorite lightweight applications. Many truly are not lightweight by standards that take into consideration memory usage, lines of code, compilation time and/or number of dependencies (libraries).

One way to find lightweight applications is to look for programs built with lightweight GUIs. I've seen a few comparisons of GUI performance. This one is particularly good because it tests the various GUIs and gives statistics:
I was rather surprised by the SDL2 results. Generally, the time it takes to build a GUI from source is one good indication of complexity. FLTK and SDL both build quickly from source compared to the other GUI frameworks mentioned. So, I was surprised that SDL2 scored so badly on the memory usage tests. I'd be curious to know if SDL 1.2.x (which many systems still use) would show a large improvement. Another surprise was how well Tcl/Tk did in the tests. I typically think interpreted languages have worse performance than compiled ones. It would be interesting to see some statistics on response times for similar applications created with these GUIs.

I often go through various source repositories such as Sourceforge, github, etc. looking for code written using specific user interfaces in order to find new and interesting applications. Standard search engines are another way to search for programs. The user interfaces I'm personally most interested in at this point are FLTK, pdcurses/ncurses, SDL and command line programs. These types of applications are typically more lightweight or designed to do one thing well. Know of any other lightweight GUIs or TUIs (text user interfaces)? Please share your recommendations and why you like them.

There are some nice blogs for finding and discussing minimalistic (or in some cases maximalistic) programs. Unfortunately, many are no longer very active. Some favorites are:

If you know of others, I'd love to hear about them.

One can also look for lightweight distributions and see what programs they have in their repositories or read their forums for more suggestions. Some of the interesting distributions to check are TinyCore Linux (uses several FLTK programs), Nanolinux (uses more interesting FLTK programs), Rogue Class Linux (uses several SDL programs), Puppy Linux, AntiX (Debian based), INX ( http://inx.maincontent.net/ ), Absolute Linux (Slackware based), 4MLinux ( https://sourceforge.net/projects/linux4m/ ), OLPC. Typically DSL and Puppy get mentioned when people list lightweight Open Source systems. There's been no active development on DSL in a long time and the forums are very quiet. I also found Puppy a little too resource intensive on one of my older machines. FreeBSD performed much better on that system. Puppy Linux has some interesting discussions in their forums.

Linux systems that work in framebuffer mode using DirectFB, nano-x and other alternatives also typically contain many interesting, unusual and lightweight applications. Nanolinux and Rogue Class Linux are in this category.

One can also look at operating systems and development projects that use more lightweight C libraries (such as uclibc and musl). Those projects typically gravitate to choosing lightweight applications, command line and console based programs and lightweight tools like Busybox and Toybox.

Alternative operating systems often offer interesting lightweight application choices. Syllable and Haiku often use SDL programs and other lightweight applications that are easier to port to those systems. Systems like Minix and ELKS are also interesting to investigate. Minix 3 uses a lot of the programs that BSD systems do, but earlier versions of Minix include some interesting alternatives. XFDOS includes many interesting FLTK applications. Plan 9 is interesting as well, but not many of the programs used on this system have been ported to other systems. Another good place to look for unusual applications is on mobile devices.
Here are some application lists from Syllable and Agenda:

I'd love to find more places to discuss lightweight applications. If you've written an article on the topic, please share it. If you know of a good blog, forum, mailing list or other resource, please let me know ( http://www.distasis.com/connect.htm ). If you'd like to discuss your favorite C/C++ applications further, you're welcome to use the CppDesign mailing list ( http://groups.yahoo.com/group/CppDesign ) as a forum.
2017-03-10 02:25 pm

Build Systems - Alternatives to GNU autotools, cmake, etc.

I've been looking for useful, efficient, user-friendly alternatives to build tools like GNU autotools and cmake. I've even created lists of build tool alternatives at various wikis and sites related to programming and compilers. If there's interest, I can post some of that information here along with pros and cons of various tools. I think, during my searches, I've found some good alternatives for my own projects and to use with my own build script generator.

If you're looking for build tools for your own projects, you may want to try the following tools. I'm also doing some active development and patching to some of the tools and would be interested in comparing notes with others on ways to use and improve them (see below for details).

GNU autotools is probably the most popular build system for most C/C++ programs. So, what's wrong with it? Just to build a C program, you need two interpreters (m4 and Perl). You also need autoconf, automake and make. The input to these programs is not necessarily intuitive to a C programmer. Many C programs, especially those using Qt make use of cmake which has less dependencies, but I've also found it less than intuitive. What if you want to build a system from scratch. You can get make to build without any other build tools. I've done it. From there, you can work to build the rest of the gnu autotools toolchain. To build cmake, you need to bootstrap the build using another instance of cmake. So where do you get the original build of cmake for the system? There is a way to build cmake with gnu autotools, but I've found it rather buggy.

After looking through several make alternatives, I really have not found a good replacement for it. The syntax takes some getting used to, but I've been able to find ways to get it do whatever I need it to do. There are situations where make is very inefficient, but there are work-arounds. Efficiency really depends on how well you craft your makefiles. I attempted to find some alternatives to the GNU version of make. BSD make has very different syntax, so only the simplest make scripts work with both tools. I find GNU make used more often than BSD make for building Open Source projects and it has features that were never added to BSD make. There's a Perl implementation of make, but that requires a Perl interpreter. My main issue with GNU make is that it is only supported on certain platforms. I sent in a bug report about GNU make not working properly on native Windows and I even sent in a patch. The developers were not interested in supporting native Windows. They thought it was enough that there were versions that worked with Cygwin and msys. Unfortunately, I've yet to find an alternative to make that handles the GNU makefile sysntax and doesn't require installation of an interpreter.

I've decided that until I can find anything better, I will continue to use GNU make. Since it is Free Software, I can modify the code to work on other systems that GNU make developers don't care to support. I can distribute the modified version if I distribute the source code. So, I use a modified version of GNU make that fixes platform specific bugs that I've encountered.

After searching through many alternatives, I've found a wonderful replacement to configure/autoconf. It's also more compact and I find it more intuitive than cmake. It's called CDetect. It's written in C and is comprised of only 3 files. The project is no longer actively developed. However, I've been making several modifications of my own to get it to suit my projects' needs. I've implemented some of the items on the original developers To Do lists. I'm still making modifications as I find features that are lacking, but it makes a useful replacement to configure. I've converted some of the Open Source projects I build from source regularly to use CDetect and custom makefiles instead of their build systems. If you're building from source, such as with a Linux from Scratch system, using CDetect means you don't need Perl, and several other programs installed just to build a C/C++ program.

Freedesktop.org advocates the use of pkg-config. It's used by many GNU projects as well. While I originally thought it was yet another complication when building from source, I've found pkg-config rather useful in simplifying build scripts. You don't have to worry about finding the right library, pkg-config can tell you where to find it. If you're cross-compiling, you can get have two sets of directories with pkg-config information, one for cross-compiled libraries and one for regular libraries. Change the path in an environment variable and your build script knows where to find the appropriate libraries. What I really didn't like about pkg-config was the complexity of the program and that it required a circular dependency with glib. I don't even want glib on my system at this point. Luckily, there is an alternative with friendly licensing called pkgconf. It's a drop in replacement for pkg-config. It's very helpful in finding where the right libraries are on a system. I've gone from never using pkg-config to use pkgconf in almost all my build scripts.

So, between CDetect, pkgconf and make, I now have all the tools I need to use with my build system in order to build the Open Source programs and libraries I use most in an automated fashion. The tools are all written in C, so they require no special interpreters or other languages just to build them.

If you'd like to try out CDetect with my patches or see some of the build scripts I'm using with these tools, check out the archive link on my LM BLD system page:

I've added several features to my patched version of CDetect. Check the notes.txt file for details. I'm currently making some new modifications and would love input from other developers/users. If you'd like to make suggestions or compare notes, please contact me through the CppDesign mailing list or my web site:
2017-02-16 04:43 pm

Open Source for Mobile Devices and Web Development

There are some great Open Source applications out there for desktops and laptop machines, but it's much harder to find Open Source for mobile devices. Most mobile devices require their users to go through a proprietary app store just to get software onto their devices. This is a setup back for the Open Source movement since more and more people are turning to mobile devices as their primary computing source.

If you have suggestions and/or recommendations for FLOSS alternatives in the mobile space, please share them with me and other Free, Libre and Open Source advocates on the Schoolforge mailing list ( https://schoolforge.net/ ).

I'll share some Open Source options I've found to date for developing mobile and cloud based applications.

Cross-platform Mobile App Development

Here are a couple of very interesting projects that let developers create cross-platform mobile applications:

Developers can use C/C++ which can be compiled to run natively on certain platforms or converted to Java bytecode output that can run on phones that only support Java applications. One can also develop HTML/CSS/JavaScript applications for phones that support those formats (similar to Cordova).

This project supports HTML/CSS/JavaScript development of applications that will work on a wide variety of mobile platforms.

C/C++ Mobile Development

If you want to port your own C/C++ programs or interesting Open Source C/C++ applications to mobile devices such as Android, these projects may be useful.

Information on using libSDL 2.x with Android.

Port of Nano-X/MicroWindows to Android. Nano-X/MicroWindows supports a subset of simple X11 and Windows applications. Many FLTK applications port easily when run with Nano-X.

Article on building and running Allegro based applications on Android.

C/C++ CGI Development

As a C/C++ programmer, I was surprised by how few CGI applications were built with these languages. Originally, I read that the reasoning behind this was that one would have to spawn a new process every time one wanted to run a C/C++ CGI program. Interpreted languages could be built into the web server and avoid spawning new processes. However, with the introduction of libraries such as FastCGI, the inefficiencies of running compiled programs as CGI applications is no longer an issue. From everything I've read, C/C++ CGI programs are much faster than other CGI programs if they avoid the process spawning issue. If you want a highly efficient CGI program, it's a good time to revisit using C/C++ with the proper libraries.

Here's some more information on FastCGI.

If anyone's interested, I have builds of FastCGI and Apache with fcgid support for Windows/MinGW (including patches to get them working on later versions of Windows).

Wt is one of the few projects I've seen that tries to provide useful CGI functionality and routines for C/C++ programmers.

C/C++ on the Web

One can also use projects like emscripten to compile C/C++ code into JavaScript that will run on the web. It's an easy way to port your own C/C++ applications or run popular Open Source applications to the web.


Another area of web development that surprises me is when I see so many language alternatives to JavaScript. JavaScript is supported (in some fashion) by most browsers. However, other programming languages are not. In order to get other languages to work in a browser, they typically need to be converted back to JavaScript. In a case like emscripten, it can be useful to convert C/C++ to JavaScript for porting purposes. However, many new web projects are choosing other languages besides JavaScript (because developers prefer the language) and then using tools to convert those languages back to JavaScript so they'll work via the Internet. Often the conversions introduce a lot of unnecessary code or provide slower implementations than the original. If I'm porting a project that's already written, it's great to be able to move it over with projects like emscripten. However, if I'm writing something new for the web that requires a programming language, I always use JavaScript. It's more efficient to write something specifically for that language than rely on generated code converted from another language.

It's also nice to be able to write CGI in JavaScript and share the same functions, objects and code between the client (browser) and the server. This is one of the major benefits provided by projects like CommonJS. The CommonJS site used to maintain a very nice list of server side JavaScript implementations. Unfortunately, the CommonJS project appears to no longer be active. It's been replaced by more implementation specific solutions like node.js.

There are alternatives for running server side JavaScript other than node.js if you don't want to be locked into a particular web server.

One option is TeaJS which uses FastCGI and works with a variety of web servers including Apache and Nginx.

Another option is jsc which is a JavaScript implementation that is a part of Webkit (one of the most commonly used browser libraries).

There are also some interesting TinyJS implementations and forks at various code archives such as github. You'll find more details under my Scripting Languages post ( http://lmemsm.dreamwidth.org/2239.html ).
2016-12-20 02:58 pm

PDF and Postscript

Some interesting Open Source command line programs output their results in Postscript and/or PDF format. That makes it useful to have lightweight PDF and PostScript viewers that don't require a lot of dependencies to build so you can view their output quickly. It's also nice on Linux systems if they work in framebuffer mode, so you don't have to start an X Windows session just to view results. Cross-platform viewers work on a variety of systems from Linux and Mac to Windows to DOS and even Android. Having tools to edit/modify PDF and PostScript output is also useful.

I've been searching a lot lately for lightweight PostScript and PDF viewer options. Thought I'd share some of what I've learned about cross-platform PostScript and PDF support. If you know of any other viewers or PDF/PostScript related programs with minimal build dependencies that I may have missed, please let me know. I'm always looking for additions for my Open Source applications list: http://www.distasis.com/cpp/osrclist.htm

First, here some of the programs I use that output to PDF or PostScript.

This does have a lot of build dependencies (needs Qt and webkit), but it's one of the few programs that I've been able to find that does a good job of converting HTML to PDF.

Other HTML to PostScript/PDF alternatives are html2ps ( http://user.it.uu.se/~jan/html2ps.html ) which requires Perl and PhantomJS ( http://phantomjs.org/ ).

I use abcm2ps to convert songs in ABC notation to sheet music. Later versions of abcm2ps have the ability to output to SVG as well as PostScript. I have a lightweight SVG viewer based on nanosvg and SDL that displays SVG, so that should be an interesting display option if a PostScript viewer isn't readily available. I can upload the build scripts for it to my archive if anyone wants to try out the SVG viewer.

lcal and pcal
Personal calendars and lunar calendars can also be output to PostScript format.

The graphics layout engine outputs charts and graphs to PostScript, PDF and some graphics formats.

Other applications can output starcharts, barcodes, graphs, etc. to PostScript format. The applications I listed above are the ones I personally use the most in this category. If you have other favorites, would be interested to hear about them.

Next, I'll mention some PDF libraries and viewers.

The two main PDF rendering libraries/applications are mupdf and poppler.

Poppler is a library based on the xpdf viewer, so in most cases you really only need one of these on your system. They both have a lot of the same utility programs. When building poppler, there's additional support for GTK and Qt frameworks if you have those on your system. I typically build with these turned off to lower dependencies.

Mupdf was written by the developers of Ghostscript. It renders very quickly and is supposed to be very lightweight. However, what I dislike about it is its growing list of dependencies. Newer versions of mupdf provide more and more functionality, but they're requiring more dependencies to do so. Later versions of mupdf now require harfbuzz which I typically don't use with any other applications. (I'm not using Opentype fonts.) Also, the developers recommend you use their third party libraries rather than system libraries. There are work-arounds to use the system libraries, but it's a nuisance to modify the build scripts each time the program is updated. The API is not very stable either. Every time I download a new version, it seems to break third party utilities built using the mupdf library.

Mupdf has example viewers that come with it. The latest example includes an OpenGL based viewer. I did some work on a SDL based viewer using mupdf, but the API kept changing, making it difficult to maintain. When the OpenGL based viewer was added, I looked into using picogl in place of OpenGL/Mesa as a backend for the viewer. Using a viewer that comes with mupdf meant the mupdf developers would be responsible for dealing with the API changes. I got the OpenGL based viewer to build with picogl, but picogl still needs additional functionality to provide clear rendering of the text displayed by the mupdf viewer. Hope to work on this further in the future and would love to hear from anyone else who'd be interested in helping on this project.

There are some SDL and framebuffer based PDF viewers. However, they were either not easy to port to other platforms or had heavy dependencies I didn't want to add to my system. I found it interesting that some viewers used SDL as the actual graphical user interface (and could run in framebuffer mode on Linux), but still required GTK or Qt as an additional interface when rendering pages.

I finally found a lightweight, cross-platform PDF viewer that uses FLTK as a front end. It's called SpRead ( https://github.com/muranoya/SpRead ). It can be run in framebuffer on Linux using FLTK built with nano-x. The viewer is very lightweight and does seem to render quickly. Dependencies are really minimal compared to other viewers I've looked at. However, mupdf renders faster in many cases. SpRead uses poppler for PDF rendering and libarchive for zipped/archived file access.

Some other interesting PDF libraries available for editing and creating PDFs are qpdf ( http://qpdf.sourceforge.net/ ) and libharu ( http://libharu.org/ ).

Poppler and xpdf both come with PDF to text conversion utilities. This is very useful if you want to work with grep to search for a word or phrase in a collection of PDF files. Mupdf has added the ability to convert to text in more recent versions. However, their help/man pages on how to use their tools could use improvement. When I was searching for a lightweight PDF to text converter, I ran across pdftxt at https://litcave.rudi.ir/ As I mentioned, the mupdf library API keeps changing with newer releases, so typically when I update my version of mupdf, the pdftxt build breaks. I've uploaded patches to build pdftxt with the version of mupdf that I've most recently updated to. You can find patches and build scripts at the archive link on this page:

Since lightweight PDF viewers are more plentiful than PostScript viewers, I searched for PostScript to PDF converters. The main option in this category is Ghostscript. Like mupdf, it requires several dependencies and uses its own versions rather than system versions of libraries. It's actually worse in this area than mupdf. The only viable option to Ghostscript that I could find for PostScript rendering was xpost ( https://github.com/luser-dr00g/xpost ). It has less dependencies that Ghostscript and is easier to build. However, it's still a work in progress. I'm hoping at some point that it will replace Ghostscript for the functionality I'm interested in, but it at this point, it still doesn't have all the functionality I require. It does look like a promising project though.

Those are some of the interesting libraries and applications I've found on my search for PostScript and PDF related utilities. Hope you find some of them useful. If you know of other lightweight, cross-platform alternatives, I'd love to hear from you about them.
2016-12-13 10:55 am


This is a list of mono-spaced fonts, Unicode fonts, font resources in general and useful FLOSS font sites.

Monospaced fonts are particularly useful for programming. They can help improve readability. Some programming editors (like PFE) only work with monospaced fonts. Other editors allow the use of more varied fonts. However, monospaced fonts are useful when working with PRE tags in HTML, ASCII art and in other areas.

Other useful fonts include Unicode fonts for internationalized projects and fonts for people with handicaps or disabilities such as fonts designed for people with dyslexia.

Using a Free, Libre or Open Source font or a public domain or Creative Commons licensed font makes it easy to reuse fonts in Open Source (or even commercial) projects.

Bitmap Fonts

Fonts for DOS and Windows Command Prompts

Free Monospaced Fonts (Bitmap)

Dina Programming Font

Electronic Font Open Laboratory

Envy Code A

Envy Code B


Liberation Fonts


MIT license

Proggy Fonts

Raize font


Terminus font
Includes a large size font especially useful for terminals in framebuffer mode on Linux systems.


Free Truetype Monospaced Fonts

Adobe Fonts


Bitstream Vera

Available with Povray software.

Cutive Mono

DejaVu Fonts

Droid Sans Mono

Envy Code R

Font Squirrel Monospaced fonts

GNU FreeFont

GNU Unifont


Unicode font for medievalists

Lucida Sans Typewriter Regular - included with Oracle's Java SDK

Matthew Welch's MIT licensed fonts
White Rabbit worked as a monospaced font


Oxygen Monospace
Oxygen Fonts

PT Mono

Roboto Mono
Google's Roboto fonts

SIL Encore Fonts
Several unicode fonts including some monospaced ones.

Source Code Pro
Adobe Font

String Literal

Ubuntu Fonts
Ubuntu Mono

Xerox Alto mono-spaced font

Other Fonts

EB Garamond


Lato Fonts

Libertine Open Fonts Project


Simon Tatham's Fonts Page

Sorts Mill fonts

Font Related Links

Cufon Type Converter
Convert TTF fonts to cufon font format for web sites. With a few text changes, fonts can be used with Raphael.js too.

Create your own fonts.

Font Library

Font Squirrel
Handpicked Free Fonts

Google Fonts

Monospace/Fixed Width Programmer's Fonts


SIL Open Font License (OFL)

Wikipedia Open Source Unicode Typefaces

Unicode Font Guide For Free/Libre Open Source Operating Systems
2016-12-08 11:50 am

FLTK applications list

I covered SDL based applications. Now, I'd like to cover FLTK based applications for desktops and/or productivity.

While there isn't as much FLTK application development going on as I would like, there are some projects that specialize in using FLTK. TinyCore Linux is probably the most well-known Linux project that uses several FLTK applications. Nanolinux is based on TinyCore but uses nano-x as a lightweight alternative to X Windows. The developer of NanoLinux uses mainly FLTK applications and has modified and updated several FLTK applications to give them new life. He's also created some of his own where good alternatives did not exist. The Equinox Desktop Environment also uses FLTK, but it typically requires another EDE specific library along with FLTK support. Also, EDE users don't always look for FLTK applications for their desktops. Some will typically use anything they consider lightweight (whether it really is lightweight or not). A few mobile devices use FLTK as their main GUI. Users of those systems have developed some interesting applications for their devices.

There are several versions of FLTK. Applications may work with one version and not another. I've spent a lot of time searching for applications and porting applications to the latest version. I did try to update the FLTK software links list at the official FLTK web site with information on what worked with the latest version of FLTK and with information on newer FLTK applications, but was unable to add some of the newer, more interesting FLTK applications out there. So, this is my definitive list at this point in time of the best FLTK applications available. For more information on FLTK and applications, see also http://www.distasis.com/cpp/scrlib.htm#fltk

I'm sure I haven't covered everything and as mentioned, check NanoLinux and Tiny Core Linux for more FLTK based applications. I'm always looking for new, portable, lightweight FLTK based applications. If you know of something I may have missed or you're working on a new FLTK based project, please contact me.


Shows disk usage. Works on POSIX systems with du command. I have patches to port this to Windows.

Calculator. The FLTK web site link includes information on some of my patches to get it to build successfully as well as link to the original source code.

Graphical diff program.

Traverse directories and find file differences. Some of it was based on fldiff.

Password manager. I ported this to work with the latest versions of FLTK and tinyxml2. It's interesting, but at this point, I'd prefer a Keepass compatible password manager. I'm looking into chkpass as a lightweight alternative for password management.


prozgui for prozilla
Fast file downloader. I have patches for building and to port this to Windows. I use a version based on GNU GPLv2 development instead of the GNU GPLv3 development.

IRC client. Based on MegaIRC, but with a lot of cleanup. I have patches to add gettext/libintl support. This is the best option for IRC using FLTK that I've found to date.

This has good potential as a threaded IRC client. It can handle multiple connections. I could get it to build on Windows either with patches or a build of MinGW with POSIX instead of Windows native threading support. It needs some work on storing IRC connections. Doesn't seem to remember any connections once you leave the application.


There are two webkit based browsers for FLTK. That's great news for FLTK applications users. What's not so great is that they don't port well to non-POSIX systems. If you want the most lightweight webkit based browser (and webkit browsers are not by nature lightweight), I'd go with either of these options instead of the many other webkit ports out there.

I was able to get the original version of netrider to port to Windows and a Windows version of it is available at Sourceforge. However, when I upgraded the version of my MinGW compiler, I was no longer able to build netrider. Seems the webkit developers took some shortcuts in the older code that really weren't up to C++ standards. Netrider upgraded to a later version of webkit (which fixed the compiler issue), but the newer version was never ported to Windows.

This was never ported to Windows although it might be easier to port that the latest version of Netrider. It uses makefiles created by the developer instead of cmake.

When people talk about FLTK based web browsers, Dillo always comes up. However, Dillo is the opposite of what I think of when I think about portable code. One developer decided to fork Dillo and make it more structured and easier to port. He's really done a wonderful job on cleaning up the code. I'd recommend this browser over Dillo if you're interested in doing anything with the source code or need a lightweight HTML viewer for FLTK. DPlus is also the lightest browser I could find that could display output from diffh properly. Most console browsers like lynx had trouble rendering the color differences in the output.

While this is meant as a utility rather than a web browser, I used DPlus as the starting point for my HTML/CSS based dialog replacement.

Mail clients

The developer of Nanolinux wrote a nice, basic, stable e-mail client. (He also reused part of my Open Source POP3 e-mail code.)

This has a lot of potential. It took a long while to get it to build with the latest version of FLTK and it's still kind of buggy. I also needed to update helper libraries fl_toggletree and fleditor to work with the latest FLTK. The interface is a lot of like sylpheed and foxmail. I would love to see some new development on this and would be happy to help update it.

There was a nice, very basic, stable e-mail client at Sourceforge. Doesn't appear to be available from there any longer.

RSS reader
Gautier's RSS reader
This one has a lot of potential. It has an attractive user interface that's easy to work with. However, it does have the ability to sort RSS posts at this point at time. It needs to be used in conjunction with a script and tools like curl to download the RSS feeds. It's basically just a reader. It uses SQLLite to store the RSS data so it can potentially provide fast access to RSS posts. I'd love to see some further development done on this project.


Cross-platform VLC based media player.

Unfortunately, this only works on POSIX systems so far. I have been able to build it on Cygwin as well as BSD and several Linux systems. This might be portable to more platforms using nano-x (and possibly SDL as the backend for nano-x), but I did not get very far in investigating this option. It is my favorite xine front end and is more lightweight than many of the other xine front end options. It provides a variety of features including a nice visualization component for use while playing music.


Simple Audio recorder and player based on Sox. I have done some work to port this one to Windows.

Midi keyboard.

Audio effects program to stretch sounds.

After more than one try to get this to build with the latest version of FLTK, I finally managed to get this working. It's a nice, lightweight audio editor. It doesn't display multiple tracks like Audacity. It does not have good support for playing or recording wave files. It's basically just a wave file editor. Was considering using libsox or another Open Source sound library to add support for playing wave files.

There is a fork of FLTK called NTK. It isn't as portable as FLTK and requires POSIX/X Windows support. A suite of audio applications were created with it.


This is a great, lightweight graphics editor. I really like this one.

Specialized graphics editor for coloring old photos.

Other graphics and drawing options include Antipaint and Cinepaint. Antipaint was updated to work with the lastest version of FLTK and to improve portability by the developer of Nanolinux. You'll find it at the Nanolinux web site. At one point Cinepaint decided to port their project from GTK to FLTK. You'll find some older versions with some FLTK support and utilities. However, the FLTK port is not actively developed.


The Daily Journal is a Personal Information Manager (PIM). It has several nice features including the ability to set alarms to remind you of appointments. I use one of the older versions (0.7) which ports well to newer versions of FLTK and, per my recommendation, so does NanoLinux.

Simple todo list. Haven't used it in a while, but if you're looking for a todo list program, it's an option.

PDF/Ebook Readers

A functional, basic PDF viewer and archived image viewer. I really like this one. It requires a compiler with later C++ support to build. Needs minimal dependencies, mainly libarchive and poppler. While poppler isn't as fast as mupdf at rendering, this still works pretty fast.

Image viewer with plugin support for mupdf and poppler.

There's also flaxpdf which is optimized for efficiency and uses mupdf. However, it's not at all portable to non-POSIX systems.

I tried BDReader as well but there are a lot of dependencies involved in building this.

File managers

I don't use any of these and personally prefer the SDL based file manager mentioned earlier. However, they're nice lightweight GUI file managers. I'm sure there are a few others not mentioned below as well.



Other POSIX only FLTK applications

GUI front end for synaptics touchpad controls.

FLTK ALSA Mixer front end.


I've yet to find a FLTK based editor I really like. For now, I'm still using SciTE, Fxite and nano.

I would love to find a Scintilla based FLTK editor. The closest I've found is https://github.com/cyantreeguo/Fl_Scintilla

There are many FLTK editor controls out there, including the one used by Postoffice.

Nanolinux offers flwriter.

The most interesting editor option I've found so far is fldev. There's a link to the original at http://www.fltk.org/wiki.php?V235+TC+Q and further development by the developer of NanoLinux at https://sourceforge.net/projects/fldev/ The main drawback is that it only opens one file at a time. The original author began to add support to use it as a debugger in conjunction with gdb. I'd love to get that support working properly and in a cross-platform manner and have been experimenting with it as time allows.

flabc is an editor specifically designed for ABC notation, but it appears to be a stable, well-written editor and may be useful for other editing purposes.
2016-11-30 09:43 am

libSDL applications list

While libSDL is typically used for games, there are some applications that can be used to replace common desktop and productivity applications. I'll list the ones I've found here. If you have other alternatives, please let me know.

I've been working on simplifying the applications I typically use on Linux so that they can run in framebuffer mode using DirectFB or Nano-x instead or requiring X Windows. I'm always looking for portable applications, so I can run them on any system I'm using (at home or at work) whether it's Linux or FreeBSD or Windows or something else. I'd love to hear from others with similar goals.

The list just covers libSDL applications. They all built with SDL 1.2.x and I'm creating or locating ports to SDL 2.x as well. I haven't listed SDL based applications that use PDCurses or OpenGL/TinyGL. There are enough of those to warrant their own lists. More information on SDL applications and other screen libraries is available at: http://www.distasis.com/cpp/scrlib.htm#sdl I've posted some of the patches/build scripts for my SDL 2.x ports and will add more over time. You can find them from the archive link at: http://www.distasis.com/cpp/lmbld.htm

URLs are accurate as of when this was posted. However, they can change over time. You can use a search engine or archive.org wayback tool to find pages that have been moved or backups of older versions of pages.

Font viewers:

Good for picking what font you want to work with quickly.

Let's you see all the characters in the font, so you can check if there's support for specific characters needed for internationalization.

Epub reader:

Includes text to speech capabilities using Festival Lite.

Word Processors:

There are a series of SDL based text editors that use a core of common code with some variations in order to port to various handheld devices. Tried building and running, but it's hard to work with since it expects an on screen keyboard rather than a real physical keyboard.

File Manager:

Completely SDL based, two pane file manager.


Lightweight, fast graphics viewer.

Slideshow viewer.

Graphics editor.

Older SVG library. Includes a simple SVG viewer.

Supports SVG rendering and is not tied to a particular screen library. I built a simple SDL SVG viewer with it. If you're looking for SVG support for SDL, this is the best option I've found to date.




Tuxpaint and lunapaint are also SDL options for graphics editors, but I'm not currently using either of them.


Mod player. Can also create music.

The project has a simple, lightweight wave file viewer. I added support so that it works with SDL in place of some of the other graphics options it used.

Sample audio applications.

AV players:



Unfortunately no one's added sound support yet.












2016-11-27 12:30 pm

my core utilities

I've listed some core utilities options besides GNU. I thought I'd share something about what utilities I personally prefer to use. My main requirement in a good set of core utilities is portability. This is rather hard to find. You would think that if a utility was efficient and lightweight, it would be easy to port. However, that's not necessarily so. Many utilities that are designed for efficiency take advantage of features of a particular operating system which makes them harder to port.

At first, I considered starting with sbase which had stated goals similar to what I was looking for, but it didn't have enough features to effectively replace the GNU core utilities when developing and building programs. While newer versions of sbase have added a lot of functionality, they've become much less portable.

My favorite source for inspiration is Minix. Earlier versions provided some interesting and fairly portable versions of a variety of utilites:
Some of the utilities don't have sufficient UTF-8 support or lack some newer functionality found in GNU utilities that makes them fail when attempting to build applications. However, they make a useful starting point.

In some cases, the OBase or BSD utilities do a better job than the older Minix ones and still do that job efficiently. I particularly like the version of patch found on BSD systems. It's an earlier variant of the Free Software Foundation's patch program. Unlike the FSF's version of patch which uses the GNU license, it uses a BSD style license.

For some utilities, I've consulted the POSIX standards ( http://pubs.opengroup.org/onlinepubs/9699919799/idx/utilities.html ) and rewritten them from scratch.

Rather than trying to port utilities such as the Free Software Foundations coreutils, I thought having a lightweight, efficient, highly portable option would be a useful alternative. Many of the FSF developers have little interest in portability making it hard to get later versions of their programs working on non-POSIX systems. I was surprised at how little interest most users have in developing portable alternatives to the core utilities that could be used to build software. Not only was their little interest, some people posted extremely negative comments when anyone suggested creating alternatives to the FSF software. I was also surprised by some of the negative reactions I read about wonderful projects like SBase.

I have a growing collection of public domain, BSD and MIT licensed alternatives to the GNU core utilities. For now, I just use them for my own projects. If you have an interest in portable utilities and tools, would like to see a viable portable alternative to the FSF's GNU coreutils or would like to further discuss related topics in a positive light, feel free to contact me. I'd enjoy talking with other developers and utility users on the topic.

You'll find some added information on my utilities and information on how to discuss the topic further at:
2016-11-27 11:44 am

core utilities

Most systems (other than BSD based ones) use GNU's core utilities. It's used by most Linux distributions. Cygwin, MinGW and gnuwin32 run ported versions of the GNU applications as well. Even Microsoft's SFU/SUA included some of the GNU utilities. However, the GNU core utilities are typically more bloated and have more feature creep than other versions of standard Unix utility programs. BSD systems have their versions of core utilities. The latest version of Minix has adopted the BSD utilities. They tend to be less bloated than the GNU versions, but are still more bloated than other options out there. The BSD utilities also tend toward adding new features similar to but not to the same extent as the GNU utilities. Also, some of their utilities aren't as well optimized as the GNU versions. Busybox seems like the most viable option for a lightweight but still comprehensive version of core utilities. I'm currently using it on my Debian system instead of the GNU core utilities. Toybox is a similar alternative to Busybox. It has a better license option than Busybox, but it's lacking some features and tools that Busybox has.

Here are some links to core utility collections:

Earlier Minix alternatives
Earlier versions of Minix put together an interesting collection of lightweight utilities from various sources.

Windows ports of Busybox:


Heirloom Project
Based on traditional implementations of standard Unix utilities. Not very portable to non-POSIX systems. Not as bloated as GNU or BSD core utilities.

Port of OpenBSD userland to Linux.

This started out as a discussion on one of the suckless.org mailing lists of how to write efficient core utilites that weren't all part of one executable like Busybox or Toybox. Some good examples were posted and the project was started. Then project development was quiet for a while. The project became active again and one of the main goals besides efficiency was UTF-8/internationalization support. Looks like they've borrowed some UTF-8 support concepts (such as Runes) from Plan 9. It's not designed to be portable to non-POSIX systems. However, it does look like they've covered replacing most of the basic core utilities with lightweight, efficient versions.

Other alternatives:
2016-11-11 12:28 pm

Patents and Open Source

--- Free Software and Patent Policy ---

Patents can inhibit and encumber legal distribution of Free and Open Source software.

One might think that the best way to avoid patent issues would be to stick with Free software and Free operating systems. However, these distributions typically deal mainly with software free of copyright issues not patent issues. Patents may only be issues in some parts of the world and many Free operating systems feel justified in distributing patented code (source code only) as part of their freedom of speech.

Note that Free software and operating systems in this article correspond to the Free Software Foundation's definition of ""Free"" and is not necessarily equivalent to Open Source. The Free Software Foundation provides a list of what they consider Free operating systems at https://www.gnu.org/distros/free-distros.en.html

Debian is a good example of a mostly Free operating system ( https://www.gnu.org/distros/common-distros.en.html ). Unlike many completely Free operating system alternatives, its package repository is so extensive, one can find a variety of useful patent unencumbered options. They've also documented information on their polices regarding patented software. Perhaps surprisingly, the best places to find patent unencumbered and royalty-free operating systems are from commercial sources.

The following links give some information on Debian's policies toward patents and patent issues:

--- Patent Encumbered Software---

It's hard to pin down a definitive list of what software may be patent encumbered. Sometimes, even the developers of the software themselves are vague as to whether the software might infringe on patents or not. So, checking a program's web site for patent information may not always help. Typically, one will find software that uses certain compression/decompression algorithms or supports certain multimedia encoding/decoding on lists of patent encumbered software. Here are some lists of software that may have distribution issues.

--- Patent Unencumbered Multimedia Formats ---

One can help to avoid patent encumbered software by using royalty-free or patent unencumbered alternatives such as some of the following:

--- GIF format and LZW ---

A good example of patent related issues with software is the GIF format. Patents on LZW compression used by the GIF format were a main reason for creation of the PNG format. Even though the related LZW patents have expired, one still sees Free software that uses alternatives like libungif (library that encodes only uncompressed GIFs).

--- MPEG-1 issues ---

According to Wikipedia's article on smpeg ( http://en.wikipedia.org/wiki/SMPEG ), it sounds as if the MPEG-1 format is patent unencumbered. However, upon further reading they contradict the information in their MPEG-1 article ( http://en.wikipedia.org/wiki/MPEG-1 ) and state that a full MPEG-1 encoder/decoder cannot be implemented royalty-free because the MP3 audio format which is part of that standard still has several active patents on it.

One possible solution might be to remove the MP3 support in MPEG-1 software. It would be enough to allow viewing of educational information such as the MPEG 1 videos at:
HubbleSOURCE - http://hubblesource.stsci.edu/sources/video/clips/

--- Multimedia Patents ---

Information on when some multimedia patents expire:

--- Patent Unencumbered Software Options ---

This list covers software and builds of software that attempt to avoid patented code or code that is not royalty free.
  • xine-lib pruned - http://pkgs.fedoraproject.org/lookaside/pkgs/xine-lib/
    Pruned older version of xine-lib that has patent encumbered code stripped from it. Some lightweight xine front ends that can be used with it include xine-ui, flxine, toxine or vplayer. (Note: When I contacted the Fedora Project offering to help update this to a later version, they were very much against updating the pruned library and very impolite about the situation. I do have a later version of this library with patented code stripped.)

  • sox - http://sox.sourceforge.net/ - Sound eXchange, the Swiss Army knife of sound processing programs, can be built with only royalty free and patent unencumbered libraries and codecs.

Graphics programs that can built with patent unencumbered libraries and codecs include:

---===== FLIC Format =====---

---===== Animated PNG =====---

---===== Animated GIF =====---

---===== Ogg Theora =====---

---===== Ogg Vorbis =====---

The OpusFAQ - https://wiki.xiph.org/OpusFAQ - says that Ogg Vorbis and speex are being replaced by Opus. I haven't seen any Open Source projects specifically switch from Ogg Vorbis to Opus. However, Xiph.Org Foundation developers created Opus to replace Ogg Vorbis and recommend that projects switch to using the newer format.

---===== WebM and VPX =====---

--- Other Possibilities ---

As mentioned, one can remove the MP3 code from smpeg. There's a pruned older version of smpeg that does this. It might then be usable with other libraries and applications such as xine-lib, dumpmpeg.

Twolame - http://www.twolame.org/ - is optimized for MP2 (MPEG Audio Layer 2) and might provide an alternative to MP3.

--- What can you do? ---

The best way to encourage royalty-free and patent unencumbered software is to support Open formats that are patent unencumbered. As consumers, pick devices that can handle these Open formats. Share multimedia in patent unencumbered formats and encourage multimedia providers to offer these formats.

Here's a list of efforts to encourage patent unencumbered options.
2016-11-10 12:00 pm

Workarounds for I/O issues with Windows console programs and terminal emulators

If you've tried to run Windows console programs (such as pdcurses based applications) using a terminal emulator (rxvt, mintty) via msys (including msys2), you'll probably have I/O issues. Output may not appear properly on the screen.

There are two work-arounds for the issue.

1. Use software from a project that deals with console/terminal incompatibilities.

Winpty - provides and interface similar to a Unix pty-master for communicating with Windows console programs.

Alternative to MinGW and Cygwin and build scripts for various Open Source tools:

2. Run the program in a console instead of a terminal.

I have an article on setting up msys to work with the standard Windows console instead of a terminal at:

For alternatives to the standard Windows console, take a look at these projects.

ConEmu Console emulator - Unlike Console 2, this has a build file for gcc.

Console 2 - Command prompt replacement. May be used with cmd.exe or rxvt or other shells.

ConsoleZ - Console 2 fork with improved support for Windows Vista/7/8/10.
2016-11-10 11:56 am

Windows and MinGW C/C++ resources

This post includes a list of resources for building applications and libraries for Windows using MinGW or similar projects.

Just wanted to note some of the native Windows porting projects I've worked on before I list some of the other projects out there. I've compiled the X server code and built a X server on Windows (similar to the Xming X server but with all my own patches). I'm looking into porting msh to Windows and other platforms (including patches for better handling of internationalization/UTF-8 character codes). I'm working on porting various core utilities based on Minix, BSD and custom implementations. I have patches and build scripts for several Open Source libraries and programs. See http://www.distasis.com/cpp/lmbld.htm for further details and a link to some of the scripts and patches. I work with regex and fnmatch libraries based on MIT licensed code from musl. musl uses regex code from the BSD licensed tre library. I also work with a BSD gettext implementation based on code put together for PostgreSQL. The libiconv and iconv implementations I use are written from scratch using custom UTF-8/internationalization code. I also have my own custom implementations of libmman, libdl and pthreads as well as other libraries I use to help with porting applications.

Places to download builds of various Open Source software and libraries for Windows and MinGW:

Many of the GNU libraries and programs patched and recompiled for MinGW.

Devpaks repository Libraries for the Dev-C++ compiler which uses MinGW. Could also be used without Dev-C++. Files should be standard tar.bz2 files renamed with .devpak extension.

OpenSuse Windows Packages has windows packages cross-compiled using MinGW on OpenSuse and the OpenSuse Build System (OBS).

ezwinports has ports of Unix and GNU software to MS-Windows.

MinGW packages repository

TakeoffGW - Cygwin style package manager and native Windows packages built with the OpenSUSE Build Service.

Patches and build scripts to build Open Source software for Windows:


MSYS and Cygwin alternatives:

Alternative to MinGW and Cygwin and build scripts for various Open Source tools:

GNU on Windows - lightweight alternative to Cygwin. Native Win32 programs.

Busybox port for Windows:
Lightweight versions of core utilities.

Busybox-w32 fork:

UnxUtils - Native Win32 ports of core utilities.

Unix Tools for Windows - Native Windows implementations emulating core utilities written in C++.

Native Win32 port of an older version of bash.

GNU Bash for Windows

WinZsh - Z shell for Windows

Xming X Server - Windows native X server

Open Source libraries patched for Windows:

Relocatable libintl for MinGW.

POSIX dynamic runtime library loading (libdl) compatibility routines.

POSIX mmap functionality.

Various POSIX functions implemented for Windows.

Winpty - provides and interface similar to a Unix pty-master for communicating with Windows console programs.

Pthreads and C/C++ 11 threads compatibility patches and implementations:

pthreads-win32 - Red Hat's POSIX thread library for Win32.

Simplified Pthread Emulation with Macros - Minimal pthread emulation.

TinyCThread - C11 threads implementation

c11threads - GNU compatibility headers for C11 thread support
2016-11-09 10:53 am

load testing

Recently needed to find Open Source load testing software. A couple of things surprised me. Most were written in interpreted languages or Java. Most were platform specific and only worked on Linux or sometimes POSIX systems. Since my preferences are cross-platform C/C++ programs, it was difficult to find load testing software to my liking. Here's a list of some load testing programs that looked useful.

openwebload (aka openload) - http://openwebload.sourceforge.net/
This was the only cross-platform C++ program in the group. The major limitation was that it didn't seem to handle load testing of secure web sites (URLs with https). It also hasn't been updated or maintained in a while, but that never deters me.

wrk - https://github.com/wg/wrk
This was the most interesting of the options I found. It's configurable using lua scripts. However, I was only able to get it to hit one page at a site. So, if you want to emulate things like logging in and navigating a site, that could be difficult. wrk uses code from nginx and other Open Source projects. There are some sample lua scripts, but documentation on how to write your own was rather sparse.

httperf - http://www.labs.hpe.com/research/linux/httperf/
This looked like a very useful tool, but I received web responses mainly in the 300s (3xx) range for the URLs I was testing.

autobench - http://www.xenoclast.org/autobench/
This is a Perl script that works with httperf. It can run httperf multiple times and increase the number of connections each time. It can also output a graph of the results. Since httperf was returning 3xx web responses for my tests, autobench did so as well.

curl-loader - http://curl-loader.sourceforge.net/doc/faq.html
This looked like a highly configurable program that could perform some interesting tests including navigating sites. However, it was very hard to set up even after reading through whatever documentation I could find. I actually froze the test machine where curl-loader was installed at one point and had to restart the machine.

Apache bench - https://httpd.apache.org/docs/2.4/programs/ab.html
This is typically a part of an Apache installation but it can be used to load test other servers including nginx which was my target. Some articles said the technology used by this program was dated and that results might not provide a true picture. However, the program seemed to work fine for my test cases and didn't have an issue with secure URLs. The nice thing about it was that it was fairly simple to use compared to some of the other options.

It's also useful to have curl around when load testing. Once can check URLs with curl and see exactly what's being returned before using the URLs with the load test software.
2016-11-08 10:29 am

Scripting languages

Scripting Languages:

Lua - http://www.lua.org/
Used by SciTE programming editor and grafx2 graphics editor.
Sample SciTE lua scripts are available at http://lua-users.org/wiki/SciteScripts

th1 - https://www.fossil-scm.org/xfer/doc/trunk/www/th1.md
Test Harness 1 is a minimal reimplementation of TCL used for testing the FOSSIL software configuration management and version control system.

microperl - https://www.foo.be/docs/tpj/issues/vol5_3/tpj0503-0003.html
A way to build a minimal implementation of Perl using the Perl source code.

tinypy - http://www.tinypy.org/
A minimalist implementation of Python in 64K of code.

Jim interpreter - http://jim.tcl.tk/index.html/doc/www/www/index.html
Small footprint implementatin of TCL.

Tiny-JS - https://github.com/gfwilliams/tiny-js
JavaScript interpreter

42tiny-JS - https://github.com/ardi69/42tiny-js
Fork on Tiny-JS with more functionality

Quad-wheel - https://code.google.com/archive/p/quad-wheel/
Small but full ECMA-262 supported javascript engine, written in ansi C.

MuPDF JavaScript - http://mupdf.com/
Some versions of MuPDF PDF viewer include a small JavaScript interpreter.

Squirrel programming language - http://squirrel-lang.org/
Scripting language with syntax similar to C/C++/Java and a dynamic nature like Python/Lua.

Squirrel modules - https://github.com/pfalcon/squirrel-modules
Module import system for Squirrel.

AndroWish - http://www.androwish.org/index.html/home
TCL/TK port for Android using SDL and AGG (Anti-Grain Geometry)
2016-10-31 02:01 pm

POSIX threads, C11 threads and thread libraries for Windows and other systems

If you need pthreads for Windows MinGW projects, most projects used pthreads-win32 from Red Hat. However, it wasn't completely compatible with POSIX versions of pthreads. It also uses LGPL licensing which can be somewhat restrictive with MIT and BSD style licensed projects. So, the MinGW64 project wrote their own pthreads implementation from scratch which is available at the MinGW64 web site. Looking at the code, I wasn't sure if the MinGW64 pthreads implementation would work with the MinGW (www.mingw.org) 32 bit compiler as is or if it would need patching. So, I decided to use another alternative for my own projects. Another option on Windows is the midipix project which uses the musl C library pthread implementation.

Many libraries including musl add C11 and C++11 thread support using their POSIX thread implementations. That's because pthreads were available long before the C11 standards were created.

I wanted a simple, cross-platform portable implementation of pthreads. I also wanted to learn more about threads especially C11 threading standards. So, I wrote my own C11 threads library and a pthreads implementation based on it. C11 threads seem more concise, so I wanted to implement C11 support independent of pthreads and then tried implementing pthreads using C11 threads. My library mainly uses semaphores (which are available on most systems). I've read that systems like Haiku exclusively use semaphores for synchronization. While the library is written for portability, I haven't finished adding code or testing for certain platforms. I have used my threading libraries when building MinGW (my own 32 bit version of the compiler) with C++11 threading support enabled.

Before I wrote my library, I used c11threads and TinyCThread with various projects I was porting to fill the gap in support for C11 and C++11 threads. Both are useful if you don't have other alternatives. Now, that I've built MinGW with my own thread libraries, I haven't had any major issues with C11, C++11 and POSIX thread support.

Here are some links I like related to thread libraries and implementations.

Interesting article on semaphores also some source code examples:

Article on benaphores with sample code:

Recursive mutex using benaphores with sample code:

Haiku - articles on Condition Variables


Bionic C library pthreads implementation

Haiku pthread implementation

C11 threads implementation

GNU compatibility headers for C11 thread support