words_extractor

Example of text file parsing in Python, Golang, Elixir, Rust and Crystal

Text source: 79.4MB in 30 files

• Python 3.9.5 (parallel) with sorting: 3.42s, without sorting: 2.47
• Rust 1.51.0 with sorting: 7s, without sorting: 5s (no parallelism)
• Go 1.16.4 (parallel) with sorting: 7.32s, without sorting: 4.06s
• Python 3.9.5 with sorting: 10s, without sorting 8.32s (no multiprocessing)
• Crystal 1.0.0 with sorting: 17s, without sorting: 7s (non optimized sort, no parallelism)
• Go 1.16.4 with sorting: 21s, without sorting: 11s (no parallelism)
• Elixir 1.12 (parallel) with sorting: 33s (without release build)

macOS 11.3.1, MacBook Pro (Retina, 15-inch, Late 2013)

cd words_extractor_py
python words.py

cd words_extractor_rs
cargo build --release
target/release/words_extractor_rs

cd words_extractor_go
make build
./main

cd words_extractor_cr
crystal build --release src/fast_words_cr.cr -o main
./main

cd words_extractor_ex
mix run "WordsExtractor.run"

Running Python

1. Install the latest Python 3.9.5
2. Create venv and dependencies

cd words_extractor_py
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

3. Run the code

python words_parallel.py