Big Data is Dead

Don't be distracted by the headline, this is very worth your time. Jordan Tigani spent ten years working on Google BigQuery, during which time he was surprised to learn that the median data storage size for regular customers was much less than 100GB. In this piece he argues that genuine Big Data solutions are relevant to a tiny fraction of companies, and there's way more value in solving problems for everyone else. I've been talking about Datasette as a tool for solving "small data" problems for a while, and this article has given me a whole bunch of new arguments I can use to support that concept.

Via Hacker News

Making SQLite extensions pip install-able

Alex Garcia figured out how to bundle a compiled SQLite extension in a Python wheel (building different wheels for different platforms) and publish them to PyPI. This is a huge leap forward in terms of the usability of SQLite extensions, which have previously been pretty difficult to actually install and run. Alex also created Datasette plugins that depend on his packages, so you can now "datasette install datasette-sqlite-regex" (or datasette-sqlite-ulid, datasette-sqlite-fastrand, datasette-sqlite-jsonschema) to gain access to his custom SQLite extensions in your Datasette instance. It even works with "datasette publish --install" to deploy to Vercel, Fly.io and Cloud Run.

Via Datasette Discord

Old technologies that have stuck around are sharks, not dinosaurs. They solve problems so well that they have survived the rapid changes that occur constantly in the technology world. Don’t bet against these technologies, and replace them only if you have a very good reason. These tools won’t be flashy, and they won’t be exciting, but they will get the job done without a lot of sleepless nights.

— Justin Etheredge

ScrollyVideo.js is a JavaScript library that makes it easier to incorporate videos in a scrollytelling layout. The examples look really straightforward, which means I’m saving this for later.

Tags: JavaScript, scrollytelling, video

The first release candidate for Nim version 2.0 is ready for testing.

Three years after Nim version 1.0, and one year since the latest minor release (Nim 1.6), we are proud to announce Nim v2.0 RC1.

With more than 200 fixed bugs and 1000 commits, version 2.0 brings lots of improvements over Nim 1.6.

Don’t panic! One of our design goals was to make it easy to write code that works with Nim version 1 and 2. Many important packages already work with version 2 and as usual many innovations are behind switches that can be enabled or disabled on a per module level thanks to the .experimental pragma.

Version 2 is based on the same codebase as version 1, it’s an evolution, not a revolution.

Why use Nim?

If you are regular Nim user, you already know Nim’s features. On the other hand, if you waited until version 2.0 to explore Nim, here are some of its key strengths:

• One language to rule them all: from shell scripting to web frontend and backend, scientific computing, deep learning, blockchain client, gamedev, embedded, see also some companies using Nim.
• Concise, readable and convenient: echo "hello world" is a 1-liner.
• Small binaries: echo "hello world" generates a 73K binary (or 5K with further options), optimized for embedded devices (Go: 2MB, Rust: 377K, C++: 56K) [1].
• Fast compile times: a full compiler rebuild takes ~12s (Rust: 15min, gcc: 30min+, clang: 1hr+, Go: 90s) [2].
• Native performance: see Web Frameworks Benchmark, ray tracing, primes.
• No need for makefiles, cmake, configure or other build scripts, thanks to compile-time function evaluation (CTFE) and dependency tracking [3].
• Target any platform with a C compiler: Android and iOS, embedded systems, micro-controllers, WASM, Nintendo Switch, Game Boy Advance.
• Zero-overhead interop lets you reuse code in C, C++ (including templates, C++ STL), JS, Objective-C, Python (via nimpy).
• Built-in documentation generator that understands Nim code and runnable examples that stay in sync.

Last but not least, macros let you manipulate/generate code at compile time instead of relying on code generators, enabling writing DSLs and language extensions in user code. Typical examples include implementing Python-like f-strings, optional chaining, command line generators, React-like Single Page Apps, protobuf serialization and binding generators.

Installing Nim 2.0

Building from source

git clone https://github.com/nim-lang/Nim
cd Nim
git checkout version-2-0
sh build_all.sh

The last command can be re-run after pulling new commits.

Using nightlies build

Download a nightly build.

Contributors to Nim 2.0

Many thanks to our recurring and new contributors. Nim is a community driven collaborative effort that welcomes all contributions, big or small.

Backward compatibility

• httpclient.contentLength default to -1 if the Content-Length header is not set in the response. It follows Apache HttpClient(Java), http(go) and .Net HttpWebResponse(C#) behavior. Previously it raised ValueError.

• addr is now available for all addressable locations, unsafeAddr is now deprecated and an alias for addr.

• Certain definitions from the default system module have been moved to the following new modules:

  • std/syncio
  • std/assertions
  • std/formatfloat
  • std/objectdollar
  • std/widestrs
  • std/typedthreads
  • std/sysatomics

  In the future, these definitions will be removed from the system module, and their respective modules will have to be imported to use them. Currently, to make these imports required, the -d:nimPreviewSlimSystem option may be used.

• Enabling -d:nimPreviewSlimSystem also removes the following deprecated symbols in the system module:
  • Aliases with Error suffix to exception types that have a Defect suffix (see exceptions): ArithmeticError, DivByZeroError, OverflowError, AccessViolationError, AssertionError, OutOfMemError, IndexError, FieldError, RangeError, StackOverflowError, ReraiseError, ObjectAssignmentError, ObjectConversionError, FloatingPointError, FloatOverflowError, FloatUnderflowError, FloatInexactError, DeadThreadError, NilAccessError
  • addQuitProc, replaced by exitprocs.addExitProc
  • Legacy unsigned conversion operations: ze, ze64, toU8, toU16, toU32
  • TaintedString, formerly a distinct alias to string
  • PInt32, PInt64, PFloat32, PFloat64, aliases to ptr int32, ptr int64, ptr float32, ptr float64

• Enabling -d:nimPreviewSlimSystem removes the import of channels_builtin in in the system module.

• Enabling -d:nimPreviewCstringConversion, ptr char, ptr array[N, char] and ptr UncheckedArray[N, char] don’t support conversion to cstring anymore.

• The gc:v2 option is removed.

• The mainmodule and m options are removed.

• The threads:on option is now the default.

• Optional parameters in combination with : body syntax (RFC #405) are now opt-in via experimental:flexibleOptionalParams.

• Automatic dereferencing (experimental feature) is removed.

• The Math.trunc polyfill for targeting Internet Explorer was previously included in most JavaScript output files. Now, it is only included with -d:nimJsMathTruncPolyfill. If you are targeting Internet Explorer, you may choose to enable this option or define your own Math.trunc polyfill using the emit pragma. Nim uses Math.trunc for the division and modulo operators for integers.

• shallowCopy and shallow are removed for ARC/ORC. Use move when possible or combine assignment and sink for optimization purposes.

• The experimental nimPreviewDotLikeOps switch is going to be removed or deprecated because it didn’t fullfill its promises.

• The {.this.} pragma, deprecated since 0.19, has been removed.
• nil literals can no longer be directly assigned to variables or fields of distinct pointer types. They must be converted instead.

  type Foo = distinct ptr int
  
  # Before:
  var x: Foo = nil
  # After:
  var x: Foo = Foo(nil)
  

• Removed two type pragma syntaxes deprecated since 0.20, namely type Foo = object {.final.}, and type Foo {.final.} [T] = object.

• foo a = b now means foo(a = b) rather than foo(a) = b. This is consistent with the existing behavior of foo a, b = c meaning foo(a, b = c). This decision was made with the assumption that the old syntax was used rarely; if your code used the old syntax, please be aware of this change.

• Overloadable enums and Unicode Operators are no longer experimental.

• Removed the nimIncrSeqV3 define.

• macros.getImpl for const symbols now returns the full definition node (as nnkConstDef) rather than the AST of the constant value.

• Lock levels are deprecated, now a noop.

• ORC is now the default memory management strategy. Use --mm:refc for a transition period.

• strictEffects are no longer experimental. Use legacy:laxEffects to keep backward compatibility.

• The gorge/staticExec calls will now return a descriptive message in the output if the execution fails for whatever reason. To get back legacy behaviour use -d:nimLegacyGorgeErrors.

• Pointer to cstring conversion now triggers a [PtrToCstringConv] warning. This warning will become an error in future versions! Use a cast operation like cast[cstring](x) instead.

• logging will default to flushing all log level messages. To get the legacy behaviour of only flushing Error and Fatal messages, use -d:nimV1LogFlushBehavior.

• Redefining templates with the same signature was previously allowed to support certain macro code. To do this explicitly, the {.redefine.} pragma has been added. Note that this is only for templates. Implicit redefinition of templates is now deprecated and will give an error in the future.

• Using an unnamed break in a block is deprecated. This warning will become an error in future versions! Use a named block with a named break instead.

• Several Standard libraries are moved to nimble packages, use nimble to install them:
  • std/punycode => punycode
  • std/asyncftpclient => asyncftpclient
  • std/smtp => smtp
  • std/db_common => db_connector/db_common
  • std/db_sqlite => db_connector/db_sqlite
  • std/db_mysql => db_connector/db_mysql
  • std/db_postgres => db_connector/db_postgres
  • std/db_odbc => db_connector/db_odbc
• Previously, calls like foo(a, b): ... or foo(a, b) do: ... where the final argument of foo had type proc () were assumed by the compiler to mean foo(a, b, proc () = ...). This behavior is now deprecated. Use foo(a, b) do (): ... or foo(a, b, proc () = ...) instead.

Major new features

Version 2.0 is a major new release with many additions and changes. We focus here on the most important aspects:

ORC is the new default

--mm:orc is now the default memory management strategy. It has been described multiple times now see, for example here or here.

Overloadable enums

Overloadable enums are no longer experimental.

For example:

  type
    E1 = enum
      value1, value2
    E2 = enum
      value1, value2 = 4

  const
    Lookuptable = [
      E1.value1: "1",
      value2: "2"
    ]

The types E1 and E2 share the names value1 and value2. These are overloaded and the usual overload disambiguation is used so that the E1 or E2 prefixes can be left out in many cases. These features are most beneficial for independently developed libraries.

Strict funcs

The definition of “strict funcs” changed and is now considered to be stable enough to become the default in later versions.

Default values for objects

Inside an object declaration, fields can now have default values:

  type
    Rational* = object
      num: int = 0
      den: int = 1

  var r = Rational()
  assert $r == "(num: 0, den: 1)"

These default values are used when the field is not initialized explicitly. See also default values for object fields for details.

Definite assignment analysis

We found Nim’s default initialization rule to be one major source of bugs. There is a new experimental switch called strictDefs that protects against these bugs. When enabled, it is enforced that a variable has been given a value explicitly before the variable can be used:

  {.experimental: "strictDefs".}

  proc main =
    var r: Rational
    echo r # Warning: use explicit initialization of 'r' for clarity [Uninit]

  main()

To turn the warning into an error, use --warningAsError:Uninit:on on the command line.

The analysis understands basic control flow so the following works because every possible code path assigns a value to r before it is used:

  {.experimental: "strictDefs".}

  proc main(cond: bool) =
    var r: Rational
    if cond:
      r = Rational(num: 3, den: 3)
    else:
      r = Rational()
    echo r

  main(false)

Even better, this feature works with let variables too:

  {.experimental: "strictDefs".}

  proc main(cond: bool) =
    let r: Rational
    if cond:
      r = Rational(num: 3, den: 3)
    else:
      r = Rational()
    echo r

  main(false)

It is checked that every let variable is assigned a value exactly once.

Out parameters

In order to make experimental:strictDefs more effective out parameters have been added to Nim. For more information consult the manual about experimental features.

Strict effects

--experimental:strictEffects are now always enabled. Strict effects require callback parameters to be annotated with effectsOf:

  func sort*[T](a: var openArray[T],
                cmp: proc (x, y: T): int {.closure.},
                order = SortOrder.Ascending) {.effectsOf: cmp.}

The meaning here is that sort has the effects of cmp: sort can raise the exceptions of cmp.

Forbid certain effects

Nim’s effect tracking can now forbid certain effects while being oblivious to others:

  type
    Effect1 = object

  proc test(callback: proc(x: int) {.forbids: [Effect1].}) =
    callback(1)

In this example test takes a callback that can have any effect except Effect1. This mechanism can be used to enforce that a subsystem cannot accidentically call into a different subsystem. For example, in a game engine it could be used to ensure that the renderer logic does not call into a scripting layer.

Unicode operators

--experimental:unicodeOperators are now always enabled: Unicode operators like ⊗ or ∘ can be used by math libraries. Note that the standard library does not use Unicode operators and will not for the foreseeable future.

Bugfixes

These reported issues were fixed:

• Fixed “SYS_getrandom undeclared compiling nim 1.6.0 stdlib on linux kernel < 3.17 (including RHEL7)” (#19052)
• Fixed “nimIdentNormalize(“”) returns “\0”” (#19067)
• Fixed “Compiler terminated with IndexDefect if --gc:arc or --gc:orc given, when proc return a global variable with lent or var type” (#18971)
• Fixed “create does not work for UncheckedArray, as sizeof(UncheckedArray[T])==0” (#19000)
• Fixed “Errors initializing an object of RootObj with the C++ backend” (#18410)
• Fixed “Stack traces broken with arc/orc” (#19078)
• Fixed “getCustomPragmaVal Error: typedesc not allowed as tuple field.” (#19020)
• Fixed “isolate happily compiles despite not being able to prove the absence of captured refs” (#19013)
• Fixed “PragmaExpr erroneously added to enum type” (#19011)
• Fixed “RVO not applied to object with large array” (#14470)
• Fixed “Compile error from backend gcc when generic int type is defined” (#19051)
• Fixed “Block expression doesn’t work in some cases” (#12274)
• Fixed “Make Math.trunc polyfill optional?” (#16144)
• Fixed “Allow adding file/line information to parseStmt/parseExpr” (#13540)
• Fixed “Varargs broken in 1.6.0 when len is 0 and preceding block arguments.” (#19015)
• Fixed “VM replaces declared type with alias” (#19198)
• Fixed “regression: effectless inner template declared as side effect” (#19159)
• Fixed “variables in closure iterators loop are not correctly unassigned” (#19193)
• Fixed “std.streams fails to compile with TCC compiler on Windows and –cpu:amd64” (#16326)
• Fixed “Unexported converters propagate through imports and affect code” (#19213)
• Fixed “[arc] of operation segfaults for a ptr object containing traced reference” (#19205)
• Fixed “Static linking with a .lib file not working” (#15955)
• Fixed “re.split unexpected results with zero-width characters” (#14468)
• Fixed “Async httpclient bodyStream reads fails when response is Http204” (#19253)
• Fixed “Object constructor fails on Windows” (#19244)
• Fixed “Out-of-bounds in strformat” (#19107)
• Fixed “Adding an empty list to a non-empty list breaks the latter list” (#19297)
• Fixed “Wrong result when using varargs with var arguments.” (#16617)
• Fixed “Adding an empty DoublyLinkedList to a non-empty DoublyLinkedList breaks the latter list” (#19314)
• Fixed “Compiler version 1.6.0 does not work on Windows XP” (#19038)
• Fixed “hasCustomPragma fails on nnkVarTy/nnkBracketExpr nodes” (#11923)
• Fixed “RST minor bugs” (#17340)
• Fixed “useNimRtl does not work for –gc:orc/arc (in windows)” (#16458)
• Fixed “Orc booting compiler doesn’t work with newSeq operations” (#19404)
• Fixed “Manual example: {.cast(uncheckedAssign).} assignment to discriminator produces [FieldDefect]” (#19266)
• Fixed “nim js ignores file write error” (#18662)
• Fixed “Nim-1.6 segfault” (#19569)
• Fixed “Nim compiler crashing when using control flow statements inside try-catch block on a closure iterator” (#19575)
• Fixed “Remove deprecated typo poDemon” (#19631)
• Fixed “useless overflowChecks runtime check generated even when one of a div b constant” (#19615)
• Fixed “[ARC] tuple unpacking leads to unnecessary copies & memory leak” (#19364)
• Fixed “pragma in unreferenced function affects subsequent code” (#19603)
• Fixed “nim dump and other information obtaining commands execute top-level exec statements in nims files” (#8219)
• Fixed “Raises pragma and generic error/exception types compiler crash” (#14318)
• Fixed “Bug using nested loops in closure iterators” (#18474)
• Fixed “Import/except doesn’t work on devel” (#18986)
• Fixed “nim check -b:js does not undefine OS symbols” (#17286)
• Fixed “Can’t check if stderr is static” (#19680)
• Fixed “View of seq[T] when T has seq attribute won’t iter with ARC/ORC, but len returns correct number of elements” (#19435)
• Fixed “Indent level ignored for first line” (#19662)
• Fixed “Method dispatch is slow” (#18612)
• Fixed “Error with anonymous tuples passed to concept function arguments.” (#19730)
• Fixed “Add link to std/tempfiles in the docs” (#19155)
• Fixed “Add –gc:arc (or –mm:arc) induce different behavior when using converter” (#19862)
• Fixed “Converting unsigned integer to float fails in VM” (#19199)
• Fixed “--genscript for vcc produces a script that does not compile” (#19883)
• Fixed “Search results are in sidebar” (#19900)
• Fixed “regression(0.20.0 => devel): var params assignment gives silently wrong results in VM” (#15974)
• Fixed “Closure iterator finishing prematurely” (#11042)
• Fixed “Crash dereferencing object via a view object” (#15897)
• Fixed “genDepend broken for duplicate module names in separate folders” (#18735)
• Fixed “asm and std=c99 incompatibility” (#20012)
• Fixed “shallowcopy string doesn’t work with arc/orc” (#20015)
• Fixed “Templates: Crash with gensym’ed proc & method call” (#20002)
• Fixed “AsyncSocket.getPeerAddr appears to not work.” (#15022)
• Fixed “hasCustomPragma and getCustomPragmaVal don’t work on fields with backticks” (#20067)
• Fixed “unmarshalling nil strings/seqs doesn’t work with ORC” (#20089)
• Fixed “Cant use uint64 in case” (#20031)
• Fixed “CI: Migration from builds.sr.ht” (#20123)
• Fixed “nim jsondoc output is broken” (#20132)
• Fixed “Underscores are unnecessarily escaped in db_mysql” (#20153)
• Fixed “Bug with effect system and forward declarations” (#6559)
• Fixed “Instant OOM in Nimsuggest” (#15316)
• Fixed “Lint/style error reported for explicit module name when there’s a type collision” (#12955)
• Fixed “Invalid codegen when block ends with lent” (#20107)
• Fixed “compiler flag --hintAsError[XDeclaredButNotUsed]:on causes unavoidable error in fatal.nim that goToBasedException is never used” (#20149)
• Fixed “jsondoc creates no files unless html-based version exist prior” (#11953)
• Fixed “locals doesn’t work with ORC” (#20162)
• Fixed “reset does not work on set” (#19967)
• Fixed “selectRead and selectWrite are dangerous to use sockets with FD numbers bigger than FD_SETSIZE (1024) on *nixes” (#19973)
• Fixed “type A* = A with B = (A,) causes compiler to run infinitely” (#18983)
• Fixed “x < 1 (and|or) b generates temp variables in js output” (#20219)
• Fixed “object fields of distinct types doesn’t work with JS” (#20227)
• Fixed “SSL certificate loading breaks after first found certificate” (#17658)
• Fixed “compiler flag –clib prefixes unnecessary path component to library name” (#16937)
• Fixed “use-after-free bugs in object variants” (#20305)
• Fixed “Nim should be able to generate a theindex.json” (#9462)
• Fixed “[ARC] C compiler error when using the result of a template in the subscript operator” (#20303)
• Fixed “Calling nullary templates without () doesn’t work inside calls inside other templates” (#13515)
• Fixed “[ARC] Sink inference prevents the usage of stdlib procedures for functional style” (#19724)
• Fixed “use-after-free bugs in object variants” (#20305)
• Fixed “Float ranges in case statement in JS crash compiler” (#20233)
• Fixed “[Regression] Incorrect captures of pegs \ident macro in nim 1.6” (#19104)
• Fixed “Windows gcc shipped with choosenim 1.6.4 with TLS emulation turned off : The application was unable to start correctly (0xc000007b). Click OK to close the application” (#19713)
• Fixed “Improve error message when instantiating generics that lack a type” (#19882)
• Fixed “of operator doesn’t consider generics under orc/arc” (#20391)
• Fixed “Tests fail in 2038” (#20285)
• Fixed “At a certain level nested generics cause causes the typechecker to get stuck” (#20348)
• Fixed “C++ backend fails when put inherited object in another object type” (#17351)
• Fixed “Static linking with a .lib file not working” (#15955)
• Fixed “sigmatch error confusing when (inferred) pragmas mismatch (eg; {.locks.}; or {.closure.} calling convention)” (#2614)
• Fixed “find and rfind differ on empty needle” (#18128)
• Fixed “-mm flag is ignored on latest Nim 1.7.1 be4bd8” (#20426)
• Fixed “Internal error on ARC/ORC when using forward declaration of finalizer proc” (#19401)
• Fixed “seqs are not properly updated in loop with ARC/ORC” (#19457)
• Fixed “dereferencing pointer to incomplete type error with gcc 9.4 with statics/cast” (#20141)
• Fixed “mutable view from immutable location” (#19986)
• Fixed “strutils.find uses cstring optimization that stops after \0” (#19500)
• Fixed “Broken behavior with string ranges in case labels” (#19678)
• Fixed “Internal error on trying to iterate on an empty array/seq” (#19224)
• Fixed “Custom pragma ignored on field of variant obj if in multiple branches” (#11415)
• Fixed “C Compiler error when initializing {.global.} with a block:” (#18645)
• Fixed “internal error: expr: var not init - in custom finalizer” (#19231)
• Fixed “Empty seq with indirection in arc” (#11267)
• Fixed “(Unintended) Destruction of Thread object causes hard to debug crash” (#7172)
• Fixed “Destructors: distinct types don’t get destructors automatically from the base type” (#9401)
• Fixed “regression(1.04): reset broken in VM; incorrect VM handling of var params” (#12994)
• Fixed “system.create doesn’t work with bitfield objects” (#20516)
• Fixed “Extra forbids: [] shown in docs and not hidden” (#20524)
• Fixed “Nimc crash on ambiguous proc cast” (#18886)
• Fixed “Generics: type mismatch “SomeunsignedInt or Natural”” (#7446)
• Fixed “Regression in proc symbol resolution; Error: attempting to call routine “ (#18990)
• Fixed “tests/proc/t17157.nim now gives SIGSEGV instead of error” (#18136)
• Fixed “Nimpretty mangles numeric literal procs” (#20553)
• Fixed “Confusing error message (methods can’t have same names as fields if UFCS is used)” (#3748)
• Fixed “JS codegen can produce extreme switch statements with case a of range” (#8821)
• Fixed “Crash when passing a template to a generic function expecting a procedure” (#19700)
• Fixed “Experimental features in normal manual instead of experimental manual/undocumented” (#19162)
• Fixed “methods inferred gcsafe is not verified” (#20515)
• Fixed “ Error: illegal context for ‘nimvm’ magic if ‘nimvm’ is used with single branch ‘when’” (#12517)
• Fixed “Compiler replaces =sink for =copy” (#20572)
• Fixed “cannot generate code for: mSlice with toOpenArray” (#19969)
• Fixed “Regression: compile error using when/elif/else and typedesc in template” (#19426)
• Fixed ““incompatible type” when mixing float32 and cfloat in generics” (#19349)
• Fixed “Illegal capture of closure iterator, when should be legal” (#20152)
• Fixed “Generic proc instantiation and tuple types” (#4466)
• Fixed “Generic proc involving generic .importcpp type with type specifier is not code-generated properly” (#4678)
• Fixed “privateAccess doesn’t work with generic ref object types.” (#19278)
• Fixed “Regression when accessing a variable generic type” (#20645)
• Fixed “Templates allowed to use ambiguous identifier” (#1027)
• Fixed “Use of _ (as var placeholder) inside a template causes XDeclaredButNotUsed hints” (#12094)
• Fixed “Can’t use empty sets as tuple field values (unless the set is a var/let value)” (#6213)
• Fixed “FAMs should not be used in the C++ backend” (#20654)
• Fixed “sink causes crash in VM” (#19201)
• Fixed “Can’t instantiate a static value of generic type” (#6637)
• Fixed “Implement Unix file regularity check (#20448)” (#20628)
• Fixed “Can not use nim 2’s new default instantiation with any object type with a DateTime field” (#20681)
• Fixed “regression(0.18.0 => devel): import times; echo low(Time) gives OverflowDefect” (#16264)
• Fixed “implicit compile time conversion int to ranged float causes compiler fatal error” (#20148)
• Fixed “range[a..b] inside object variant fails” (#20715)
• Fixed “range of uint64 shows signed upper bound” (#20272)
• Fixed “attempting to call undeclared routine: ‘case’” (#20283)
• Fixed “Threads and channels modules’ docs leak into system module docs” (#20526)
• Fixed “Wrong assignment for tuples in some contexts.” (#16331)
• Fixed “Returning procedures with different noSideEffect pragmas” (#14216)
• Fixed “Range types don’t work with BackwardsIndex (and possibly others)” (#13618)
• Fixed “pre-existing field visibility of VM object passed to runtime” (#20740)
• Fixed “Nim compiler crashes when trailing whitespace are too many (>=128)” (#15688)
• Fixed “Illegal storage access compiling call with nested ref/deref’d types” (#18079)
• Fixed “function return enum type cause wrong.” (#12589)
• Fixed “Invalid codegen when returning var tuple from a template” (#19149)
• Fixed “regression (0.19=> 0.20 onwards): adding doc comment in importc proc makes it silently noop at CT” (#17121)
• Fixed “Invalid type in `importc: “exit”’” (#20694)
• Fixed “multiReplace / replace taking a long time to execute in VM since commit: ae050b05e9ce6f4e356c46de8722724a2f706e18 “ (#20746)
• Fixed “Default value of object parameterized with a RootRef generates incorrect C” (#20699)
• Fixed “gc:arc cannot fully support threadpool with FlowVar” (#13781)
• Fixed “Add stew to important_packages” (#20798)
• Fixed “Regression of type inference when using templates and a proc with the same name as one of them” (#20807)
• Fixed “Nim doesn’t catch wrong var {.global.} initialization” (#3505)
• Fixed “Object variants + UncheckedArray[T] causes unsafeNew() act like new() and ignore the size parameter” (#20836)
• Fixed “getImpl can no longer be used with nkObjConstr’s bound type” (#20856)
• Fixed “utf-8 (windows 7)” (#20835)
• Fixed “dochack.nim uses wrong path to find theindex.html => search doesn’t work unless file is in same dir as theindex.html” (#14476)
• Fixed “[ORC] Bad codegen for global pointer to iterator” (#20866)
• Fixed “doAssertRaises cannot handle IndexDefect with goto exceptions” (#20026)
• Fixed “Small string case with else statement first in AST evaluates wrongly” (#18964)
• Fixed “ptr char implicitly converts to cstring, resulting in undefined behavior” (#13790)
• Fixed “arc/orc is broken for vcc (devel)” (#20873)
• Fixed “sizeof object containing a set is wrong” (#20914)
• Fixed “Error: internal error: yield in expr not lowered” (#13583)
• Fixed “Invalid type in slice generated by parallel transform” (#20958)
• Fixed “arc/orc is broken for cpp backend using vcc (devel)” (#20969)
• Fixed “Unspecified generic on default value segfaults the compiler” (#20883)
• Fixed “[ICE] Combination of concept, dotOperators & static leads to internal error” (#20996)
• Fixed “warn on overloaded = with refc” (#20846)
• Fixed “Missing bounds check for len(toOpenArray..)” (#20954)
• Fixed “getImpl on types return incorrect tree in when branches” (#16639)
• Fixed “tyInt tyUint fit target int bit width” (#20829)
• Fixed “SIGSEGV when cast is Improperly Formatted” (#21027)
• Fixed “Modules not linked in the main stdlib documentation” (#16656)
• Fixed “noReturn pragma doesn’t work when we add a doc comment” (#9839)
• Fixed “Extremely confusing error message with invalid syntax of: '+':” (#20922)
• Fixed “Dangerous implicit type conversion from auto + generics” (#15836)
• Fixed “SIGSEGV in alloc.nim addToSharedFreeList() in heavily threaded code” (#21062)
• Fixed “getTime with vmopsDanger is broken” (#21045)
• Fixed “Nim crashes in fixAbstractType” (#16758)
• Fixed “Dangerous implicit type conversion from auto + generics” (#15836)
• Fixed “Strict func does not catch mutation” (#20808)
• Fixed “Named except clauses and experimental strictDefs don’t work together.” (#21043)
• Fixed “Compiler quits SILENTLY when compiling code with generic types.” (#20416)
• Fixed “Add warning for bare except: clause” (#19580)
• Fixed “Error: internal error: getTypeDescAux(tyFromExpr) when using auto + arc, works with refc” (#20588)
• Fixed “ByteAddress broken for its intended purpose” (#12122)
• Fixed “os.walkDir breaks if called in a function with a parameter named ‘glob’” (#21116)
• Fixed “static arg for [] causes deref to fail in typeof within template” (#11705)

Footnotes

Tested on a 2.3 GHz 8-Core Intel Core i9, 2019 macOS 11.5 with 64GB RAM.

• [1] command used: nim c -d:danger. The binary size can be further reduced to 49K with stripping (--passL:-s) and link-time optimization (--passC:-flto). Statically linking against musl brings it under 5K - see here for more details.
• [2] commands used:
  • for Nim: nim c --forceBuild compiler/nim
  • for Rust: ./x.py build, details
  • for GCC: see 1 2
  • for Clang: details
  • for Go: ./make.bash
• [3] a separate nimscript file can be used if needed to execute code at compile time before compiling the main program but it’s in the same language

Elon Musk laid off about half of Twitter this morning. There are many terrible stories emerging about how this went down, but one that particularly struck me was that he laid off the entire accessibility team. For me this feels like a microcosm of the whole situation. Twitter's priorities are no longer even remotely aligned with my own.

I've been using Twitter since November 2006 - wow, that's 16 years! I've accumulated 42,804 followers there. It's been really good to me, and I've invested a lot of work generating content there to feed the machine.

I can't see myself putting the same work in to help the world's (current) richest man pay the billion dollar annual interest on the loans he took out to buy the place on a weird narcissistic whim.

So I've started to explore Mastodon - and so far it's exceeding all of my expectations.

My new profile is at https://fedi.simonwillison.net/@simon - you can follow @simon@simonwillison.net in your Mastodon client of choice.

Why Mastodon?

The lesson I have learned from Twitter is that, even if a service you trust makes it past an IPO and becomes a public company, there's always a risk that it can be bought by someone who very much doesn't share your values.

Mastodon has been designed to avoid this from the start. It operates as a federated network of independent servers, each of which is run by a different person or organization with the ability to set their own rules and standards.

You can also host your own instance on your own domain.

My initial nudge to try this out was from Jacob and Andrew, who figured out how to do exactly that:

• The Fediverse, And Custom Domains - Andrew Godwin
• Setting up a personal Fediverse ID / Mastodon instance - Jacob Kaplan-Moss

Andrew and Jacob both opted to pay masto.host to run their instance for them. I've decided to do the same. It's on my domain, which means if I ever want to run it myself I can do so without any visible disruption.

I'm paying $9/month. I find it darkly amusing that this is a dollar more than Elon has been planning to charge for users to keep their verified status on Twitter!

If you don't want to use your own domain there are plenty of good free options, though I recommend reading Ash Furrow's post about his shutdown of mastodon.technology to help understand how much of a commitment it is for the admins who run a free instance.

This post by @klillington@mastodon.ie has some good links for getting started understanding the system. I particularly enjoyed Nikodemus’ Guide to Mastodon as it matched most closely the questions I had at first.

Initial impressions

Despite taking the second hardest route to joining Mastodon (the hardest route is spinning up a new server from scratch) it took me just less than an hour to get started. I wrote up a TIL describing what I did - more or less directly following the steps described by Andrew and Jacob.

I signed into my new account and started following people, by pasting in their full Mastodon names (mine is @simon@simonwillison.net). I was initially surprised that this did nothing: your timeline won't be populated until the people you follow have said something.

And then people started to toot, and my timeline slowly kicked into life.

And it was really, really pleasant.

My fear was that everyone on Mastodon would spend all of their time talking about Mastodon - especially given the current news. And sure, there's some of that. (I'm obviously guilty here.)

But there's lots of stuff that isn't that. The 500 character limit gives people a bit more space, and replies work much like they do on Twitter. I followed a bunch of people, replied to a few things, posted some pelican photos and it all worked pretty much exactly as I hoped it would.

It's also attracting very much the kind of people I want to hang out with. Mastodon is, unsurprisingly, entirely populated by nerds. But the variety of nerds is highly pleasing to me.

I've been checking in on the #introduction hashtag (hashtag search is much more useful than regular search on Mastodon because it spans servers in a way I've not quite understood yet) and I'm seeing artists, academics, writers, historians. It's not just programmers. The variety of interest areas on Twitter is the thing I'll miss most about it, so seeing that start to become true on Mastodon too is a huge relief.

Considering how complicated a federated network is, the fact that it's this smooth to use is really impressive. It helps that they've had six years to iron out the wrinkles - the network seems to be coping with the massive influx of new users over the past few days really well.

I'm also appreciating how much thought has been put into the design of the system. Quote tweeting isn't supported, for reasons explained by Eugen Rochko in this 2018 post:

Another feature that has been requested almost since the start, and which I keep rejecting is quoting messages. Coming back to my disclaimer, of course it’s impossible to prevent people from sharing screenshots or linking to public resources, but quoting messages is immediately actionable. It makes it a lot easier for people to immediately engage with the quoted content… and it usually doesn’t lead to anything good. When people use quotes to reply to other people, conversations become performative power plays. “Heed, my followers, how I dunk on this fool!” When you use the reply function, your message is broadcast only to people who happen to follow you both. It means one person’s follower count doesn’t play a massive role in the conversation. A quote, on the other hand, very often invites the followers to join in on the conversation, and whoever has got more of them ends up having the upper hand and massively stressing out the other person.

Mastodon so far feels much more chilled out than Twitter. I get the impression this is by design. When there's no profit motive to "maximize engagement" you can design features to optimize for a different set of goals.

And there's an API

Unsurprisingly, Mastodon has a powerful API. It's necessary for the system itself to work - those toots aren't going to federate themselves!

Poking around with it is really fun.

First, a friendly note. @pamela@bsd.network wrote the following:

Hacky folks, please resist finding ways to scrape the fediverse, build archives, automate tools and connect to people via bot without their consent.

[...]

Whatever your thing is, make it 100% opt-in. Make it appropriate for a significantly more at-risk user than you are. Make sure it forgets things, purges info about servers it can't contact, can't operate in any sort of logged-in mode where consent is an issue.

We will straight up help advertise your cool thing if it respects users properly and takes the time to consider the safety and preferences of every person involved. There are a lot of fun, thoughtfully-designed toys! And there are a lot of people really tired of having to come and tell you off when you wanted to help, honestly. Help yourself and ask around before you flip on your cool new thing, let folks point out what you're missing.

(Read the whole thing, it's great.)

So far I've done a couple of things.

I built a Git scraper to track the list of peer instances that various servers have picked up. This feels like a reasonable piece of public information to track, and it's a fun way to get a feel for how the network is growing.

I also figured out how to Export a Mastodon timeline to SQLite using the timelines API and my paginate-json and sqlite-utils CLI tools, so I could explore it in Datasette.

Running my own instance means I have no ethical qualms at all about hammering away at my own API endpoint as fast as I like!

I like to follow a lot of different people, and I don't like to feel committed to reading everything that crosses my timeline - so I expect that the feature I'll miss most from Twitter will be the algorithmic timeline! This is very much not in the spirit of Mastodon, which is firmly committed to a reverse chronological sort order.

But with access to the raw data I can start experimenting with alternative timeline solutions myself.

I'm somewhat intrigued by the idea of iterating on my own algorithmic timeline, to try and keep the variety of content high while hopefully ensuring I'm most likely to catch the highlights (whatever that means.)

Past experience building recommendation systems has taught me that one of the smartest seeming things you can do is pick the top 100 most interesting looking things based on very loose criteria and then apply random.shuffle() to produce a final feed!

I have a hunch that this is going to be a lot of fun.